1055 lines
118 KiB
Plaintext
1055 lines
118 KiB
Plaintext
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:55:48 [api_server.py:1351] vLLM API server version 0.11.2.dev690+g67475a6e8.d20251209
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:55:48 [utils.py:253] non-default args: {'model_tag': 'cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', 'host': '127.0.0.1', 'model': 'cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', 'trust_remote_code': True, 'max_model_len': 24576, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 64}
|
|
[0;36m(APIServer pid=16712)[0;0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:55:52 [model.py:629] Resolved architecture: Qwen3MoeForCausalLM
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:55:52 [model.py:1755] Using max model len 24576
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:55:52 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
|
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:55:56 [core.py:93] Initializing a V1 LLM engine (v0.11.2.dev690+g67475a6e8.d20251209) with config: model='cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', speculative_config=None, tokenizer='cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=24576, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:55:56 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.1.122:38381 backend=nccl
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:55:56 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:55:57 [gpu_model_runner.py:3544] Starting to load model cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit...
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:55:57 [compressed_tensors_wNa16.py:114] Using ConchLinearKernel for CompressedTensorsWNA16
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:55:57 [rocm.py:320] Using Triton Attention backend on V1 engine.
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:55:57 [layer.py:379] Enabled separate cuda stream for MoE shared_experts
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:55:57 [compressed_tensors_moe.py:189] Using CompressedTensorsWNA16MoEMethod
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m WARNING 12-09 19:55:57 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m
|
|
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m
|
|
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:01, 1.94it/s]
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m
|
|
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:02<00:02, 1.37s/it]
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m
|
|
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:04<00:01, 1.59s/it]
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m
|
|
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:06<00:00, 1.78s/it]
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m
|
|
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:06<00:00, 1.60s/it]
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:56:04 [default_loader.py:308] Loading weights took 6.43 seconds
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:56:05 [gpu_model_runner.py:3626] Model loading took 16.2266 GiB memory and 7.385149 seconds
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:56:10 [backends.py:616] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/d30966f362/rank_0_0/backbone for vLLM's torch.compile
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:56:10 [backends.py:676] Dynamo bytecode transform time: 5.01 s
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:56:16 [backends.py:243] Cache the graph of compile range (1, 2048) for later use
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m WARNING 12-09 19:56:19 [fused_moe.py:888] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/opt/venv/lib/python3.13/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=768,device_name=AMD-gfx1201,dtype=int4_w4a16.json']
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:57:02 [backends.py:260] Compiling a graph for compile range (1, 2048) takes 49.43 s
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:57:02 [monitor.py:34] torch.compile takes 54.43 s in total
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:57:05 [gpu_worker.py:364] Available KV cache memory: 11.81 GiB
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:57:05 [kv_cache_utils.py:1287] GPU KV cache size: 128,976 tokens
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:57:05 [kv_cache_utils.py:1292] Maximum concurrency for 24,576 tokens per request: 5.25x
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/19 [00:00<?, ?it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 11%|█ | 2/19 [00:00<00:00, 18.81it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 26%|██▋ | 5/19 [00:00<00:00, 20.79it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 42%|████▏ | 8/19 [00:00<00:00, 21.40it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 58%|█████▊ | 11/19 [00:00<00:00, 22.59it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 74%|███████▎ | 14/19 [00:00<00:00, 22.79it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 89%|████████▉ | 17/19 [00:00<00:00, 23.23it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:00<00:00, 22.38it/s]
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m
|
|
Capturing CUDA graphs (decode, FULL): 0%| | 0/11 [00:00<?, ?it/s]
|
|
Capturing CUDA graphs (decode, FULL): 27%|██▋ | 3/11 [00:00<00:00, 26.17it/s]
|
|
Capturing CUDA graphs (decode, FULL): 55%|█████▍ | 6/11 [00:00<00:00, 27.79it/s]
|
|
Capturing CUDA graphs (decode, FULL): 82%|████████▏ | 9/11 [00:00<00:00, 26.71it/s]
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 26.90it/s]
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:57:07 [gpu_model_runner.py:4548] Graph capturing finished in 2 secs, took 1.24 GiB
|
|
[0;36m(EngineCore_DP0 pid=16880)[0;0m INFO 12-09 19:57:07 [core.py:256] init engine (profile, create kv cache, warmup model) took 62.29 seconds
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:09 [api_server.py:1099] Supported tasks: ['generate']
|
|
[0;36m(APIServer pid=16712)[0;0m WARNING 12-09 19:57:09 [model.py:1581] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:09 [serving_responses.py:197] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [serving_chat.py:133] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [serving_completion.py:73] Using default completion sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [serving_chat.py:133] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [api_server.py:1425] Starting vLLM API server 0 on http://127.0.0.1:8000
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:38] Available routes are:
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /docs, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /tokenize, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /detokenize, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /pause, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /resume, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /is_paused, Methods: GET
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /metrics, Methods: GET
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /health, Methods: GET
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /load, Methods: GET
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /v1/models, Methods: GET
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /version, Methods: GET
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /v1/responses, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /v1/messages, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /v1/completions, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /ping, Methods: GET
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /ping, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /invocations, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /classify, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /v1/embeddings, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /score, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /v1/score, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /rerank, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /v1/rerank, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /v2/rerank, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:10 [launcher.py:46] Route: /pooling, Methods: POST
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: Started server process [16712]
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: Waiting for application startup.
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: Application startup complete.
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48882 - "GET /v1/models HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60086 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:30 [loggers.py:248] Engine 000: Avg prompt throughput: 43.9 tokens/s, Avg generation throughput: 53.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:40 [loggers.py:248] Engine 000: Avg prompt throughput: 176.9 tokens/s, Avg generation throughput: 123.5 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:57:50 [loggers.py:248] Engine 000: Avg prompt throughput: 207.4 tokens/s, Avg generation throughput: 180.4 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35306 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:58:00 [loggers.py:248] Engine 000: Avg prompt throughput: 215.3 tokens/s, Avg generation throughput: 188.3 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60086 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35306 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60086 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:58:10 [loggers.py:248] Engine 000: Avg prompt throughput: 285.9 tokens/s, Avg generation throughput: 174.8 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60086 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43806 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43822 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:50952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43822 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:58:20 [loggers.py:248] Engine 000: Avg prompt throughput: 310.2 tokens/s, Avg generation throughput: 215.5 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43806 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43822 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:57206 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60086 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:57222 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60086 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:57236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:58:30 [loggers.py:248] Engine 000: Avg prompt throughput: 452.9 tokens/s, Avg generation throughput: 220.2 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:57236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:57236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:58:40 [loggers.py:248] Engine 000: Avg prompt throughput: 185.7 tokens/s, Avg generation throughput: 199.3 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:57236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:50728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46646 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46646 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:57206 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:58:50 [loggers.py:248] Engine 000: Avg prompt throughput: 289.2 tokens/s, Avg generation throughput: 225.9 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:50728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:57206 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37884 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:57236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46646 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:59:00 [loggers.py:248] Engine 000: Avg prompt throughput: 320.1 tokens/s, Avg generation throughput: 264.1 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43806 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46646 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:57236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43806 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46646 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:59:10 [loggers.py:248] Engine 000: Avg prompt throughput: 131.0 tokens/s, Avg generation throughput: 283.9 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:57206 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:59:20 [loggers.py:248] Engine 000: Avg prompt throughput: 111.2 tokens/s, Avg generation throughput: 271.3 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:50728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43806 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55238 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:50728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55250 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:59:30 [loggers.py:248] Engine 000: Avg prompt throughput: 245.5 tokens/s, Avg generation throughput: 224.7 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55238 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:50728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:50728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43806 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:59:40 [loggers.py:248] Engine 000: Avg prompt throughput: 184.9 tokens/s, Avg generation throughput: 241.5 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46646 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 19:59:50 [loggers.py:248] Engine 000: Avg prompt throughput: 130.9 tokens/s, Avg generation throughput: 237.1 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46646 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43806 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:41424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:00:00 [loggers.py:248] Engine 000: Avg prompt throughput: 80.4 tokens/s, Avg generation throughput: 148.3 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55250 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:00:10 [loggers.py:248] Engine 000: Avg prompt throughput: 140.2 tokens/s, Avg generation throughput: 181.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55250 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55250 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:41424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37072 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37084 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:41424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:41424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:00:20 [loggers.py:248] Engine 000: Avg prompt throughput: 243.2 tokens/s, Avg generation throughput: 179.2 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37094 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37094 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46646 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37094 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:37062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:00:30 [loggers.py:248] Engine 000: Avg prompt throughput: 82.2 tokens/s, Avg generation throughput: 249.9 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:00:40 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 146.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:00:50 [loggers.py:248] Engine 000: Avg prompt throughput: 1.2 tokens/s, Avg generation throughput: 26.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42772 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42780 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34928 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34948 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34956 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34948 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34966 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34928 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34972 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34972 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35016 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:01:00 [loggers.py:248] Engine 000: Avg prompt throughput: 727.9 tokens/s, Avg generation throughput: 220.9 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 15.3%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34928 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42772 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35016 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35028 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35028 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35028 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44926 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44930 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44930 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44968 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45016 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45016 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:01:10 [loggers.py:248] Engine 000: Avg prompt throughput: 1149.0 tokens/s, Avg generation throughput: 467.9 tokens/s, Running: 37 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.7%, Prefix cache hit rate: 31.7%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35016 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45032 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45052 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35016 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45068 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45094 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45098 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45052 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35028 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34956 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48428 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34928 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34956 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48428 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34928 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34956 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34948 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:01:20 [loggers.py:248] Engine 000: Avg prompt throughput: 851.5 tokens/s, Avg generation throughput: 612.7 tokens/s, Running: 52 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.7%, Prefix cache hit rate: 39.9%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44926 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34948 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39710 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39732 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39748 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39750 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34948 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39764 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39710 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39764 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:01:30 [loggers.py:248] Engine 000: Avg prompt throughput: 641.7 tokens/s, Avg generation throughput: 700.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.6%, Prefix cache hit rate: 44.9%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35016 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34948 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44968 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44926 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44930 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39748 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34972 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44968 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44930 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55768 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34972 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55770 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55784 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55800 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55768 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44926 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55816 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34956 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55830 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48428 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55842 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:01:40 [loggers.py:248] Engine 000: Avg prompt throughput: 589.3 tokens/s, Avg generation throughput: 739.3 tokens/s, Running: 64 reqs, Waiting: 12 reqs, GPU KV cache usage: 19.5%, Prefix cache hit rate: 47.2%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55844 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55854 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55800 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45068 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55864 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55878 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55880 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55894 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42772 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39750 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43824 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55768 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44926 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34956 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55816 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42780 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34948 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45098 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45052 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55830 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48428 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43834 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43838 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42772 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34966 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:01:50 [loggers.py:248] Engine 000: Avg prompt throughput: 640.6 tokens/s, Avg generation throughput: 684.7 tokens/s, Running: 64 reqs, Waiting: 23 reqs, GPU KV cache usage: 18.8%, Prefix cache hit rate: 43.6%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39750 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55878 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34928 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55880 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55768 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34956 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45032 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45094 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45052 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43834 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55816 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42780 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51174 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43824 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:02:00 [loggers.py:248] Engine 000: Avg prompt throughput: 806.4 tokens/s, Avg generation throughput: 665.4 tokens/s, Running: 64 reqs, Waiting: 25 reqs, GPU KV cache usage: 19.7%, Prefix cache hit rate: 39.8%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34966 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35028 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55880 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55830 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51190 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51206 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42772 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55864 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34956 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55894 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54094 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54100 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34948 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45016 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55784 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44968 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39748 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55816 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45032 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39732 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54128 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51174 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54158 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54174 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43824 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:02:10 [loggers.py:248] Engine 000: Avg prompt throughput: 585.7 tokens/s, Avg generation throughput: 678.4 tokens/s, Running: 63 reqs, Waiting: 41 reqs, GPU KV cache usage: 19.0%, Prefix cache hit rate: 37.5%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55842 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43834 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55800 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54182 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54190 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44926 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54196 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34956 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34616 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34632 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34642 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34646 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34658 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34668 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35028 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55894 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35016 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45094 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55854 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48428 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34686 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34966 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45068 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34696 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:02:20 [loggers.py:248] Engine 000: Avg prompt throughput: 403.5 tokens/s, Avg generation throughput: 710.4 tokens/s, Running: 63 reqs, Waiting: 59 reqs, GPU KV cache usage: 17.3%, Prefix cache hit rate: 36.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34714 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34716 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34724 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39710 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55770 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39732 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55768 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44930 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51174 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34972 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46690 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46704 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43838 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42772 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42780 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46750 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:02:30 [loggers.py:248] Engine 000: Avg prompt throughput: 391.7 tokens/s, Avg generation throughput: 742.4 tokens/s, Running: 64 reqs, Waiting: 73 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 34.6%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46772 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51206 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46800 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46810 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46816 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46830 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46844 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46846 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54100 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34956 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54190 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34948 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34632 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34928 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34668 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34646 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54174 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55834 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34642 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44968 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55864 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:02:40 [loggers.py:248] Engine 000: Avg prompt throughput: 642.3 tokens/s, Avg generation throughput: 691.1 tokens/s, Running: 64 reqs, Waiting: 78 reqs, GPU KV cache usage: 14.3%, Prefix cache hit rate: 32.7%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39764 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55844 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45052 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45032 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55880 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54182 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55830 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55878 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34724 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55894 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34696 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34716 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43834 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54158 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55816 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39750 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35016 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44930 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55768 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34972 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54196 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:02:50 [loggers.py:248] Engine 000: Avg prompt throughput: 672.0 tokens/s, Avg generation throughput: 697.6 tokens/s, Running: 64 reqs, Waiting: 72 reqs, GPU KV cache usage: 14.8%, Prefix cache hit rate: 30.8%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54128 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43838 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45068 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55784 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42772 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34616 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46772 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34966 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34714 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46846 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51174 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51206 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34632 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:03:00 [loggers.py:248] Engine 000: Avg prompt throughput: 915.7 tokens/s, Avg generation throughput: 640.0 tokens/s, Running: 61 reqs, Waiting: 77 reqs, GPU KV cache usage: 16.6%, Prefix cache hit rate: 28.6%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34956 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46810 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35028 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34948 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54190 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34668 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45098 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55834 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45016 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34658 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34642 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55800 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44926 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39748 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34646 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45032 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55878 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34696 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55854 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45094 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39750 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34686 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55770 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39710 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:03:10 [loggers.py:248] Engine 000: Avg prompt throughput: 689.9 tokens/s, Avg generation throughput: 691.1 tokens/s, Running: 63 reqs, Waiting: 84 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 27.2%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35016 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54128 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55830 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54182 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46844 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54196 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51190 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45068 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55880 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55816 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55864 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:03:20 [loggers.py:248] Engine 000: Avg prompt throughput: 771.4 tokens/s, Avg generation throughput: 678.3 tokens/s, Running: 64 reqs, Waiting: 81 reqs, GPU KV cache usage: 16.4%, Prefix cache hit rate: 25.7%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46772 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39732 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55894 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34714 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46750 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39764 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55842 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48428 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34956 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34716 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60372 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60380 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60394 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60398 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60408 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60412 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34616 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54158 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54174 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34658 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:03:30 [loggers.py:248] Engine 000: Avg prompt throughput: 562.2 tokens/s, Avg generation throughput: 697.6 tokens/s, Running: 64 reqs, Waiting: 79 reqs, GPU KV cache usage: 15.8%, Prefix cache hit rate: 24.9%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55784 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55800 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34646 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46830 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45052 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34668 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45032 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51206 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55878 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43824 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55844 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34686 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46810 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34928 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46800 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48014 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44926 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44968 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39710 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39748 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55770 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35016 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:03:40 [loggers.py:248] Engine 000: Avg prompt throughput: 1037.6 tokens/s, Avg generation throughput: 659.2 tokens/s, Running: 62 reqs, Waiting: 73 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 23.3%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46690 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51188 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45016 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34948 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44930 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:51174 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55834 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:42728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46728 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55864 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:45094 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:43838 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34714 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34642 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55894 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:54196 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:55816 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34966 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36360 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:48428 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36374 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:46762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39764 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60372 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36382 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36390 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36394 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:34616 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36402 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:44988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36416 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36420 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:35042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36430 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36444 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36460 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:03:50 [loggers.py:248] Engine 000: Avg prompt throughput: 638.7 tokens/s, Avg generation throughput: 684.7 tokens/s, Running: 64 reqs, Waiting: 100 reqs, GPU KV cache usage: 17.4%, Prefix cache hit rate: 22.4%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:36466 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:60412 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: 127.0.0.1:39778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:04:00 [loggers.py:248] Engine 000: Avg prompt throughput: 312.9 tokens/s, Avg generation throughput: 710.4 tokens/s, Running: 62 reqs, Waiting: 83 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 21.9%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:04:10 [loggers.py:248] Engine 000: Avg prompt throughput: 819.3 tokens/s, Avg generation throughput: 684.8 tokens/s, Running: 62 reqs, Waiting: 51 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 20.9%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:04:20 [loggers.py:248] Engine 000: Avg prompt throughput: 680.3 tokens/s, Avg generation throughput: 697.6 tokens/s, Running: 64 reqs, Waiting: 12 reqs, GPU KV cache usage: 16.6%, Prefix cache hit rate: 20.2%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:04:30 [loggers.py:248] Engine 000: Avg prompt throughput: 139.3 tokens/s, Avg generation throughput: 704.4 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.2%, Prefix cache hit rate: 20.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:04:40 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 471.3 tokens/s, Running: 21 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.0%, Prefix cache hit rate: 20.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:04:50 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 259.9 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 20.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:05:00 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 128.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 20.0%
|
|
[0;36m(APIServer pid=16712)[0;0m INFO 12-09 20:05:01 [launcher.py:110] Shutting down FastAPI HTTP server.
|
|
[rank0]:[W1209 20:05:01.182181928 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: Shutting down
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: Waiting for application shutdown.
|
|
[0;36m(APIServer pid=16712)[0;0m INFO: Application shutdown complete.
|