1048 lines
117 KiB
Plaintext
1048 lines
117 KiB
Plaintext
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:31:49 [api_server.py:1351] vLLM API server version 0.11.2.dev690+g67475a6e8.d20251209
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:31:49 [utils.py:253] non-default args: {'model_tag': 'RedHatAI/Qwen3-14B-FP8-dynamic', 'host': '127.0.0.1', 'model': 'RedHatAI/Qwen3-14B-FP8-dynamic', 'trust_remote_code': True, 'max_model_len': 32768, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 64}
|
|
[0;36m(APIServer pid=11550)[0;0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:31:53 [model.py:629] Resolved architecture: Qwen3ForCausalLM
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:31:53 [model.py:1755] Using max model len 32768
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:31:53 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
|
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:31:57 [core.py:93] Initializing a V1 LLM engine (v0.11.2.dev690+g67475a6e8.d20251209) with config: model='RedHatAI/Qwen3-14B-FP8-dynamic', speculative_config=None, tokenizer='RedHatAI/Qwen3-14B-FP8-dynamic', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=RedHatAI/Qwen3-14B-FP8-dynamic, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:31:57 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.1.122:56787 backend=nccl
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:31:57 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:31:58 [gpu_model_runner.py:3544] Starting to load model RedHatAI/Qwen3-14B-FP8-dynamic...
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:31:58 [rocm.py:320] Using Triton Attention backend on V1 engine.
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m
|
|
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m
|
|
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:00, 4.01it/s]
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m
|
|
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:01<00:01, 1.56it/s]
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m
|
|
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:02<00:00, 1.27it/s]
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m
|
|
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00, 1.18it/s]
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m
|
|
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00, 1.30it/s]
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:32:02 [default_loader.py:308] Loading weights took 3.14 seconds
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:32:02 [gpu_model_runner.py:3626] Model loading took 15.4180 GiB memory and 3.928491 seconds
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:32:08 [backends.py:616] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/1372a8f1b2/rank_0_0/backbone for vLLM's torch.compile
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:32:08 [backends.py:676] Dynamo bytecode transform time: 5.15 s
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:32:14 [backends.py:243] Cache the graph of compile range (1, 2048) for later use
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:32:38 [backends.py:260] Compiling a graph for compile range (1, 2048) takes 27.87 s
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:32:38 [monitor.py:34] torch.compile takes 33.02 s in total
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:32:41 [gpu_worker.py:364] Available KV cache memory: 11.80 GiB
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:32:41 [kv_cache_utils.py:1287] GPU KV cache size: 77,312 tokens
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:32:41 [kv_cache_utils.py:1292] Maximum concurrency for 32,768 tokens per request: 2.36x
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/19 [00:00<?, ?it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 11%|█ | 2/19 [00:00<00:01, 16.45it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 21%|██ | 4/19 [00:00<00:00, 17.79it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 32%|███▏ | 6/19 [00:00<00:00, 18.46it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 42%|████▏ | 8/19 [00:00<00:00, 18.86it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 53%|█████▎ | 10/19 [00:00<00:00, 18.66it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 63%|██████▎ | 12/19 [00:00<00:00, 18.71it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 74%|███████▎ | 14/19 [00:00<00:00, 18.88it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 84%|████████▍ | 16/19 [00:00<00:00, 19.09it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 95%|█████████▍| 18/19 [00:00<00:00, 19.35it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:01<00:00, 18.87it/s]
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m
|
|
Capturing CUDA graphs (decode, FULL): 0%| | 0/11 [00:00<?, ?it/s]
|
|
Capturing CUDA graphs (decode, FULL): 18%|█▊ | 2/11 [00:00<00:00, 17.28it/s]
|
|
Capturing CUDA graphs (decode, FULL): 36%|███▋ | 4/11 [00:00<00:00, 18.65it/s]
|
|
Capturing CUDA graphs (decode, FULL): 64%|██████▎ | 7/11 [00:00<00:00, 19.57it/s]
|
|
Capturing CUDA graphs (decode, FULL): 82%|████████▏ | 9/11 [00:00<00:00, 18.94it/s]
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 19.12it/s]
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 19.00it/s]
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:32:44 [gpu_model_runner.py:4548] Graph capturing finished in 2 secs, took 1.74 GiB
|
|
[0;36m(EngineCore_DP0 pid=11713)[0;0m INFO 12-09 19:32:44 [core.py:256] init engine (profile, create kv cache, warmup model) took 41.14 seconds
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [api_server.py:1099] Supported tasks: ['generate']
|
|
[0;36m(APIServer pid=11550)[0;0m WARNING 12-09 19:32:45 [model.py:1581] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [serving_responses.py:197] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [serving_chat.py:133] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [serving_completion.py:73] Using default completion sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [serving_chat.py:133] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [api_server.py:1425] Starting vLLM API server 0 on http://127.0.0.1:8000
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:38] Available routes are:
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /docs, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /tokenize, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /detokenize, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /pause, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /resume, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /is_paused, Methods: GET
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /metrics, Methods: GET
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /health, Methods: GET
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /load, Methods: GET
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /v1/models, Methods: GET
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /version, Methods: GET
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /v1/responses, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /v1/messages, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /v1/completions, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /ping, Methods: GET
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /ping, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /invocations, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /classify, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /v1/embeddings, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /score, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /v1/score, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /rerank, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /v1/rerank, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /v2/rerank, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:32:45 [launcher.py:46] Route: /pooling, Methods: POST
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: Started server process [11550]
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: Waiting for application startup.
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: Application startup complete.
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:42186 - "GET /v1/models HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:48102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:48102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:33:06 [loggers.py:248] Engine 000: Avg prompt throughput: 4.8 tokens/s, Avg generation throughput: 16.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46142 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46148 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:48102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:48102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:33:16 [loggers.py:248] Engine 000: Avg prompt throughput: 133.8 tokens/s, Avg generation throughput: 102.9 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51374 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51374 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51382 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51374 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:33:26 [loggers.py:248] Engine 000: Avg prompt throughput: 224.6 tokens/s, Avg generation throughput: 142.8 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35902 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:48102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:34506 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51382 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:33:36 [loggers.py:248] Engine 000: Avg prompt throughput: 279.4 tokens/s, Avg generation throughput: 198.1 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:48102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:34506 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51374 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:34506 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35902 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51374 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:33:46 [loggers.py:248] Engine 000: Avg prompt throughput: 217.3 tokens/s, Avg generation throughput: 185.0 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46142 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51374 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:48102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46142 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36576 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:33:56 [loggers.py:248] Engine 000: Avg prompt throughput: 374.5 tokens/s, Avg generation throughput: 205.5 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.9%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51382 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:48102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:48102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:48102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51374 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36576 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:48102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43240 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:34:06 [loggers.py:248] Engine 000: Avg prompt throughput: 343.4 tokens/s, Avg generation throughput: 232.9 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.2%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43240 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:34506 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:34:16 [loggers.py:248] Engine 000: Avg prompt throughput: 278.0 tokens/s, Avg generation throughput: 213.0 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46142 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36576 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51382 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:34506 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36576 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51374 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:34:26 [loggers.py:248] Engine 000: Avg prompt throughput: 295.0 tokens/s, Avg generation throughput: 198.9 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.0%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:34:36 [loggers.py:248] Engine 000: Avg prompt throughput: 176.4 tokens/s, Avg generation throughput: 279.2 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:34506 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36576 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36576 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:34506 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36576 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:34:46 [loggers.py:248] Engine 000: Avg prompt throughput: 284.7 tokens/s, Avg generation throughput: 292.9 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.2%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51374 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51382 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:34:56 [loggers.py:248] Engine 000: Avg prompt throughput: 40.2 tokens/s, Avg generation throughput: 289.2 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46142 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:35:06 [loggers.py:248] Engine 000: Avg prompt throughput: 229.6 tokens/s, Avg generation throughput: 186.7 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51920 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51928 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51920 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51950 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51382 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:35:16 [loggers.py:248] Engine 000: Avg prompt throughput: 219.1 tokens/s, Avg generation throughput: 250.1 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.7%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51928 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51950 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51950 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:35:26 [loggers.py:248] Engine 000: Avg prompt throughput: 156.4 tokens/s, Avg generation throughput: 232.8 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.4%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46142 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51950 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:35:36 [loggers.py:248] Engine 000: Avg prompt throughput: 70.6 tokens/s, Avg generation throughput: 158.7 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51950 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:35904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51018 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50998 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51010 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:35:46 [loggers.py:248] Engine 000: Avg prompt throughput: 172.9 tokens/s, Avg generation throughput: 180.7 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.9%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:36546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46160 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:35:56 [loggers.py:248] Engine 000: Avg prompt throughput: 73.3 tokens/s, Avg generation throughput: 175.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46166 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46182 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46190 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:36:06 [loggers.py:248] Engine 000: Avg prompt throughput: 263.0 tokens/s, Avg generation throughput: 245.3 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.2%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:36:16 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 176.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:36:26 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 74.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:55034 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:36:36 [loggers.py:248] Engine 000: Avg prompt throughput: 1.2 tokens/s, Avg generation throughput: 9.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:55034 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51054 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51084 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51152 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51168 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51180 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51152 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:55034 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:36:46 [loggers.py:248] Engine 000: Avg prompt throughput: 516.5 tokens/s, Avg generation throughput: 155.4 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 3.8%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51152 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37170 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51152 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37228 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51054 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37290 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51168 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43822 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43826 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:36:56 [loggers.py:248] Engine 000: Avg prompt throughput: 1210.6 tokens/s, Avg generation throughput: 535.3 tokens/s, Running: 36 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.8%, Prefix cache hit rate: 18.6%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43826 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43822 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51180 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51054 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43834 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51180 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37170 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43880 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51180 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51152 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43822 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37290 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43822 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:37:06 [loggers.py:248] Engine 000: Avg prompt throughput: 909.5 tokens/s, Avg generation throughput: 760.7 tokens/s, Running: 38 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 29.5%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:55034 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37228 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43834 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:55034 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51054 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43826 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50916 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50932 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50936 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50942 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51054 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50916 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50936 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:47596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50936 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:37:16 [loggers.py:248] Engine 000: Avg prompt throughput: 653.2 tokens/s, Avg generation throughput: 869.0 tokens/s, Running: 51 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.3%, Prefix cache hit rate: 35.6%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50936 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37290 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37228 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:55034 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37170 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51054 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51084 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43822 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43880 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:47596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50916 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51180 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43880 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:47596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:37:26 [loggers.py:248] Engine 000: Avg prompt throughput: 684.2 tokens/s, Avg generation throughput: 840.2 tokens/s, Running: 46 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.8%, Prefix cache hit rate: 39.1%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37290 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37228 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:55000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43826 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51152 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50942 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51152 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43826 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43822 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:37:36 [loggers.py:248] Engine 000: Avg prompt throughput: 946.5 tokens/s, Avg generation throughput: 783.9 tokens/s, Running: 50 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.7%, Prefix cache hit rate: 34.9%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51168 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45486 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45508 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51168 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45538 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45508 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43834 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45554 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:55000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45554 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51168 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51084 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:37:46 [loggers.py:248] Engine 000: Avg prompt throughput: 1051.5 tokens/s, Avg generation throughput: 876.2 tokens/s, Running: 56 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.4%, Prefix cache hit rate: 31.2%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37228 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51054 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51084 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45554 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:55000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51152 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50942 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:55034 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37290 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45554 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37170 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43822 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51054 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37290 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:37:56 [loggers.py:248] Engine 000: Avg prompt throughput: 376.9 tokens/s, Avg generation throughput: 905.8 tokens/s, Running: 50 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.3%, Prefix cache hit rate: 30.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50942 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37228 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45554 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51180 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43880 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43828 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50932 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50916 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51180 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37290 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:47596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43880 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51180 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43834 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45486 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56568 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56584 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56598 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56608 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56614 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56568 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56584 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:55000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56608 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51152 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50936 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:38:06 [loggers.py:248] Engine 000: Avg prompt throughput: 960.4 tokens/s, Avg generation throughput: 956.4 tokens/s, Running: 64 reqs, Waiting: 2 reqs, GPU KV cache usage: 22.9%, Prefix cache hit rate: 27.4%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51168 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45554 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43880 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51054 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56598 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45508 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51084 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56584 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45554 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50942 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56598 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45538 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45486 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51054 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50932 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45554 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:38:16 [loggers.py:248] Engine 000: Avg prompt throughput: 632.8 tokens/s, Avg generation throughput: 1012.1 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 25.9%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:55034 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56584 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43822 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50942 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45538 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37170 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37290 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43826 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56584 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:55034 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45538 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43880 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43834 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56568 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56598 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50942 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56552 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:38:26 [loggers.py:248] Engine 000: Avg prompt throughput: 1205.1 tokens/s, Avg generation throughput: 825.3 tokens/s, Running: 56 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 23.5%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45486 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56614 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37290 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:47596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50936 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56598 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45554 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37170 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45508 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50936 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43822 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:38:36 [loggers.py:248] Engine 000: Avg prompt throughput: 534.7 tokens/s, Avg generation throughput: 792.4 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.4%, Prefix cache hit rate: 22.6%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37170 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51084 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50942 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:47596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37170 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56584 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43826 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50916 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45554 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:38:46 [loggers.py:248] Engine 000: Avg prompt throughput: 765.4 tokens/s, Avg generation throughput: 734.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.8%, Prefix cache hit rate: 21.4%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:47596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51180 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45554 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51054 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56598 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:55000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45508 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46480 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46494 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45508 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50942 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45508 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50942 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51168 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43826 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45508 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:38:56 [loggers.py:248] Engine 000: Avg prompt throughput: 697.9 tokens/s, Avg generation throughput: 832.7 tokens/s, Running: 49 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 20.5%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51180 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46480 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50936 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46480 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56614 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51180 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46494 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:39:06 [loggers.py:248] Engine 000: Avg prompt throughput: 877.0 tokens/s, Avg generation throughput: 706.3 tokens/s, Running: 42 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.1%, Prefix cache hit rate: 19.4%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:55000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45508 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43834 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51084 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:47596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45508 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46494 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50916 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:47596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37170 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51084 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51054 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56598 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:39:16 [loggers.py:248] Engine 000: Avg prompt throughput: 721.0 tokens/s, Avg generation throughput: 777.6 tokens/s, Running: 44 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.1%, Prefix cache hit rate: 18.6%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37202 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56614 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43826 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56598 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56614 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50942 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51042 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51168 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43834 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43862 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45508 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:39:26 [loggers.py:248] Engine 000: Avg prompt throughput: 555.0 tokens/s, Avg generation throughput: 818.7 tokens/s, Running: 50 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.5%, Prefix cache hit rate: 18.0%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45508 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50916 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43848 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45508 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56584 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50916 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45524 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43886 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49398 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49404 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49406 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37170 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46480 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50916 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49404 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50942 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:56598 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49406 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49398 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51180 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49404 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49418 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50942 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49404 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51180 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:54992 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49406 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49418 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43826 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51104 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:39:36 [loggers.py:248] Engine 000: Avg prompt throughput: 1128.5 tokens/s, Avg generation throughput: 789.6 tokens/s, Running: 52 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.7%, Prefix cache hit rate: 16.8%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:45554 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:50942 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:37216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:49418 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:51092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:46480 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43130 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43146 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43160 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: 127.0.0.1:43832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:39:46 [loggers.py:248] Engine 000: Avg prompt throughput: 242.3 tokens/s, Avg generation throughput: 916.4 tokens/s, Running: 37 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.9%, Prefix cache hit rate: 16.6%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:39:56 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 444.1 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.3%, Prefix cache hit rate: 16.6%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:40:06 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 157.8 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 16.6%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:40:16 [launcher.py:110] Shutting down FastAPI HTTP server.
|
|
[rank0]:[W1209 19:40:16.062937543 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: Shutting down
|
|
[0;36m(APIServer pid=11550)[0;0m INFO 12-09 19:40:16 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 72.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 16.6%
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: Waiting for application shutdown.
|
|
[0;36m(APIServer pid=11550)[0;0m INFO: Application shutdown complete.
|