1043 lines
116 KiB
Plaintext
1043 lines
116 KiB
Plaintext
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:38:31 [api_server.py:1351] vLLM API server version 0.11.2.dev690+g67475a6e8.d20251209
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:38:31 [utils.py:253] non-default args: {'model_tag': 'RedHatAI/Qwen3-14B-FP8-dynamic', 'host': '127.0.0.1', 'model': 'RedHatAI/Qwen3-14B-FP8-dynamic', 'trust_remote_code': True, 'max_model_len': 32768, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 64}
|
|
[0;36m(APIServer pid=15688)[0;0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:38:35 [model.py:629] Resolved architecture: Qwen3ForCausalLM
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:38:35 [model.py:1755] Using max model len 32768
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:38:35 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
|
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:38:39 [core.py:93] Initializing a V1 LLM engine (v0.11.2.dev690+g67475a6e8.d20251209) with config: model='RedHatAI/Qwen3-14B-FP8-dynamic', speculative_config=None, tokenizer='RedHatAI/Qwen3-14B-FP8-dynamic', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=RedHatAI/Qwen3-14B-FP8-dynamic, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:38:39 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.1.122:56369 backend=nccl
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:38:39 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:38:39 [gpu_model_runner.py:3544] Starting to load model RedHatAI/Qwen3-14B-FP8-dynamic...
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:38:40 [rocm.py:320] Using Triton Attention backend on V1 engine.
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m
|
|
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m
|
|
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:00, 4.28it/s]
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m
|
|
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:01<00:01, 1.55it/s]
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m
|
|
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:02<00:00, 1.26it/s]
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m
|
|
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00, 1.21it/s]
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m
|
|
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00, 1.33it/s]
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:38:43 [default_loader.py:308] Loading weights took 3.08 seconds
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:38:44 [gpu_model_runner.py:3626] Model loading took 15.4180 GiB memory and 3.812286 seconds
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:38:49 [backends.py:616] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/1372a8f1b2/rank_0_0/backbone for vLLM's torch.compile
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:38:49 [backends.py:676] Dynamo bytecode transform time: 5.17 s
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:38:55 [backends.py:243] Cache the graph of compile range (1, 2048) for later use
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:39:19 [backends.py:260] Compiling a graph for compile range (1, 2048) takes 27.34 s
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:39:19 [monitor.py:34] torch.compile takes 32.51 s in total
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:39:21 [gpu_worker.py:364] Available KV cache memory: 13.76 GiB
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:39:22 [kv_cache_utils.py:1287] GPU KV cache size: 90,160 tokens
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:39:22 [kv_cache_utils.py:1292] Maximum concurrency for 32,768 tokens per request: 2.75x
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/19 [00:00<?, ?it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 11%|█ | 2/19 [00:00<00:00, 17.99it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 26%|██▋ | 5/19 [00:00<00:00, 20.06it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 42%|████▏ | 8/19 [00:00<00:00, 20.85it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 58%|█████▊ | 11/19 [00:00<00:00, 20.46it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 74%|███████▎ | 14/19 [00:00<00:00, 20.64it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 89%|████████▉ | 17/19 [00:00<00:00, 21.07it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:00<00:00, 20.83it/s]
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m
|
|
Capturing CUDA graphs (decode, FULL): 0%| | 0/11 [00:00<?, ?it/s]
|
|
Capturing CUDA graphs (decode, FULL): 18%|█▊ | 2/11 [00:00<00:00, 18.50it/s]
|
|
Capturing CUDA graphs (decode, FULL): 45%|████▌ | 5/11 [00:00<00:00, 20.67it/s]
|
|
Capturing CUDA graphs (decode, FULL): 73%|███████▎ | 8/11 [00:00<00:00, 20.73it/s]
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 21.18it/s]
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 20.88it/s]
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:39:24 [gpu_model_runner.py:4548] Graph capturing finished in 2 secs, took 1.61 GiB
|
|
[0;36m(EngineCore_DP0 pid=15850)[0;0m INFO 12-12 17:39:24 [core.py:256] init engine (profile, create kv cache, warmup model) took 40.43 seconds
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [api_server.py:1099] Supported tasks: ['generate']
|
|
[0;36m(APIServer pid=15688)[0;0m WARNING 12-12 17:39:26 [model.py:1581] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [serving_responses.py:197] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [serving_chat.py:133] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [serving_completion.py:73] Using default completion sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [serving_chat.py:133] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [api_server.py:1425] Starting vLLM API server 0 on http://127.0.0.1:8000
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:38] Available routes are:
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /docs, Methods: HEAD, GET
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /tokenize, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /detokenize, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /pause, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /resume, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /is_paused, Methods: GET
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /metrics, Methods: GET
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /health, Methods: GET
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /load, Methods: GET
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /v1/models, Methods: GET
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /version, Methods: GET
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /v1/responses, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /v1/messages, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /v1/completions, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /ping, Methods: GET
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /ping, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /invocations, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /classify, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /v1/embeddings, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /score, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /v1/score, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /rerank, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /v1/rerank, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /v2/rerank, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:26 [launcher.py:46] Route: /pooling, Methods: POST
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: Started server process [15688]
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: Waiting for application startup.
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: Application startup complete.
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53572 - "GET /v1/models HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35226 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:46 [loggers.py:248] Engine 000: Avg prompt throughput: 8.3 tokens/s, Avg generation throughput: 28.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35250 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:39:56 [loggers.py:248] Engine 000: Avg prompt throughput: 170.5 tokens/s, Avg generation throughput: 115.7 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35226 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:40:06 [loggers.py:248] Engine 000: Avg prompt throughput: 249.4 tokens/s, Avg generation throughput: 172.1 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35226 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:40:16 [loggers.py:248] Engine 000: Avg prompt throughput: 214.4 tokens/s, Avg generation throughput: 207.3 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35250 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:40:26 [loggers.py:248] Engine 000: Avg prompt throughput: 217.3 tokens/s, Avg generation throughput: 176.2 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35226 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:40:36 [loggers.py:248] Engine 000: Avg prompt throughput: 379.7 tokens/s, Avg generation throughput: 213.2 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35250 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35226 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:40:46 [loggers.py:248] Engine 000: Avg prompt throughput: 400.9 tokens/s, Avg generation throughput: 230.5 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35226 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35226 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:40:56 [loggers.py:248] Engine 000: Avg prompt throughput: 215.3 tokens/s, Avg generation throughput: 199.2 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35226 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38490 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38512 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38512 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38512 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:42860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:41:06 [loggers.py:248] Engine 000: Avg prompt throughput: 298.4 tokens/s, Avg generation throughput: 221.9 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35226 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:42868 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:42878 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38490 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38490 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:41:16 [loggers.py:248] Engine 000: Avg prompt throughput: 301.7 tokens/s, Avg generation throughput: 290.1 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.4%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38490 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38490 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38512 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38490 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38512 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:41:26 [loggers.py:248] Engine 000: Avg prompt throughput: 160.7 tokens/s, Avg generation throughput: 310.8 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:42860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35226 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:42868 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:42860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:41:36 [loggers.py:248] Engine 000: Avg prompt throughput: 36.4 tokens/s, Avg generation throughput: 243.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38512 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:42860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38512 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50310 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:41:46 [loggers.py:248] Engine 000: Avg prompt throughput: 232.2 tokens/s, Avg generation throughput: 194.9 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50324 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50338 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38512 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50324 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50310 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:41:56 [loggers.py:248] Engine 000: Avg prompt throughput: 219.2 tokens/s, Avg generation throughput: 242.5 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:46960 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50324 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:38512 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:42:06 [loggers.py:248] Engine 000: Avg prompt throughput: 157.6 tokens/s, Avg generation throughput: 242.6 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50310 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50324 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:42:16 [loggers.py:248] Engine 000: Avg prompt throughput: 65.8 tokens/s, Avg generation throughput: 149.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35214 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:42860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50338 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:41468 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:41472 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:51062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:42:26 [loggers.py:248] Engine 000: Avg prompt throughput: 180.0 tokens/s, Avg generation throughput: 183.9 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:35248 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:41468 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:41468 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:42860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45780 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:42860 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:42:36 [loggers.py:248] Engine 000: Avg prompt throughput: 189.6 tokens/s, Avg generation throughput: 176.4 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50324 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50324 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50324 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:41458 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:42:46 [loggers.py:248] Engine 000: Avg prompt throughput: 139.6 tokens/s, Avg generation throughput: 256.8 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:42:56 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 143.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:43:06 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:55674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:55674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37434 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37446 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37462 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37478 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:43:16 [loggers.py:248] Engine 000: Avg prompt throughput: 137.1 tokens/s, Avg generation throughput: 49.3 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 3.2%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37514 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37530 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37552 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:55674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37462 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37514 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:55674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37560 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37578 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37446 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53958 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53958 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37560 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37434 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53994 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37434 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37434 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:43:26 [loggers.py:248] Engine 000: Avg prompt throughput: 1062.8 tokens/s, Avg generation throughput: 406.1 tokens/s, Running: 30 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.5%, Prefix cache hit rate: 22.9%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:55674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37578 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54126 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37552 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37530 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:55674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54126 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37446 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53994 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36258 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36264 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36278 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:43:36 [loggers.py:248] Engine 000: Avg prompt throughput: 1072.7 tokens/s, Avg generation throughput: 742.0 tokens/s, Running: 37 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.8%, Prefix cache hit rate: 35.7%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37514 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37552 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:55674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37514 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37530 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37578 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:43:46 [loggers.py:248] Engine 000: Avg prompt throughput: 825.1 tokens/s, Avg generation throughput: 825.9 tokens/s, Running: 49 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 42.9%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37578 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37560 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37478 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37434 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53958 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37462 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53994 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37552 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37434 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37478 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37530 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:43:56 [loggers.py:248] Engine 000: Avg prompt throughput: 439.4 tokens/s, Avg generation throughput: 894.3 tokens/s, Running: 39 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.2%, Prefix cache hit rate: 46.1%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37514 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36264 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37446 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37578 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37478 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36258 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:55674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36278 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37478 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37552 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37434 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37530 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37462 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:55674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:44:06 [loggers.py:248] Engine 000: Avg prompt throughput: 1004.6 tokens/s, Avg generation throughput: 782.2 tokens/s, Running: 44 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.1%, Prefix cache hit rate: 43.9%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37552 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37560 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37552 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37478 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54126 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37478 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40328 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40332 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37462 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40328 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53994 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40344 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:44:16 [loggers.py:248] Engine 000: Avg prompt throughput: 1023.4 tokens/s, Avg generation throughput: 865.6 tokens/s, Running: 55 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.8%, Prefix cache hit rate: 39.2%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:55674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37514 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37434 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:55674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:55674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40344 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54126 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37578 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:44:26 [loggers.py:248] Engine 000: Avg prompt throughput: 551.7 tokens/s, Avg generation throughput: 921.0 tokens/s, Running: 43 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.9%, Prefix cache hit rate: 37.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53958 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37514 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37552 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36278 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37530 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36258 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36264 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37446 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40328 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40332 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36264 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37434 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37478 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45062 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40328 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37462 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48218 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53994 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:44:36 [loggers.py:248] Engine 000: Avg prompt throughput: 891.5 tokens/s, Avg generation throughput: 873.5 tokens/s, Running: 58 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.0%, Prefix cache hit rate: 33.9%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40328 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48240 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48254 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48266 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48276 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37578 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48254 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37552 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48266 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48240 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37530 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48240 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40332 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40344 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36278 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:44:46 [loggers.py:248] Engine 000: Avg prompt throughput: 652.9 tokens/s, Avg generation throughput: 1085.5 tokens/s, Running: 56 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.4%, Prefix cache hit rate: 32.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53958 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45092 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37514 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37560 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54070 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:55674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37446 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37530 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40328 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37434 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37514 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48254 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:55674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48276 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:44:56 [loggers.py:248] Engine 000: Avg prompt throughput: 907.6 tokens/s, Avg generation throughput: 903.2 tokens/s, Running: 49 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.9%, Prefix cache hit rate: 29.7%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37462 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54114 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:55674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36264 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40328 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48266 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37560 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37184 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37434 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48276 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48254 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36258 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53994 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53958 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37578 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:45:06 [loggers.py:248] Engine 000: Avg prompt throughput: 916.0 tokens/s, Avg generation throughput: 751.3 tokens/s, Running: 35 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.5%, Prefix cache hit rate: 27.6%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36264 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40328 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36258 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54126 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37446 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37578 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37552 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48276 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36258 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37514 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40332 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48266 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36264 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40328 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36258 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:45:16 [loggers.py:248] Engine 000: Avg prompt throughput: 471.8 tokens/s, Avg generation throughput: 760.2 tokens/s, Running: 36 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.6%, Prefix cache hit rate: 26.7%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53994 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37560 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48266 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40332 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37478 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37446 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37560 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53958 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48266 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48254 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48276 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37560 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48240 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37578 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40344 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48254 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37446 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53994 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:33978 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:45:26 [loggers.py:248] Engine 000: Avg prompt throughput: 847.9 tokens/s, Avg generation throughput: 755.8 tokens/s, Running: 49 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.8%, Prefix cache hit rate: 25.3%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:33988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:34004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:34010 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:34020 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37578 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:34004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48266 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37514 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53994 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37578 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37478 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48276 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:34004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:33978 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:33988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36264 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48276 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:34020 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36258 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37446 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48254 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:45:36 [loggers.py:248] Engine 000: Avg prompt throughput: 971.8 tokens/s, Avg generation throughput: 775.8 tokens/s, Running: 34 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.6%, Prefix cache hit rate: 23.7%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36264 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40328 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37578 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:34004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:34020 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54126 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36258 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53958 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37446 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54080 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:34004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37462 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37552 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40332 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48240 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:34004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37478 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:45:46 [loggers.py:248] Engine 000: Avg prompt throughput: 509.6 tokens/s, Avg generation throughput: 749.1 tokens/s, Running: 40 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.8%, Prefix cache hit rate: 23.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40344 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48240 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54126 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40332 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37514 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36264 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54048 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40344 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:34004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37514 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37550 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36264 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:33988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48266 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40344 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48276 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40328 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36264 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36258 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37462 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40328 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:45:56 [loggers.py:248] Engine 000: Avg prompt throughput: 741.8 tokens/s, Avg generation throughput: 780.6 tokens/s, Running: 44 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 22.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53994 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37552 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48240 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:34004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37446 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:53984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:34004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37446 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48254 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:34004 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37446 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:33988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48254 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54126 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:34020 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37478 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54126 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37560 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48254 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54000 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:46:06 [loggers.py:248] Engine 000: Avg prompt throughput: 949.2 tokens/s, Avg generation throughput: 797.3 tokens/s, Running: 45 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 20.8%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48276 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48254 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50212 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50212 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48276 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40344 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54024 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:48266 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37566 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36264 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37462 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37578 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45118 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50220 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50222 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50212 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54036 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:40332 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:50216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:36258 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:54038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:45046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO: 127.0.0.1:37514 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:46:16 [loggers.py:248] Engine 000: Avg prompt throughput: 692.8 tokens/s, Avg generation throughput: 925.9 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.7%, Prefix cache hit rate: 20.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:46:26 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 632.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.6%, Prefix cache hit rate: 20.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:46:36 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 210.0 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.5%, Prefix cache hit rate: 20.0%
|
|
[0;36m(APIServer pid=15688)[0;0m INFO 12-12 17:46:45 [launcher.py:110] Shutting down FastAPI HTTP server.
|
|
[rank0]:[W1212 17:46:45.021549801 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|