1049 lines
117 KiB
Plaintext
1049 lines
117 KiB
Plaintext
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:45:29 [api_server.py:1351] vLLM API server version 0.11.2.dev690+g67475a6e8.d20251209
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:45:29 [utils.py:253] non-default args: {'model_tag': 'RedHatAI/gemma-3-12b-it-FP8-dynamic', 'host': '127.0.0.1', 'model': 'RedHatAI/gemma-3-12b-it-FP8-dynamic', 'trust_remote_code': True, 'max_model_len': 32768, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 64}
|
|
[0;36m(APIServer pid=5124)[0;0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:45:33 [model.py:629] Resolved architecture: Gemma3ForConditionalGeneration
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:45:33 [model.py:1755] Using max model len 32768
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:45:33 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
|
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:45:38 [core.py:93] Initializing a V1 LLM engine (v0.11.2.dev690+g67475a6e8.d20251209) with config: model='RedHatAI/gemma-3-12b-it-FP8-dynamic', speculative_config=None, tokenizer='RedHatAI/gemma-3-12b-it-FP8-dynamic', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=RedHatAI/gemma-3-12b-it-FP8-dynamic, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:45:40 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.1.122:45401 backend=nccl
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:45:40 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:45:45 [gpu_model_runner.py:3544] Starting to load model RedHatAI/gemma-3-12b-it-FP8-dynamic...
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m WARNING 12-11 18:45:46 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:45:46 [layer.py:524] Using AttentionBackendEnum.TORCH_SDPA for MultiHeadAttention in multimodal encoder.
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m WARNING 12-11 18:45:46 [activation.py:544] [ROCm] PyTorch's native GELU with tanh approximation is unstable. Falling back to GELU(approximate='none').
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:45:46 [rocm.py:320] Using Triton Attention backend on V1 engine.
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m WARNING 12-11 18:45:46 [activation.py:220] [ROCm] PyTorch's native GELU with tanh approximation is unstable with torch.compile. For native implementation, fallback to 'none' approximation. The custom kernel implementation is unaffected.
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m
|
|
Loading safetensors checkpoint shards: 0% Completed | 0/3 [00:00<?, ?it/s]
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m
|
|
Loading safetensors checkpoint shards: 33% Completed | 1/3 [00:01<00:03, 1.57s/it]
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m
|
|
Loading safetensors checkpoint shards: 67% Completed | 2/3 [00:02<00:01, 1.36s/it]
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m
|
|
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:03<00:00, 1.15s/it]
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m
|
|
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:03<00:00, 1.23s/it]
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:45:50 [default_loader.py:308] Loading weights took 3.74 seconds
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:45:50 [gpu_model_runner.py:3626] Model loading took 13.5273 GiB memory and 4.353595 seconds
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:45:51 [gpu_model_runner.py:4388] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 7 image items of the maximum feature size.
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:45:58 [backends.py:616] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/55ce54d8f2/rank_0_0/backbone for vLLM's torch.compile
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:45:58 [backends.py:676] Dynamo bytecode transform time: 6.40 s
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:46:02 [backends.py:243] Cache the graph of compile range (1, 2048) for later use
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:46:06 [backends.py:260] Compiling a graph for compile range (1, 2048) takes 4.76 s
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:46:06 [monitor.py:34] torch.compile takes 11.16 s in total
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:46:09 [gpu_worker.py:364] Available KV cache memory: 16.25 GiB
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:46:09 [kv_cache_utils.py:1287] GPU KV cache size: 44,368 tokens
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:46:09 [kv_cache_utils.py:1292] Maximum concurrency for 32,768 tokens per request: 5.52x
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/19 [00:00<?, ?it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 11%|█ | 2/19 [00:00<00:00, 19.11it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 26%|██▋ | 5/19 [00:00<00:00, 21.64it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 42%|████▏ | 8/19 [00:00<00:00, 22.45it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 58%|█████▊ | 11/19 [00:00<00:00, 23.48it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 74%|███████▎ | 14/19 [00:00<00:00, 24.20it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 89%|████████▉ | 17/19 [00:00<00:00, 24.89it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:00<00:00, 23.98it/s]
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m
|
|
Capturing CUDA graphs (decode, FULL): 0%| | 0/11 [00:00<?, ?it/s]
|
|
Capturing CUDA graphs (decode, FULL): 27%|██▋ | 3/11 [00:00<00:00, 21.95it/s]
|
|
Capturing CUDA graphs (decode, FULL): 55%|█████▍ | 6/11 [00:00<00:00, 24.62it/s]
|
|
Capturing CUDA graphs (decode, FULL): 82%|████████▏ | 9/11 [00:00<00:00, 24.80it/s]
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 24.71it/s]
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:46:11 [gpu_model_runner.py:4548] Graph capturing finished in 2 secs, took 1.38 GiB
|
|
[0;36m(EngineCore_DP0 pid=5288)[0;0m INFO 12-11 18:46:11 [core.py:256] init engine (profile, create kv cache, warmup model) took 20.95 seconds
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:13 [api_server.py:1099] Supported tasks: ['generate']
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [api_server.py:1425] Starting vLLM API server 0 on http://127.0.0.1:8000
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:38] Available routes are:
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /docs, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /tokenize, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /detokenize, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /pause, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /resume, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /is_paused, Methods: GET
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /metrics, Methods: GET
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /health, Methods: GET
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /load, Methods: GET
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /v1/models, Methods: GET
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /version, Methods: GET
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /v1/responses, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /v1/messages, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /v1/completions, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /ping, Methods: GET
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /ping, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /invocations, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /classify, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /v1/embeddings, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /score, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /v1/score, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /rerank, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /v1/rerank, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /v2/rerank, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:14 [launcher.py:46] Route: /pooling, Methods: POST
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: Started server process [5124]
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: Waiting for application startup.
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: Application startup complete.
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:46170 - "GET /v1/models HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45752 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:34 [loggers.py:248] Engine 000: Avg prompt throughput: 42.1 tokens/s, Avg generation throughput: 39.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45752 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:44 [loggers.py:248] Engine 000: Avg prompt throughput: 138.6 tokens/s, Avg generation throughput: 117.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:46:54 [loggers.py:248] Engine 000: Avg prompt throughput: 205.5 tokens/s, Avg generation throughput: 199.3 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.8%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45752 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:47:04 [loggers.py:248] Engine 000: Avg prompt throughput: 297.3 tokens/s, Avg generation throughput: 183.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.5%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45752 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45752 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:47:14 [loggers.py:248] Engine 000: Avg prompt throughput: 202.2 tokens/s, Avg generation throughput: 184.9 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45752 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:50720 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:50734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:47:24 [loggers.py:248] Engine 000: Avg prompt throughput: 397.0 tokens/s, Avg generation throughput: 183.5 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:50734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:50734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45752 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:50734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:40082 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:40096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:50734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:40096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:47:34 [loggers.py:248] Engine 000: Avg prompt throughput: 380.0 tokens/s, Avg generation throughput: 204.8 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.8%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43126 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:50720 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:50734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45752 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:50734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:47:44 [loggers.py:248] Engine 000: Avg prompt throughput: 170.5 tokens/s, Avg generation throughput: 216.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45752 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:40096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:40096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60838 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:47:54 [loggers.py:248] Engine 000: Avg prompt throughput: 397.2 tokens/s, Avg generation throughput: 183.9 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:40082 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:40096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:55596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:55600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:40096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:55612 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:48:04 [loggers.py:248] Engine 000: Avg prompt throughput: 230.9 tokens/s, Avg generation throughput: 256.9 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60838 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60838 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:48:14 [loggers.py:248] Engine 000: Avg prompt throughput: 292.5 tokens/s, Avg generation throughput: 284.6 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45752 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:40082 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:55600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:48:24 [loggers.py:248] Engine 000: Avg prompt throughput: 38.6 tokens/s, Avg generation throughput: 252.6 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:55596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:40096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:40096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:40082 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:41336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:41348 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:48:34 [loggers.py:248] Engine 000: Avg prompt throughput: 265.7 tokens/s, Avg generation throughput: 198.2 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.8%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60832 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59604 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59614 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59618 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59604 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:40096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59604 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:55596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:48:44 [loggers.py:248] Engine 000: Avg prompt throughput: 290.7 tokens/s, Avg generation throughput: 235.3 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:40096 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45752 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:41348 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:41336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:41348 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59618 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:48:54 [loggers.py:248] Engine 000: Avg prompt throughput: 106.1 tokens/s, Avg generation throughput: 217.9 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:41336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59614 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:41348 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:49:04 [loggers.py:248] Engine 000: Avg prompt throughput: 109.9 tokens/s, Avg generation throughput: 195.2 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.9%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:41348 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59614 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45752 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:41336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59604 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:49:14 [loggers.py:248] Engine 000: Avg prompt throughput: 110.6 tokens/s, Avg generation throughput: 176.9 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:43116 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:41336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59572 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:41336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:41336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59578 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59590 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:49:24 [loggers.py:248] Engine 000: Avg prompt throughput: 242.3 tokens/s, Avg generation throughput: 192.2 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:45752 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:41348 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59590 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59604 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:49574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:49:34 [loggers.py:248] Engine 000: Avg prompt throughput: 126.3 tokens/s, Avg generation throughput: 228.8 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:49:44 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 130.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:49:54 [loggers.py:248] Engine 000: Avg prompt throughput: 1.1 tokens/s, Avg generation throughput: 17.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58538 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58558 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58564 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58572 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58590 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58612 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58564 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58590 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58656 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58590 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58666 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:50:04 [loggers.py:248] Engine 000: Avg prompt throughput: 459.6 tokens/s, Avg generation throughput: 178.1 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58558 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58666 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37708 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37722 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58666 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37764 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37806 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58590 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37810 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37830 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37842 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37842 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:50:14 [loggers.py:248] Engine 000: Avg prompt throughput: 1338.0 tokens/s, Avg generation throughput: 420.2 tokens/s, Running: 37 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59582 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59598 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59610 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59642 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59648 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37722 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58564 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59668 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37810 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58558 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58558 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37810 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59642 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:50:24 [loggers.py:248] Engine 000: Avg prompt throughput: 975.7 tokens/s, Avg generation throughput: 601.7 tokens/s, Running: 46 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59668 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37806 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58666 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58612 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37764 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59668 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37764 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34782 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34798 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37842 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58612 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59668 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34782 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58656 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:50:34 [loggers.py:248] Engine 000: Avg prompt throughput: 681.3 tokens/s, Avg generation throughput: 656.1 tokens/s, Running: 51 reqs, Waiting: 0 reqs, GPU KV cache usage: 39.3%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37842 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58656 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60844 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60864 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60876 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60844 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37842 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59598 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60914 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60924 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58666 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59598 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58666 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60998 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:50:44 [loggers.py:248] Engine 000: Avg prompt throughput: 684.8 tokens/s, Avg generation throughput: 709.3 tokens/s, Running: 62 reqs, Waiting: 5 reqs, GPU KV cache usage: 51.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58666 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54644 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54648 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58558 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54678 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34782 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58666 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58656 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54644 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54704 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54706 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59668 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54716 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54732 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54748 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60844 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60876 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58558 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59648 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37722 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:50:54 [loggers.py:248] Engine 000: Avg prompt throughput: 565.3 tokens/s, Avg generation throughput: 697.6 tokens/s, Running: 63 reqs, Waiting: 19 reqs, GPU KV cache usage: 56.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58538 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56656 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56664 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56670 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37806 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58666 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56672 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56686 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54706 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54704 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58572 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60914 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54716 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54648 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37810 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37830 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37764 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58564 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56670 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59610 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:51:04 [loggers.py:248] Engine 000: Avg prompt throughput: 1034.7 tokens/s, Avg generation throughput: 620.8 tokens/s, Running: 62 reqs, Waiting: 24 reqs, GPU KV cache usage: 55.3%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58666 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59668 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54704 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54706 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58538 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59582 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37708 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54716 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58656 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36630 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36642 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56672 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36656 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36672 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37830 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36704 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59648 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36718 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36724 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:51:14 [loggers.py:248] Engine 000: Avg prompt throughput: 503.9 tokens/s, Avg generation throughput: 691.2 tokens/s, Running: 64 reqs, Waiting: 38 reqs, GPU KV cache usage: 60.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60924 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36748 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54644 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56670 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58140 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58154 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58158 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58190 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58200 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37806 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58206 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60864 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58218 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59668 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58224 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34782 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37764 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58666 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59642 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58230 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60844 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58246 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58590 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54732 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56672 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58612 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36672 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37708 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54748 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58272 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58282 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58538 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:51:24 [loggers.py:248] Engine 000: Avg prompt throughput: 654.6 tokens/s, Avg generation throughput: 684.7 tokens/s, Running: 64 reqs, Waiting: 54 reqs, GPU KV cache usage: 53.9%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56664 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58564 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47090 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54706 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47130 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47140 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47146 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47160 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56656 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37842 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59598 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58154 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58158 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47212 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47228 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:51:34 [loggers.py:248] Engine 000: Avg prompt throughput: 390.7 tokens/s, Avg generation throughput: 723.2 tokens/s, Running: 62 reqs, Waiting: 67 reqs, GPU KV cache usage: 47.8%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60998 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54648 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58206 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58572 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47426 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47432 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47440 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47444 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37866 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47450 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47468 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47470 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47480 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47500 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59610 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36656 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36642 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34798 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60914 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54704 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58590 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58666 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58246 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37810 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36630 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56686 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60876 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56672 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:51:44 [loggers.py:248] Engine 000: Avg prompt throughput: 608.1 tokens/s, Avg generation throughput: 755.2 tokens/s, Running: 63 reqs, Waiting: 78 reqs, GPU KV cache usage: 44.9%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37708 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60844 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58272 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58612 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58224 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58190 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54706 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47130 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59582 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47146 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58538 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54644 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37830 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60864 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:51:54 [loggers.py:248] Engine 000: Avg prompt throughput: 640.7 tokens/s, Avg generation throughput: 768.0 tokens/s, Running: 63 reqs, Waiting: 75 reqs, GPU KV cache usage: 46.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58140 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36724 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47212 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54678 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59642 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54648 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37744 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36718 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36704 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54716 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47444 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47426 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58572 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58218 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59648 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:52:04 [loggers.py:248] Engine 000: Avg prompt throughput: 820.3 tokens/s, Avg generation throughput: 748.8 tokens/s, Running: 63 reqs, Waiting: 72 reqs, GPU KV cache usage: 46.9%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47480 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36748 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47468 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34798 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37764 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58590 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58206 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54704 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47228 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58558 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37842 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56672 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60876 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34782 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54732 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58246 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58612 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47192 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37722 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56686 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36672 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36630 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58190 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59668 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37708 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47130 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58272 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47146 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56670 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58538 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60998 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34812 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:52:14 [loggers.py:248] Engine 000: Avg prompt throughput: 1096.4 tokens/s, Avg generation throughput: 678.4 tokens/s, Running: 64 reqs, Waiting: 75 reqs, GPU KV cache usage: 47.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58230 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47470 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58158 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60864 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54748 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47212 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60924 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58154 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47440 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36718 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36642 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54716 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47160 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58666 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59652 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47500 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47124 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:52:24 [loggers.py:248] Engine 000: Avg prompt throughput: 752.2 tokens/s, Avg generation throughput: 742.4 tokens/s, Running: 63 reqs, Waiting: 74 reqs, GPU KV cache usage: 49.9%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54648 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58218 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59648 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36748 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36724 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58224 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58656 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47450 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34798 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59610 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58590 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58558 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47228 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54636 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37806 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59642 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47090 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58612 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58564 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60914 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:52:34 [loggers.py:248] Engine 000: Avg prompt throughput: 659.8 tokens/s, Avg generation throughput: 761.5 tokens/s, Running: 64 reqs, Waiting: 70 reqs, GPU KV cache usage: 46.1%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37818 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54644 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60876 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47432 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47130 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56672 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56664 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36704 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60998 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56670 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56686 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60984 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59582 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37842 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58230 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54732 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58162 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56656 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60924 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47146 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37762 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37778 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47468 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:52:44 [loggers.py:248] Engine 000: Avg prompt throughput: 1211.2 tokens/s, Avg generation throughput: 691.2 tokens/s, Running: 64 reqs, Waiting: 62 reqs, GPU KV cache usage: 45.1%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34782 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58154 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58200 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36718 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47110 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47212 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:56682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54716 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58282 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58538 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37708 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58218 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58190 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47106 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58572 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58158 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54678 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47444 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60898 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:34798 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58246 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:36726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47450 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58656 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60944 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60890 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59668 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37806 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58558 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59648 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54704 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58612 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54648 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:58176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60914 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60876 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:59598 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60592 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:60608 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:54706 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:52:54 [loggers.py:248] Engine 000: Avg prompt throughput: 749.0 tokens/s, Avg generation throughput: 768.0 tokens/s, Running: 63 reqs, Waiting: 78 reqs, GPU KV cache usage: 47.7%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:38474 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:38484 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:38492 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:38494 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37830 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:38498 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:38508 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: 127.0.0.1:37764 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:53:04 [loggers.py:248] Engine 000: Avg prompt throughput: 435.1 tokens/s, Avg generation throughput: 800.0 tokens/s, Running: 64 reqs, Waiting: 67 reqs, GPU KV cache usage: 48.5%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:53:14 [loggers.py:248] Engine 000: Avg prompt throughput: 762.0 tokens/s, Avg generation throughput: 723.2 tokens/s, Running: 63 reqs, Waiting: 35 reqs, GPU KV cache usage: 51.3%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:53:24 [loggers.py:248] Engine 000: Avg prompt throughput: 784.6 tokens/s, Avg generation throughput: 701.9 tokens/s, Running: 56 reqs, Waiting: 0 reqs, GPU KV cache usage: 46.7%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:53:34 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 640.0 tokens/s, Running: 30 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.2%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:53:44 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 350.8 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.6%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:53:54 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 101.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 0.1%, MM cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=5124)[0;0m INFO 12-11 18:53:57 [launcher.py:110] Shutting down FastAPI HTTP server.
|
|
[rank0]:[W1211 18:53:58.272992418 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: Shutting down
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: Waiting for application shutdown.
|
|
[0;36m(APIServer pid=5124)[0;0m INFO: Application shutdown complete.
|