1075 lines
124 KiB
Plaintext
1075 lines
124 KiB
Plaintext
|
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
||
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
||
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
||
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
||
|
|
dispatch key: ADInplaceOrView
|
||
|
|
previous kernel: no debug info
|
||
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
||
|
|
self.m.impl(
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:27:42 [api_server.py:1351] vLLM API server version 0.11.2.dev690+g67475a6e8.d20251209
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:27:42 [utils.py:253] non-default args: {'model_tag': 'RedHatAI/gemma-3-12b-it-FP8-dynamic', 'host': '127.0.0.1', 'model': 'RedHatAI/gemma-3-12b-it-FP8-dynamic', 'trust_remote_code': True, 'max_model_len': 9900, 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 64}
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:27:46 [model.py:629] Resolved architecture: Gemma3ForConditionalGeneration
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:27:46 [model.py:1755] Using max model len 9900
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:27:46 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
||
|
|
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
||
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
||
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
||
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
||
|
|
dispatch key: ADInplaceOrView
|
||
|
|
previous kernel: no debug info
|
||
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
||
|
|
self.m.impl(
|
||
|
|
[0;36m(EngineCore_DP0 pid=10431)[0;0m INFO 12-11 19:27:51 [core.py:93] Initializing a V1 LLM engine (v0.11.2.dev690+g67475a6e8.d20251209) with config: model='RedHatAI/gemma-3-12b-it-FP8-dynamic', speculative_config=None, tokenizer='RedHatAI/gemma-3-12b-it-FP8-dynamic', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=9900, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=RedHatAI/gemma-3-12b-it-FP8-dynamic, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
|
||
|
|
[0;36m(EngineCore_DP0 pid=10431)[0;0m WARNING 12-11 19:27:51 [multiproc_executor.py:880] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
|
||
|
|
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
||
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
||
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
||
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
||
|
|
dispatch key: ADInplaceOrView
|
||
|
|
previous kernel: no debug info
|
||
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
||
|
|
self.m.impl(
|
||
|
|
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
||
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
||
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
||
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
||
|
|
dispatch key: ADInplaceOrView
|
||
|
|
previous kernel: no debug info
|
||
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
||
|
|
self.m.impl(
|
||
|
|
INFO 12-11 19:27:56 [parallel_state.py:1203] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:56779 backend=nccl
|
||
|
|
INFO 12-11 19:27:56 [parallel_state.py:1203] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:56779 backend=nccl
|
||
|
|
INFO 12-11 19:27:56 [pynccl.py:111] vLLM is using nccl==2.27.3
|
||
|
|
INFO 12-11 19:27:57 [parallel_state.py:1411] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
|
||
|
|
INFO 12-11 19:27:57 [parallel_state.py:1411] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank 1
|
||
|
|
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
|
||
|
|
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m INFO 12-11 19:28:03 [gpu_model_runner.py:3544] Starting to load model RedHatAI/gemma-3-12b-it-FP8-dynamic...
|
||
|
|
[0;36m(Worker_TP1 pid=10514)[0;0m WARNING 12-11 19:28:03 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
|
||
|
|
[0;36m(Worker_TP1 pid=10514)[0;0m INFO 12-11 19:28:03 [layer.py:524] Using AttentionBackendEnum.TORCH_SDPA for MultiHeadAttention in multimodal encoder.
|
||
|
|
[0;36m(Worker_TP1 pid=10514)[0;0m WARNING 12-11 19:28:03 [activation.py:544] [ROCm] PyTorch's native GELU with tanh approximation is unstable. Falling back to GELU(approximate='none').
|
||
|
|
[0;36m(Worker_TP1 pid=10514)[0;0m INFO 12-11 19:28:03 [rocm.py:320] Using Triton Attention backend on V1 engine.
|
||
|
|
[0;36m(Worker_TP1 pid=10514)[0;0m WARNING 12-11 19:28:03 [activation.py:220] [ROCm] PyTorch's native GELU with tanh approximation is unstable with torch.compile. For native implementation, fallback to 'none' approximation. The custom kernel implementation is unaffected.
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m WARNING 12-11 19:28:03 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m INFO 12-11 19:28:03 [layer.py:524] Using AttentionBackendEnum.TORCH_SDPA for MultiHeadAttention in multimodal encoder.
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m WARNING 12-11 19:28:03 [activation.py:544] [ROCm] PyTorch's native GELU with tanh approximation is unstable. Falling back to GELU(approximate='none').
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m INFO 12-11 19:28:03 [rocm.py:320] Using Triton Attention backend on V1 engine.
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m WARNING 12-11 19:28:03 [activation.py:220] [ROCm] PyTorch's native GELU with tanh approximation is unstable with torch.compile. For native implementation, fallback to 'none' approximation. The custom kernel implementation is unaffected.
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 0% Completed | 0/3 [00:00<?, ?it/s]
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 33% Completed | 1/3 [00:01<00:02, 1.19s/it]
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 67% Completed | 2/3 [00:02<00:01, 1.06s/it]
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:02<00:00, 1.11it/s]
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:02<00:00, 1.05it/s]
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m INFO 12-11 19:28:07 [default_loader.py:308] Loading weights took 2.89 seconds
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m INFO 12-11 19:28:07 [gpu_model_runner.py:3626] Model loading took 7.1152 GiB memory and 3.610627 seconds
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m INFO 12-11 19:28:07 [gpu_model_runner.py:4388] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 7 image items of the maximum feature size.
|
||
|
|
[0;36m(Worker_TP1 pid=10514)[0;0m INFO 12-11 19:28:07 [gpu_model_runner.py:4388] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 7 image items of the maximum feature size.
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m INFO 12-11 19:28:16 [backends.py:616] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/8a51c8e3ac/rank_0_0/backbone for vLLM's torch.compile
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m INFO 12-11 19:28:16 [backends.py:676] Dynamo bytecode transform time: 7.15 s
|
||
|
|
[0;36m(Worker_TP1 pid=10514)[0;0m INFO 12-11 19:28:19 [backends.py:243] Cache the graph of compile range (1, 2048) for later use
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m INFO 12-11 19:28:19 [backends.py:243] Cache the graph of compile range (1, 2048) for later use
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m INFO 12-11 19:28:36 [backends.py:260] Compiling a graph for compile range (1, 2048) takes 17.45 s
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m INFO 12-11 19:28:36 [monitor.py:34] torch.compile takes 24.59 s in total
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m INFO 12-11 19:28:40 [gpu_worker.py:364] Available KV cache memory: 22.78 GiB
|
||
|
|
[0;36m(EngineCore_DP0 pid=10431)[0;0m INFO 12-11 19:28:40 [kv_cache_utils.py:1287] GPU KV cache size: 124,416 tokens
|
||
|
|
[0;36m(EngineCore_DP0 pid=10431)[0;0m INFO 12-11 19:28:40 [kv_cache_utils.py:1292] Maximum concurrency for 9,900 tokens per request: 29.46x
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/19 [00:00<?, ?it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 5%|▌ | 1/19 [00:00<00:07, 2.52it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 11%|█ | 2/19 [00:00<00:06, 2.53it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 16%|█▌ | 3/19 [00:01<00:06, 2.52it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 21%|██ | 4/19 [00:01<00:05, 2.51it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 26%|██▋ | 5/19 [00:01<00:05, 2.50it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 32%|███▏ | 6/19 [00:02<00:05, 2.49it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 37%|███▋ | 7/19 [00:02<00:04, 2.48it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 42%|████▏ | 8/19 [00:03<00:04, 2.48it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 47%|████▋ | 9/19 [00:03<00:04, 2.47it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 53%|█████▎ | 10/19 [00:04<00:03, 2.48it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 58%|█████▊ | 11/19 [00:04<00:03, 2.47it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 63%|██████▎ | 12/19 [00:04<00:02, 2.47it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 68%|██████▊ | 13/19 [00:05<00:02, 2.53it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 74%|███████▎ | 14/19 [00:05<00:01, 2.53it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 79%|███████▉ | 15/19 [00:06<00:01, 2.49it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 84%|████████▍ | 16/19 [00:06<00:01, 2.50it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 89%|████████▉ | 17/19 [00:06<00:00, 2.52it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 95%|█████████▍| 18/19 [00:07<00:00, 2.50it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:07<00:00, 2.50it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:07<00:00, 2.50it/s]
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m
|
||
|
|
Capturing CUDA graphs (decode, FULL): 0%| | 0/11 [00:00<?, ?it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 9%|▉ | 1/11 [00:01<00:11, 1.12s/it]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 18%|█▊ | 2/11 [00:02<00:10, 1.14s/it]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 27%|██▋ | 3/11 [00:02<00:06, 1.27it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 36%|███▋ | 4/11 [00:03<00:04, 1.58it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 45%|████▌ | 5/11 [00:03<00:03, 1.84it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 55%|█████▍ | 6/11 [00:04<00:03, 1.47it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 64%|██████▎ | 7/11 [00:05<00:03, 1.30it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 73%|███████▎ | 8/11 [00:05<00:01, 1.56it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 82%|████████▏ | 9/11 [00:06<00:01, 1.80it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 91%|█████████ | 10/11 [00:06<00:00, 2.01it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:07<00:00, 1.58it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:07<00:00, 1.49it/s]
|
||
|
|
[0;36m(Worker_TP0 pid=10513)[0;0m INFO 12-11 19:28:56 [gpu_model_runner.py:4548] Graph capturing finished in 16 secs, took 0.98 GiB
|
||
|
|
[0;36m(EngineCore_DP0 pid=10431)[0;0m INFO 12-11 19:28:56 [core.py:256] init engine (profile, create kv cache, warmup model) took 48.99 seconds
|
||
|
|
[0;36m(EngineCore_DP0 pid=10431)[0;0m Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:04 [api_server.py:1099] Supported tasks: ['generate']
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [api_server.py:1425] Starting vLLM API server 0 on http://127.0.0.1:8000
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:38] Available routes are:
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /docs, Methods: HEAD, GET
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /tokenize, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /detokenize, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /pause, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /resume, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /is_paused, Methods: GET
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /metrics, Methods: GET
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /health, Methods: GET
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /load, Methods: GET
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /v1/models, Methods: GET
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /version, Methods: GET
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /v1/responses, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /v1/messages, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /v1/completions, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /ping, Methods: GET
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /ping, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /invocations, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /classify, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /v1/embeddings, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /score, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /v1/score, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /rerank, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /v1/rerank, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /v2/rerank, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:05 [launcher.py:46] Route: /pooling, Methods: POST
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: Started server process [10267]
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: Waiting for application startup.
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: Application startup complete.
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:54092 - "GET /v1/models HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48850 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48866 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:25 [loggers.py:248] Engine 000: Avg prompt throughput: 45.0 tokens/s, Avg generation throughput: 69.3 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:35 [loggers.py:248] Engine 000: Avg prompt throughput: 178.1 tokens/s, Avg generation throughput: 146.2 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55576 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55584 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55576 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48850 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:45 [loggers.py:248] Engine 000: Avg prompt throughput: 163.1 tokens/s, Avg generation throughput: 239.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:46922 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:29:55 [loggers.py:248] Engine 000: Avg prompt throughput: 298.2 tokens/s, Avg generation throughput: 131.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51822 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51822 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51822 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51832 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51822 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:30:05 [loggers.py:248] Engine 000: Avg prompt throughput: 231.5 tokens/s, Avg generation throughput: 199.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:46922 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51822 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:59884 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:46922 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:46922 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:46922 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:30:15 [loggers.py:248] Engine 000: Avg prompt throughput: 366.8 tokens/s, Avg generation throughput: 222.1 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:59884 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51822 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51832 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:46922 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:59884 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:59884 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51822 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:46922 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:30:25 [loggers.py:248] Engine 000: Avg prompt throughput: 399.0 tokens/s, Avg generation throughput: 232.9 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:46922 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:59418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:59418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51832 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51832 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:30:35 [loggers.py:248] Engine 000: Avg prompt throughput: 153.9 tokens/s, Avg generation throughput: 178.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:59418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51832 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:59418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:34954 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:34954 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:34962 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:59418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:59418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:59418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:59418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:30:45 [loggers.py:248] Engine 000: Avg prompt throughput: 417.8 tokens/s, Avg generation throughput: 239.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:34954 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42630 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42630 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:34962 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:59418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:59418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:30:55 [loggers.py:248] Engine 000: Avg prompt throughput: 207.9 tokens/s, Avg generation throughput: 354.1 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:59418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:34954 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51832 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:48848 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:34954 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51832 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:31:05 [loggers.py:248] Engine 000: Avg prompt throughput: 292.5 tokens/s, Avg generation throughput: 203.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51086 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51832 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:31:15 [loggers.py:248] Engine 000: Avg prompt throughput: 45.8 tokens/s, Avg generation throughput: 199.3 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51832 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51086 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51086 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:60876 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51832 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:31:25 [loggers.py:248] Engine 000: Avg prompt throughput: 305.6 tokens/s, Avg generation throughput: 211.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51832 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51086 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51832 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57056 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51086 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51086 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:31:35 [loggers.py:248] Engine 000: Avg prompt throughput: 247.2 tokens/s, Avg generation throughput: 255.3 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57056 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:55562 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:56662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:51832 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:60876 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:31:45 [loggers.py:248] Engine 000: Avg prompt throughput: 103.5 tokens/s, Avg generation throughput: 261.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:56662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:60876 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:60876 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:31:55 [loggers.py:248] Engine 000: Avg prompt throughput: 145.2 tokens/s, Avg generation throughput: 96.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:56662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38666 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38666 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38674 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38680 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38684 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:32:05 [loggers.py:248] Engine 000: Avg prompt throughput: 90.8 tokens/s, Avg generation throughput: 192.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:60876 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38666 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:60876 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38684 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:60876 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:35480 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:35490 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:32:15 [loggers.py:248] Engine 000: Avg prompt throughput: 233.8 tokens/s, Avg generation throughput: 209.8 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:35490 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:56662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38684 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:35480 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38684 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:32:25 [loggers.py:248] Engine 000: Avg prompt throughput: 118.3 tokens/s, Avg generation throughput: 243.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:32:35 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44608 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44608 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44630 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44608 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44672 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44696 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44630 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:32:45 [loggers.py:248] Engine 000: Avg prompt throughput: 386.2 tokens/s, Avg generation throughput: 205.0 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 8.2%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57740 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57742 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57756 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57740 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44696 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57742 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44696 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57772 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57788 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44696 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57800 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57808 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57836 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57852 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57742 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57852 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44672 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44608 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44630 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57788 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57740 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44630 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57740 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57808 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38692 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:32:55 [loggers.py:248] Engine 000: Avg prompt throughput: 1295.5 tokens/s, Avg generation throughput: 643.6 tokens/s, Running: 26 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.4%, Prefix cache hit rate: 28.2%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38700 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38700 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44672 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38692 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57800 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44672 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44696 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38708 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57742 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57742 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57800 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38722 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57836 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57742 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57808 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57852 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38722 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38752 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38768 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57808 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38700 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38768 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57742 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57756 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38700 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57742 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:33:05 [loggers.py:248] Engine 000: Avg prompt throughput: 1072.0 tokens/s, Avg generation throughput: 692.3 tokens/s, Running: 27 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.9%, Prefix cache hit rate: 39.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38768 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57756 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44630 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38692 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44630 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57800 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45344 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45356 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45356 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45362 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45376 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45380 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45392 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45408 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57740 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45362 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45376 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57740 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45410 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45414 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45420 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45376 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45362 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:33:15 [loggers.py:248] Engine 000: Avg prompt throughput: 701.1 tokens/s, Avg generation throughput: 784.5 tokens/s, Running: 44 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.7%, Prefix cache hit rate: 44.4%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44198 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38700 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45362 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57772 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45356 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45408 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38692 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57788 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45414 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57742 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45362 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57756 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57772 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57836 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38692 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38700 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38722 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38752 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57836 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38722 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38708 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45376 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45362 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57800 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45420 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:33:25 [loggers.py:248] Engine 000: Avg prompt throughput: 671.8 tokens/s, Avg generation throughput: 832.7 tokens/s, Running: 37 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.0%, Prefix cache hit rate: 47.6%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38700 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45376 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38768 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57852 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44696 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45376 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45414 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57742 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44696 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45414 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45392 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57702 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44608 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57714 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57740 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45408 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57702 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57730 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57808 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38752 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57740 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57730 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:33:35 [loggers.py:248] Engine 000: Avg prompt throughput: 1031.9 tokens/s, Avg generation throughput: 719.6 tokens/s, Running: 46 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.1%, Prefix cache hit rate: 42.2%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57742 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44198 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33178 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57800 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33196 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57730 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33210 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33212 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44630 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33226 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33234 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33240 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33210 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33210 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33254 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45380 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57800 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45408 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33268 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44672 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33210 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33268 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33284 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42508 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:33:45 [loggers.py:248] Engine 000: Avg prompt throughput: 1086.0 tokens/s, Avg generation throughput: 803.4 tokens/s, Running: 56 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.0%, Prefix cache hit rate: 37.8%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33210 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45410 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38722 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57788 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57702 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42508 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33234 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57808 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44696 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57756 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57836 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38692 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44672 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44630 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57740 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57742 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38752 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:33:55 [loggers.py:248] Engine 000: Avg prompt throughput: 509.4 tokens/s, Avg generation throughput: 883.0 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 36.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45392 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45376 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44630 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57836 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57772 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57730 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57808 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57714 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45344 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33234 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33196 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45362 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45420 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33212 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38708 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38700 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57714 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38768 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44608 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45362 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57714 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33254 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33234 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36866 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33212 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33226 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33268 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36868 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36874 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36876 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36888 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36890 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57788 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36892 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36906 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36922 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45410 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33268 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45380 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58568 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58578 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45408 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33240 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:34:05 [loggers.py:248] Engine 000: Avg prompt throughput: 911.4 tokens/s, Avg generation throughput: 927.9 tokens/s, Running: 64 reqs, Waiting: 10 reqs, GPU KV cache usage: 16.5%, Prefix cache hit rate: 33.2%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58610 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57788 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36890 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57702 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36888 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36906 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45410 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58578 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57714 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57800 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58568 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57772 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58610 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33268 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45392 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33284 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33226 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45356 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36922 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45410 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57740 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57800 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33196 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:34:15 [loggers.py:248] Engine 000: Avg prompt throughput: 684.7 tokens/s, Avg generation throughput: 1015.7 tokens/s, Running: 58 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 31.4%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57772 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57756 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44696 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45392 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57788 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57852 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38768 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45414 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36876 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38708 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33226 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45344 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57742 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36890 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44696 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57772 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33196 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57800 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45420 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44608 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57836 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57852 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38722 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38708 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38692 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45356 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57714 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57730 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44630 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:34:25 [loggers.py:248] Engine 000: Avg prompt throughput: 952.5 tokens/s, Avg generation throughput: 900.2 tokens/s, Running: 57 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.8%, Prefix cache hit rate: 29.1%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57742 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45344 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38752 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33226 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:42508 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45420 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57800 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57808 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33254 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45376 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44198 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33284 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33240 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57836 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38722 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45380 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58578 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33226 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44198 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36888 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33210 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36906 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57808 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57788 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57756 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45376 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57836 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36876 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:34:35 [loggers.py:248] Engine 000: Avg prompt throughput: 1029.1 tokens/s, Avg generation throughput: 754.8 tokens/s, Running: 43 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.1%, Prefix cache hit rate: 27.0%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45408 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38722 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33212 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36922 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44630 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44608 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45380 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58578 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57740 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44672 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38700 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33196 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45376 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33284 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57756 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57836 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38708 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57788 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:34:45 [loggers.py:248] Engine 000: Avg prompt throughput: 476.6 tokens/s, Avg generation throughput: 809.6 tokens/s, Running: 43 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.0%, Prefix cache hit rate: 26.2%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33196 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44672 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36890 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57740 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57836 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38722 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33268 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45376 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33240 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44672 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45356 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57852 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57800 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45408 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57836 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45376 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45380 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33210 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57800 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33212 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36906 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57852 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45380 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33226 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33212 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45380 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33268 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45414 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:34:55 [loggers.py:248] Engine 000: Avg prompt throughput: 980.0 tokens/s, Avg generation throughput: 842.8 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.4%, Prefix cache hit rate: 24.7%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36866 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33284 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33210 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45408 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57702 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38722 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45380 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33284 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33210 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57702 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44644 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33284 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58568 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57836 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45414 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57852 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33212 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44630 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45344 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36906 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57702 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:35:05 [loggers.py:248] Engine 000: Avg prompt throughput: 944.0 tokens/s, Avg generation throughput: 757.0 tokens/s, Running: 29 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.6%, Prefix cache hit rate: 23.3%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45408 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33196 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57756 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33210 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45376 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44672 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57788 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44198 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33226 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36866 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33212 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57852 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36906 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45408 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57702 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57772 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38700 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44198 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36890 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33226 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33254 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36922 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57788 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57702 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58596 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38700 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44672 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57788 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57800 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44618 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:35:15 [loggers.py:248] Engine 000: Avg prompt throughput: 776.3 tokens/s, Avg generation throughput: 699.0 tokens/s, Running: 44 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.8%, Prefix cache hit rate: 22.3%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33254 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44198 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57800 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45380 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38700 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57756 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33268 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33210 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33196 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45380 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45344 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38700 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38700 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57824 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33210 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33268 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33212 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45414 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44630 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38700 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:38722 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44608 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58578 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO 12-11 19:35:25 [loggers.py:248] Engine 000: Avg prompt throughput: 640.6 tokens/s, Avg generation throughput: 754.1 tokens/s, Running: 43 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.4%, Prefix cache hit rate: 21.5%, MM cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45414 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36890 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:52250 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:52266 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:52282 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:52286 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57702 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:52302 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:52306 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:52316 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33226 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:52316 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:36866 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33226 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57740 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:52324 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:52338 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:52324 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57740 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:52338 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:52340 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57794 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57788 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:58590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45408 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33196 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:52338 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:33182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57728 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45408 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:52354 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45344 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:44648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:45408 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=10267)[0;0m INFO: 127.0.0.1:57770 - "POST /v1/completions HTTP/1.1" 200 OK
|