1035 lines
117 KiB
Plaintext
1035 lines
117 KiB
Plaintext
|
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
|
||
|
|
WARNING 12-19 15:40:52 [attention.py:82] Using VLLM_V1_USE_PREFILL_DECODE_ATTENTION environment variable is deprecated and will be removed in v0.14.0 or v1.0.0, whichever is soonest. Please use --attention-config.use_prefill_decode_attention command line argument or AttentionConfig(use_prefill_decode_attention=...) config field instead.
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:40:52 [api_server.py:1351] vLLM API server version 0.13.0rc2.dev112+g763963aa7.d20251213
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:40:52 [utils.py:253] non-default args: {'model_tag': 'meta-llama/Meta-Llama-3.1-8B-Instruct', 'host': '127.0.0.1', 'model': 'meta-llama/Meta-Llama-3.1-8B-Instruct', 'max_model_len': 65536, 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 64}
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:40:56 [model.py:514] Resolved architecture: LlamaForCausalLM
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:40:56 [model.py:1636] Using max model len 65536
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:40:57 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
||
|
|
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
|
||
|
|
[0;36m(EngineCore_DP0 pid=62228)[0;0m INFO 12-19 15:41:01 [core.py:93] Initializing a V1 LLM engine (v0.13.0rc2.dev112+g763963aa7.d20251213) with config: model='meta-llama/Meta-Llama-3.1-8B-Instruct', speculative_config=None, tokenizer='meta-llama/Meta-Llama-3.1-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=65536, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=meta-llama/Meta-Llama-3.1-8B-Instruct, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
|
||
|
|
[0;36m(EngineCore_DP0 pid=62228)[0;0m WARNING 12-19 15:41:01 [multiproc_executor.py:884] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
|
||
|
|
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
|
||
|
|
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
|
||
|
|
INFO 12-19 15:41:04 [parallel_state.py:1203] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:40771 backend=nccl
|
||
|
|
INFO 12-19 15:41:04 [parallel_state.py:1203] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:40771 backend=nccl
|
||
|
|
INFO 12-19 15:41:04 [pynccl.py:111] vLLM is using nccl==2.27.3
|
||
|
|
INFO 12-19 15:41:05 [parallel_state.py:1411] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
|
||
|
|
INFO 12-19 15:41:05 [parallel_state.py:1411] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank 1
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m INFO 12-19 15:41:05 [gpu_model_runner.py:3562] Starting to load model meta-llama/Meta-Llama-3.1-8B-Instruct...
|
||
|
|
[0;36m(Worker_TP1 pid=62311)[0;0m INFO 12-19 15:41:06 [rocm.py:306] Using Rocm Attention backend on V1 engine.
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m INFO 12-19 15:41:06 [rocm.py:306] Using Rocm Attention backend on V1 engine.
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:00, 7.96it/s]
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:01<00:01, 1.60it/s]
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:02<00:00, 1.27it/s]
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00, 1.24it/s]
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00, 1.37it/s]
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m INFO 12-19 15:41:10 [default_loader.py:308] Loading weights took 2.92 seconds
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m INFO 12-19 15:41:11 [gpu_model_runner.py:3659] Model loading took 7.5820 GiB memory and 4.299710 seconds
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m INFO 12-19 15:41:13 [backends.py:634] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/2936335609/rank_0_0/backbone for vLLM's torch.compile
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m INFO 12-19 15:41:13 [backends.py:694] Dynamo bytecode transform time: 2.19 s
|
||
|
|
[0;36m(Worker_TP1 pid=62311)[0;0m INFO 12-19 15:41:14 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m INFO 12-19 15:41:14 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m INFO 12-19 15:41:16 [backends.py:278] Compiling a graph for compile range (1, 2048) takes 1.36 s
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m INFO 12-19 15:41:16 [monitor.py:34] torch.compile takes 3.55 s in total
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m INFO 12-19 15:41:17 [gpu_worker.py:375] Available KV cache memory: 22.92 GiB
|
||
|
|
[0;36m(EngineCore_DP0 pid=62228)[0;0m INFO 12-19 15:41:18 [kv_cache_utils.py:1291] GPU KV cache size: 375,536 tokens
|
||
|
|
[0;36m(EngineCore_DP0 pid=62228)[0;0m INFO 12-19 15:41:18 [kv_cache_utils.py:1296] Maximum concurrency for 65,536 tokens per request: 5.73x
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/19 [00:00<?, ?it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 5%|▌ | 1/19 [00:00<00:05, 3.43it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 11%|█ | 2/19 [00:00<00:04, 3.52it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 16%|█▌ | 3/19 [00:00<00:04, 3.55it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 21%|██ | 4/19 [00:01<00:04, 3.56it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 26%|██▋ | 5/19 [00:01<00:03, 3.57it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 32%|███▏ | 6/19 [00:01<00:03, 3.59it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 37%|███▋ | 7/19 [00:01<00:03, 3.61it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 42%|████▏ | 8/19 [00:02<00:03, 3.62it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 47%|████▋ | 9/19 [00:02<00:02, 3.63it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 53%|█████▎ | 10/19 [00:02<00:02, 3.65it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 58%|█████▊ | 11/19 [00:03<00:02, 3.66it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 63%|██████▎ | 12/19 [00:03<00:01, 3.68it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 68%|██████▊ | 13/19 [00:03<00:01, 3.71it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 74%|███████▎ | 14/19 [00:03<00:01, 3.73it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 79%|███████▉ | 15/19 [00:04<00:01, 3.75it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 84%|████████▍ | 16/19 [00:04<00:00, 3.74it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 89%|████████▉ | 17/19 [00:04<00:00, 3.76it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 95%|█████████▍| 18/19 [00:04<00:00, 3.76it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:05<00:00, 3.78it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:05<00:00, 3.68it/s]
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m
|
||
|
|
Capturing CUDA graphs (decode, FULL): 0%| | 0/11 [00:00<?, ?it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 9%|▉ | 1/11 [00:00<00:04, 2.37it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 18%|█▊ | 2/11 [00:00<00:02, 3.05it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 27%|██▋ | 3/11 [00:00<00:02, 3.38it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 36%|███▋ | 4/11 [00:01<00:01, 3.55it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 45%|████▌ | 5/11 [00:01<00:01, 3.67it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 55%|█████▍ | 6/11 [00:01<00:01, 3.76it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 64%|██████▎ | 7/11 [00:01<00:01, 3.80it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 73%|███████▎ | 8/11 [00:02<00:00, 3.82it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 82%|████████▏ | 9/11 [00:02<00:00, 3.85it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 91%|█████████ | 10/11 [00:02<00:00, 3.84it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:03<00:00, 3.85it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:03<00:00, 3.66it/s]
|
||
|
|
[0;36m(Worker_TP0 pid=62310)[0;0m INFO 12-19 15:41:26 [gpu_model_runner.py:4610] Graph capturing finished in 9 secs, took 0.62 GiB
|
||
|
|
[0;36m(EngineCore_DP0 pid=62228)[0;0m INFO 12-19 15:41:26 [core.py:259] init engine (profile, create kv cache, warmup model) took 15.91 seconds
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:28 [api_server.py:1099] Supported tasks: ['generate']
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m WARNING 12-19 15:41:28 [model.py:1462] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:28 [serving_responses.py:201] Using default chat sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:28 [serving_chat.py:137] Using default chat sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [serving_completion.py:77] Using default completion sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [serving_chat.py:137] Using default chat sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [api_server.py:1425] Starting vLLM API server 0 on http://127.0.0.1:8000
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:38] Available routes are:
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /docs, Methods: GET, HEAD
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /tokenize, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /detokenize, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /pause, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /resume, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /is_paused, Methods: GET
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /metrics, Methods: GET
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /health, Methods: GET
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /load, Methods: GET
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /v1/models, Methods: GET
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /version, Methods: GET
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /v1/responses, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /v1/messages, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /v1/completions, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /ping, Methods: GET
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /ping, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /invocations, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /classify, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /v1/embeddings, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /score, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /v1/score, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /rerank, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /v1/rerank, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /v2/rerank, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:29 [launcher.py:46] Route: /pooling, Methods: POST
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: Started server process [62066]
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: Waiting for application startup.
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: Application startup complete.
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:36254 - "GET /v1/models HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:32896 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:32896 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:32900 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:32896 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35678 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35694 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:49 [loggers.py:248] Engine 000: Avg prompt throughput: 41.7 tokens/s, Avg generation throughput: 66.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35710 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35710 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:32896 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35678 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35710 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:32896 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35678 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:41:59 [loggers.py:248] Engine 000: Avg prompt throughput: 175.6 tokens/s, Avg generation throughput: 160.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:32896 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:32900 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56050 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:32896 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:42:09 [loggers.py:248] Engine 000: Avg prompt throughput: 143.0 tokens/s, Avg generation throughput: 206.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35678 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:32896 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59902 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59902 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59908 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:42:19 [loggers.py:248] Engine 000: Avg prompt throughput: 278.9 tokens/s, Avg generation throughput: 142.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59902 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:32896 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:32896 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35678 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59902 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59908 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:42:29 [loggers.py:248] Engine 000: Avg prompt throughput: 220.0 tokens/s, Avg generation throughput: 186.1 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59902 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59908 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59908 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59986 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59908 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35678 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59902 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59988 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:42:39 [loggers.py:248] Engine 000: Avg prompt throughput: 335.5 tokens/s, Avg generation throughput: 239.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59908 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59986 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59902 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59988 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59908 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59986 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35678 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59986 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35678 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:35678 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59908 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:42:49 [loggers.py:248] Engine 000: Avg prompt throughput: 366.2 tokens/s, Avg generation throughput: 227.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59902 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59902 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:42:59 [loggers.py:248] Engine 000: Avg prompt throughput: 151.0 tokens/s, Avg generation throughput: 171.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59902 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59988 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59988 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37682 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59988 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59902 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:43:09 [loggers.py:248] Engine 000: Avg prompt throughput: 373.9 tokens/s, Avg generation throughput: 272.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59988 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59902 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37682 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37682 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:43:19 [loggers.py:248] Engine 000: Avg prompt throughput: 223.3 tokens/s, Avg generation throughput: 353.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59902 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59988 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37682 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:59988 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37682 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:43:29 [loggers.py:248] Engine 000: Avg prompt throughput: 276.0 tokens/s, Avg generation throughput: 204.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:39232 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:43:39 [loggers.py:248] Engine 000: Avg prompt throughput: 43.3 tokens/s, Avg generation throughput: 199.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:39232 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:43:49 [loggers.py:248] Engine 000: Avg prompt throughput: 231.3 tokens/s, Avg generation throughput: 197.3 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:39232 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:48642 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:39232 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:39232 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:48642 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:48642 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:39232 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:43:59 [loggers.py:248] Engine 000: Avg prompt throughput: 277.9 tokens/s, Avg generation throughput: 255.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:39232 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56040 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37690 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:58222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:44:09 [loggers.py:248] Engine 000: Avg prompt throughput: 94.4 tokens/s, Avg generation throughput: 274.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:36756 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:44:19 [loggers.py:248] Engine 000: Avg prompt throughput: 124.0 tokens/s, Avg generation throughput: 67.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:36756 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50582 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50582 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50586 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50606 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:44:29 [loggers.py:248] Engine 000: Avg prompt throughput: 75.8 tokens/s, Avg generation throughput: 198.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50586 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:36756 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50582 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50586 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50582 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:36756 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50582 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:44:39 [loggers.py:248] Engine 000: Avg prompt throughput: 230.9 tokens/s, Avg generation throughput: 196.2 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50582 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:57768 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:57784 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:36756 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50586 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:44:49 [loggers.py:248] Engine 000: Avg prompt throughput: 122.7 tokens/s, Avg generation throughput: 238.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:44:59 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 29.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33564 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33664 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:45:09 [loggers.py:248] Engine 000: Avg prompt throughput: 439.7 tokens/s, Avg generation throughput: 263.7 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 9.9%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33664 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33664 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54146 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54158 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:45:19 [loggers.py:248] Engine 000: Avg prompt throughput: 1232.0 tokens/s, Avg generation throughput: 770.6 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 29.4%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33564 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33564 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33664 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54158 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33664 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:45:29 [loggers.py:248] Engine 000: Avg prompt throughput: 926.1 tokens/s, Avg generation throughput: 840.9 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 39.1%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33664 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54146 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33664 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46272 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46284 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46302 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46284 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54158 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:45:39 [loggers.py:248] Engine 000: Avg prompt throughput: 634.2 tokens/s, Avg generation throughput: 971.0 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 44.2%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33564 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46284 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54146 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33664 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46284 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54158 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54146 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33564 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54158 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:45:49 [loggers.py:248] Engine 000: Avg prompt throughput: 720.9 tokens/s, Avg generation throughput: 731.9 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 47.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46284 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54146 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33564 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56292 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33564 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56314 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56322 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46272 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56292 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46302 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56292 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33664 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:45:59 [loggers.py:248] Engine 000: Avg prompt throughput: 914.1 tokens/s, Avg generation throughput: 897.0 tokens/s, Running: 27 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 42.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46272 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33564 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56314 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56314 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37246 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37246 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46302 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46272 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54158 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54146 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56322 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56292 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56314 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46272 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:46:09 [loggers.py:248] Engine 000: Avg prompt throughput: 890.5 tokens/s, Avg generation throughput: 892.2 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 38.1%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56292 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54158 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56292 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37246 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56314 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46302 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:46:19 [loggers.py:248] Engine 000: Avg prompt throughput: 469.5 tokens/s, Avg generation throughput: 790.4 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 36.3%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56322 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54146 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56292 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37246 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56322 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54158 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56322 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50054 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46302 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50058 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50070 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50072 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50074 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50080 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50092 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50098 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50128 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54146 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50098 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50058 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50098 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50092 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46272 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50058 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54158 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50098 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:46:29 [loggers.py:248] Engine 000: Avg prompt throughput: 1005.2 tokens/s, Avg generation throughput: 1015.6 tokens/s, Running: 35 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 33.0%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54146 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50058 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46302 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56314 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33600 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50074 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54158 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56314 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50098 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50054 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50128 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56292 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37246 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54158 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:46:39 [loggers.py:248] Engine 000: Avg prompt throughput: 527.3 tokens/s, Avg generation throughput: 1004.4 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 31.5%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54152 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50070 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56314 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46272 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50092 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50072 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37242 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50128 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50058 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37246 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46302 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46272 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50070 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56292 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50072 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56314 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33662 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37246 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54158 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56322 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46302 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:46:49 [loggers.py:248] Engine 000: Avg prompt throughput: 948.7 tokens/s, Avg generation throughput: 764.8 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 29.7%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54146 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50070 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50092 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50070 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50128 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56314 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56292 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50070 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50098 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:46:59 [loggers.py:248] Engine 000: Avg prompt throughput: 907.5 tokens/s, Avg generation throughput: 628.2 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 27.7%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50092 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56292 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50128 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50098 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50072 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56314 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50092 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:47:09 [loggers.py:248] Engine 000: Avg prompt throughput: 529.9 tokens/s, Avg generation throughput: 681.8 tokens/s, Running: 21 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 26.6%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50098 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50098 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40296 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50098 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40304 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40304 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40316 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50072 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50092 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56314 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:47:19 [loggers.py:248] Engine 000: Avg prompt throughput: 795.3 tokens/s, Avg generation throughput: 889.5 tokens/s, Running: 28 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 25.3%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50072 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56292 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40316 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50072 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56292 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50092 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40316 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50098 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56314 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50128 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50072 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:47:29 [loggers.py:248] Engine 000: Avg prompt throughput: 876.4 tokens/s, Avg generation throughput: 697.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 23.9%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40304 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50098 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56314 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50128 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45388 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56292 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40296 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50072 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:47:39 [loggers.py:248] Engine 000: Avg prompt throughput: 654.3 tokens/s, Avg generation throughput: 733.1 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 22.9%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40304 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50098 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40296 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56314 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40316 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45396 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33628 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33548 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33626 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33590 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO 12-19 15:47:49 [loggers.py:248] Engine 000: Avg prompt throughput: 595.1 tokens/s, Avg generation throughput: 764.6 tokens/s, Running: 24 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 22.1%
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50072 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40296 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40316 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45446 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45412 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54176 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54140 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:40304 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46310 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50114 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:54172 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33634 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:37236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:56300 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:46290 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:45444 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:50128 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:33648 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:60026 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:60042 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:60050 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=62066)[0;0m INFO: 127.0.0.1:60062 - "POST /v1/completions HTTP/1.1" 200 OK
|