1028 lines
114 KiB
Plaintext
1028 lines
114 KiB
Plaintext
|
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
|
||
|
|
WARNING 12-19 14:22:47 [attention.py:82] Using VLLM_V1_USE_PREFILL_DECODE_ATTENTION environment variable is deprecated and will be removed in v0.14.0 or v1.0.0, whichever is soonest. Please use --attention-config.use_prefill_decode_attention command line argument or AttentionConfig(use_prefill_decode_attention=...) config field instead.
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:22:47 [api_server.py:1351] vLLM API server version 0.13.0rc2.dev112+g763963aa7.d20251213
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:22:47 [utils.py:253] non-default args: {'model_tag': 'meta-llama/Meta-Llama-3.1-8B-Instruct', 'host': '127.0.0.1', 'model': 'meta-llama/Meta-Llama-3.1-8B-Instruct', 'max_model_len': 65536, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 64}
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:22:51 [model.py:514] Resolved architecture: LlamaForCausalLM
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:22:51 [model.py:1636] Using max model len 65536
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:22:51 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
||
|
|
Skipping import of cpp extensions due to incompatible torch version 2.10.0a0+rocm7.11.0a20251210 for torchao version 0.14.1 Please see https://github.com/pytorch/ao/issues/2919 for more info
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:22:56 [core.py:93] Initializing a V1 LLM engine (v0.13.0rc2.dev112+g763963aa7.d20251213) with config: model='meta-llama/Meta-Llama-3.1-8B-Instruct', speculative_config=None, tokenizer='meta-llama/Meta-Llama-3.1-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=65536, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=meta-llama/Meta-Llama-3.1-8B-Instruct, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:22:56 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.1.122:59373 backend=nccl
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:22:56 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:22:56 [gpu_model_runner.py:3562] Starting to load model meta-llama/Meta-Llama-3.1-8B-Instruct...
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:22:56 [rocm.py:306] Using Rocm Attention backend on V1 engine.
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:00, 3.23it/s]
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:01<00:01, 1.08it/s]
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:03<00:01, 1.16s/it]
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:04<00:00, 1.24s/it]
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m
|
||
|
|
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:04<00:00, 1.12s/it]
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:23:02 [default_loader.py:308] Loading weights took 4.48 seconds
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:23:02 [gpu_model_runner.py:3659] Model loading took 15.0586 GiB memory and 5.596868 seconds
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:23:05 [backends.py:634] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/2ba6637f7a/rank_0_0/backbone for vLLM's torch.compile
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:23:05 [backends.py:694] Dynamo bytecode transform time: 2.06 s
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:23:06 [backends.py:261] Cache the graph of compile range (1, 2048) for later use
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:23:07 [backends.py:278] Compiling a graph for compile range (1, 2048) takes 1.25 s
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:23:07 [monitor.py:34] torch.compile takes 3.31 s in total
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:23:08 [gpu_worker.py:375] Available KV cache memory: 15.33 GiB
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:23:09 [kv_cache_utils.py:1291] GPU KV cache size: 125,616 tokens
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:23:09 [kv_cache_utils.py:1296] Maximum concurrency for 65,536 tokens per request: 1.92x
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/19 [00:00<?, ?it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 16%|█▌ | 3/19 [00:00<00:00, 21.43it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 32%|███▏ | 6/19 [00:00<00:00, 22.97it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 47%|████▋ | 9/19 [00:00<00:00, 23.69it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 63%|██████▎ | 12/19 [00:00<00:00, 24.30it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 79%|███████▉ | 15/19 [00:00<00:00, 24.85it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 95%|█████████▍| 18/19 [00:00<00:00, 25.44it/s]
|
||
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:00<00:00, 24.62it/s]
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m
|
||
|
|
Capturing CUDA graphs (decode, FULL): 0%| | 0/11 [00:00<?, ?it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 9%|▉ | 1/11 [00:00<00:01, 5.60it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 36%|███▋ | 4/11 [00:00<00:00, 15.89it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 64%|██████▎ | 7/11 [00:00<00:00, 20.87it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 24.73it/s]
|
||
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 21.00it/s]
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:23:11 [gpu_model_runner.py:4610] Graph capturing finished in 2 secs, took 0.94 GiB
|
||
|
|
[0;36m(EngineCore_DP0 pid=56723)[0;0m INFO 12-19 14:23:11 [core.py:259] init engine (profile, create kv cache, warmup model) took 8.30 seconds
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [api_server.py:1099] Supported tasks: ['generate']
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m WARNING 12-19 14:23:13 [model.py:1462] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [serving_responses.py:201] Using default chat sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [serving_chat.py:137] Using default chat sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [serving_completion.py:77] Using default completion sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [serving_chat.py:137] Using default chat sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [api_server.py:1425] Starting vLLM API server 0 on http://127.0.0.1:8000
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:38] Available routes are:
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /docs, Methods: GET, HEAD
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /tokenize, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /detokenize, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /pause, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /resume, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /is_paused, Methods: GET
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /metrics, Methods: GET
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /health, Methods: GET
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /load, Methods: GET
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /v1/models, Methods: GET
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /version, Methods: GET
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /v1/responses, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /v1/messages, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /v1/completions, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /ping, Methods: GET
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /ping, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /invocations, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /classify, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /v1/embeddings, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /score, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /v1/score, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /rerank, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /v1/rerank, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /v2/rerank, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:13 [launcher.py:46] Route: /pooling, Methods: POST
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: Started server process [56561]
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: Waiting for application startup.
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: Application startup complete.
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:36670 - "GET /v1/models HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35640 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35640 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43716 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43722 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35640 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:33 [loggers.py:248] Engine 000: Avg prompt throughput: 41.7 tokens/s, Avg generation throughput: 49.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35640 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43722 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43716 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35640 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:43 [loggers.py:248] Engine 000: Avg prompt throughput: 175.6 tokens/s, Avg generation throughput: 125.0 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43716 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35894 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54262 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:23:53 [loggers.py:248] Engine 000: Avg prompt throughput: 143.0 tokens/s, Avg generation throughput: 219.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35894 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54262 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43716 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35640 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:24:03 [loggers.py:248] Engine 000: Avg prompt throughput: 278.9 tokens/s, Avg generation throughput: 146.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35640 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43716 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54262 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43716 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:24:13 [loggers.py:248] Engine 000: Avg prompt throughput: 190.2 tokens/s, Avg generation throughput: 192.8 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35640 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35894 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43716 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43716 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:44998 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:44998 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35894 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:24:23 [loggers.py:248] Engine 000: Avg prompt throughput: 365.3 tokens/s, Avg generation throughput: 216.0 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:44998 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35640 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43704 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54262 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:44998 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35894 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:44998 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:24:33 [loggers.py:248] Engine 000: Avg prompt throughput: 366.2 tokens/s, Avg generation throughput: 225.6 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54262 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35640 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43716 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43716 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:24:43 [loggers.py:248] Engine 000: Avg prompt throughput: 151.0 tokens/s, Avg generation throughput: 189.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35894 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43716 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:55530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:55532 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:55532 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:55530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49106 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49112 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49112 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:24:53 [loggers.py:248] Engine 000: Avg prompt throughput: 373.9 tokens/s, Avg generation throughput: 246.1 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35894 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:55530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49106 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:55532 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49106 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49112 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:36198 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:36204 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:36204 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:25:03 [loggers.py:248] Engine 000: Avg prompt throughput: 223.3 tokens/s, Avg generation throughput: 303.4 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43738 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:55530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:55530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:25:13 [loggers.py:248] Engine 000: Avg prompt throughput: 276.0 tokens/s, Avg generation throughput: 292.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49112 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49106 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:36204 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:25:23 [loggers.py:248] Engine 000: Avg prompt throughput: 43.3 tokens/s, Avg generation throughput: 187.1 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:55530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49106 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54632 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54638 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54646 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:25:33 [loggers.py:248] Engine 000: Avg prompt throughput: 231.3 tokens/s, Avg generation throughput: 205.5 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:36204 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:43744 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49112 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:36204 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49106 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:55530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49112 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:36204 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54638 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49112 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:36204 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:25:43 [loggers.py:248] Engine 000: Avg prompt throughput: 274.2 tokens/s, Avg generation throughput: 238.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54632 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49112 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:55530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:46606 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:55530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:25:53 [loggers.py:248] Engine 000: Avg prompt throughput: 98.1 tokens/s, Avg generation throughput: 250.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54646 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54632 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49106 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54646 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:26:03 [loggers.py:248] Engine 000: Avg prompt throughput: 93.5 tokens/s, Avg generation throughput: 143.9 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49106 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54632 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54646 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:55530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54646 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:51166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:51182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:26:13 [loggers.py:248] Engine 000: Avg prompt throughput: 106.3 tokens/s, Avg generation throughput: 151.7 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:51166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54646 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:51166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:56432 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:51166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54646 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:55530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:35880 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:56442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:26:23 [loggers.py:248] Engine 000: Avg prompt throughput: 230.9 tokens/s, Avg generation throughput: 195.8 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:51166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:56442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:56448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:49106 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:51182 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54646 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:26:33 [loggers.py:248] Engine 000: Avg prompt throughput: 122.7 tokens/s, Avg generation throughput: 241.3 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:26:43 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 70.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:26:53 [loggers.py:248] Engine 000: Avg prompt throughput: 1.3 tokens/s, Avg generation throughput: 12.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47532 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47546 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47550 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47550 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47550 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47550 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41424 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47532 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41424 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41474 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41474 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41500 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41500 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41508 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:27:03 [loggers.py:248] Engine 000: Avg prompt throughput: 761.3 tokens/s, Avg generation throughput: 313.8 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 16.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41520 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41538 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41540 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47406 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47406 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47532 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41500 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47406 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41520 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47406 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47550 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47464 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41474 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47464 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41538 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41520 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:27:13 [loggers.py:248] Engine 000: Avg prompt throughput: 1216.9 tokens/s, Avg generation throughput: 732.4 tokens/s, Running: 28 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.6%, Prefix cache hit rate: 33.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41424 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47550 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41540 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47532 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47464 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53216 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41474 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47550 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47550 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41424 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47550 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47546 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47406 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41538 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:27:23 [loggers.py:248] Engine 000: Avg prompt throughput: 728.3 tokens/s, Avg generation throughput: 853.2 tokens/s, Running: 27 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.0%, Prefix cache hit rate: 40.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41508 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47406 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53250 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53266 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53266 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53266 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53154 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41424 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41508 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53250 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47532 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:27:33 [loggers.py:248] Engine 000: Avg prompt throughput: 657.9 tokens/s, Avg generation throughput: 930.2 tokens/s, Running: 26 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.8%, Prefix cache hit rate: 45.2%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41520 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53154 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53216 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41540 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41508 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47550 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47464 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41538 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41500 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53216 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41474 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41500 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53154 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47546 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47532 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41424 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:27:43 [loggers.py:248] Engine 000: Avg prompt throughput: 948.9 tokens/s, Avg generation throughput: 760.8 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.5%, Prefix cache hit rate: 44.9%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53154 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41540 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47550 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47532 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53266 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41540 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47546 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47532 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41520 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47532 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41508 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60556 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60588 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60602 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60556 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41474 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60608 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47550 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60556 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:27:53 [loggers.py:248] Engine 000: Avg prompt throughput: 922.4 tokens/s, Avg generation throughput: 928.9 tokens/s, Running: 42 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.5%, Prefix cache hit rate: 40.3%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47464 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47406 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60608 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47464 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53266 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47550 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41424 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41540 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53154 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53250 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:28:03 [loggers.py:248] Engine 000: Avg prompt throughput: 733.2 tokens/s, Avg generation throughput: 867.5 tokens/s, Running: 26 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.4%, Prefix cache hit rate: 37.2%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60602 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53266 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41424 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60588 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53250 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60556 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41474 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41520 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53266 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60588 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47464 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60556 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60608 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:28:13 [loggers.py:248] Engine 000: Avg prompt throughput: 586.3 tokens/s, Avg generation throughput: 796.1 tokens/s, Running: 33 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.4%, Prefix cache hit rate: 35.1%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60556 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47406 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60468 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60482 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60496 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60512 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53250 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48636 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48646 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48652 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48674 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48684 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48684 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53154 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41520 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48684 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53154 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48674 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41424 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48684 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60468 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41540 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41508 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41520 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:28:23 [loggers.py:248] Engine 000: Avg prompt throughput: 811.1 tokens/s, Avg generation throughput: 1103.1 tokens/s, Running: 41 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.5%, Prefix cache hit rate: 32.6%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60588 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60602 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47464 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41474 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48652 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60588 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48684 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53266 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60512 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60482 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48674 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48636 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60556 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:28:33 [loggers.py:248] Engine 000: Avg prompt throughput: 567.3 tokens/s, Avg generation throughput: 921.2 tokens/s, Running: 27 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.9%, Prefix cache hit rate: 31.0%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60440 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60608 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60496 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48684 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53154 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41508 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53250 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60496 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60608 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47406 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60602 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53266 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48652 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41508 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60588 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:28:43 [loggers.py:248] Engine 000: Avg prompt throughput: 1090.6 tokens/s, Avg generation throughput: 753.2 tokens/s, Running: 27 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.4%, Prefix cache hit rate: 28.4%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53154 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47464 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60588 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41540 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53250 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60556 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60588 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:28:53 [loggers.py:248] Engine 000: Avg prompt throughput: 605.3 tokens/s, Avg generation throughput: 650.7 tokens/s, Running: 23 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.2%, Prefix cache hit rate: 27.1%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60482 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53250 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60496 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41540 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60556 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47464 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41540 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53154 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60588 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:29:03 [loggers.py:248] Engine 000: Avg prompt throughput: 790.5 tokens/s, Avg generation throughput: 684.6 tokens/s, Running: 30 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.5%, Prefix cache hit rate: 25.6%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48684 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41540 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45070 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45082 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53154 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48652 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:46782 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45082 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60588 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60556 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60496 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53250 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45082 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60482 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:29:13 [loggers.py:248] Engine 000: Avg prompt throughput: 761.7 tokens/s, Avg generation throughput: 889.6 tokens/s, Running: 30 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.5%, Prefix cache hit rate: 24.4%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41540 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60482 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60496 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45082 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60556 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60482 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60496 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45082 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45070 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:29:23 [loggers.py:248] Engine 000: Avg prompt throughput: 876.0 tokens/s, Avg generation throughput: 662.3 tokens/s, Running: 29 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.2%, Prefix cache hit rate: 23.1%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45096 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60482 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:46782 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45082 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45070 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41540 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60496 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:46782 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53250 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45082 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47464 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41540 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60574 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:29:33 [loggers.py:248] Engine 000: Avg prompt throughput: 483.5 tokens/s, Avg generation throughput: 766.6 tokens/s, Running: 25 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 22.4%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45082 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48684 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60556 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53154 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60496 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60556 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47464 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41448 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60482 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:46782 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47462 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:45070 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41540 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53250 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO 12-19 14:29:43 [loggers.py:248] Engine 000: Avg prompt throughput: 719.3 tokens/s, Avg generation throughput: 763.7 tokens/s, Running: 34 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.6%, Prefix cache hit rate: 21.4%
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53154 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60558 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47402 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:48676 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60614 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41530 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41486 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53236 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60496 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54650 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47428 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54650 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47478 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47418 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54660 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41494 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:60456 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53154 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53222 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47452 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47504 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:41422 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:47442 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:53166 - "POST /v1/completions HTTP/1.1" 200 OK
|
||
|
|
[0;36m(APIServer pid=56561)[0;0m INFO: 127.0.0.1:54674 - "POST /v1/completions HTTP/1.1" 200 OK
|