wehub-resource-sync
596 lines
46 KiB
Plaintext
596 lines
46 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "211cffab",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Token Dropping Example\n",
|
|
"---\n",
|
|
"\n",
|
|
"Long prompts create large KV caches that eat up GPU memory and limit how many \n",
|
|
"requests fit in a batch, and a smaller batch means lower decode throughput. \n",
|
|
"Token dropping shrinks each request's KV cache (in this example by half), so more\n",
|
|
"requests fits in a batch. This notebook shows improvement in decode \n",
|
|
"throughput by 1.5-1.7x by using LMCache's SDK API that enables ML engineers get\n",
|
|
"a requests' KV, do optimizations such as token dropping, and put it back to use\n",
|
|
"during LLM serving.\n",
|
|
"\n",
|
|
"Requirements:\n",
|
|
"* This experiment requires a GPU. To demonstrate how token dropping increases \n",
|
|
" the decoding batch size, adjust the GPU memory utilization together with the \n",
|
|
" number of requests. This example uses one RTX 6000 PRO.\n",
|
|
"* This example uses shared memory for data transfer between LMCache server and\n",
|
|
" the SDK. If shared memory is unavailable, the SDK automatically falls back to pickle."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1cd06b97",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Setup vLLM and LMCache server\n",
|
|
"Follow below instructions (1-4), then proceed to below Python cell.\n",
|
|
"\n",
|
|
"**1. Start LMCache server first**\n",
|
|
"\n",
|
|
"To use shared memory, specify the `--shm-name` and `--no-l1-use-lazy`\n",
|
|
"\n",
|
|
"```sh\n",
|
|
"lmcache server \\\n",
|
|
" --l1-size-gb 150 \\\n",
|
|
" --eviction-policy LRU \\\n",
|
|
" --chunk-size 256 \\\n",
|
|
" --port 6555 \\\n",
|
|
" --http-port 8080 \\\n",
|
|
" --shm-name lmcache_kvcache_sdk_e2e \\\n",
|
|
" --no-l1-use-lazy\n",
|
|
"```\n",
|
|
"\n",
|
|
"**2. Wait until LMCache server is ready**\n",
|
|
"\n",
|
|
"```sh\n",
|
|
"curl -sf http://localhost:8080/healthcheck && echo \" LMCache ready\"\n",
|
|
"```\n",
|
|
"\n",
|
|
"**3. Start vLLM once LMCache server is ready**\n",
|
|
"\n",
|
|
"We pass --no-enable-prefix-caching to disable vLLM's built-in prefix caching. \n",
|
|
"This ensures the prefilled KV cache is always served from LMCache rather than \n",
|
|
"vLLM, so the decoding throughput improvement can be attributed entirely to the \n",
|
|
"tokens dropped through LMCache.\n",
|
|
"\n",
|
|
"```sh\n",
|
|
"vllm serve Qwen/Qwen3-8B \\\n",
|
|
" --port 8000 \\\n",
|
|
" --served-model-name Qwen/Qwen3-8B \\\n",
|
|
" --no-enable-prefix-caching \\\n",
|
|
" --enforce-eager \\\n",
|
|
" --gpu-memory-utilization 0.65 \\\n",
|
|
" --kv-transfer-config '{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"lmcache.mp.port\":6555}}' \\\n",
|
|
" --trust-remote-code \\\n",
|
|
" --return-tokens-as-token-ids\n",
|
|
"```\n",
|
|
"\n",
|
|
"**4. Wait until vLLM is ready**\n",
|
|
"\n",
|
|
"```sh\n",
|
|
"curl -sf http://localhost:8000/v1/models && echo \" vLLM ready\"\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c2d81202",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Run E2E KV Edit Example"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "aa87a653",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"/home/rani/LMCache/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
|
|
" from .autonotebook import tqdm as notebook_tqdm\n",
|
|
"\u001b[32;20m[2026-07-04 06:19:57,388] LMCache INFO:\u001b[0m torch_dev=<module 'torch.cuda' from '/home/rani/LMCache/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py'>, torch_device_type=cuda \u001b[3m(__init__.py:63:lmcache)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:19:57,485] LMCache INFO:\u001b[0m CudaPinMemoryBackend: using torch cudart \u001b[3m(pin_memory.py:89:lmcache.v1.platform.cuda.pin_memory)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:19:57,873] LMCache INFO:\u001b[0m Skipping backend lmcache.v1.platform.musa.ops: predicate returned False \u001b[3m(__init__.py:114:lmcache)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:19:57,874] LMCache INFO:\u001b[0m Skipping backend lmcache.xpu_ops: predicate returned False \u001b[3m(__init__.py:114:lmcache)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:19:57,876] LMCache INFO:\u001b[0m Using backend: lmcache.c_ops \u001b[3m(__init__.py:132:lmcache)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:19:57,929] LMCache INFO:\u001b[0m multi_layer_block_kv_transfer mode: ptr \u001b[3m(base.py:94:lmcache.v1.multiprocess.transfer_context.base)\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
" _ __ __ ____ _ \n",
|
|
"| | | \\/ | / ___|__ _ ___| |__ ___ LMCache v0.4.8rc5.dev14 (geb5dfeb9)\n",
|
|
"| | | |\\/| | | | / _` |/ __| '_ \\ / _ \\ Website: https://lmcache.ai/\n",
|
|
"| |___| | | | | |__| (_| | (__| | | | __/ Recipes: https://docs.lmcache.ai/recipes\n",
|
|
"|_____|_| |_| \\____\\__,_|\\___|_| |_|\\___| LinkedIn: https://www.linkedin.com/company/lmcache-lab\n",
|
|
"Set LMCACHE_DISABLE_BANNER=1 to hide this banner.\n",
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# SPDX-License-Identifier: Apache-2.0\n",
|
|
"\"\"\"End-to-end KV cache remapping driver for the SDK example.\"\"\"\n",
|
|
"\n",
|
|
"# Standard\n",
|
|
"from datasets import load_dataset\n",
|
|
"from matplotlib import pyplot as plt\n",
|
|
"import sys\n",
|
|
"from transformers import AutoTokenizer, AutoConfig\n",
|
|
"\n",
|
|
"# Third Party\n",
|
|
"import torch\n",
|
|
"\n",
|
|
"# First Party\n",
|
|
"from lmcache.logging import init_logger\n",
|
|
"import lmcache.sdk.kvcache as lmc_sdk\n",
|
|
"import lmcache.sdk.stream as lmc_stream\n",
|
|
"import lmcache.sdk.batch as lmc_batch\n",
|
|
"from lmcache.banner import print_banner_once\n",
|
|
"from utils import rerotate_k_cache, make_post_completion\n",
|
|
"\n",
|
|
"print_banner_once(sys.stdout)\n",
|
|
"logger = init_logger(__name__)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5342c6c6",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Setting up hyperparameters"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "8f1065b0",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"model_name = \"Qwen/Qwen3-8B\"\n",
|
|
"vllm_url = \"http://localhost:8000\"\n",
|
|
"lmcache_url = \"http://localhost:8081\"\n",
|
|
"lmcache_mq_url = \"tcp://localhost:6555\"\n",
|
|
"chunk_size = 256\n",
|
|
"max_tokens = 5120\n",
|
|
"timeout = 60 # timeout for context retrieval (seconds)\n",
|
|
"trust_remote_code = True"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "34eb19b4",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Configuring the model and LMCache endpoint"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "eac8beea",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[32;20m[2026-07-04 06:19:59,379] LMCache INFO:\u001b[0m Initialized LMCacheKVCacheContext with instance_id=3193593, model_name=Qwen/Qwen3-8B, chunk_size=256, shm_name=lmcache_l1_pool_lmcache_kvcache_sdk_e2e \u001b[3m(kvcache.py:114:lmcache.sdk.kvcache)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:19:59,381] LMCache INFO:\u001b[0m Creating transfer context (device_type=cpu, mode=auto) \u001b[3m(worker_transfer.py:551:lmcache.v1.multiprocess.transfer_context.worker_transfer)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:19:59,382] LMCache INFO:\u001b[0m Engine KV Format: EngineKVFormat.NL_X_NB_TWO_NH_BS_HS NL x [NB, 2, NH, BS, HS] \u001b[3m(detection.py:44:lmcache.v1.gpu_connector.kv_format.detection)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:19:59,383] LMCache INFO:\u001b[0m Creating EngineDrivenContextShm (shm_name=lmcache_l1_pool_lmcache_kvcache_sdk_e2e, pool_size=161061273600) \u001b[3m(base.py:235:lmcache.v1.multiprocess.transfer_context.base)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:20:24,553] LMCache INFO:\u001b[0m SHM pinned=True for shm_name=lmcache_l1_pool_lmcache_kvcache_sdk_e2e \u001b[3m(shm.py:116:lmcache.v1.multiprocess.transfer_context.shm)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:20:24,557] LMCache INFO:\u001b[0m Worker non-GPU transfer context registered (instance_id=3193593, mode=SHM) \u001b[3m(worker_transfer.py:420:lmcache.v1.multiprocess.transfer_context.worker_transfer)\u001b[0m\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"tokenizer = AutoTokenizer.from_pretrained(\n",
|
|
" model_name, trust_remote_code=trust_remote_code\n",
|
|
")\n",
|
|
"config = AutoConfig.from_pretrained(model_name, trust_remote_code=trust_remote_code)\n",
|
|
"head_size = getattr(\n",
|
|
" config, \"head_dim\", config.hidden_size // config.num_attention_heads\n",
|
|
")\n",
|
|
"work_device = torch.device(\"cpu\")\n",
|
|
"post_completion = make_post_completion(vllm_url, model_name, timeout)\n",
|
|
"ctx = lmc_sdk.connect(\n",
|
|
" url=lmcache_mq_url,\n",
|
|
" http_url=lmcache_url,\n",
|
|
" model_name=model_name,\n",
|
|
" timeout=timeout,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c2aaffa7",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Constructing Prompt"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "030145fc",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Loaded 30 prompts from the dataset.\n",
|
|
"Total prompts length: 418566 tokens.\n",
|
|
"Mean prompt length: 13952.20 tokens.\n",
|
|
"Max prompt length: 16630 tokens.\n",
|
|
"Min prompt length: 10007 tokens.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"prompts = []\n",
|
|
"ds = load_dataset(\"raniayu/token-dropping-demo\", split=\"train\")\n",
|
|
"for i, example in enumerate(ds):\n",
|
|
" prompt = tokenizer.encode(ds[i][\"prompt\"], return_tensors=\"pt\").squeeze(0).tolist()\n",
|
|
" prompts.append(prompt)\n",
|
|
"\n",
|
|
"print(f\"Loaded {len(prompts)} prompts from the dataset.\")\n",
|
|
"print(f\"Total prompts length: {sum(len(p) for p in prompts)} tokens.\")\n",
|
|
"print(f\"Mean prompt length: {sum(len(p) for p in prompts) / len(prompts):.2f} tokens.\")\n",
|
|
"print(f\"Max prompt length: {max(len(p) for p in prompts)} tokens.\")\n",
|
|
"print(f\"Min prompt length: {min(len(p) for p in prompts)} tokens.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "23c992fa",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Baseline (without token dropping)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "b49aecc9",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"======================= Batched Stream Metrics (prefill) =======================\n",
|
|
"-------------------------------- Configuration ---------------------------------\n",
|
|
"Number of Streams: 30\n",
|
|
"----------------------------------- Results ------------------------------------\n",
|
|
"Total Duration (s): 26.45\n",
|
|
"Total Input Tokens: 418566\n",
|
|
"Input Throughput (tokens/s): 15824.59\n",
|
|
"================================================================================\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"======================= Batched Stream Metrics (decode) ========================\n",
|
|
"-------------------------------- Configuration ---------------------------------\n",
|
|
"Number of Streams: 30\n",
|
|
"----------------------------------- Results ------------------------------------\n",
|
|
"Total Duration (s): 649.28\n",
|
|
"Total Output Tokens: 153600\n",
|
|
"Decode Throughput (tokens/s): 236.57\n",
|
|
"================================================================================\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"batch = lmc_batch.LMCacheBatchedStream()\n",
|
|
"for i, prompt in enumerate(prompts):\n",
|
|
" stream = lmc_stream.create_request(\n",
|
|
" ctx=ctx,\n",
|
|
" post_completion=post_completion,\n",
|
|
" prompt_token_ids=prompt,\n",
|
|
" )\n",
|
|
" batch.add(stream)\n",
|
|
"results = batch.prefill(\n",
|
|
" sampling_params={\"max_tokens\": 1, \"temperature\": 1.0, \"ignore_eos\": True}\n",
|
|
")\n",
|
|
"results.emit()\n",
|
|
"\n",
|
|
"print(\"\\n\\n\")\n",
|
|
"\n",
|
|
"results = batch.decode(\n",
|
|
" sampling_params={\"max_tokens\": max_tokens, \"temperature\": 1.0, \"ignore_eos\": True}\n",
|
|
")\n",
|
|
"results.emit()\n",
|
|
"before_drop_data = results.to_dict()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f164ce86",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Clear LMCache"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "a9c1dc14",
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "shellscript"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{\"status\":\"ok\"}"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"!curl -X POST {lmcache_url}/clear-cache"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b7fb3744",
|
|
"metadata": {},
|
|
"source": [
|
|
"## With Token Dropping"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "a495be98",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def drop_tokens_fn(\n",
|
|
" kv_tensor: torch.Tensor, token_source: list[int]\n",
|
|
") -> tuple[torch.Tensor, list[int]]:\n",
|
|
" \"\"\"Drop the middle half of the chunks, keeping the first and last intact.\n",
|
|
"\n",
|
|
" Args:\n",
|
|
" kv_tensor: KV cache with shape [2, L, T, D]\n",
|
|
" token_source: Token ids\n",
|
|
" Returns:\n",
|
|
" A tuple of the compacted KV tensor (on CPU) and the token ids.\n",
|
|
" Raises:\n",
|
|
" ValueError: If there are fewer than 3 chunks.\n",
|
|
" \"\"\"\n",
|
|
" h = kv_tensor.shape[2]\n",
|
|
"\n",
|
|
" num_chunks = (h + chunk_size - 1) // chunk_size\n",
|
|
" drop_count = num_chunks // 2\n",
|
|
" if num_chunks < 3:\n",
|
|
" raise ValueError(\"Not enough chunks to drop.\")\n",
|
|
"\n",
|
|
" drop_start = (num_chunks - drop_count) // 2\n",
|
|
" drop_start = max(1, min(drop_start, num_chunks - 1 - drop_count))\n",
|
|
"\n",
|
|
" lo = drop_start * chunk_size\n",
|
|
" hi = (drop_start + drop_count) * chunk_size\n",
|
|
" keep_idx = torch.cat([torch.arange(lo), torch.arange(hi, h)])\n",
|
|
" kept_ids = token_source[:lo] + token_source[hi:]\n",
|
|
" logger.info(f\"compacting {drop_count}/{num_chunks} chunks\")\n",
|
|
"\n",
|
|
" e_kv = rerotate_k_cache(\n",
|
|
" kv_tensor[:, :, keep_idx, :].clone().to(work_device),\n",
|
|
" old_positions=keep_idx.to(work_device),\n",
|
|
" new_positions=torch.arange(\n",
|
|
" keep_idx.numel(), device=work_device, dtype=torch.long\n",
|
|
" ),\n",
|
|
" model_config=config,\n",
|
|
" )\n",
|
|
"\n",
|
|
" assert keep_idx[0].item() == 0 and keep_idx[-1].item() == h - 1\n",
|
|
" return e_kv.cpu(), kept_ids"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "e0675ef4",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"======================= Batched Stream Metrics (prefill) =======================\n",
|
|
"-------------------------------- Configuration ---------------------------------\n",
|
|
"Number of Streams: 30\n",
|
|
"----------------------------------- Results ------------------------------------\n",
|
|
"Total Duration (s): 26.96\n",
|
|
"Total Input Tokens: 418566\n",
|
|
"Input Throughput (tokens/s): 15523.32\n",
|
|
"================================================================================\n",
|
|
"\n",
|
|
"\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[32;20m[2026-07-04 06:32:10,001] LMCache INFO:\u001b[0m compacting 19/39 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,183] LMCache INFO:\u001b[0m compacting 19/39 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,298] LMCache INFO:\u001b[0m compacting 21/42 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,299] LMCache INFO:\u001b[0m compacting 31/62 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,303] LMCache INFO:\u001b[0m compacting 31/62 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,310] LMCache INFO:\u001b[0m compacting 31/62 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,390] LMCache INFO:\u001b[0m compacting 22/44 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,405] LMCache INFO:\u001b[0m compacting 21/42 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,417] LMCache INFO:\u001b[0m compacting 21/43 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,515] LMCache INFO:\u001b[0m compacting 29/58 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,572] LMCache INFO:\u001b[0m compacting 21/42 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,658] LMCache INFO:\u001b[0m compacting 24/49 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,692] LMCache INFO:\u001b[0m compacting 28/57 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,696] LMCache INFO:\u001b[0m compacting 25/50 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,721] LMCache INFO:\u001b[0m compacting 23/47 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,758] LMCache INFO:\u001b[0m compacting 25/50 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,848] LMCache INFO:\u001b[0m compacting 29/59 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,897] LMCache INFO:\u001b[0m compacting 27/54 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,936] LMCache INFO:\u001b[0m compacting 32/64 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,938] LMCache INFO:\u001b[0m compacting 26/53 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,941] LMCache INFO:\u001b[0m compacting 30/61 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,965] LMCache INFO:\u001b[0m compacting 30/60 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,971] LMCache INFO:\u001b[0m compacting 25/51 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,974] LMCache INFO:\u001b[0m compacting 29/59 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,986] LMCache INFO:\u001b[0m compacting 30/61 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,988] LMCache INFO:\u001b[0m compacting 29/59 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,989] LMCache INFO:\u001b[0m compacting 32/64 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,991] LMCache INFO:\u001b[0m compacting 31/63 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:10,993] LMCache INFO:\u001b[0m compacting 31/62 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n",
|
|
"\u001b[32;20m[2026-07-04 06:32:11,234] LMCache INFO:\u001b[0m compacting 32/64 chunks \u001b[3m(3563635992.py:28:__main__)\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"======================== Batched Stream Modify Metrics =========================\n",
|
|
"----------------------------------- Results ------------------------------------\n",
|
|
"Total Duration (s): 11.46\n",
|
|
"================================================================================\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"======================= Batched Stream Metrics (decode) ========================\n",
|
|
"-------------------------------- Configuration ---------------------------------\n",
|
|
"Number of Streams: 30\n",
|
|
"----------------------------------- Results ------------------------------------\n",
|
|
"Total Duration (s): 406.73\n",
|
|
"Total Output Tokens: 153542\n",
|
|
"Decode Throughput (tokens/s): 377.50\n",
|
|
"================================================================================\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"batch = lmc_batch.LMCacheBatchedStream()\n",
|
|
"for i, prompt in enumerate(prompts):\n",
|
|
" stream = lmc_stream.create_request(\n",
|
|
" ctx=ctx,\n",
|
|
" post_completion=post_completion,\n",
|
|
" prompt_token_ids=prompt,\n",
|
|
" )\n",
|
|
" batch.add(stream)\n",
|
|
"\n",
|
|
"results = batch.prefill(\n",
|
|
" sampling_params={\"max_tokens\": 1, \"temperature\": 1.0, \"ignore_eos\": True}\n",
|
|
")\n",
|
|
"results.emit()\n",
|
|
"\n",
|
|
"print(\"\\n\\n\")\n",
|
|
"\n",
|
|
"results = batch.modify(drop_tokens_fn)\n",
|
|
"results.emit()\n",
|
|
"\n",
|
|
"print(\"\\n\\n\")\n",
|
|
"\n",
|
|
"results = batch.decode(\n",
|
|
" sampling_params={\"max_tokens\": max_tokens, \"temperature\": 1.0, \"ignore_eos\": True}\n",
|
|
")\n",
|
|
"results.emit()\n",
|
|
"after_drop_data = results.to_dict()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "26ac7108",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Result Plot"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "f6316471",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAQAAAAEiCAYAAAAWFgxbAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlcelbwAAAAlwSFlzAAAPYQAAD2EBqD+naQAAQcNJREFUeJzt3XlYE1f7N/AvEIgIyCJuIOICgghSiqjgBoorLhVEUKtVH7uitm5Fu1Dt09a1VX8uj7W11h0Bxa0o4kK1KoqiFVEoorKIICKyGyC53z94MyUmYDKE/XyuK9dFTs5M7hmSOzNnzpyjQUQEhmFaJM2GDoBhmIbDEgDDtGAsATBMC8YSAMO0YCwBMEwLxhIAw7RgLAEwTAvGEgDDtGAsATBMC8YSAMO0YCwBvOb27ds4evRoQ4fRLCUlJSEsLAxisbihQ2ky0tPTERERgbCwMOTm5uLevXsICwuDRCJRy/oFallLA8vIyEBMTIxSdceMGQM9Pb1qX//111/x+++/o6ioSF3h8SYWixEeHs4919LSgoGBASwsLGBtbQ1NzaaVv8PDw7F8+XIUFhZCX1+/3t//4cOHiIuL455raGjAxMQEVlZWsLCwqPd43mTz5s34/PPPMXToUOjr68Pe3h4hISFYuXIlSktL0apVKwDAvXv3cO/ePXh7e6v8mWgWCSA1NRXBwcEyZUePHkXbtm0xePBgmfJBgwbVmAAak9LSUvj6+sLS0hJ9+/YFEaGwsBD37t1DcXExpk+fjhUrVsDU1LShQ1WKra0tfHx8IBA0zMfuzJkz+PjjjzF48GC0b98eRITs7Gxcu3YNY8aMwW+//dZo9mVxcTGWLVuGr776Cl9++SVXbmdnBx8fH2hpaXFlipKC0qiZ0tPTo+HDh6u8XEBAAOnp6dVBRKorLCwkAPTee+/JvXb27Fnq0qULmZub0+PHj+s/uCbof//7HwGgqKgomfIrV66Qpqamwv3cUO7cuUMAKDQ09I11v/nmGwJApaWlKr9PszgCUMWzZ89w69YtiMVi2Nvbo0uXLkotFxMTg4yMDAwePBgdOnTgyhMSEvDw4UMYGBigX79+aN26NfdaQUEBzpw5g379+qFLly6Ij49HamoqbGxsYG1tXavtGD58OCIiIuDo6IiAgACcPHlS4XvevXsXDx48gJubG9q3bw8AyM3NRVxcHMrKymBnZ4du3brJrPvu3btITEzE5MmTUVJSgitXroCI4OrqKnforkrdpKQkxMfHY9KkSdwvWNXlRSIRrly5grKyMri6uqJNmzYKtz0lJQX379+Hubk5nJyckJaWhuvXr2PkyJHVLlMTV1dXdOvWDTdu3FD4emlpKW7cuIG8vDxYWlqiT58+0NDQULgPlN2GmtZ59epVnDt3DgAQGxurcHnp4f6NGzdw7949AJWnWNra2gCAESNGwNDQ8M0br3puahpePwKoqKigBQsWkLa2Njk7O9PAgQNJR0eHpkyZQoWFhVw9RUcAa9asIU1NTfrmm2+4svj4eHJ0dCRjY2Py9PQkBwcHMjY2puDgYJk6AGjnzp3k7+9Prq6u5OrqShoaGrR06dI3bkNNRwBSXl5eBICys7Nl3nPHjh3k7e1N/fv3p549e1JUVBSJxWIKDAwkHR0dcnJyoiFDhpBQKKSJEydSXl4et87AwEACQLGxsWRnZ0cjRoygLl26kLGxMR0/flzm/VWpu2rVKgIgs7+ly9+9e5feeustbnlDQ0O6ePGizPLl5eX0n//8h7S0tKhfv37k5uZGEyZMoB07dhAAio+Pr3F/VncEkJ+fT3p6ejR+/Hi5ZbZt20aGhobUs2dPGjVqFLVv35769u1LqampvLZBmXWuWbOG3N3dCQC5urqSj48P9+jVq5fMr/2WLVu4skmTJnH1Hj16VOO+kGoxCUB6mFT1kOrq1avUunVrmjZtGldWNQGUlZXRnDlzSCgU0v79+7k62dnZ1L59e/Lw8JD54vzwww8kEAgoLi6OiP79Mjo4OMh8EL777jvS0NCg27dv17gNyiSAlStXEgA6deqUzHv26tWLzp8/T0REIpGIUlNTad26dQSAfv/9d275uLg4MjQ0pAkTJnBl0g/0lClTKDc3l4gqv3xTpkwhXV1dSklJ4VW3pgQwY8YMevHiBRERlZSU0Ntvv019+vQhiUQit99OnDjBlV28eJH7AiibAIKCgig0NJRCQ0Np69at5OzsTFZWVnT37l2Z+vv27SMAtH79eq4sPz+fXF1dycnJiYtNlW1Qdp2XLl0iABQeHi4Tk6LD/dqcArSIBFBRUUFGRkY0atQouXrz588nDQ0NevLkCRH9mwBevHhBHh4e1K5dO7p8+bLMMitWrCAAlJSUJFMuFoupc+fONHfuXCL698s4e/ZsmXp5eXkEgFavXl3jNiiTADZv3kwAuAQlfU9/f3+5umZmZjRw4EC58uXLlxMA+ueff4jo3w+0NKlIPXnyhLS0tOjzzz/nylSpW1MCOHfunMLtysjI4Mo6duxIo0ePlov/3XffVSkBDB48mPul9PDwIGNjY5o1axY9e/ZMpr61tTW5uLjIrefcuXMEgKKjo1XeBmXXWV8JoEW0ATx+/BgvX75E//795V4bMGAANm/ejNu3b8PMzAwAUF5ejgEDBkAikSAmJgbdu3eXWebq1avQ09PjLr9QZSIFEcHY2Bh37tyRqe/s7Czz3MjICAYGBkhPT6/1tpWWlgKA3JUNFxcXmefZ2dnIzMyEv7+/3DoGDBgAALh165ZM20Tfvn1l6pmZmcHc3By3bt2SW4cqdRV5fR9J22bS09Nhbm6OrKwsZGVlyb2PdNl9+/Yp9T4AEBQUBE9PT+75s2fPMGDAAAwfPhy3bt2ClpYWnj9/juTkZHh5eeHo0aOg/z9yHhHhxYsXAIA7d+5g6NChSm8Dn3XWtRaRAMrKygDIf0mqlolEIq5MU1MTHTt2xK1bt5CWliaXAEpLS6GhoaHwQ9ezZ0+5hkVFjTE6Ojp49eqV6hvzmsTERO59q2rXrp3Mc1X3AQCZBs2qZa/XU7WuIq/vIx0dHQDg9pE0fl1dXaXeWxXt27fH7NmzERQUhIsXL8LDw4NLrI8ePVL4f/bx8ZHrO/CmbeCzzrrWIhKAhYUFtLS0kJycLPeatKxqS7iWlhZOnz4NX19fjBkzBiEhIRg/fjz3eo8ePXD9+nUcOHCA+yc3hJKSEhw/fhzW1tbo1auXzGtVW6kBoEOHDmjVqpXS+wCobG13cHDgnpeXlyMtLU3hkZQqdfno2LEjhEIhHj58KPeaojJVSa9YPHv2jHs/PT09ODs7Y8+ePbVef12tE5D/X6uiaXUl40lfXx9jxozBoUOHkJ2dzZWXlpbi559/hq2tLRwdHWWW0dXVxdGjR+Ht7Q1vb2+ZjP2f//wHIpEIP/30k9x7icViZGZm1t3G/H8ikQizZ8/GixcvFMbxOh0dHbzzzjs4evQo0tLSuPKysjJs27YNlpaW3KmA1G+//Sb3vKSkBFOmTJFbvyp1+dDR0cHEiRMRGhqK58+fc+XFxcUIDQ2t9fqjoqKgoaEBJycnAIC2tjbee+89HD58GElJSXL1nz9/zv2iK6su1gkAbdu2BQDk5+ervGyLOAIAgK1bt2Lo0KEYMGAA5s2bh1atWmHnzp148eIFIiMjFWZRgUCAffv2wdjYGDNnzkR+fj4CAgIwcOBA/N///R8WLlyIa9euYcSIEdyv65EjR7B8+XLMmjVLbbGnpqYiLCwMAFBYWIiEhAQcOnQIYrEYYWFhGDdunFLr2bBhA+Li4uDm5oZPP/0UBgYG+P3335Gamoo//vhDroeehYUFpk+fDk9PTyQkJGDTpk2YPXs2xo4dK7duVery9eOPP+Lq1atwc3PDJ598AoFAgMOHD2P69On473//q/Qv4aVLl/Dy5UsAQF5eHv744w9ERkYiKChI5lRq3bp1ePDgAfr164cPPvgA9vb2yM/Px+3bt3H+/Hlcv35d4SlJTepincOGDYOWlhYWL16McePGQSAQKN0PoNkmgEmTJsk0aHXp0gV37tzBnj17EBsbC7FYDD8/P8yePZvrIAMATk5OMplUQ0MDW7ZsQZcuXRAdHY1+/frBxcUF8+bNw5gxYxAcHIyrV6+iVatW6NmzJ86cOQNLS0sAleeEPj4+3POqJkyYoLBBqyqBQAAfHx8AQHBwMLS0tKCvrw8LCwvs2LEDnp6eXMcPqZreU9qusXfvXsTExKC8vBwTJkzAkSNHuAbQqhYtWoQTJ07g2LFjkEgkOHDgACZPnqwwVmXqKuoK7ODgwG3j67H6+PjItGV07twZt27dwvbt2xEbG4vOnTvjt99+wx9//AFAcftGVT169ICPjw8SEhKQkJAAAGjTpg3efvttrF69Gra2tjL1W7dujdOnT+PMmTOIjIxEZGQkOnToAHd3d2zbto3rdqvKNii7TlNTU/j4+MDc3FxmnYq6AtvZ2eHs2bMIDQ3FkSNHIJFI0Ldv35bdEYjhT3pZS91168qnn35KQqGQysrKGjSOpqhFtAEwzQNVuVwmlZubi71798LLy0vuaIh5s2Z7CsA0P0SEgQMHYvz48ejduzcyMzOxfft26OrqYv369Q0dXpPEjgAYOdWd09a2bm1pamrir7/+gpmZGS5duoSkpCQsXLgQ8fHxcpcwGeVoELHJQRmmpWJHAAzTgrEEwDAtWLNLAESEb7/9Fj169IBQKJTr3cbIGjBgAEaNGlXv76uvr4/FixfX+/uq05EjRyAQCHDz5s2GDoW3ZpcAQkND8c0332DLli0oLCzE5cuX6/T97O3tIRAIIBAIoKOjAxMTE/Tq1QtTp07F4cOH1TZ6a12pqKhQyyi9ixcv5vZDTY+OHTuq9X0bkkQigVgsRkM2o0VEREAgEPD+nDe7BBAdHY127dphzJgx0NHRkekxVRcqKipga2uLV69eoaSkBI8ePcKBAwfQs2dPzJw5E4MGDZK7dt0crVu3Dq9eveIeN2/ehFgsRmBgoEz5kydPGjrUZqW2SajZ9QPIycmp9e2hfEi7txoaGsLJyQlOTk4YO3YshgwZgunTp+PUqVP1HlN90tTUlBmSWpp4NTQ0GmwUYObNms0RQGpqKndziPRvgUDA3aVWUVGB1atXo3fv3tDT00OnTp0wc+ZMpKamyqxHem56+fJlDBo0CPr6+li7di2vmPr3749p06bh9OnT+Pvvv2Ve27dvHzdopLGxMcaOHSszZr3UgQMHMGjQIBgZGaFjx46YPHky7t+/z72u7HaJRCIsXboUnTt3hpGREcaNGydzV+DrlI2vtmJjY7n93LNnT+zatUuuTk3/k9TUVMycOROdOnWCnp4eevfujdWrV8ucXvzf//0fBAKBzJ2gQOWArgKBAAcOHJArHzNmDNq0aQNzc3MEBQXh/v37EAgE1d7Gu3HjRnTr1g0GBgYYPny4zP8I+Le94MaNG/jhhx/QpUsXtGnTBiNHjkR8fLxMXWXj3bZtGyZOnAgAcHd3l/vMK6UBuyGrXXl5OXl7e5OlpSWVl5dTeXk5N8aav78/6erq0m+//UbPnz+n69evk6OjI3Xo0EFmyCahUEhjx46lCRMm0N27dyklJYWOHTtW7Xva2NhQ7969q3394MGDBIA2bNjAlUkH5vzxxx8pIyODUlNT6YMPPqDWrVvLjBO4ePFiEggE9O2339LDhw/p2bNndPjwYZkhxpTdrkmTJlGbNm3o0KFDlJubS3/++SeNHj2a7O3t5YZPVza+mkiHJvvyyy8Vvi4UCmn8+PHk6+tLCQkJlJ2dTQsWLCAAFBMTI1dX0f/kyZMn1KFDB+rTpw9du3aNnj9/Trt27SJdXV2ZIdE2bNhAAOjp06cKY9y7dy9XlpqaSsbGxjRw4EC6c+cOPXv2jDZv3kz+/v4EgHbt2sXVDQ0N5cYCXLt2LWVlZdHdu3epd+/eZGNjQ2KxWK6ur68vrVmzhqs7dOhQMjQ0lBk7Udl4JRIJHTt2jBtK7PXPvDKaVQIgIvLx8SFLS0uZsqtXrxIA+u6772TKU1JSSCAQ0Mcff8yVCYVCMjAwoPz8fKXe700JQDq2m3QU4ISEBNLQ0KCvv/5app5YLCYHBwduZNq///6bAFBgYGC161Z2u65cuUIAaNOmTTL1Ll68SABkEoCy8b2JMgnA1NRUZnxAkUhEJiYmNGfOHLm6iv4nAQEBpKWlRQ8ePJAp//777wkAN5ajKgng448/JqFQKFc3ICCg2gQgHQNSKjw8nABwg7JWrTtz5kyZujk5OaSnp0ezZs3iylSJ98SJEwSALl26RHw0m1OAmpw9exYA5Lqsdu/eHc7OztzrUsOGDeM1vnxNpPeqnzhxAkQkN1CGpqYmhg0bhgsXLgAAd4vrjBkzql2nstslHWP+nXfekak3ePBguZlwlI1PHTw8PGTmDtDR0YGNjY3CEX4U/U/Onj2Lt956Cz169JApl8b++v9VGefPn0e/fv24qxVSEyZMqHaZqqNFAUCfPn0AKB6pSHrILmVqaorBgwfzilUdWkTrTG5uLgCgU6dOcq917NhRbnSW1+/Brg1pq7f0fvusrCwA/w6iSf9/MFGgcjQhiUSCsrIybmiqmmJRdrukI+hUndCkar2qlI1PHUOhKYq7TZs2cue+gOL9kJubq3CCFek25eTk1Pj+pKDl/Pnz5zJDm0kp2ndSr2+HNFFJBx1503o6duyI6OjoGmMFFMdbWy3iCMDExAQAFH6wsrOzuSGVpNR5W2lUVBQAcCO9St/r3r17KCoqQnFxMUpKSlBSUgKRSITy8nLo6Ohwg0jUNLyYstslrSdNKq/Xq0rZ+NShuhF8FH3QFf1PTExMqt124N9tkQ6M8fqEr4r2rYmJicL9pKhMSpXtqO5/UPUzqEq8tdUiEoCHhwcA4NixYzLlaWlpuHnzJoYNG1Yn7xsXF4d9+/Zh5MiReOuttwAAXl5eACqncaquswwAbiitgwcPVrt+ZbdLWk86fZhUTEyM3K+ksvE1Bh4eHrh9+7bc1QzpjMrS7ZeO6nz37l2ZetLTrNfXGRsbKzPuYHV1+Xj9f5CXl4e//vqL+x+pGq90FCTpqMkq49Vy0IgpagQkIpo4cSIZGBjQoUOHqKioiOLj46lfv35kYmIiM7mmUCikTz/9VOn3e70RUCQS0f3792nVqlVkYGBALi4ulJOTI7PMwoULqVWrVvTTTz/RkydPSCQSUWJiIm3YsIEWLFjA1Zs3bx7p6OjQunXr6OnTp1RQUEARERH00UcfqbxdY8aMIRMTEzpx4gQVFxfT9evXacyYMWRnZyd3FUDZ+GqiTCOgov08atQocnR0VKru48ePycTEhPr160fx8fFUVFREhw4dIgMDA5o4cSJXTywWk42NDTk6OlJycjK9fPmStm3bRtOmTZNrVEtJSaE2bdrQ8OHDKTk5mQoKCmjnzp00efLkahsBY2NjZeLKyckhALRu3Tq5upMmTaLNmzdTfn4+paSk0KhRo0hPT48SExN5xfvkyRMSCAT0+eef8xoRqcUkAJFIRF9//TV17dqVNDU1ydDQkHx8fLjZcKT4JAAApKWlRVpaWtSqVSuytLSkd955h/bv308VFRUKl9u3bx+5ubmRrq4uCYVCsrW1pSVLllBaWppMvZ9//pkcHR1JW1ubTExMaPz48TKX4pTdrqKiIvrkk0/I2NiYhEIheXh4UHJyMjk7OyucRVnZ+KpTHwmAiCgpKYl8fHzI0NCQNDU1qWvXrvT111+TSCSSqXfnzh0aNGgQCQQCateuHa1YsYKbgbfqF4qocrq0oUOHcvt86dKldPPmTQJABw4c4OrxSQDXrl2j5cuXk6mpKWlra9OgQYPo+vXrctulSrybN28mCwsL0tDQIC0tLdq5c6fCfaVIsxsPQCKRgIh4dwEWi8XQ0NCQ6dX2pvrSXfh6b7imQNXtVUVFRUW1+6S691X0/6vrGLW0tN44onBERAS8vLxw/vx57nCdiCAWixWeFr2+3rCwMPj6+iI2NvaNg8HyjVcajzLbI9V4TujUpLYfElUTR13fa1DX6jL+mtoLqntfRf+/hoqxqiNHjkBXV1fmy1tTN+e6aiupab18ul03rZ8rhqkH8+bNw/nz51FUVISsrCysWbMGu3fvxuLFi2FgYNDQ4amVymmqpKQEp0+fxsWLF5GRkQGgclKIIUOGYNSoUQ1yIw7DqNO0adOwcuVK3Lx5E/n5+bC2tsb69euxYMGChg5N7ZRuAygoKMAPP/yAn3/+GWKxGE5OTlynhuzsbMTFxUFbWxsffvghli9frvaedAzTlNXUXtCQlI6mV69eGDJkCA4fPoyhQ4fKnZeJxWJER0fj119/Ra9evdh93wxTRWO9LVrpI4CEhAT07t1bqZWqUpdhmIajtsuAFRUVjTLDKUsikSAzMxMGBga1mm6ZYWqLiFBYWAgzM7O6v6ysdI+BKpKTk2VuF12+fDkJBALq0aMHJSQk8Fllg0tPTycA7MEejeaRnp5e5597XkcA48ePxyeffIIxY8YgJSUFDg4O2LVrFy5cuID09HS19ZuuT/n5+TAyMkJ6ejprwGQaVEFBASwsLPDy5UvlZvitBV4JwMjICBkZGdDX18e2bdtw8eJFBAcH48WLF7CysmqSg2AWFBTA0NAQ+fn5LAEwDao+P4u8TjC0tbW5u6VOnz7N3XUlkUiafM84hmlJeLXajRo1Cv7+/ujXrx/Onz+Pn3/+GUDlkNx1dWstwzDqx+sIYOvWrXB1dUVGRgZCQ0O5EVFOnTqFFStWqDM+hmHqkEptAEVFRTJjuDUnrA2AaSwabRuAqakpRo0ahc2bN+PRo0d1FRPDMPVEpQRw7949jB07FsePH4etrS169+6NwMBAXLp0qcnP88YwLRHvnoCFhYU4c+YMTp48iVOnTqG8vByjR4/GuHHjMHr0aBgbG6s71jrFTgEav9zcXOzYsQOxsbFo3bo1PDw8MGPGDG6Q0hs3bmDZsmUKl12zZg2cnZ2VqqOIj48P8vPzZcpmzpyJmTNn1mKLFKvPz6JaugITEa5fv46TJ0/i5MmTuHv3Ltzc3PDnn3+qI8Z6wRJA41ZUVAQXFxf4+fnByckJWVlZ+P7779G3b18cOXIEQGWCuHXrlsxyv/76K44fP46nT5/C0NBQqTqKmJqa4qOPPoK7uztX1r17d24AT3Wq18+iuroUlpeXc39nZGTQ9u3bVVo+MTGR1q1bJzObilRxcTGFhITQ5s2bKTo6WuHyytSpSX5+PgFQekYgpn5VVFRQUVGRTFlwcDBpaGhQQUGBwmUkEgl1796d3nvvvWrXq0wdIqK2bdvSwYMHVQ2bl/r8LDaKewFKSkqod+/epKenJzf4Y2ZmJllZWZGjoyPNmjWL2rVrR9OnT1e5zpuwBND0fP7552RpaSkzB19V586dIwD0119/VbsOZeoQVSaAwYMHk5eXF33yySd08eLFWsVek0afAMaNG0cRERFERPTgwQPS1dWl4OBg+vDDD2ns2LEqr2/u3Lm0YMECcnR0lEsAM2bMICcnJ3r16hURVY42q6mpSeHh4SrVeROWAJqGH3/8kYYNG0ZWVlbUt29fuXkBq5o2bRr16tWrxvUpU4eI6O2336Z169ZReHg4LVq0iHR0dGjLli0qx6+MRp8ADA0NuUkdt27dSn5+fkRElJubS8bGxiqt69ChQ2Rvb0+lpaVyCaCiooL09PRo48aNMssMGTKEpk6dqnQdZbAE0DTcu3ePIiMjacuWLdSlSxe5yTal8vLyuLkNqqNMHanXTz++++470tXVpeLiYtU2QAn1+Vls0HsBHj16hPnz52P//v1o1aqV3Ovp6ekoLi6GjY2NTLmtrS03/7oydRQRiUQoKCiQeTCNX69evTBy5EgEBAQgJCQEe/bswfXr1+Xq7d+/H0RUYyu9MnWkpDPwSI0dOxalpaVITExUfSMaEV4JQHovwIIFC3D+/HludlRV7gUoLy/H1KlTsWzZMm421dcVFhYCqLz7sCojIyPuNWXqKLJq1SoYGhpyDwsLC6XiZhoPS0tLAP9OaFrVzp074e3tLTfvo6p1qiP9AVT0w9WUNNi9APv27UNiYiIqKiqwfv16rF+/Hjk5OYiLi8P69etBRNwIw6//Oufn53MZWZk6iixfvhz5+fncIz09Xam4mYZx+fJlXLt2jXsuFouxfv166OnpwdXVVaburVu3cOvWLbz//vvVru9Ndd555x3s2bMHAHD16lVcuXKFe+3Fixf45ptvYGdnh169etVmsxocr7sBDQ0NsWHDBrnynTt3Kr2Onj17Yu7cuTKzu5aXl6O4uBhZWVkgIlhaWkIoFCIlJUVm2ZSUFG5aaGXqKCIUCiEUCpWOl2lYHTt2xIcffoiUlBSYm5sjJSUFRkZGCA8P52ZSltq5cyesrKxkrtm/7k11oqOjuQldO3bsiA8++ADJycno1KkTEhIS4OzsjGPHjjX94eNq04CQm5tLycnJcg++FF0F8Pb2poEDB3KXeh49ekTa2tq0b98+leq8CWsEbBoyMjLo8uXLlJKSUu3lv6tXr9K9e/dqXM+b6kRHR1NKSopMWXp6Ol25coWePHmieuAqqM/PIq+egDdv3sS7775bbQMIj1UCAN566y24u7tj48aNXFlKSgrc3Nzg4OCAfv36ITg4GNbW1oiIiOAaHJWp8yasJyDTWDT6rsAuLi6wtbXFokWLFPb579q1K69gNmzYgB49emDChAky5Tk5OTh48CCys7Ph4OAAX19fuS+2MnVqwhIA01g0+gSgp6eHzMzMOh+wsD6xBMA0Fo12PACpbt26IS8vT92xMAxTz3glgMDAQHz00UdsUBCGaeJ4nQIIBAJuABBNTU25SyEVFRXqia4esVMAprGoz88ir34AJ0+eVHccDMM0AF4JYPTo0eqOg2lJDjTxzjPqMq3WY/HUGu+ZByUSCf7++2+Eh4dzZSKRSC1BMQxTP3glgKdPn8LV1RVvv/02vL29uXJvb29ERkaqLTiGYeoWrwSwcOFC9OzZU26QxGXLluGHH35QS2AMw9Q9Xm0A586dQ0JCgtwkIY6OjgrvzWYYpnHidQRQUlLCDcVc9RJgTk5Ok78/mmFaEl4JYNCgQdi1axeAfxNAWVkZvvjiixpvwWQYpnHhdQqwbt06uLu7IzIyEkSE2bNn48KFCygoKJAZOIFhmMaN1xFAnz598Pfff8PR0REeHh54/PgxfHx8cPv2bdja2qo7RoZh6givI4DDhw/Dx8cHa9askXvtq6++wnfffVfrwBiGqXu8jgBmz56Nc+fOyZV/9dVX2LZtW62DYhimfvBKANLRVGNjY7myL7/8Ev/73/8UJgaGYRonXqcAvr6+yM3NxdixY3Hx4kXs3bsXP//8M86dO8cNpMgwTOPHKwEAwEcffYTnz5/DxcUFrVq1Yl9+hmmClE4A1TXsGRsbw9XVlZsaHKhsC2AYpvFTOgFUNwaAubk50tLSkJaWxpWxBMAwTYPSCSAmJqYu42AYpgHwHg+AYZimj3cj4NOnT7F161bcv38fRAQ7OzsEBARw8wQyDNP48ToCuHz5MqytrRESEgJdXV3o6ekhNDQU1tbWuHz5srpjZBimjvA6AliyZAkCAgKwevVq7m5AIkJgYCCWLFmCq1evqjVIhmHqBq9hwYVCIbKysuSmBXvx4gXMzMzw6tUrtQVYX9iw4PWIDQpaqZpBQRv9zECGhoZ4/PixXPnjx4/Zl4dhmhBeCcDf3x9+fn44evQoMjMzkZmZifDwcEyZMgX+/v7qjpFhmDrCqw1g7dq1qKiogK+vLzcLkEAgwPvvv4+1a9eqNUCGYeoOrwTw/PlzbNu2DT/88AOSkpKgoaGBnj17wsjICBkZGejcubO642QYpg7wSgAWFhYgIhgZGaF///4KX2MYpvFTa0/AkpIS6OrqqnOVDMPUIZWOAKre5PP6DT8SiQRxcXHslmCGaUJUSgB//fWXwr8BQFtbG127dsXSpUvVExnDMHVOpQQQHR0NAJg8eTLCwsLqIh6GYeqR0m0AOTk53N9v+vJXrVuTsrIy7NixAz4+Phg1ahQWLVqER48eydW7ePEi/Pz84O7ujvnz5+Pp06e86jAMI0vpBGBvb4+vv/5aZuCP1z169AhffvklevfurdQ6Z8+ejXv37mHWrFn47LPP8OjRI7i4uCA9PZ2rc+HCBQwfPhzW1tZYunQpkpOTMXDgQBQWFqpUh2EYeUrfC/D48WMEBgYiPDwc9vb2cHZ2RocOHUBEyMrKQmxsLO7fvw9vb2+sWbMGXbt2feM6S0pK0Lp1a+55WVkZ9PT08Msvv2DWrFkAgIEDB8LCwgLBwcHcMp06dUJQUBAWL16sdJ03YfcC1CN2L0ClpnQvQNeuXXHo0CH8888/8Pf3x7NnzxAREYHTp0/j+fPnmDFjBh48eIBDhw4p9eUHIPPlB4CrV69CLBbDwcEBQOUXOSYmBuPHj5dZxtPTE2fPnlW6DsMwiqncEahr1674/PPP1RZATEwMPvvsMxQUFCArKwtHjx6Fs7MzACA9PR0SiQRmZmYyy5iZmXHzDyhTRxGRSASRSMQ9LygoUNcmMUyT0eBDgvXq1QsbN27E2rVrMWTIEMyfP59rZygvLwdQeftxVbq6utxrytRRZNWqVTA0NOQeFhYWatsmhmkqGjwBGBoaYsCAARg3bhwOHz4MHR0dbNq0CQBgYmICoHKcgapyc3PRtm1bpesosnz5cuTn53OPqg2PDNNSNHgCqEpLSwvt2rXD8+fPAVQexnfs2FFmCjIAuHbtGpycnJSuo4hQKESbNm1kHgzT0vBKABkZGbxeq6q0tBQ//vijzGH68ePHERsbizFjxnBlc+bMwa+//oonT54AqJyZ+N69e5gzZ45KdRiGkVeruwFVfa0qoVCIvLw8mJmZoVOnTnj58iVKS0uxZs0amUFFgoKCkJycDCsrK1hYWCAjIwNbtmyBi4uLSnUYhpHHa0xADQ0NhV/ykpISmJqaoqSkROl1lZeX459//oGenh46d+4MgUBxTsrMzER2djasrKxgYGDAu051WD+AesT6AVRqBP0AGvxuQG1tbaV6DpqZmcld6uNTh2GYf7G7ARmmBWN3AzJMC8brKgD78jNM88DrKsBnn31W4+sbN27ks1qGYeoZrwTw4MEDmecSiQQPHjxAcnIyPD091RIYwzB1j9cpwMmTJ2UeERERSEpKwrJly9iYgK9JSkrC3Llz0b17d3Tq1Anjxo1DXFycTJ28vDx89dVX6N27N0xNTdG/f3/s379fbl25ubkICAhAly5dYG5ujoULF6K4uLjG9+ezDNNyqK0rsIaGBpYuXYqQkBB1rbJZWLx4Mdzc3HD27FncvHkT5ubm8PDwkJlabfPmzWjbti3Cw8ORmJiIuXPn4r333sPx48e5OsXFxRg6dCju3buHiIgIxMfHw8rKCuHh4dW+N59lmJaFV0eg6qSkpMDZ2RkvX75U1yrrTX11vhCLxTA0NMSaNWsQEBBQbT07Ozt4eXlh3bp1AIDvv/8e69evx6NHj2BkZKTUe/FZpl6wjkCVmlpHIKnt27fLleXl5WH37t0YNWpUrYNqzoqKilBeXg59fX2Fr1dUVCAiIgKpqanw8vLiysPDw+Hl5aXSF5nPMkzLwisBrF+/Xq7M2NgYY8eORVBQUK2Das4CAwPRpk0bmRGMACA1NRU2NjYoLy+HQCDA1q1b4e7uzr2ekpKC4cOHw9/fH2fOnEHbtm0xceJErFixotpkwmcZpmVRy1UARjk//fQTfv/9d0RERHDjGEh16dIFL1++REFBAY4ePYqAgAAYGxvDx8cHQOWpw08//YQtW7Zg69at+OeffzB9+nRkZ2dj7969Ct+PzzJMy1LrNgDp7bza2tpqCaih1PV51+bNm7F06VIcOXIEY8eOfWP9adOm4cmTJ/jzzz8BAD179oSZmRnXGxOoPBVbuHAhioqKoKWlJbcOPsvUC9YGUKkRtAHwugpARNi+fTtsbGygq6sLXV1d2NraYseOHWxiUAW2bt2KpUuXIiwsTKkvP1C5j6vuy4EDB8p9YbW0tOTqVcVnGaZl4ZUAgoKCsHz5ckydOhXHjh3DsWPH4O/vj8DAQKxYsULNITZt27dvx+LFi3H48GGMGzdOYZ3Zs2cjJiYGr169QlFREXbt2oWwsDDMmDGDq/PZZ5/hypUrOHToECoqKpCYmIgff/wRPj4+3C3UGzdulDm3V2YZpmXjdQpgamqKgwcPYsSIETLlZ86cwfTp05WeGagxqavDrnbt2iE3Nxc6Ojoy5QsXLsSqVasAAJcuXcLKlStx7do1EBFsbW2xcOFCTJ8+XWaZU6dOYcmSJUhKSkLbtm3h6+uLVatWcWMfrF+/HsuWLUNFRYXSyzQIdgpQqRGcAvBOAA8fPpQLLj8/Hz169ODG9GtK6mqni0QihYfbAoFA7leYiKCh8eYvh0Qigaam/MGbWCxGRUWF3AjJNS3TIFgCqNQIEgCvT8TgwYOxZ88eufI9e/ZgyJAhtQ6qOREKhWjVqpXcQ9EhuDJffgDVfpG1tLQUfvlrWoZp2XidCJqZmWH+/Pk4cuQIXFxcQES4ceMGLly4gE8++QTfffcdV/f1kYMYhmk8eJ0CDBgwQOm6MTExqq6+QbAxAesROwWo1AhOAXgdATSVLzXDMDVjJ4YM04LxOgIgIoSEhODy5ctyU3IBwL59+2odGMMwdY/XEcCSJUswZ84cpKamcpezqj4YhmkaeH1b9+7di6ioKLi5uak7nkZLySt0zR7rQdy88L4XwNHRUd2xMAxTz3glgGHDhuGPP/5QdywMw9QzpU8BqnbuadeuHWbMmIE//vgDVlZWcj3YWOcfhmkalO4I1Bw7/1T1ps4XrA2gklraAFhHoEpNqSNQU/xSMwxTM9YRiGFaMLVPDSYUCtG9e3dMmjQJ7du35xsXwzD1gFcCuH//Ps6cOYN27dqhd+/e0NDQQHx8PJ4/f44hQ4bgwIEDCAwMxKVLl+Dg4KDumBmGURNeCcDKygpWVlbYsGEDN9KNSCTCZ599Bi0tLZw/fx7z5s3D4sWLcebMGbUGzDCM+vC6Hbhz5874+++/0bZtW5ny58+fw8nJCenp6cjMzISDgwNyc3PVFmxdYlcBlMOuAqhRI7gKwKsRMC8vD9nZ2XLlWVlZyMvLA1A5TLiurq5S6ysoKEBJSUmNdV69eoWsrCxIJJJa1WEY5l+8EsDEiRMxZcoUnDp1Cjk5OXj27BlOnToFPz8/TJgwAUDltFQTJ06scT27d++Gg4MDunTpgnbt2sHZ2VnucqNEIsGnn34KIyMj2NjYwMzMDGFhYSrXYRhGHq8EsH37dvTt2xfjx49H+/bt0aFDB4wfPx4uLi7cvIHm5uZYs2ZNtesQi8W4cOECgoOD8fLlS+Tl5aFfv37w8vKSucV43bp12L9/P27cuIGXL18iKCgIU6dORUJCgkp1GIaRV6uZgV68eIF//vkHGhoasLa2lpvuSlWZmZkwNzfHqVOnMHr0aACApaUlpk6ditWrV3P1rK2t4eXlhY0bNypd501YG4ByWBuAGjWCNoBa3bxvYmKiUhfhN0lOTgZQOegoADx79gxpaWlytx0PHDgQsbGxStdhGEYxtXcEAqD0r25VxcXFmD9/Pjw9PdGnTx8A4CYYef1qg6mpKa5cuaJ0HUVEIhFEIhH3vKCgQOWYGaapU8vswBKJBA8ePEBycjI8PT1VXp9IJIK3tzfKysqwf/9+rlw6ln3VmW6AyglJpXPeKVNHkVWrVmHlypUqx8owzQmvBHDy5Em5MiLCF198IfdFfJOysjJ4e3vj0aNHiI6Oluk+3LlzZwCVlxerysrKgrm5udJ1FFm+fDkWLVrEPS8oKICFhYVKsTNMU6e2m4E0NDSwdOlShISEKL2M9MufnJyM6Oho7txfysDAAG+//TYiIyO5svLycpw7dw5Dhw5Vuo4iQqEQbdq0kXkwTEuj1hE88/LykJ+fr1RdsVgMX19fxMbGIjw8HGVlZXj8+DGAyvN36Sy3QUFBmDx5MpydneHq6ooff/wRWlpa+Pjjj7l1KVOHYRh5vC4DSq/1V5WXl4fdu3fD0dERhw4deuM68vLy4OTkpPC1NWvWwM/Pj3t++PBhbNq0CdnZ2XBwcMD3338PGxsbmWWUqVMTdhlQOewyoBo1gsuAvBKAlZWVXJmxsTEGDx6MoKAgGBkZqSO2esUSgHJYAlCjRpAA1HIVgGGYpomNCMQwLRjvRsCnT59i69atuH//PogIdnZ2CAgIQKdOndQZH8MwdYjXEcDly5dhbW2NkJAQ6OrqQk9PD6GhobC2tsbly5fVHSPDMHWE1xHAkiVLEBAQgNWrV3NzAhARAgMDsWTJEly9elWtQTIMUzd4XQUQCoXIysqCsbGxTPmLFy9gZmaGV69eqS3A+sKuAiiHXQVQo0ZwFYDXKYChoSHXaaeqx48fsx51DNOE8EoA/v7+8PPzw9GjR5GZmYnMzEyEh4djypQp8Pf3V3eMDMPUEV5tAGvXrkVFRQV8fX25m38EAgHef/99rF27Vq0BMgxTd3i1AVRUVEAgEODly5dISkqChoYGevbs2SR7AEqxNgDlsDYANWoEbQC8jgB0dHQgkUhgZGSE/v37qzsmhmHqCa82gPbt2yscFpxhmKaFVwL48MMP8cUXX6C0tFTd8TAMU494nQIcP34ct2/fRkhICLp168ZNDyZ148YNtQTHMEzd4pUA/Pz8ZO7XZximaeKVAJYtW6buOBiGaQDsdmCGacF4JYAnT55g6tSp6NKlC/T19eUeDMM0DbxOAf7zn/8gPz8fK1eulLshiGGYpoNXArh8+TKSkpLkhvFmGKZp4d0RSFtbW92xMAxTz3glgFmzZmHFihUqzwLEMEzjovQpQN++fbm/xWIxbt++jUOHDsHS0pIbFUiKdQRimKZB6QQwefJkmeesIxDDNH1KJwDW+Ydhmh+V2gAUzQjEMEzTpVICSElJqas4GIZpAKwrMMO0YCp3BPrrr7/eWGfQoEG8gmEYpn6pnAAGDx78xjo8hhlkGKYBqJwA7t+/XxdxMAzTAFROALa2tnURB8MwDYA1AjJMC6ZSAmBDgDNM86JSAoiJiamrOBiGaQC8xgNQJ7FYjFOnTiExMRF+fn6wsLCQq1NSUoI//vgD2dnZcHBwwNChQ3nVYRhGVoO2AYSFhcHKygpr167F0qVLFfY0fPr0KRwdHfH999/j5s2b8PX1xbvvvqtyHYZh5DXoEUC7du0QHR0NLS0thb/8ABAYGAgDAwNcvXoVQqEQd+/ehaOjIyZPnox33nlH6ToMw8ir1RFARUUFHj16xHv5oUOHwtLSstrXxWIxjhw5gvfeew9CoRAAYG9vj0GDBiEkJETpOgzDKMYrAZSUlGDu3Llo3bo1unfvzpVPnz4dt2/fVldsSE9PR3FxMWxsbGTKbW1tuQ5JytRRRCQSoaCgQObBMC0NrwTw5ZdfIjExEdHR0TLl/v7+WLlypTriAgAUFhYCgNy040ZGRtxrytRRZNWqVTA0NOQe1Z2CMExzxisBhIWF4ffff4ebm5tMuZubG86dO6eWwACgdevWACD365yfnw89PT2l6yiyfPly5Ofnc4/09HS1xc0wTQWvRsCcnBx06tQJAGTGAxSJRBCLxeqJDIClpSWEQqHc1YGUlBRYW1srXUcRoVDItRkwTEvF6wigT58+iIyMBCCbALZt2wYXFxf1RAZAIBDAy8sL+/fvh0QiAQA8fvwYf/75JyZNmqR0HYZhFON1BPDtt9/Cz88P169fBwBs2rQJp0+fRlRUFKKiopReT0JCAk6dOoX8/HwAwKFDh3Djxg24ublxpxdr166Fm5sbRo4ciX79+iE4OBgeHh7w9/fn1qNMHYZh5PE6Ahg9ejSOHTuGW7duwcjICCtWrEBZWRnOnj0LDw8PpddTWlqKrKwslJaWYvHixdDT00NWVhaKioq4Oj169MDdu3cxYcIEaGho4IcffkBERAS0tLRUqsMwjDwNYqN3AKhsRDQ0NER+fj7atGkj9/prUx+0WGr5tBxgOxMAME3xznzTZ1Gd2O3ADNOCKd0G0LVrV6VX+vjxYx6hMAxT35ROACtWrOD+Tk5Oxrp16zBlyhSu1T82NhYhISFYunSp2oNkGKZuKJ0AZs2axf09YsQI7Nq1C9OnT5epM2bMGOzevVttwTEMU7d4tQHExsZi/PjxcuXjx4/nLg0yDNP48UoArVu35joCVRUZGVlj91uGYRoXXh2BPv/8c8ycORNRUVFwcXEBEeHGjRvYu3cv1q5dq+4YGYapI7wSwGeffQYrKyts2LABJ06cAADY2dkhLCwMXl5eag2QYZi6w3tEoHHjxmHcuHHqjIVhmHpW645AYrFYrXcAMgxTf3gngL1798Le3h66urrQ1dWFvb099u7dq87YGIapY7wSwKZNm/Dxxx9j9OjROHDgAA4ePIjRo0fjo48+wqZNm9QdI8MwdYTXzUDdunXDxo0bMXHiRJnyo0ePYtGiRXj48KHaAqwv7GYg5bCbgdSoqd4M9OTJEwwbNkyufNiwYcjIyKh1UAzD1A9eCaBbt244fvy4XPmxY8fQrVu3WgfFMEz94HUZ8IsvvsCcOXMQFRWFfv36AQCuXbuGgwcP4pdfflFrgAzD1B1eCeC9995Dp06dsHbtWkRGRkJDQwN2dnY4efIkRo4cqe4YGYapI7w7Ao0cOZJ92RmmiePdDyA7O1upMoZhGi9eCeB///ufwhmAVq5cie3bt9c6KIZh6gevfgA9evTA2bNn5Vr8Hz58iNGjR+Off/5RW4D1hfUDUA7rB6BGTbUfQGZmJgwMDOTKDQwM2BRbDNOE8EoAjo6OCof+2rVrFxwcHGodFMMw9YPXVYAVK1Zg3LhxuHbtGoYMGQIiwsWLFxEeHs6ND8AwTOPHe2agyMhI5Obm4uuvv8Y333yDFy9eIDIyEmPGjFF3jAzD1BHe/QCGDx+O4cOHqzMWhmHqWa0GBKmoqMCjR4/UFQvDMPWMVwIoKSnB3Llz0bp1a3Tv3p0rnz59Om7fvq2u2BiGqWO8EsCXX36JxMREREdHy5T7+/sr7CDEMEzjxKsNICwsDBcuXICVlZVMuZubm9xsQQzDNF68jgBycnLQqVMnAIBGlS5yIpGIDRDKME0IrwTQp08fbmagqglg27Zt3GShDMM0frxOAb799lv4+flx8wBu2rQJp0+fRlRUFKKiotQaIMMwdYd3R6Bjx47h1q1bMDIywooVK1BWVoazZ8/Cw8ND3TEyDFNHeHcEcnd3h7u7uxpDYRimvvHuCPTy5UvcuXMH8fHxyM/PV2dMvFy8eBF+fn5wd3fH/Pnz8fTp04YOiWEaPZUTQFJSEkaMGIG2bdvC0dERffr0gYmJCUaNGoXk5OS6iPGNLly4gOHDh8Pa2hpLly5FcnIyBg4ciMLCwgaJh2GaCpUGBMnOzoaDgwPMzc0xd+5c2NjYAAASExPxyy+/IDs7G/Hx8WjXrl2dBazIwIEDYWFhgeDgYACVPRU7deqEoKAgLF68WKl1sAFBlMMGBFGjpjYgyMaNG2Fvb4/r168jICAAnp6e8PT0xLx58xAbGwsbG5t6nxqspKQEMTExGD9+PFfWunVreHp64uzZs/UaC8M0NSo1AkZGRmL9+vXQ1taWe01HRwdBQUEIDAzEd999p7YA3yQ9PR0SiQRmZmYy5WZmZjh37ly1y4lEIohEIu65tB2joKCgbgJtJtSye0rUsI7moJqdKf0M8hitT2UqJYCUlBT07du32tf79u2LBw8e1DooVZSXlwMAhEKhTLmuri73miKrVq1SeN+ChYWFegNsZgwNGzqCZuT9mndmYWEhDOt4h6uUAAoLC2s8JzE0NKz3hjcTExMAwIsXL2TKc3Nz0bZt22qXW758ORYtWsQ9l0gkePHiBdq2bSvTu7GxKCgogIWFBdLT0+v8vLC5a+z7kohQWFgod1RbF1RKAESEioqKGutIJJJaBaQqMzMzdOzYEbGxsRg3bhxXfu3aNQwePLja5YRCodxRg5GRUV2FqTZt2rRplB/apqgx78u6/uWXUvkyoLa2do2PhjBnzhz8+uuvePLkCQDg8OHDuHfvHubMmdMg8TBMU6HSEcDevXvrKo5aCQoKQnJyMqysrGBhYYGMjAxs2bKF3ZjEMG+gUgJ499136yqOWhEKhQgJCUFmZiays7NhZWWlcN6CpkwoFOKbb76RO21hVMf25b94zQzEMEzzUKtBQRmGadpYAmCYFowlgAaSmZmJ0NDQGuukpaUhPDy8niJqGbKyshASElJjnYyMDBw5cqSeImpYLAHUUlRUFGJjY2XK8vLyEBwcjKSkJJny1NRUBAcHo6KiAnFxcZgxYwb32pMnT3D48GGZ+leuXMH7779fd8FXoej9G7Nz585xI1JJvXz5EsHBwUhMTJQpT0tLQ3BwMMrLy3H79m1MmzaNe01RIo6JiWk5l5CJqZU5c+aQq6urTNnvv/9OAGj27Nky5cuWLSNLS0siIoqLi6MZM2Zwr4WHh5Oenp5M/YMHD1Lbtm3rJvDXKHr/xuyDDz4gFxcXmbJ9+/YRAJn9SkT01Vdfkbm5ORER3b59m6ZNm8a9duLECRIKhTL1Q0NDydDQsG4Cb2R4jwjEVPLw8MCePXtQXFwMPT09AP+OT3DhwgWZuhcuXMCwYcMAAB06dODuYHz+/Dn++usvVFRUcLc029racstJJBLcvXsXaWlpsLe3R9euXeXiSExMxP3792FqagpXV1cIBP/+a5OTk5GcnIyxY8dyZdnZ2bhw4QL8/f2rff+33nqr9juojnh4eGDnzp0oKCjgevMpu98nTpwIoLK7+KVLlyCRSLjtlt7iDlT2fI2Pj0daWhp69+6Nbt261cem1a+GzkBNXUZGBgGg06dPc2WWlpYUGRlJrVu3psePHxMRUWFhIQkEAtqzZw8Ryf7yJCYm0qBBg0ggEJCfnx/5+fnRvn376ODBg6Svr08eHh7k5uZGnp6epKOjQ7t375aJYfbs2WRgYECjRo2ibt26ka2tLaWmpnKvb968mWxsbGSWiYqKIum/v7r3b8yysrIIAJ08eZIr69GjB0VERJC+vj49ePCAiIiKi4tJR0eHfvvtNyIiOnXqFGlpaRERUXJyMg0ePJg0NTW57d69ezeFhoZS69atydPTk1xdXWnEiBEy62hOWAJQg549e1JgYCARET18+JCEQiGVlpbS8OHDadeuXUREFBERQQAoIyODiOQPPas7BQBA27dv58pWr15NZmZm3PPg4GDS1dWl+/fvExHRq1evaNCgQTRp0iSuzpsSQHXv39jZ2dnR4sWLiYgoLS2NtLW1qaioiEaNGkW//PILERGdOXOGAHCJuGoCIKr+FAAAbdmyhStbv349tW/fvq43qd6xRkA18PDw4A47L1y4gP79+6NVq1YYOnSoTLm1tTXMzc1VWre2tjbmzp3LPXd3d0dmZiaKiooAAMHBwZg0aRJ3yiAUCrFkyRIcPXoUr169UsfmNVqv73cXFxfo6enJ7fdu3brB0tJSpXVraWnhgw8+4J67u7vj2bNnePnypdribwxYAlADDw8P3Lx5EwUFBbhw4QI3WvLrH0Tpeagq2rRpAy0tLe65tPuq9MudmpoqM0ErAPTo0QNEhPT0dD6b02R4eHjg9u3byMvLU/t+19fXl7m57fX93lywBKAGHh4eEIvFuHTpEqKjo7kPYv/+/ZGTk4O4uDjcunWL1wfxTUxNTeXGQpA+NzU1BQBoamrK3abdHD7I7u7uICJcvHhRJgG4uLggPz8fN27cwI0bN+pkvzcXLAGoQfv27WFnZ4cdO3YgJycHrq6uACp/Nfr374+VK1dCIpHUOI+Cvr4+RCKRysNADRo0CCdPnkRZWRlXFhoaCjs7OxgbGwMAzM3NkZmZidLSUq7O6y3lfN+/IbVt2xYODg747bffkJmZCTc3NwCVp02urq7473//i4qKihonq9HX10d5eXm9j2PRWLDLgGoybNgwbNmyBUOGDEGrVq248qFDh+Lbb7+Fvb092rdvX+3yDg4O0NLSwrJly+Dk5CRzGbAmixYtwu7duzF8+HC8++67uHPnDn755RecPHmSq+Pp6QlDQ0P4+vrC29sbcXFxcj3dFL1/Y74MKDVs2DBs3LgRbm5u3GVYoHK/BwUFwdbWlpvIVpHevXtDR0cHgYGBcHZ2lrkM2BKwIwA1mTJlCvz8/OR67k2cOBF+fn6YN2+eTLm5uTmmTJnCPe/QoQOioqJQWlqKY8eOISEhAZaWlvD29pZZztjYGH5+ftw5qb6+Pm7cuAEvLy9cvnwZQqEQ165dw8iRI7ll9PT0cO3aNfTp0wcxMTGwt7fHyZMn4efnV+P7NwW+vr7w8/PDhx9+KFM+YcIE+Pn5YcGCBTLlnTp1ktnudu3a4ezZsxCJRDh27Bji4+NhYWEBHx8fmeWMjIzg5+cHXV3dutuYBsBuB2aYFowdATBMC8YSAMO0YCwBMEwLxhIAw7RgLAEwTAvGEgDDtGAsATBMC8YSAMO0YCwBMEwLxhIAw7RgLAEwTAvGEgDDtGD/D1udNk89koLuAAAAAElFTkSuQmCC",
|
|
"text/plain": [
|
|
"<Figure size 250x300 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"labels = [\"Without\", \"With\"]\n",
|
|
"data = [\n",
|
|
" before_drop_data[\"metrics\"][\"results\"][\"output_tput\"],\n",
|
|
" after_drop_data[\"metrics\"][\"results\"][\"output_tput\"],\n",
|
|
"]\n",
|
|
"\n",
|
|
"fig, ax = plt.subplots(figsize=(2.5, 3))\n",
|
|
"bars = ax.bar(labels, data, color=[\"blue\", \"orange\"])\n",
|
|
"ax.bar_label(bars, fmt=\"%.1f\", padding=3)\n",
|
|
"ax.set_ylabel(\"Decode Throughput (tokens/s)\")\n",
|
|
"ax.set_title(\"Token Dropping Benefit\\nfor Decode Throughput\")\n",
|
|
"ax.margins(y=0.15)\n",
|
|
"plt.tight_layout()\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "f83db925",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Only close the context when done.\n",
|
|
"lmc_sdk.close(ctx)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "LMCache (3.12.3)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.12.3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|