wehub-resource-sync
1167 lines
40 KiB
Plaintext
1167 lines
40 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c146ddf0",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001753,
|
|
"end_time": "2026-05-03T02:00:41.332681+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:41.330928+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"# FRED Macro Indicators Dataset\n",
|
|
"\n",
|
|
"Treasury yields and economic indicators for regime filtering.\n",
|
|
"\n",
|
|
"| Property | Value |\n",
|
|
"|----------|-------|\n",
|
|
"| **Provider** | FRED (Federal Reserve) |\n",
|
|
"| **Asset Class** | Macro/Economic |\n",
|
|
"| **Frequency** | Daily (treasury), Monthly (economic) |\n",
|
|
"| **Series** | 17+ indicators |\n",
|
|
"| **Coverage** | 2000-2025 |\n",
|
|
"| **Size** | ~5 MB |\n",
|
|
"| **API Key** | `FRED_API_KEY` (free) |\n",
|
|
"| **Loader** | `load_macro()` |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "3495765b",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-05-03T02:00:41.335687Z",
|
|
"iopub.status.busy": "2026-05-03T02:00:41.335542Z",
|
|
"iopub.status.idle": "2026-05-03T02:00:41.385967Z",
|
|
"shell.execute_reply": "2026-05-03T02:00:41.385566Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.052462,
|
|
"end_time": "2026-05-03T02:00:41.386266+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:41.333804+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"True"
|
|
]
|
|
},
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"\"\"\"FRED Macro Indicators - download, explore, and update workflow.\"\"\"\n",
|
|
"\n",
|
|
"import json\n",
|
|
"import os\n",
|
|
"from pathlib import Path\n",
|
|
"\n",
|
|
"import polars as pl\n",
|
|
"import yaml\n",
|
|
"from dotenv import load_dotenv\n",
|
|
"\n",
|
|
"# Load environment variables\n",
|
|
"load_dotenv()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a8ccb2da",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001036,
|
|
"end_time": "2026-05-03T02:00:41.388583+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:41.387547+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"## 1. Configuration\n",
|
|
"\n",
|
|
"The macro series are defined in `config.yaml`. Primary use: Treasury yields\n",
|
|
"for regime filtering (risk-on/risk-off based on yield curve slope)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "2eb98059",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-05-03T02:00:41.391141Z",
|
|
"iopub.status.busy": "2026-05-03T02:00:41.391062Z",
|
|
"iopub.status.idle": "2026-05-03T02:00:41.395527Z",
|
|
"shell.execute_reply": "2026-05-03T02:00:41.395247Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.006307,
|
|
"end_time": "2026-05-03T02:00:41.395913+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:41.389606+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=== Macro Configuration ===\n",
|
|
"Provider: fred\n",
|
|
"Date range: 2000-01-01 to 2025-12-31\n",
|
|
"\n",
|
|
"Series groups:\n",
|
|
" daily: Daily rates and volatility indicators\n",
|
|
" - DFF\n",
|
|
" - DGS2\n",
|
|
" - DGS5\n",
|
|
" - DGS10\n",
|
|
" - DGS30\n",
|
|
" - T10Y2Y\n",
|
|
" - VIXCLS\n",
|
|
" weekly: Weekly labor and Federal Reserve balance sheet indicators\n",
|
|
" - ICSA\n",
|
|
" - WALCL\n",
|
|
" monthly: Monthly inflation, labor, production, and money supply indicators\n",
|
|
" - CPIAUCSL\n",
|
|
" - CPILFESL\n",
|
|
" - PCEPI\n",
|
|
" - UNRATE\n",
|
|
" - PAYEMS\n",
|
|
" - CIVPART\n",
|
|
" - INDPRO\n",
|
|
" - M2SL\n",
|
|
" quarterly: Quarterly growth indicators\n",
|
|
" - GDP\n",
|
|
" - GDPC1\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Load and display configuration\n",
|
|
"config_path = Path(\"config.yaml\")\n",
|
|
"config = yaml.safe_load(config_path.read_text())\n",
|
|
"macro_config = config[\"macro\"]\n",
|
|
"\n",
|
|
"print(\"=== Macro Configuration ===\")\n",
|
|
"print(f\"Provider: {macro_config['provider']}\")\n",
|
|
"print(f\"Date range: {macro_config['start']} to {macro_config['end']}\")\n",
|
|
"print(\"\\nSeries groups:\")\n",
|
|
"for group_name, info in macro_config[\"series\"].items():\n",
|
|
" if isinstance(info, dict) and \"symbols\" in info:\n",
|
|
" symbols = info[\"symbols\"]\n",
|
|
" print(f\" {group_name}: {info.get('description', '')}\")\n",
|
|
" for s in symbols:\n",
|
|
" print(f\" - {s}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7612d806",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.000982,
|
|
"end_time": "2026-05-03T02:00:41.398049+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:41.397067+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"## 2. API Key Setup\n",
|
|
"\n",
|
|
"FRED requires a free API key.\n",
|
|
"\n",
|
|
"### Getting a FRED API Key\n",
|
|
"\n",
|
|
"1. Go to [FRED API Key Signup](https://fredaccount.stlouisfed.org/login/secure/)\n",
|
|
"2. Create a free account or sign in\n",
|
|
"3. Navigate to **API Keys** and create a new key\n",
|
|
"4. Add to your `.env` file in the repository root:\n",
|
|
"\n",
|
|
"```bash\n",
|
|
"FRED_API_KEY=your-32-character-api-key\n",
|
|
"```\n",
|
|
"\n",
|
|
"FRED is free with generous rate limits (120 requests/minute)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "32321b74",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-05-03T02:00:41.400687Z",
|
|
"iopub.status.busy": "2026-05-03T02:00:41.400623Z",
|
|
"iopub.status.idle": "2026-05-03T02:00:41.402371Z",
|
|
"shell.execute_reply": "2026-05-03T02:00:41.402071Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.003534,
|
|
"end_time": "2026-05-03T02:00:41.402647+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:41.399113+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"FRED_API_KEY: 9d734c55... (configured)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Verify API key is configured\n",
|
|
"api_key = os.getenv(\"FRED_API_KEY\")\n",
|
|
"if api_key:\n",
|
|
" print(f\"FRED_API_KEY: {api_key[:8]}... (configured)\")\n",
|
|
"else:\n",
|
|
" print(\"WARNING: FRED_API_KEY not set in environment\")\n",
|
|
" print(\"Get free key at: https://fredaccount.stlouisfed.org/login/secure/\")\n",
|
|
" print(\"Add to .env file: FRED_API_KEY=your-key-here\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2eb8e9cf",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.000992,
|
|
"end_time": "2026-05-03T02:00:41.404754+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:41.403762+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"## 3. Download Data\n",
|
|
"\n",
|
|
"The download fetches multiple economic series and aligns them to a daily calendar.\n",
|
|
"Different series have different native frequencies (daily, weekly, monthly, quarterly)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "29940c17",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-05-03T02:00:41.407400Z",
|
|
"iopub.status.busy": "2026-05-03T02:00:41.407333Z",
|
|
"iopub.status.idle": "2026-05-03T02:00:41.412335Z",
|
|
"shell.execute_reply": "2026-05-03T02:00:41.412009Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.006746,
|
|
"end_time": "2026-05-03T02:00:41.412586+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:41.405840+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Key macro indicators with native frequency\n",
|
|
"FRED_SERIES = {\n",
|
|
" # Daily series\n",
|
|
" \"DFF\": (\"Fed Funds Rate\", \"daily\"),\n",
|
|
" \"DGS10\": (\"10-Year Treasury\", \"daily\"),\n",
|
|
" \"DGS2\": (\"2-Year Treasury\", \"daily\"),\n",
|
|
" \"DGS5\": (\"5-Year Treasury\", \"daily\"),\n",
|
|
" \"DGS30\": (\"30-Year Treasury\", \"daily\"),\n",
|
|
" \"T10Y2Y\": (\"10Y-2Y Spread\", \"daily\"),\n",
|
|
" \"VIXCLS\": (\"VIX Volatility Index\", \"daily\"),\n",
|
|
" # Weekly series\n",
|
|
" \"ICSA\": (\"Initial Jobless Claims\", \"weekly\"),\n",
|
|
" # Monthly series\n",
|
|
" \"CPIAUCSL\": (\"CPI All Urban Consumers\", \"monthly\"),\n",
|
|
" \"UNRATE\": (\"Unemployment Rate\", \"monthly\"),\n",
|
|
" \"PAYEMS\": (\"Non-Farm Payrolls\", \"monthly\"),\n",
|
|
" \"INDPRO\": (\"Industrial Production\", \"monthly\"),\n",
|
|
" # Quarterly series\n",
|
|
" \"GDP\": (\"Gross Domestic Product\", \"quarterly\"),\n",
|
|
"}\n",
|
|
"\n",
|
|
"\n",
|
|
"def download_macro_data(\n",
|
|
" dry_run: bool = False, force: bool = False, series: list[str] | None = None\n",
|
|
"):\n",
|
|
" \"\"\"Download macro data from FRED.\n",
|
|
"\n",
|
|
" Args:\n",
|
|
" dry_run: If True, show what would be downloaded without doing it\n",
|
|
" force: If True, re-download even if data exists\n",
|
|
" series: Specific series to download (default: all from FRED_SERIES)\n",
|
|
" \"\"\"\n",
|
|
" from ml4t.data.providers import FREDProvider\n",
|
|
"\n",
|
|
" from utils import ML4T_DATA_PATH\n",
|
|
"\n",
|
|
" api_key = os.getenv(\"FRED_API_KEY\")\n",
|
|
" if not api_key and not dry_run:\n",
|
|
" raise ValueError(\"FRED_API_KEY not set. See API Key Setup section.\")\n",
|
|
"\n",
|
|
" # Load config for date range (resolved relative to this script for cwd-independence;\n",
|
|
" # __file__ is undefined in papermill/notebook execution, so fall back to cwd).\n",
|
|
" try:\n",
|
|
" here = Path(__file__).parent\n",
|
|
" except NameError:\n",
|
|
" here = Path.cwd()\n",
|
|
" config = yaml.safe_load((here / \"config.yaml\").read_text())\n",
|
|
" macro_config = config[\"macro\"]\n",
|
|
"\n",
|
|
" if series is None:\n",
|
|
" series_to_download = FRED_SERIES\n",
|
|
" else:\n",
|
|
" series_to_download = {s: FRED_SERIES[s] for s in series if s in FRED_SERIES}\n",
|
|
"\n",
|
|
" output_dir = ML4T_DATA_PATH / \"macro\"\n",
|
|
" output_path = output_dir / \"fred_macro.parquet\"\n",
|
|
"\n",
|
|
" print(\"=== Macro Download ===\")\n",
|
|
" print(f\"Series: {len(series_to_download)}\")\n",
|
|
" print(f\"Date range: {macro_config['start']} to {macro_config['end']}\")\n",
|
|
" print(f\"Output: {output_path}\")\n",
|
|
"\n",
|
|
" if dry_run:\n",
|
|
" print(\"\\n[DRY RUN] Would download:\")\n",
|
|
" for series_id, (name, freq) in series_to_download.items():\n",
|
|
" print(f\" {series_id:12s} ({freq:9s}) {name}\")\n",
|
|
" return\n",
|
|
"\n",
|
|
" # Check existing\n",
|
|
" if output_path.exists() and not force:\n",
|
|
" existing = pl.read_parquet(output_path)\n",
|
|
" print(f\"\\nData already exists ({len(existing):,} rows).\")\n",
|
|
" print(\"Use force=True to re-download.\")\n",
|
|
" return existing\n",
|
|
"\n",
|
|
" # Initialize provider\n",
|
|
" provider = FREDProvider(api_key=api_key)\n",
|
|
"\n",
|
|
" # Download each series\n",
|
|
" all_series = []\n",
|
|
" print(f\"\\nDownloading {len(series_to_download)} series...\")\n",
|
|
" for series_id, (name, frequency) in series_to_download.items():\n",
|
|
" print(f\" {series_id}...\", end=\" \", flush=True)\n",
|
|
" try:\n",
|
|
" df = provider.fetch_ohlcv(\n",
|
|
" series_id,\n",
|
|
" start=macro_config[\"start\"],\n",
|
|
" end=macro_config[\"end\"],\n",
|
|
" frequency=frequency,\n",
|
|
" )\n",
|
|
" # Rename close to series_id\n",
|
|
" series_df = df.select(\n",
|
|
" [\n",
|
|
" pl.col(\"timestamp\").cast(pl.Date).alias(\"date\"),\n",
|
|
" pl.col(\"close\").alias(series_id.lower()),\n",
|
|
" ]\n",
|
|
" )\n",
|
|
" all_series.append(series_df)\n",
|
|
" print(f\"OK ({len(df):,} obs)\")\n",
|
|
" except Exception as e:\n",
|
|
" print(f\"ERROR: {e}\")\n",
|
|
"\n",
|
|
" provider.close()\n",
|
|
"\n",
|
|
" if not all_series:\n",
|
|
" raise RuntimeError(\"No series downloaded!\")\n",
|
|
"\n",
|
|
" # Create daily date range for alignment\n",
|
|
" from datetime import datetime\n",
|
|
"\n",
|
|
" dates = pl.date_range(\n",
|
|
" datetime.strptime(macro_config[\"start\"], \"%Y-%m-%d\"),\n",
|
|
" datetime.strptime(macro_config[\"end\"], \"%Y-%m-%d\"),\n",
|
|
" eager=True,\n",
|
|
" )\n",
|
|
" result = pl.DataFrame({\"date\": dates})\n",
|
|
"\n",
|
|
" # Join all series and forward-fill\n",
|
|
" for series_df in all_series:\n",
|
|
" series_col = [c for c in series_df.columns if c != \"date\"][0]\n",
|
|
" result = result.join(series_df, on=\"date\", how=\"left\")\n",
|
|
" result = result.with_columns(pl.col(series_col).forward_fill())\n",
|
|
"\n",
|
|
" # Save\n",
|
|
" output_dir.mkdir(parents=True, exist_ok=True)\n",
|
|
" result.write_parquet(output_path)\n",
|
|
"\n",
|
|
" print(\"\\n=== Complete ===\")\n",
|
|
" print(f\"Total rows: {len(result):,}\")\n",
|
|
" print(f\"Columns: {len(result.columns)}\")\n",
|
|
" print(f\"Saved to: {output_path}\")\n",
|
|
"\n",
|
|
" return result"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "eb1b1b8a",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001023,
|
|
"end_time": "2026-05-03T02:00:41.414739+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:41.413716+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"### Download All Series"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "2e614d33",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-05-03T02:00:41.417445Z",
|
|
"iopub.status.busy": "2026-05-03T02:00:41.417380Z",
|
|
"iopub.status.idle": "2026-05-03T02:00:41.418743Z",
|
|
"shell.execute_reply": "2026-05-03T02:00:41.418458Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.00311,
|
|
"end_time": "2026-05-03T02:00:41.418948+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:41.415838+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Uncomment to download all macro data\n",
|
|
"# download_macro_data()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c996a633",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001033,
|
|
"end_time": "2026-05-03T02:00:41.421098+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:41.420065+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"### Dry Run (Preview)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "2509b870",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-05-03T02:00:41.423687Z",
|
|
"iopub.status.busy": "2026-05-03T02:00:41.423621Z",
|
|
"iopub.status.idle": "2026-05-03T02:00:42.533014Z",
|
|
"shell.execute_reply": "2026-05-03T02:00:42.532665Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 1.111362,
|
|
"end_time": "2026-05-03T02:00:42.533556+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:41.422194+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=== Macro Download ===\n",
|
|
"Series: 13\n",
|
|
"Date range: 2000-01-01 to 2025-12-31\n",
|
|
"Output: data/macro/fred_macro.parquet\n",
|
|
"\n",
|
|
"[DRY RUN] Would download:\n",
|
|
" DFF (daily ) Fed Funds Rate\n",
|
|
" DGS10 (daily ) 10-Year Treasury\n",
|
|
" DGS2 (daily ) 2-Year Treasury\n",
|
|
" DGS5 (daily ) 5-Year Treasury\n",
|
|
" DGS30 (daily ) 30-Year Treasury\n",
|
|
" T10Y2Y (daily ) 10Y-2Y Spread\n",
|
|
" VIXCLS (daily ) VIX Volatility Index\n",
|
|
" ICSA (weekly ) Initial Jobless Claims\n",
|
|
" CPIAUCSL (monthly ) CPI All Urban Consumers\n",
|
|
" UNRATE (monthly ) Unemployment Rate\n",
|
|
" PAYEMS (monthly ) Non-Farm Payrolls\n",
|
|
" INDPRO (monthly ) Industrial Production\n",
|
|
" GDP (quarterly) Gross Domestic Product\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"download_macro_data(dry_run=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e7928409",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001901,
|
|
"end_time": "2026-05-03T02:00:42.537917+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.536016+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"## 4. Load and Explore\n",
|
|
"\n",
|
|
"Once downloaded, use the loader throughout the book:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "8e388413",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-05-03T02:00:42.540784Z",
|
|
"iopub.status.busy": "2026-05-03T02:00:42.540593Z",
|
|
"iopub.status.idle": "2026-05-03T02:00:42.552530Z",
|
|
"shell.execute_reply": "2026-05-03T02:00:42.552228Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.013754,
|
|
"end_time": "2026-05-03T02:00:42.552815+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.539061+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Shape: (9497, 22)\n",
|
|
"Columns: ['timestamp', 'dff', 'dgs2', 'dgs5', 'dgs10', 'dgs30', 't10y2y', 'vixcls', 'icsa', 'walcl', 'cpiaucsl', 'cpilfesl', 'pcepi', 'unrate', 'payems', 'civpart', 'indpro', 'm2sl', 'gdp', 'gdpc1', 'YIELD_CURVE_SLOPE', 'YIELD_CURVE_5_10']\n",
|
|
"Date range: 2000-01-01 to 2025-12-31\n",
|
|
"Memory: 1.6 MB\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from data import load_macro\n",
|
|
"\n",
|
|
"# Load all macro data\n",
|
|
"df = load_macro()\n",
|
|
"\n",
|
|
"print(f\"Shape: {df.shape}\")\n",
|
|
"print(f\"Columns: {df.columns}\")\n",
|
|
"print(f\"Date range: {df['timestamp'].min()} to {df['timestamp'].max()}\")\n",
|
|
"print(f\"Memory: {df.estimated_size('mb'):.1f} MB\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "7ee95b3e",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-05-03T02:00:42.555733Z",
|
|
"iopub.status.busy": "2026-05-03T02:00:42.555659Z",
|
|
"iopub.status.idle": "2026-05-03T02:00:42.557869Z",
|
|
"shell.execute_reply": "2026-05-03T02:00:42.557650Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.004145,
|
|
"end_time": "2026-05-03T02:00:42.558220+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.554075+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Schema([('timestamp', Date),\n",
|
|
" ('dff', Float64),\n",
|
|
" ('dgs2', Float64),\n",
|
|
" ('dgs5', Float64),\n",
|
|
" ('dgs10', Float64),\n",
|
|
" ('dgs30', Float64),\n",
|
|
" ('t10y2y', Float64),\n",
|
|
" ('vixcls', Float64),\n",
|
|
" ('icsa', Float64),\n",
|
|
" ('walcl', Float64),\n",
|
|
" ('cpiaucsl', Float64),\n",
|
|
" ('cpilfesl', Float64),\n",
|
|
" ('pcepi', Float64),\n",
|
|
" ('unrate', Float64),\n",
|
|
" ('payems', Float64),\n",
|
|
" ('civpart', Float64),\n",
|
|
" ('indpro', Float64),\n",
|
|
" ('m2sl', Float64),\n",
|
|
" ('gdp', Float64),\n",
|
|
" ('gdpc1', Float64),\n",
|
|
" ('YIELD_CURVE_SLOPE', Float64),\n",
|
|
" ('YIELD_CURVE_5_10', Float64)])"
|
|
]
|
|
},
|
|
"execution_count": 8,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"# Schema\n",
|
|
"df.schema"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "365f2ebc",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-05-03T02:00:42.560996Z",
|
|
"iopub.status.busy": "2026-05-03T02:00:42.560928Z",
|
|
"iopub.status.idle": "2026-05-03T02:00:42.563512Z",
|
|
"shell.execute_reply": "2026-05-03T02:00:42.563276Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.004334,
|
|
"end_time": "2026-05-03T02:00:42.563781+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.559447+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<div><style>\n",
|
|
".dataframe > thead > tr,\n",
|
|
".dataframe > tbody > tr {\n",
|
|
" text-align: right;\n",
|
|
" white-space: pre-wrap;\n",
|
|
"}\n",
|
|
"</style>\n",
|
|
"<small>shape: (10, 22)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>timestamp</th><th>dff</th><th>dgs2</th><th>dgs5</th><th>dgs10</th><th>dgs30</th><th>t10y2y</th><th>vixcls</th><th>icsa</th><th>walcl</th><th>cpiaucsl</th><th>cpilfesl</th><th>pcepi</th><th>unrate</th><th>payems</th><th>civpart</th><th>indpro</th><th>m2sl</th><th>gdp</th><th>gdpc1</th><th>YIELD_CURVE_SLOPE</th><th>YIELD_CURVE_5_10</th></tr><tr><td>date</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>2000-01-01</td><td>3.99</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>286000.0</td><td>null</td><td>169.3</td><td>179.3</td><td>72.961</td><td>4.0</td><td>131011.0</td><td>67.3</td><td>91.538</td><td>4667.6</td><td>10002.179</td><td>13878.147</td><td>null</td><td>null</td></tr><tr><td>2000-01-02</td><td>3.99</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>286000.0</td><td>null</td><td>169.3</td><td>179.3</td><td>72.961</td><td>4.0</td><td>131011.0</td><td>67.3</td><td>91.538</td><td>4667.6</td><td>10002.179</td><td>13878.147</td><td>null</td><td>null</td></tr><tr><td>2000-01-03</td><td>5.43</td><td>6.38</td><td>6.5</td><td>6.58</td><td>6.61</td><td>0.2</td><td>24.21</td><td>286000.0</td><td>null</td><td>169.3</td><td>179.3</td><td>72.961</td><td>4.0</td><td>131011.0</td><td>67.3</td><td>91.538</td><td>4667.6</td><td>10002.179</td><td>13878.147</td><td>0.2</td><td>0.08</td></tr><tr><td>2000-01-04</td><td>5.38</td><td>6.3</td><td>6.4</td><td>6.49</td><td>6.53</td><td>0.19</td><td>27.01</td><td>286000.0</td><td>null</td><td>169.3</td><td>179.3</td><td>72.961</td><td>4.0</td><td>131011.0</td><td>67.3</td><td>91.538</td><td>4667.6</td><td>10002.179</td><td>13878.147</td><td>0.19</td><td>0.09</td></tr><tr><td>2000-01-05</td><td>5.41</td><td>6.38</td><td>6.51</td><td>6.62</td><td>6.64</td><td>0.24</td><td>26.41</td><td>286000.0</td><td>null</td><td>169.3</td><td>179.3</td><td>72.961</td><td>4.0</td><td>131011.0</td><td>67.3</td><td>91.538</td><td>4667.6</td><td>10002.179</td><td>13878.147</td><td>0.24</td><td>0.11</td></tr><tr><td>2000-01-06</td><td>5.54</td><td>6.35</td><td>6.46</td><td>6.57</td><td>6.58</td><td>0.22</td><td>25.73</td><td>286000.0</td><td>null</td><td>169.3</td><td>179.3</td><td>72.961</td><td>4.0</td><td>131011.0</td><td>67.3</td><td>91.538</td><td>4667.6</td><td>10002.179</td><td>13878.147</td><td>0.22</td><td>0.11</td></tr><tr><td>2000-01-07</td><td>5.61</td><td>6.31</td><td>6.42</td><td>6.52</td><td>6.55</td><td>0.21</td><td>21.72</td><td>286000.0</td><td>null</td><td>169.3</td><td>179.3</td><td>72.961</td><td>4.0</td><td>131011.0</td><td>67.3</td><td>91.538</td><td>4667.6</td><td>10002.179</td><td>13878.147</td><td>0.21</td><td>0.1</td></tr><tr><td>2000-01-08</td><td>5.61</td><td>6.31</td><td>6.42</td><td>6.52</td><td>6.55</td><td>0.21</td><td>21.72</td><td>298000.0</td><td>null</td><td>169.3</td><td>179.3</td><td>72.961</td><td>4.0</td><td>131011.0</td><td>67.3</td><td>91.538</td><td>4667.6</td><td>10002.179</td><td>13878.147</td><td>0.21</td><td>0.1</td></tr><tr><td>2000-01-09</td><td>5.61</td><td>6.31</td><td>6.42</td><td>6.52</td><td>6.55</td><td>0.21</td><td>21.72</td><td>298000.0</td><td>null</td><td>169.3</td><td>179.3</td><td>72.961</td><td>4.0</td><td>131011.0</td><td>67.3</td><td>91.538</td><td>4667.6</td><td>10002.179</td><td>13878.147</td><td>0.21</td><td>0.1</td></tr><tr><td>2000-01-10</td><td>5.74</td><td>6.38</td><td>6.49</td><td>6.57</td><td>6.59</td><td>0.19</td><td>21.71</td><td>298000.0</td><td>null</td><td>169.3</td><td>179.3</td><td>72.961</td><td>4.0</td><td>131011.0</td><td>67.3</td><td>91.538</td><td>4667.6</td><td>10002.179</td><td>13878.147</td><td>0.19</td><td>0.08</td></tr></tbody></table></div>"
|
|
],
|
|
"text/plain": [
|
|
"shape: (10, 22)\n",
|
|
"┌────────────┬──────┬──────┬──────┬───┬───────────┬───────────┬──────────────────┬─────────────────┐\n",
|
|
"│ timestamp ┆ dff ┆ dgs2 ┆ dgs5 ┆ … ┆ gdp ┆ gdpc1 ┆ YIELD_CURVE_SLOP ┆ YIELD_CURVE_5_1 │\n",
|
|
"│ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ E ┆ 0 │\n",
|
|
"│ date ┆ f64 ┆ f64 ┆ f64 ┆ ┆ f64 ┆ f64 ┆ --- ┆ --- │\n",
|
|
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ f64 ┆ f64 │\n",
|
|
"╞════════════╪══════╪══════╪══════╪═══╪═══════════╪═══════════╪══════════════════╪═════════════════╡\n",
|
|
"│ 2000-01-01 ┆ 3.99 ┆ null ┆ null ┆ … ┆ 10002.179 ┆ 13878.147 ┆ null ┆ null │\n",
|
|
"│ 2000-01-02 ┆ 3.99 ┆ null ┆ null ┆ … ┆ 10002.179 ┆ 13878.147 ┆ null ┆ null │\n",
|
|
"│ 2000-01-03 ┆ 5.43 ┆ 6.38 ┆ 6.5 ┆ … ┆ 10002.179 ┆ 13878.147 ┆ 0.2 ┆ 0.08 │\n",
|
|
"│ 2000-01-04 ┆ 5.38 ┆ 6.3 ┆ 6.4 ┆ … ┆ 10002.179 ┆ 13878.147 ┆ 0.19 ┆ 0.09 │\n",
|
|
"│ 2000-01-05 ┆ 5.41 ┆ 6.38 ┆ 6.51 ┆ … ┆ 10002.179 ┆ 13878.147 ┆ 0.24 ┆ 0.11 │\n",
|
|
"│ 2000-01-06 ┆ 5.54 ┆ 6.35 ┆ 6.46 ┆ … ┆ 10002.179 ┆ 13878.147 ┆ 0.22 ┆ 0.11 │\n",
|
|
"│ 2000-01-07 ┆ 5.61 ┆ 6.31 ┆ 6.42 ┆ … ┆ 10002.179 ┆ 13878.147 ┆ 0.21 ┆ 0.1 │\n",
|
|
"│ 2000-01-08 ┆ 5.61 ┆ 6.31 ┆ 6.42 ┆ … ┆ 10002.179 ┆ 13878.147 ┆ 0.21 ┆ 0.1 │\n",
|
|
"│ 2000-01-09 ┆ 5.61 ┆ 6.31 ┆ 6.42 ┆ … ┆ 10002.179 ┆ 13878.147 ┆ 0.21 ┆ 0.1 │\n",
|
|
"│ 2000-01-10 ┆ 5.74 ┆ 6.38 ┆ 6.49 ┆ … ┆ 10002.179 ┆ 13878.147 ┆ 0.19 ┆ 0.08 │\n",
|
|
"└────────────┴──────┴──────┴──────┴───┴───────────┴───────────┴──────────────────┴─────────────────┘"
|
|
]
|
|
},
|
|
"execution_count": 9,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"# Preview\n",
|
|
"df.head(10)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3e671a01",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001335,
|
|
"end_time": "2026-05-03T02:00:42.566389+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.565054+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"### Treasury Yield Statistics"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "6166ff69",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-05-03T02:00:42.569257Z",
|
|
"iopub.status.busy": "2026-05-03T02:00:42.569197Z",
|
|
"iopub.status.idle": "2026-05-03T02:00:42.572177Z",
|
|
"shell.execute_reply": "2026-05-03T02:00:42.571951Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.004842,
|
|
"end_time": "2026-05-03T02:00:42.572500+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.567658+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Treasury Yield Summary:\n",
|
|
" DGS2: mean=2.26%, min=0.09%, max=6.93%\n",
|
|
" DGS5: mean=2.77%, min=0.19%, max=6.83%\n",
|
|
" DGS10: mean=3.32%, min=0.52%, max=6.79%\n",
|
|
" DGS30: mean=3.91%, min=0.99%, max=6.75%\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Treasury yield summary\n",
|
|
"yield_cols = [c for c in df.columns if c.startswith(\"dgs\")]\n",
|
|
"if yield_cols:\n",
|
|
" print(\"Treasury Yield Summary:\")\n",
|
|
" for col in yield_cols:\n",
|
|
" series = df[col].drop_nulls()\n",
|
|
" print(\n",
|
|
" f\" {col.upper()}: mean={series.mean():.2f}%, min={series.min():.2f}%, max={series.max():.2f}%\"\n",
|
|
" )"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b519fa87",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001197,
|
|
"end_time": "2026-05-03T02:00:42.574983+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.573786+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"### Yield Curve Slope"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "60094da0",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-05-03T02:00:42.577985Z",
|
|
"iopub.status.busy": "2026-05-03T02:00:42.577920Z",
|
|
"iopub.status.idle": "2026-05-03T02:00:42.580365Z",
|
|
"shell.execute_reply": "2026-05-03T02:00:42.580095Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.004388,
|
|
"end_time": "2026-05-03T02:00:42.580639+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.576251+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"Yield Curve Slope (10Y - 2Y):\n",
|
|
" Mean: 1.06%\n",
|
|
" Current: 0.71%\n",
|
|
" % Inverted (< 0): 15.5%\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Yield curve slope (10Y - 2Y)\n",
|
|
"if all(c in df.columns for c in [\"dgs10\", \"dgs2\"]):\n",
|
|
" df_with_slope = df.with_columns((pl.col(\"dgs10\") - pl.col(\"dgs2\")).alias(\"yield_curve_slope\"))\n",
|
|
"\n",
|
|
" slope = df_with_slope[\"yield_curve_slope\"].drop_nulls()\n",
|
|
" print(\"\\nYield Curve Slope (10Y - 2Y):\")\n",
|
|
" print(f\" Mean: {slope.mean():.2f}%\")\n",
|
|
" print(f\" Current: {slope[-1]:.2f}%\")\n",
|
|
" print(f\" % Inverted (< 0): {(slope < 0).sum() / len(slope) * 100:.1f}%\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5e39f442",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001235,
|
|
"end_time": "2026-05-03T02:00:42.583187+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.581952+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"## 5. Data Profile"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"id": "ff3104a9",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-05-03T02:00:42.586167Z",
|
|
"iopub.status.busy": "2026-05-03T02:00:42.586097Z",
|
|
"iopub.status.idle": "2026-05-03T02:00:42.588181Z",
|
|
"shell.execute_reply": "2026-05-03T02:00:42.587902Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.003953,
|
|
"end_time": "2026-05-03T02:00:42.588418+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.584465+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Profile not found at data/macro/profile.json\n",
|
|
"Generate with: python generate_profiles.py --dataset macro\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from utils import ML4T_DATA_PATH\n",
|
|
"\n",
|
|
"# Check for existing profile\n",
|
|
"profile_path = ML4T_DATA_PATH / \"macro\" / \"profile.json\"\n",
|
|
"\n",
|
|
"if profile_path.exists():\n",
|
|
" profile = json.loads(profile_path.read_text())\n",
|
|
" print(\"=== Macro Profile ===\")\n",
|
|
" print(f\"Rows: {profile['rows']:,}\")\n",
|
|
" print(f\"Columns: {profile['columns']}\")\n",
|
|
" print(f\"Memory: {profile['memory_mb']:.1f} MB\")\n",
|
|
"else:\n",
|
|
" print(f\"Profile not found at {profile_path}\")\n",
|
|
" print(\"Generate with: python generate_profiles.py --dataset macro\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "502f5a73",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001238,
|
|
"end_time": "2026-05-03T02:00:42.591027+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.589789+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"## 6. Loader Options\n",
|
|
"\n",
|
|
"The loader supports filtering by series and date range:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"id": "f11b34ce",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-05-03T02:00:42.593989Z",
|
|
"iopub.status.busy": "2026-05-03T02:00:42.593922Z",
|
|
"iopub.status.idle": "2026-05-03T02:00:42.598218Z",
|
|
"shell.execute_reply": "2026-05-03T02:00:42.597863Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.006074,
|
|
"end_time": "2026-05-03T02:00:42.598419+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.592345+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Treasury yields only: (9497, 1)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Specific series\n",
|
|
"yields_only = load_macro(series=[\"DGS2\", \"DGS10\", \"DGS30\"])\n",
|
|
"print(f\"Treasury yields only: {yields_only.shape}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"id": "86b97ba4",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-05-03T02:00:42.601516Z",
|
|
"iopub.status.busy": "2026-05-03T02:00:42.601449Z",
|
|
"iopub.status.idle": "2026-05-03T02:00:42.605197Z",
|
|
"shell.execute_reply": "2026-05-03T02:00:42.604941Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.005694,
|
|
"end_time": "2026-05-03T02:00:42.605491+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.599797+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"2020 onwards: (2192, 22)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Date range\n",
|
|
"recent = load_macro(start_date=\"2020-01-01\")\n",
|
|
"print(f\"2020 onwards: {recent.shape}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"id": "c6edbb72",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-05-03T02:00:42.608648Z",
|
|
"iopub.status.busy": "2026-05-03T02:00:42.608576Z",
|
|
"iopub.status.idle": "2026-05-03T02:00:42.612752Z",
|
|
"shell.execute_reply": "2026-05-03T02:00:42.612531Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.006215,
|
|
"end_time": "2026-05-03T02:00:42.613099+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.606884+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Yields + VIX, 2020-2023: (1461, 1)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Combined filters\n",
|
|
"filtered = load_macro(\n",
|
|
" series=[\"DGS10\", \"DGS2\", \"VIXCLS\"], start_date=\"2020-01-01\", end_date=\"2023-12-31\"\n",
|
|
")\n",
|
|
"print(f\"Yields + VIX, 2020-2023: {filtered.shape}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "aba04cd2",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001314,
|
|
"end_time": "2026-05-03T02:00:42.615834+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.614520+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"## 7. Documentation\n",
|
|
"\n",
|
|
"### FRED API\n",
|
|
"- [FRED API Documentation](https://fred.stlouisfed.org/docs/api/)\n",
|
|
"- [API Key Request](https://fredaccount.stlouisfed.org/login/secure/)\n",
|
|
"- Rate limit: 120 requests/minute (generous)\n",
|
|
"\n",
|
|
"### Regime Filtering\n",
|
|
"\n",
|
|
"The yield curve slope is commonly used for regime detection:\n",
|
|
"\n",
|
|
"| Slope Range | Regime | Interpretation |\n",
|
|
"|-------------|--------|----------------|\n",
|
|
"| > 0.5% | Risk-on | Normal economic expansion |\n",
|
|
"| 0% to 0.5% | Caution | Late cycle |\n",
|
|
"| < 0% | Risk-off | Inverted curve, recession signal |\n",
|
|
"\n",
|
|
"Chapter 6 strategies use this for conditional signal weighting.\n",
|
|
"\n",
|
|
"### Data Quality Notes\n",
|
|
"- Treasury yields are daily (excluding weekends/holidays)\n",
|
|
"- Economic series are forward-filled to daily alignment\n",
|
|
"- VIX is close price (not intraday high)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "edbbd0fe",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001308,
|
|
"end_time": "2026-05-03T02:00:42.618509+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.617201+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"## 8. Updating Data\n",
|
|
"\n",
|
|
"To update with the latest data:\n",
|
|
"\n",
|
|
"```python\n",
|
|
"# Update all series\n",
|
|
"download_macro_data()\n",
|
|
"\n",
|
|
"# Force full re-download\n",
|
|
"download_macro_data(force=True)\n",
|
|
"```\n",
|
|
"\n",
|
|
"**Tip**: Update the `end` date in `config.yaml` before re-downloading."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d519c064",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001386,
|
|
"end_time": "2026-05-03T02:00:42.621287+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-05-03T02:00:42.619901+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"## Summary\n",
|
|
"\n",
|
|
"| Item | Value |\n",
|
|
"|------|-------|\n",
|
|
"| Series | 13+ (treasury yields, economic indicators) |\n",
|
|
"| Frequency | Daily (aligned from native frequencies) |\n",
|
|
"| Coverage | 2000-2025 |\n",
|
|
"| Provider | FRED (free API key) |\n",
|
|
"| Config | `config.yaml` |\n",
|
|
"| Loader | `load_macro(series, start_date, end_date)` |\n",
|
|
"\n",
|
|
"**Primary use**: Yield curve slope for regime filtering in strategy signals."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"jupytext": {
|
|
"cell_metadata_filter": "-all"
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.14.3"
|
|
},
|
|
"papermill": {
|
|
"default_parameters": {},
|
|
"duration": 2.743025,
|
|
"end_time": "2026-05-03T02:00:43.437146+00:00",
|
|
"environment_variables": {},
|
|
"exception": null,
|
|
"input_path": "data/macro/dataset_card.ipynb",
|
|
"output_path": "data/macro/dataset_card.ipynb",
|
|
"parameters": {},
|
|
"start_time": "2026-05-03T02:00:40.694121+00:00",
|
|
"version": "2.7.0"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|