chore: import upstream snapshot with attribution

2026-07-13 12:39:48 +08:00
commit 9b395f5cc3
2400 changed files with 376116 additions and 0 deletions
@@ -0,0 +1,178 @@
+# Using OpenCLI with Android Chrome
+
+OpenCLI can control Chrome on a connected Android device via **ADB port forwarding** and **CDPBridge** — no extra tools or custom builds required. The same adapters that run on desktop Chrome work identically on Android, reusing whatever cookies are already in the mobile browser.
+
+---
+
+## How It Works
+
+Android Chrome supports [remote debugging via CDP](https://developer.chrome.com/docs/devtools/remote-debugging/). The device exposes a local Unix socket that ADB can forward to a TCP port on your machine. OpenCLI's `CDPBridge` then connects to that port exactly as it would to any other CDP endpoint.
+
+```
+OpenCLI (CDPBridge)
+    │  WebSocket (CDP)
+    ▼
+localhost:9222                ← ADB forward
+    │  adb forward
+    ▼
+Android device
+  chrome_devtools_remote      ← Chrome's Unix debug socket
+```
+
+No Chrome extension, no daemon process — just a direct CDP WebSocket connection.
+
+---
+
+## Prerequisites
+
+**On the Android device:**
+1. Settings → About Phone → tap **Build Number** 7 times to enable Developer Options
+2. Settings → Developer Options → enable **USB Debugging**
+3. In Chrome for Android, open `chrome://flags`, search for `DevTools remote debugging`, and enable it (Chrome 119+). On older versions this is on by default when USB debugging is active.
+
+**On your machine:**
+- [Android Debug Bridge (ADB)](https://developer.android.com/tools/adb) installed and on `$PATH`
+- OpenCLI installed (`npm install -g opencli`)
+
+---
+
+## Step-by-Step Setup
+
+### 1. Connect the device
+
+```bash
+adb devices
+```
+
+Expected output:
+```
+List of devices attached
+R5CT443TRDM    device
+```
+
+If the device shows as `unauthorized`, check for a "Allow USB Debugging?" prompt on the phone and tap **Allow**.
+
+### 2. Forward the CDP port
+
+```bash
+adb forward tcp:9222 localabstract:chrome_devtools_remote
+```
+
+### 3. Verify the connection
+
+```bash
+curl http://localhost:9222/json
+```
+
+A successful response lists the open tabs:
+```json
+[
+  {
+    "id": "3941",
+    "title": "Hacker News",
+    "type": "page",
+    "url": "https://news.ycombinator.com",
+    "webSocketDebuggerUrl": "ws://localhost:9222/devtools/page/3941"
+  }
+]
+```
+
+### 4. Run any OpenCLI command
+
+```bash
+export OPENCLI_CDP_ENDPOINT=http://localhost:9222
+opencli hackernews top --limit 5
+```
+
+---
+
+## Targeting a Specific Tab
+
+When multiple tabs are open, `CDPBridge` picks the best one automatically using a scoring algorithm (prefer `type=page`, real URLs over `about:blank`, etc.). To override this, set `OPENCLI_CDP_TARGET` to a substring of the tab's title or URL:
+
+```bash
+OPENCLI_CDP_TARGET="twitter" opencli twitter trending
+```
+
+You can also connect directly to a specific tab's WebSocket URL (from `/json`):
+
+```bash
+OPENCLI_CDP_ENDPOINT=ws://localhost:9222/devtools/page/3941 opencli ...
+```
+
+---
+
+## Using Login-Required Adapters
+
+Adapters that use the `cookie` strategy (most social/content sites) need you to be logged in on the Android device. The cookies are already in Android Chrome — OpenCLI reads them automatically over CDP.
+
+To check whether an adapter requires login:
+
+```bash
+opencli zhihu hot --help
+# Strategy: cookie | Browser: yes | Domain: www.zhihu.com
+```
+
+If you see `Strategy: cookie`, log into the site on the phone first, then run the command.
+
+---
+
+## Teardown
+
+Remove the port forward when done:
+
+```bash
+adb forward --remove tcp:9222
+# or remove all forwards:
+adb forward --remove-all
+```
+
+---
+
+## Differences from Desktop Chrome
+
+| Feature | Desktop Chrome (BrowserBridge) | Android Chrome (CDPBridge) |
+|---------|-------------------------------|---------------------------|
+| Chrome extension required | Yes | No |
+| Daemon process | Yes (auto-started) | No |
+| Multi-tab management | Full (`tabs`, `selectTab`) | Not supported |
+| Cookie session | Desktop browser's cookies | Android browser's cookies |
+| Touch events | N/A | Not needed (CDP uses DOM events) |
+| Concurrent devices | N/A | Use different local ports per device |
+
+---
+
+## Multiple Devices
+
+To connect to more than one Android device simultaneously, assign each a different local port:
+
+```bash
+# Device 1
+adb -s <device1-serial> forward tcp:9222 localabstract:chrome_devtools_remote
+
+# Device 2
+adb -s <device2-serial> forward tcp:9223 localabstract:chrome_devtools_remote
+
+# Run commands targeting each device
+OPENCLI_CDP_ENDPOINT=http://localhost:9222 opencli twitter trending
+OPENCLI_CDP_ENDPOINT=http://localhost:9223 opencli twitter trending
+```
+
+---
+
+## Troubleshooting
+
+**`adb devices` shows nothing**
+- Check USB cable (data cable, not charge-only)
+- Revoke USB debugging authorizations on the device and re-approve
+
+**`curl http://localhost:9222/json` returns empty array `[]`**
+- Chrome for Android is not open, or has no visible tabs — open a tab and retry
+- Remote debugging flag in `chrome://flags` may be disabled
+
+**`curl http://localhost:9222/json` returns connection refused**
+- Port forward may have dropped (happens after device screen lock on some ROMs) — re-run `adb forward`
+
+**Adapter returns `(no data)` despite a working connection**
+- The site's API requires authentication: log into the site in Android Chrome first
+- Confirm with `--verbose` to see which pipeline step returns 0 items
@@ -0,0 +1,103 @@
+# Connecting OpenCLI via CDP (Remote/Headless Servers)
+
+If you cannot use the opencli Browser Bridge extension (e.g., in a remote headless server environment without a UI), OpenCLI provides an alternative: connecting directly to Chrome via **CDP (Chrome DevTools Protocol)**.
+
+Because CDP binds to `localhost` by default for security reasons, accessing it from a remote server requires an additional networking tunnel.
+
+This guide is broken down into three phases:
+1. **Preparation**: Start Chrome with CDP enabled locally.
+2. **Network Tunnels**: Expose that CDP port to your remote server using either **SSH Tunnels** or **Reverse Proxies**.
+3. **Execution**: Run OpenCLI on your server.
+
+---
+
+## Phase 1: Preparation (Local Machine)
+
+First, you need to start a Chrome browser on your local machine with remote debugging enabled.
+
+**macOS:**
+```bash
+/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
+  --remote-debugging-port=9222 \
+  --user-data-dir="$HOME/chrome-debug-profile" \
+  --remote-allow-origins="*"
+```
+
+**Linux:**
+```bash
+google-chrome \
+  --remote-debugging-port=9222 \
+  --user-data-dir="$HOME/chrome-debug-profile" \
+  --remote-allow-origins="*"
+```
+
+**Windows:**
+```cmd
+"C:\Program Files\Google\Chrome\Application\chrome.exe" ^
+  --remote-debugging-port=9222 ^
+  --user-data-dir="%USERPROFILE%\chrome-debug-profile" ^
+  --remote-allow-origins="*"
+```
+
+> **Note**: The `--remote-allow-origins="*"` flag is often required for modern Chrome versions to accept cross-origin CDP WebSocket connections (e.g. from reverse proxies like ngrok).
+
+Once this browser instance opens, **log into the target websites you want to use** (e.g., bilibili.com, zhihu.com) so that the session contains the correct cookies.
+
+---
+
+## Phase 2: Remote Access Methods
+
+Once CDP is running locally on port `9222`, you must securely expose this port to your remote server. Choose one of the two methods below depending on your network conditions.
+
+### Method A: SSH Tunnel (Recommended)
+
+If your local machine has SSH access to the remote server, this is the most secure and straightforward method.
+
+Run this command on your **Local Machine** to forward the remote server's port `9222` back to your local port `9222`:
+
+```bash
+ssh -R 9222:localhost:9222 your-server-user@your-server-ip
+```
+
+Leave this SSH session running in the background.
+
+### Method B: Reverse Proxy (ngrok / frp / socat)
+
+If you cannot establish a direct SSH connection (e.g., due to NAT or firewalls), you can use an intranet penetration tool like `ngrok`.
+
+Run this command on your **Local Machine** to expose your local port `9222` to the public internet securely via ngrok:
+
+```bash
+ngrok http 9222
+```
+
+This will print a forwarding URL, such as `https://abcdef.ngrok.app`. **Copy this URL**.
+
+---
+
+## Phase 3: Execution (Remote Server)
+
+Now switch to your **Remote Server** where OpenCLI is installed. 
+
+Depending on the network tunnel method you chose in Phase 2, set the `OPENCLI_CDP_ENDPOINT` environment variable and run your commands.
+
+### If you used Method A (SSH Tunnel):
+
+```bash
+export OPENCLI_CDP_ENDPOINT="http://localhost:9222"
+opencli doctor                    # Verify connection
+opencli bilibili hot --limit 5    # Test a command
+```
+
+### If you used Method B (Reverse Proxy like ngrok):
+
+```bash
+# Use the URL you copied from ngrok earlier
+export OPENCLI_CDP_ENDPOINT="https://abcdef.ngrok.app"
+opencli doctor                    # Verify connection
+opencli bilibili hot --limit 5    # Test a command
+```
+
+> *Tip: If you provide a standard HTTP/HTTPS CDP endpoint, OpenCLI requests the `/json` target list and picks the most likely inspectable app/page target automatically. If multiple app targets exist, you can further narrow selection with `OPENCLI_CDP_TARGET` (for example `antigravity` or `codex`).*
+
+If you plan to use this setup frequently, you can persist the environment variable by adding the `export` line to your `~/.bashrc` or `~/.zshrc` on the server.
@@ -0,0 +1,81 @@
+# Download Support
+
+OpenCLI supports downloading images, videos, and articles from supported platforms.
+
+## Supported Platforms
+
+| Platform | Content Types | Notes |
+|----------|---------------|-------|
+| **xiaohongshu** | Images, Videos | Downloads all media from a note |
+| **bilibili** | Videos | Requires `yt-dlp` installed |
+| **twitter** | Images, Videos | Downloads from user media tab or single tweet |
+| **douban** | Images | Downloads poster / still image lists from movie subjects |
+| **xiaoyuzhou** | Audio, Transcript | Downloads episode audio and transcript JSON/text with local credentials |
+| **zhihu** | Articles (Markdown) | Exports articles with optional image download |
+| **weixin** | Articles (Markdown) | Exports WeChat Official Account articles |
+
+## Prerequisites
+
+For video downloads from streaming platforms, install `yt-dlp`:
+
+```bash
+# Install yt-dlp
+pip install yt-dlp
+# or
+brew install yt-dlp
+```
+
+## Usage Examples
+
+```bash
+# Download images/videos from Xiaohongshu note
+opencli xiaohongshu download "https://www.xiaohongshu.com/search_result/<id>?xsec_token=..." --output ./xhs
+opencli xiaohongshu download "https://xhslink.com/..." --output ./xhs
+
+# Download Bilibili video (requires yt-dlp)
+opencli bilibili download --bvid BV1xxx --output ./bilibili
+opencli bilibili download --bvid BV1xxx --quality 1080p
+
+# Download Twitter media from user
+opencli twitter download elonmusk --limit 20 --output ./twitter
+
+# Download single tweet media
+opencli twitter download --tweet-url "https://x.com/user/status/123" --output ./twitter
+
+# Download Douban posters / stills
+opencli douban download 30382501 --output ./douban
+
+# Download Xiaoyuzhou episode audio
+opencli xiaoyuzhou download 69b3b675772ac2295bfc01d0 --output ./xiaoyuzhou
+
+# Download Xiaoyuzhou transcript JSON + text
+opencli xiaoyuzhou transcript 69dd0c98e2c8be31551f6a33 --output ./xiaoyuzhou-transcripts
+
+# Export Zhihu article to Markdown
+opencli zhihu download "https://zhuanlan.zhihu.com/p/xxx" --output ./zhihu
+
+# Export with local images
+opencli zhihu download "https://zhuanlan.zhihu.com/p/xxx" --download-images
+
+# Export WeChat article to Markdown
+opencli weixin download --url "https://mp.weixin.qq.com/s/xxx" --output ./weixin
+```
+
+`opencli xiaoyuzhou download` and `transcript` require local Xiaoyuzhou credentials in `~/.opencli/xiaoyuzhou.json`.
+
+## Pipeline Step
+
+The `download` step can be used in pipeline adapters:
+
+::: v-pre
+```yaml
+pipeline:
+  - fetch: https://api.example.com/media
+  - download:
+      url: ${{ item.imageUrl }}
+      dir: ./downloads
+      filename: ${{ item.title | sanitize }}.jpg
+      concurrency: 5
+      skip_existing: true
+```
+:::
@@ -0,0 +1,126 @@
+---
+description: How to CLI-ify and automate any Electron Desktop Application via CDP
+---
+
+# CLI-ifying Electron Applications (Skill Guide)
+
+Based on the successful automation of **Cursor**, **Codex**, **Antigravity**, **ChatWise**, and **Discord** desktop apps, this guide serves as the standard operating procedure (SOP) for adapting ANY Electron-based application into an OpenCLI adapter.
+
+## Core Concept
+
+Electron apps are essentially local Chromium browser instances. By exposing a debugging port (CDP — Chrome DevTools Protocol) at launch time, we can use the Browser Bridge to pierce through the UI layer, accessing and controlling all underlying state including React/Vue components and Shadow DOM.
+
+> **Note:** Not all desktop apps are Electron. WeChat (native Cocoa) and Feishu/Lark (custom Lark Framework) embed Chromium but do NOT expose CDP. For those apps, use the AppleScript + clipboard approach instead (see [Non-Electron Pattern](#non-electron-pattern-applescript)).
+
+### Launching the Target App
+```bash
+/Applications/AppName.app/Contents/MacOS/AppName --remote-debugging-port=<unique-port>
+```
+
+### Verifying Electron
+```bash
+# Check for Electron Framework in the app bundle
+ls /Applications/AppName.app/Contents/Frameworks/Electron\ Framework.framework
+# If this directory exists → Electron → CDP works
+# If not → check for libEGL.dylib (embedded Chromium/CEF, CDP may not work)
+```
+
+## The 5-Command Pattern (CDP / Electron)
+
+Every new Electron adapter should implement these 5 commands in `clis/<app_name>/`:
+
+### 1. `status.ts` — Connection Test
+```typescript
+export const statusCommand = cli({
+  site: 'myapp',
+  name: 'status',
+  domain: 'localhost',
+  strategy: Strategy.UI,
+  browser: true,       // Requires CDP connection
+  args: [],
+  columns: ['Status', 'Url', 'Title'],
+  func: async (page: IPage) => {
+    const url = await page.evaluate('window.location.href');
+    const title = await page.evaluate('document.title');
+    return [{ Status: 'Connected', Url: url, Title: title }];
+  },
+});
+```
+
+### 2. `dump.ts` — Reverse Engineering Core
+Modern app DOMs are huge and obfuscated. **Never guess selectors.** Dump first, then extract precise class names with AI or `grep`:
+```typescript
+const dom = await page.evaluate('document.body.innerHTML');
+fs.writeFileSync('/tmp/app-dom.html', dom);
+const snap = await page.snapshot({ interactive: false });
+fs.writeFileSync('/tmp/app-snapshot.json', JSON.stringify(snap, null, 2));
+```
+
+### 3. `send.ts` — Advanced Text Injection
+Electron apps often use complex rich-text editors (Monaco, Lexical, ProseMirror). Setting `.value` directly is ignored by React state.
+
+**Best practice:** Use `document.execCommand('insertText')` to perfectly simulate real user input, fully piercing React state:
+```javascript
+const composer = document.querySelector('[contenteditable="true"]');
+composer.focus();
+document.execCommand('insertText', false, 'Hello');
+```
+Then submit with `await page.pressKey('Enter')`.
+
+### 4. `read.ts` — Context Extraction
+Don't extract the entire page text. Use `dump.ts` output to find the real "conversation container":
+- Look for semantic selectors: `[role="log"]`, `[data-testid="conversation"]`, `[data-content-search-turn-key]`
+- Format output as Markdown — readable by both humans and LLMs
+
+### 5. `new.ts` — Keyboard Shortcuts
+Many GUI actions respond to native shortcuts rather than button clicks:
+```typescript
+const isMac = process.platform === 'darwin';
+await page.pressKey(isMac ? 'Meta+N' : 'Control+N');
+await page.wait(1); // Wait for re-render
+```
+
+## Environment Variable
+```bash
+export OPENCLI_CDP_ENDPOINT="http://127.0.0.1:<unique-port>"
+```
+
+## Non-Electron Pattern (AppleScript)
+
+For native macOS apps (WeChat, Feishu) that don't expose CDP:
+```typescript
+export const statusCommand = cli({
+  site: 'myapp',
+  strategy: Strategy.PUBLIC,
+  browser: false,       // No browser needed
+  func: async (page: IPage | null) => {
+    const output = execSync("osascript -e 'application \"MyApp\" is running'", { encoding: 'utf-8' }).trim();
+    return [{ Status: output === 'true' ? 'Running' : 'Stopped' }];
+  },
+});
+```
+
+Core techniques:
+- **status**: `osascript -e 'application "AppName" is running'`
+- **send**: `pbcopy` → activate window → `Cmd+V` → `Enter`
+- **read**: `Cmd+A` → `Cmd+C` → `pbpaste`
+- **search**: Activate → `Cmd+F`/`Cmd+K` → `keystroke "query"`
+
+## Pitfalls & Gotchas
+
+1. **Port conflicts (EADDRINUSE)**: Only one app per port. Use unique ports matching the builtin registry: Codex=9238, Doubao=9225, Cursor=9226, ChatWise=9228, Discord=9232, Antigravity=9234, ChatGPT=9236. Avoid `9222`, the default Chrome DevTools port the opencli browser bridge already binds.
+2. **IPage abstraction**: OpenCLI wraps the browser page as `IPage` (`src/types.ts`). Use `page.pressKey()` and `page.evaluate()`, NOT direct DOM APIs
+3. **Timing**: Always add `await page.wait(0.5)` to `1.0` after DOM mutations. Returning too early disconnects prematurely
+4. **AppleScript requires Accessibility**: Terminal app must be granted permission in System Settings → Privacy & Security → Accessibility
+
+## Port Assignment Table
+
+| App | Port | Mode |
+|-----|------|------|
+| Codex | 9238 | CDP |
+| Doubao | 9225 | CDP |
+| Cursor | 9226 | CDP |
+| ChatWise | 9228 | CDP |
+| Discord App | 9232 | CDP |
+| Antigravity | 9234 | CDP |
+| ChatGPT | 9236 | CDP / AppleScript |
@@ -0,0 +1,99 @@
+# Rate Limiter Plugin
+
+An optional plugin that adds a random sleep between browser-based commands to reduce the risk of platform rate-limiting or bot detection.
+
+## Install
+
+```bash
+opencli plugin install github:jackwener/opencli-plugin-rate-limiter
+```
+
+Or copy the example below into `~/.opencli/plugins/rate-limiter/` to use it locally without installing from GitHub.
+
+## What it does
+
+After every command targeting a browser platform (xiaohongshu, weibo, bilibili, douyin, tiktok, …), the plugin sleeps for a random duration — 5–30 seconds by default — before returning control to the caller.
+
+## Configuration
+
+| Variable | Default | Description |
+|---|---|---|
+| `OPENCLI_RATE_MIN` | `5` | Minimum sleep in seconds |
+| `OPENCLI_RATE_MAX` | `30` | Maximum sleep in seconds |
+| `OPENCLI_NO_RATE` | — | Set to `1` to disable entirely (local dev) |
+
+```bash
+# Shorter delays for light scraping
+OPENCLI_RATE_MIN=3 OPENCLI_RATE_MAX=10 opencli xiaohongshu search "AI眼镜"
+
+# Skip delays when iterating locally
+OPENCLI_NO_RATE=1 opencli bilibili comments BV1WtAGzYEBm
+```
+
+## Local installation (without GitHub)
+
+1. Create the plugin directory:
+
+   ```bash
+   mkdir -p ~/.opencli/plugins/rate-limiter
+   ```
+
+2. Create `~/.opencli/plugins/rate-limiter/package.json`:
+
+   ```json
+   { "type": "module" }
+   ```
+
+3. Create `~/.opencli/plugins/rate-limiter/index.js`:
+
+   ```js
+   import { onAfterExecute } from '@jackwener/opencli/hooks'
+
+   const BROWSER_DOMAINS = [
+     'xiaohongshu', 'weibo', 'bilibili', 'douyin', 'tiktok',
+     'instagram', 'twitter', 'youtube', 'zhihu', 'douban',
+     'jike', 'weixin', 'xiaoyuzhou',
+   ]
+
+   onAfterExecute(async (ctx) => {
+     if (process.env.OPENCLI_NO_RATE === '1') return
+
+     const site = ctx.command?.split('/')?.[0] ?? ''
+     if (!BROWSER_DOMAINS.includes(site)) return
+
+     const min = Number(process.env.OPENCLI_RATE_MIN ?? 5)
+     const max = Number(process.env.OPENCLI_RATE_MAX ?? 30)
+     const ms = Math.floor(Math.random() * (max - min + 1) + min) * 1000
+
+     process.stderr.write(`[rate-limiter] ${site}: sleeping ${(ms / 1000).toFixed(0)}s\n`)
+     await new Promise(r => setTimeout(r, ms))
+   })
+   ```
+
+4. Verify it loaded:
+
+   ```bash
+   OPENCLI_NO_RATE=1 opencli xiaohongshu search "test" 2>&1 | grep rate-limiter
+   # → (no output — plugin loaded but rate limit skipped)
+
+   opencli xiaohongshu search "test" 2>&1 | grep rate-limiter
+   # → [rate-limiter] xiaohongshu: sleeping 12s
+   ```
+
+## Writing your own plugin
+
+Plugins are plain JS/TS files in `~/.opencli/plugins/<name>/`. A plugin file must export a hook registration call that matches the pattern `onStartup(`, `onBeforeExecute(`, or `onAfterExecute(` — opencli's discovery engine uses this pattern to identify hook files vs. command files.
+
+```js
+// ~/.opencli/plugins/my-plugin/index.js
+import { onAfterExecute } from '@jackwener/opencli/hooks'
+
+onAfterExecute(async (ctx) => {
+  // ctx.command — e.g. "bilibili/comments"
+  // ctx.args    — coerced command arguments
+  // ctx.error   — set if the command threw
+  console.error(`[my-plugin] finished: ${ctx.command}`)
+})
+```
+
+See [hooks.ts](../../src/hooks.ts) for the full `HookContext` type.
@@ -0,0 +1,64 @@
+# Remote Chrome
+
+Run OpenCLI on a server or headless environment by connecting to a remote Chrome instance.
+
+## Use Cases
+
+- Running CLI commands on a remote server
+- CI/CD automation with headed browser
+- Shared team browser sessions
+
+## Setup
+
+### 1. Start Chrome on the Remote Machine
+
+```bash
+# On the remote machine (or your Mac)
+/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
+  --remote-debugging-port=9222
+```
+
+### 2. SSH Tunnel (If Needed)
+
+If the remote Chrome is on a different machine, create an SSH tunnel:
+
+```bash
+# On your local machine or server
+ssh -L 9222:127.0.0.1:9222 user@remote-host
+```
+
+::: warning
+Use `127.0.0.1` instead of `localhost` in the SSH command to avoid IPv6 resolution issues that can cause timeouts.
+:::
+
+### 3. Configure OpenCLI
+
+```bash
+export OPENCLI_CDP_ENDPOINT="http://127.0.0.1:9222"
+```
+
+### 4. Verify
+
+```bash
+# Test the connection
+curl http://127.0.0.1:9222/json/version
+
+# Run a diagnostic
+opencli doctor
+```
+
+## CI/CD Integration
+
+For CI/CD environments, use a real Chrome instance with `xvfb`:
+
+::: v-pre
+```yaml
+steps:
+  - uses: browser-actions/setup-chrome@latest
+    id: setup-chrome
+  - run: |
+      xvfb-run --auto-servernum \
+        ${{ steps.setup-chrome.outputs.chrome-path }} \
+        --remote-debugging-port=9222 &
+```
+:::