chore: import upstream snapshot with attribution

This commit is contained in:
wehub-resource-sync
2026-07-13 13:11:38 +08:00
commit 03e739d01e
86 changed files with 11352 additions and 0 deletions
+8
View File
@@ -0,0 +1,8 @@
worker/node_modules/
worker/.dev.vars
.DS_Store
*.xcuserstate
build/
releases/
.claude/
coding-plans/
+167
View File
@@ -0,0 +1,167 @@
# Clicky - Agent Instructions
<!-- This is the single source of truth for all AI coding agents. CLAUDE.md is a symlink to this file. -->
<!-- AGENTS.md spec: https://github.com/agentsmd/agents.md — supported by Claude Code, Cursor, Copilot, Gemini CLI, and others. -->
## Overview
macOS menu bar companion app. Lives entirely in the macOS status bar (no dock icon, no main window). Clicking the menu bar icon opens a custom floating panel with companion voice controls. Uses push-to-talk (ctrl+option) to capture voice input, transcribes it via AssemblyAI streaming, and sends the transcript + a screenshot of the user's screen to Claude. Claude responds with text (streamed via SSE) and voice (ElevenLabs TTS). A blue cursor overlay can fly to and point at UI elements Claude references on any connected monitor.
All API keys live on a Cloudflare Worker proxy — nothing sensitive ships in the app.
## Architecture
- **App Type**: Menu bar-only (`LSUIElement=true`), no dock icon or main window
- **Framework**: SwiftUI (macOS native) with AppKit bridging for menu bar panel and cursor overlay
- **Pattern**: MVVM with `@StateObject` / `@Published` state management
- **AI Chat**: Claude (Sonnet 4.6 default, Opus 4.6 optional) via Cloudflare Worker proxy with SSE streaming
- **Speech-to-Text**: AssemblyAI real-time streaming (`u3-rt-pro` model) via websocket, with OpenAI and Apple Speech as fallbacks
- **Text-to-Speech**: ElevenLabs (`eleven_flash_v2_5` model) via Cloudflare Worker proxy
- **Screen Capture**: ScreenCaptureKit (macOS 14.2+), multi-monitor support
- **Voice Input**: Push-to-talk via `AVAudioEngine` + pluggable transcription-provider layer. System-wide keyboard shortcut via listen-only CGEvent tap.
- **Element Pointing**: Claude embeds `[POINT:x,y:label:screenN]` tags in responses. The overlay parses these, maps coordinates to the correct monitor, and animates the blue cursor along a bezier arc to the target.
- **Concurrency**: `@MainActor` isolation, async/await throughout
- **Analytics**: PostHog via `ClickyAnalytics.swift`
### API Proxy (Cloudflare Worker)
The app never calls external APIs directly. All requests go through a Cloudflare Worker (`worker/src/index.ts`) that holds the real API keys as secrets.
| Route | Upstream | Purpose |
|-------|----------|---------|
| `POST /chat` | `api.anthropic.com/v1/messages` | Claude vision + streaming chat |
| `POST /tts` | `api.elevenlabs.io/v1/text-to-speech/{voiceId}` | ElevenLabs TTS audio |
| `POST /transcribe-token` | `streaming.assemblyai.com/v3/token` | Fetches a short-lived (480s) AssemblyAI websocket token |
Worker secrets: `ANTHROPIC_API_KEY`, `ASSEMBLYAI_API_KEY`, `ELEVENLABS_API_KEY`
Worker vars: `ELEVENLABS_VOICE_ID`
### Key Architecture Decisions
**Menu Bar Panel Pattern**: The companion panel uses `NSStatusItem` for the menu bar icon and a custom borderless `NSPanel` for the floating control panel. This gives full control over appearance (dark, rounded corners, custom shadow) and avoids the standard macOS menu/popover chrome. The panel is non-activating so it doesn't steal focus. A global event monitor auto-dismisses it on outside clicks.
**Cursor Overlay**: A full-screen transparent `NSPanel` hosts the blue cursor companion. It's non-activating, joins all Spaces, and never steals focus. The cursor position, response text, waveform, and pointing animations all render in this overlay via SwiftUI through `NSHostingView`.
**Global Push-To-Talk Shortcut**: Background push-to-talk uses a listen-only `CGEvent` tap instead of an AppKit global monitor so modifier-based shortcuts like `ctrl + option` are detected more reliably while the app is running in the background.
**Shared URLSession for AssemblyAI**: A single long-lived `URLSession` is shared across all AssemblyAI streaming sessions (owned by the provider, not the session). Creating and invalidating a URLSession per session corrupts the OS connection pool and causes "Socket is not connected" errors after a few rapid reconnections.
**Transient Cursor Mode**: When "Show Clicky" is off, pressing the hotkey fades in the cursor overlay for the duration of the interaction (recording → response → TTS → optional pointing), then fades it out automatically after 1 second of inactivity.
## Key Files
| File | Lines | Purpose |
|------|-------|---------|
| `leanring_buddyApp.swift` | ~89 | Menu bar app entry point. Uses `@NSApplicationDelegateAdaptor` with `CompanionAppDelegate` which creates `MenuBarPanelManager` and starts `CompanionManager`. No main window — the app lives entirely in the status bar. |
| `CompanionManager.swift` | ~1026 | Central state machine. Owns dictation, shortcut monitoring, screen capture, Claude API, ElevenLabs TTS, and overlay management. Tracks voice state (idle/listening/processing/responding), conversation history, model selection, and cursor visibility. Coordinates the full push-to-talk → screenshot → Claude → TTS → pointing pipeline. |
| `MenuBarPanelManager.swift` | ~243 | NSStatusItem + custom NSPanel lifecycle. Creates the menu bar icon, manages the floating companion panel (show/hide/position), installs click-outside-to-dismiss monitor. |
| `CompanionPanelView.swift` | ~761 | SwiftUI panel content for the menu bar dropdown. Shows companion status, push-to-talk instructions, model picker (Sonnet/Opus), permissions UI, DM feedback button, and quit button. Dark aesthetic using `DS` design system. |
| `OverlayWindow.swift` | ~881 | Full-screen transparent overlay hosting the blue cursor, response text, waveform, and spinner. Handles cursor animation, element pointing with bezier arcs, multi-monitor coordinate mapping, and fade-out transitions. |
| `CompanionResponseOverlay.swift` | ~217 | SwiftUI view for the response text bubble and waveform displayed next to the cursor in the overlay. |
| `CompanionScreenCaptureUtility.swift` | ~132 | Multi-monitor screenshot capture using ScreenCaptureKit. Returns labeled image data for each connected display. |
| `BuddyDictationManager.swift` | ~866 | Push-to-talk voice pipeline. Handles microphone capture via `AVAudioEngine`, provider-aware permission checks, keyboard/button dictation sessions, transcript finalization, shortcut parsing, contextual keyterms, and live audio-level reporting for waveform feedback. |
| `BuddyTranscriptionProvider.swift` | ~100 | Protocol surface and provider factory for voice transcription backends. Resolves provider based on `VoiceTranscriptionProvider` in Info.plist — AssemblyAI, OpenAI, or Apple Speech. |
| `AssemblyAIStreamingTranscriptionProvider.swift` | ~478 | Streaming transcription provider. Fetches temp tokens from the Cloudflare Worker, opens an AssemblyAI v3 websocket, streams PCM16 audio, tracks turn-based transcripts, and delivers finalized text on key-up. Shares a single URLSession across all sessions. |
| `OpenAIAudioTranscriptionProvider.swift` | ~317 | Upload-based transcription provider. Buffers push-to-talk audio locally, uploads as WAV on release, returns finalized transcript. |
| `AppleSpeechTranscriptionProvider.swift` | ~147 | Local fallback transcription provider backed by Apple's Speech framework. |
| `BuddyAudioConversionSupport.swift` | ~108 | Audio conversion helpers. Converts live mic buffers to PCM16 mono audio and builds WAV payloads for upload-based providers. |
| `GlobalPushToTalkShortcutMonitor.swift` | ~132 | System-wide push-to-talk monitor. Owns the listen-only `CGEvent` tap and publishes press/release transitions. |
| `ClaudeAPI.swift` | ~291 | Claude vision API client with streaming (SSE) and non-streaming modes. TLS warmup optimization, image MIME detection, conversation history support. |
| `OpenAIAPI.swift` | ~142 | OpenAI GPT vision API client. |
| `ElevenLabsTTSClient.swift` | ~81 | ElevenLabs TTS client. Sends text to the Worker proxy, plays back audio via `AVAudioPlayer`. Exposes `isPlaying` for transient cursor scheduling. |
| `ElementLocationDetector.swift` | ~335 | Detects UI element locations in screenshots for cursor pointing. |
| `DesignSystem.swift` | ~880 | Design system tokens — colors, corner radii, shared styles. All UI references `DS.Colors`, `DS.CornerRadius`, etc. |
| `ClickyAnalytics.swift` | ~121 | PostHog analytics integration for usage tracking. |
| `WindowPositionManager.swift` | ~262 | Window placement logic, Screen Recording permission flow, and accessibility permission helpers. |
| `AppBundleConfiguration.swift` | ~28 | Runtime configuration reader for keys stored in the app bundle Info.plist. |
| `worker/src/index.ts` | ~142 | Cloudflare Worker proxy. Three routes: `/chat` (Claude), `/tts` (ElevenLabs), `/transcribe-token` (AssemblyAI temp token). |
## Build & Run
```bash
# Open in Xcode
open leanring-buddy.xcodeproj
# Select the leanring-buddy scheme, set signing team, Cmd+R to build and run
# Known non-blocking warnings: Swift 6 concurrency warnings,
# deprecated onChange warning in OverlayWindow.swift. Do NOT attempt to fix these.
```
**Do NOT run `xcodebuild` from the terminal** — it invalidates TCC (Transparency, Consent, and Control) permissions and the app will need to re-request screen recording, accessibility, etc.
## Cloudflare Worker
```bash
cd worker
npm install
# Add secrets
npx wrangler secret put ANTHROPIC_API_KEY
npx wrangler secret put ASSEMBLYAI_API_KEY
npx wrangler secret put ELEVENLABS_API_KEY
# Deploy
npx wrangler deploy
# Local dev (create worker/.dev.vars with your keys)
npx wrangler dev
```
## Code Style & Conventions
### Variable and Method Naming
IMPORTANT: Follow these naming rules strictly. Clarity is the top priority.
- Be as clear and specific with variable and method names as possible
- **Optimize for clarity over concision.** A developer with zero context on the codebase should immediately understand what a variable or method does just from reading its name
- Use longer names when it improves clarity. Do NOT use single-character variable names
- Example: use `originalQuestionLastAnsweredDate` instead of `originalAnswered`
- When passing props or arguments to functions, keep the same names as the original variable. Do not shorten or abbreviate parameter names. If you have `currentCardData`, pass it as `currentCardData`, not `card` or `cardData`
### Code Clarity
- **Clear is better than clever.** Do not write functionality in fewer lines if it makes the code harder to understand
- Write more lines of code if additional lines improve readability and comprehension
- Make things so clear that someone with zero context would completely understand the variable names, method names, what things do, and why they exist
- When a variable or method name alone cannot fully explain something, add a comment explaining what is happening and why
### Swift/SwiftUI Conventions
- Use SwiftUI for all UI unless a feature is only supported in AppKit (e.g., `NSPanel` for floating windows)
- All UI state updates must be on `@MainActor`
- Use async/await for all asynchronous operations
- Comments should explain "why" not just "what", especially for non-obvious AppKit bridging
- AppKit `NSPanel`/`NSWindow` bridged into SwiftUI via `NSHostingView`
- All buttons must show a pointer cursor on hover
- For any interactive element, explicitly think through its hover behavior (cursor, visual feedback, and whether hover should communicate clickability)
### Do NOT
- Do not add features, refactor code, or make "improvements" beyond what was asked
- Do not add docstrings, comments, or type annotations to code you did not change
- Do not try to fix the known non-blocking warnings (Swift 6 concurrency, deprecated onChange)
- Do not rename the project directory or scheme (the "leanring" typo is intentional/legacy)
- Do not run `xcodebuild` from the terminal — it invalidates TCC permissions
## Git Workflow
- Branch naming: `feature/description` or `fix/description`
- Commit messages: imperative mood, concise, explain the "why" not the "what"
- Do not force-push to main
## Self-Update Instructions
<!-- AI agents: follow these instructions to keep this file accurate. -->
When you make changes to this project that affect the information in this file, update this file to reflect those changes. Specifically:
1. **New files**: Add new source files to the "Key Files" table with their purpose and approximate line count
2. **Deleted files**: Remove entries for files that no longer exist
3. **Architecture changes**: Update the architecture section if you introduce new patterns, frameworks, or significant structural changes
4. **Build changes**: Update build commands if the build process changes
5. **New conventions**: If the user establishes a new coding convention during a session, add it to the appropriate conventions section
6. **Line count drift**: If a file's line count changes significantly (>50 lines), update the approximate count in the Key Files table
Do NOT update this file for minor edits, bug fixes, or changes that don't affect the documented architecture or conventions.
Symlink
+1
View File
@@ -0,0 +1 @@
AGENTS.md
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Farza
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+162
View File
@@ -0,0 +1,162 @@
Update: April 27, 2026.
Hi there! I'm Farza, the guy that made Clicky.
The existing codebase remains open source. Tinker with it, make it yours, start a company out of it, do whatever you want I don't mind. But, for all the new stuff I'm hacking on, gonna keep it private. To get the latest Clicky, you can go [here](https://www.heyclicky.com/).
I also tweeted about this [here](https://x.com/FarzaTV/status/2043402737828962489).
Go crazy with this repo!! It's an MIT license.
# Hi, this is Clicky.
It's an AI teacher that lives as a buddy next to your cursor. It can see your screen, talk to you, and even point at stuff. Kinda like having a real teacher next to you.
Download it [here](https://www.clicky.so/) for free.
Here's the [original tweet](https://x.com/FarzaTV/status/2041314633978659092) that kinda blew up for a demo for more context.
![Clicky — an ai buddy that lives on your mac](clicky-demo.gif)
This is the open-source version of Clicky for those that want to hack on it, build their own features, or just see how it works under the hood.
## Get started with Claude Code
The fastest way to get this running is with [Claude Code](https://docs.anthropic.com/en/docs/claude-code).
Once you get Claude running, paste this:
```
Hi Claude.
Clone https://github.com/farzaa/clicky.git into my current directory.
Then read the CLAUDE.md. I want to get Clicky running locally on my Mac.
Help me set up everything — the Cloudflare Worker with my own API keys, the proxy URLs, and getting it building in Xcode. Walk me through it.
```
That's it. It'll clone the repo, read the docs, and walk you through the whole setup. Once you're running you can just keep talking to it — build features, fix bugs, whatever. Go crazy.
## Manual setup
If you want to do it yourself, here's the deal.
### Prerequisites
- macOS 14.2+ (for ScreenCaptureKit)
- Xcode 15+
- Node.js 18+ (for the Cloudflare Worker)
- A [Cloudflare](https://cloudflare.com) account (free tier works)
- API keys for: [Anthropic](https://console.anthropic.com), [AssemblyAI](https://www.assemblyai.com), [ElevenLabs](https://elevenlabs.io)
### 1. Set up the Cloudflare Worker
The Worker is a tiny proxy that holds your API keys. The app talks to the Worker, the Worker talks to the APIs. This way your keys never ship in the app binary.
```bash
cd worker
npm install
```
Now add your secrets. Wrangler will prompt you to paste each one:
```bash
npx wrangler secret put ANTHROPIC_API_KEY
npx wrangler secret put ASSEMBLYAI_API_KEY
npx wrangler secret put ELEVENLABS_API_KEY
```
For the ElevenLabs voice ID, open `wrangler.toml` and set it there (it's not sensitive):
```toml
[vars]
ELEVENLABS_VOICE_ID = "your-voice-id-here"
```
Deploy it:
```bash
npx wrangler deploy
```
It'll give you a URL like `https://your-worker-name.your-subdomain.workers.dev`. Copy that.
### 2. Run the Worker locally (for development)
If you want to test changes to the Worker without deploying:
```bash
cd worker
npx wrangler dev
```
This starts a local server (usually `http://localhost:8787`) that behaves exactly like the deployed Worker. You'll need to create a `.dev.vars` file in the `worker/` directory with your keys:
```
ANTHROPIC_API_KEY=sk-ant-...
ASSEMBLYAI_API_KEY=...
ELEVENLABS_API_KEY=...
ELEVENLABS_VOICE_ID=...
```
Then update the proxy URLs in the Swift code to point to `http://localhost:8787` instead of the deployed Worker URL while developing. Grep for `clicky-proxy` to find them all.
### 3. Update the proxy URLs in the app
The app has the Worker URL hardcoded in a few places. Search for `your-worker-name.your-subdomain.workers.dev` and replace it with your Worker URL:
```bash
grep -r "clicky-proxy" leanring-buddy/
```
You'll find it in:
- `CompanionManager.swift` — Claude chat + ElevenLabs TTS
- `AssemblyAIStreamingTranscriptionProvider.swift` — AssemblyAI token endpoint
### 4. Open in Xcode and run
```bash
open leanring-buddy.xcodeproj
```
In Xcode:
1. Select the `leanring-buddy` scheme (yes, the typo is intentional, long story)
2. Set your signing team under Signing & Capabilities
3. Hit **Cmd + R** to build and run
The app will appear in your menu bar (not the dock). Click the icon to open the panel, grant the permissions it asks for, and you're good.
### Permissions the app needs
- **Microphone** — for push-to-talk voice capture
- **Accessibility** — for the global keyboard shortcut (Control + Option)
- **Screen Recording** — for taking screenshots when you use the hotkey
- **Screen Content** — for ScreenCaptureKit access
## Architecture
If you want the full technical breakdown, read `CLAUDE.md`. But here's the short version:
**Menu bar app** (no dock icon) with two `NSPanel` windows — one for the control panel dropdown, one for the full-screen transparent cursor overlay. Push-to-talk streams audio over a websocket to AssemblyAI, sends the transcript + screenshot to Claude via streaming SSE, and plays the response through ElevenLabs TTS. Claude can embed `[POINT:x,y:label:screenN]` tags in its responses to make the cursor fly to specific UI elements across multiple monitors. All three APIs are proxied through a Cloudflare Worker.
## Project structure
```
leanring-buddy/ # Swift source (yes, the typo stays)
CompanionManager.swift # Central state machine
CompanionPanelView.swift # Menu bar panel UI
ClaudeAPI.swift # Claude streaming client
ElevenLabsTTSClient.swift # Text-to-speech playback
OverlayWindow.swift # Blue cursor overlay
AssemblyAI*.swift # Real-time transcription
BuddyDictation*.swift # Push-to-talk pipeline
worker/ # Cloudflare Worker proxy
src/index.ts # Three routes: /chat, /tts, /transcribe-token
CLAUDE.md # Full architecture doc (agents read this)
```
## Contributing
PRs welcome. If you're using Claude Code, it already knows the codebase — just tell it what you want to build and point it at `CLAUDE.md`.
Got feedback? DM me on X [@farzatv](https://x.com/farzatv).
+7
View File
@@ -0,0 +1,7 @@
# WeHub 来源说明
- 原始项目:`farzaa/clicky`
- 原始仓库:https://github.com/farzaa/clicky
- 导入方式:上游默认分支的最新快照
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
- 本文件仅用于记录来源,不代表 WeHub 是原项目作者
+30
View File
@@ -0,0 +1,30 @@
<?xml version="1.0" standalone="yes"?>
<rss xmlns:sparkle="http://www.andymatuschak.org/xml-namespaces/sparkle" version="2.0">
<channel>
<title>makesomething</title>
<item>
<title>2.0</title>
<pubDate>Sat, 14 Mar 2026 09:56:02 -0700</pubDate>
<sparkle:version>11</sparkle:version>
<sparkle:shortVersionString>2.0</sparkle:shortVersionString>
<sparkle:minimumSystemVersion>14.0</sparkle:minimumSystemVersion>
<enclosure url="https://github.com/julianjear/makesomething-mac-app/releases/download/v2.0/makesomething.dmg" length="11852050" type="application/octet-stream" sparkle:edSignature="Y9o+YBln/BXAwYeJLuGV0E7B9q9/+kuhhFlNRyekZuYLPwbTYoL7aUYhurAQFfmiSuWuCUNHKXgtzaKCyaMGCQ=="/>
</item>
<item>
<title>1.9</title>
<pubDate>Sat, 14 Mar 2026 08:33:19 -0700</pubDate>
<sparkle:version>10</sparkle:version>
<sparkle:shortVersionString>1.9</sparkle:shortVersionString>
<sparkle:minimumSystemVersion>14.0</sparkle:minimumSystemVersion>
<enclosure url="https://github.com/julianjear/makesomething-mac-app/releases/download/v1.9/makesomething.dmg" length="11851619" type="application/octet-stream" sparkle:edSignature="l0OE/1XAsVgbyyP0vty+50oqzkTeR1yTRsuKtCU/m8UHduT/9cafKI/x1Vy5p8RW2t5JKDFIqmar/jRf6NRuDg=="/>
</item>
<item>
<title>1.8</title>
<pubDate>Wed, 11 Mar 2026 10:17:12 -0700</pubDate>
<sparkle:version>9</sparkle:version>
<sparkle:shortVersionString>1.8</sparkle:shortVersionString>
<sparkle:minimumSystemVersion>14.0</sparkle:minimumSystemVersion>
<enclosure url="https://github.com/julianjear/makesomething-mac-app/releases/download/v1.8/makesomething.dmg" length="11489099" type="application/octet-stream" sparkle:edSignature="K7n75BUbzhzTbrnYDQDcsmMQGb3bEjvl5yxSIdBGtbanNWrEUym2sozah5hGkAQ0sLShrPhx8FDBunmd2T3rAw=="/>
</item>
</channel>
</rss>
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 2.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.4 KiB

+635
View File
@@ -0,0 +1,635 @@
// !$*UTF8*$!
{
archiveVersion = 1;
classes = {
};
objectVersion = 77;
objects = {
/* Begin PBXBuildFile section */
AA00BB032F6500030039DA55 /* Sparkle in Frameworks */ = {isa = PBXBuildFile; productRef = AA00BB022F6500020039DA55 /* Sparkle */; };
AA00BB062F6500060039DA55 /* PostHog in Frameworks */ = {isa = PBXBuildFile; productRef = AA00BB052F6500050039DA55 /* PostHog */; };
/* End PBXBuildFile section */
/* Begin PBXContainerItemProxy section */
28F22CCD2F56440300A0FC59 /* PBXContainerItemProxy */ = {
isa = PBXContainerItemProxy;
containerPortal = 28F22CB72F56440300A0FC59 /* Project object */;
proxyType = 1;
remoteGlobalIDString = 28F22CBE2F56440300A0FC59;
remoteInfo = "leanring-buddy";
};
28F22CD72F56440300A0FC59 /* PBXContainerItemProxy */ = {
isa = PBXContainerItemProxy;
containerPortal = 28F22CB72F56440300A0FC59 /* Project object */;
proxyType = 1;
remoteGlobalIDString = 28F22CBE2F56440300A0FC59;
remoteInfo = "leanring-buddy";
};
/* End PBXContainerItemProxy section */
/* Begin PBXFileReference section */
28F22CBF2F56440300A0FC59 /* Clicky.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = Clicky.app; sourceTree = BUILT_PRODUCTS_DIR; };
28F22CCC2F56440300A0FC59 /* leanring-buddyTests.xctest */ = {isa = PBXFileReference; explicitFileType = wrapper.cfbundle; includeInIndex = 0; path = "leanring-buddyTests.xctest"; sourceTree = BUILT_PRODUCTS_DIR; };
28F22CD62F56440300A0FC59 /* leanring-buddyUITests.xctest */ = {isa = PBXFileReference; explicitFileType = wrapper.cfbundle; includeInIndex = 0; path = "leanring-buddyUITests.xctest"; sourceTree = BUILT_PRODUCTS_DIR; };
/* End PBXFileReference section */
/* Begin PBXFileSystemSynchronizedRootGroup section */
28F22CC12F56440300A0FC59 /* leanring-buddy */ = {
isa = PBXFileSystemSynchronizedRootGroup;
path = "leanring-buddy";
sourceTree = "<group>";
};
28F22CCF2F56440300A0FC59 /* leanring-buddyTests */ = {
isa = PBXFileSystemSynchronizedRootGroup;
path = "leanring-buddyTests";
sourceTree = "<group>";
};
28F22CD92F56440300A0FC59 /* leanring-buddyUITests */ = {
isa = PBXFileSystemSynchronizedRootGroup;
path = "leanring-buddyUITests";
sourceTree = "<group>";
};
/* End PBXFileSystemSynchronizedRootGroup section */
/* Begin PBXFrameworksBuildPhase section */
28F22CBC2F56440300A0FC59 /* Frameworks */ = {
isa = PBXFrameworksBuildPhase;
buildActionMask = 2147483647;
files = (
AA00BB032F6500030039DA55 /* Sparkle in Frameworks */,
AA00BB062F6500060039DA55 /* PostHog in Frameworks */,
);
runOnlyForDeploymentPostprocessing = 0;
};
28F22CC92F56440300A0FC59 /* Frameworks */ = {
isa = PBXFrameworksBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
28F22CD32F56440300A0FC59 /* Frameworks */ = {
isa = PBXFrameworksBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
/* End PBXFrameworksBuildPhase section */
/* Begin PBXGroup section */
28F22CB62F56440300A0FC59 = {
isa = PBXGroup;
children = (
28F22CC12F56440300A0FC59 /* leanring-buddy */,
28F22CCF2F56440300A0FC59 /* leanring-buddyTests */,
28F22CD92F56440300A0FC59 /* leanring-buddyUITests */,
28F22CC02F56440300A0FC59 /* Products */,
);
sourceTree = "<group>";
};
28F22CC02F56440300A0FC59 /* Products */ = {
isa = PBXGroup;
children = (
28F22CBF2F56440300A0FC59 /* Clicky.app */,
28F22CCC2F56440300A0FC59 /* leanring-buddyTests.xctest */,
28F22CD62F56440300A0FC59 /* leanring-buddyUITests.xctest */,
);
name = Products;
sourceTree = "<group>";
};
/* End PBXGroup section */
/* Begin PBXNativeTarget section */
28F22CBE2F56440300A0FC59 /* leanring-buddy */ = {
isa = PBXNativeTarget;
buildConfigurationList = 28F22CE02F56440300A0FC59 /* Build configuration list for PBXNativeTarget "leanring-buddy" */;
buildPhases = (
28F22CBB2F56440300A0FC59 /* Sources */,
28F22CBC2F56440300A0FC59 /* Frameworks */,
28F22CBD2F56440300A0FC59 /* Resources */,
);
buildRules = (
);
dependencies = (
);
fileSystemSynchronizedGroups = (
28F22CC12F56440300A0FC59 /* leanring-buddy */,
);
name = "leanring-buddy";
packageProductDependencies = (
AA00BB022F6500020039DA55 /* Sparkle */,
AA00BB052F6500050039DA55 /* PostHog */,
);
productName = "leanring-buddy";
productReference = 28F22CBF2F56440300A0FC59 /* Clicky.app */;
productType = "com.apple.product-type.application";
};
28F22CCB2F56440300A0FC59 /* leanring-buddyTests */ = {
isa = PBXNativeTarget;
buildConfigurationList = 28F22CE32F56440300A0FC59 /* Build configuration list for PBXNativeTarget "leanring-buddyTests" */;
buildPhases = (
28F22CC82F56440300A0FC59 /* Sources */,
28F22CC92F56440300A0FC59 /* Frameworks */,
28F22CCA2F56440300A0FC59 /* Resources */,
);
buildRules = (
);
dependencies = (
28F22CCE2F56440300A0FC59 /* PBXTargetDependency */,
);
fileSystemSynchronizedGroups = (
28F22CCF2F56440300A0FC59 /* leanring-buddyTests */,
);
name = "leanring-buddyTests";
packageProductDependencies = (
);
productName = "leanring-buddyTests";
productReference = 28F22CCC2F56440300A0FC59 /* leanring-buddyTests.xctest */;
productType = "com.apple.product-type.bundle.unit-test";
};
28F22CD52F56440300A0FC59 /* leanring-buddyUITests */ = {
isa = PBXNativeTarget;
buildConfigurationList = 28F22CE62F56440300A0FC59 /* Build configuration list for PBXNativeTarget "leanring-buddyUITests" */;
buildPhases = (
28F22CD22F56440300A0FC59 /* Sources */,
28F22CD32F56440300A0FC59 /* Frameworks */,
28F22CD42F56440300A0FC59 /* Resources */,
);
buildRules = (
);
dependencies = (
28F22CD82F56440300A0FC59 /* PBXTargetDependency */,
);
fileSystemSynchronizedGroups = (
28F22CD92F56440300A0FC59 /* leanring-buddyUITests */,
);
name = "leanring-buddyUITests";
packageProductDependencies = (
);
productName = "leanring-buddyUITests";
productReference = 28F22CD62F56440300A0FC59 /* leanring-buddyUITests.xctest */;
productType = "com.apple.product-type.bundle.ui-testing";
};
/* End PBXNativeTarget section */
/* Begin PBXProject section */
28F22CB72F56440300A0FC59 /* Project object */ = {
isa = PBXProject;
attributes = {
BuildIndependentTargetsInParallel = 1;
LastSwiftUpdateCheck = 2620;
LastUpgradeCheck = 2620;
TargetAttributes = {
28F22CBE2F56440300A0FC59 = {
CreatedOnToolsVersion = 26.2;
};
28F22CCB2F56440300A0FC59 = {
CreatedOnToolsVersion = 26.2;
TestTargetID = 28F22CBE2F56440300A0FC59;
};
28F22CD52F56440300A0FC59 = {
CreatedOnToolsVersion = 26.2;
TestTargetID = 28F22CBE2F56440300A0FC59;
};
};
};
buildConfigurationList = 28F22CBA2F56440300A0FC59 /* Build configuration list for PBXProject "leanring-buddy" */;
developmentRegion = en;
hasScannedForEncodings = 0;
knownRegions = (
en,
Base,
);
mainGroup = 28F22CB62F56440300A0FC59;
minimizedProjectReferenceProxies = 1;
packageReferences = (
AA00BB012F6500010039DA55 /* XCRemoteSwiftPackageReference "Sparkle" */,
AA00BB042F6500040039DA55 /* XCRemoteSwiftPackageReference "posthog-ios" */,
);
preferredProjectObjectVersion = 77;
productRefGroup = 28F22CC02F56440300A0FC59 /* Products */;
projectDirPath = "";
projectRoot = "";
targets = (
28F22CBE2F56440300A0FC59 /* leanring-buddy */,
28F22CCB2F56440300A0FC59 /* leanring-buddyTests */,
28F22CD52F56440300A0FC59 /* leanring-buddyUITests */,
);
};
/* End PBXProject section */
/* Begin PBXResourcesBuildPhase section */
28F22CBD2F56440300A0FC59 /* Resources */ = {
isa = PBXResourcesBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
28F22CCA2F56440300A0FC59 /* Resources */ = {
isa = PBXResourcesBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
28F22CD42F56440300A0FC59 /* Resources */ = {
isa = PBXResourcesBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
/* End PBXResourcesBuildPhase section */
/* Begin PBXSourcesBuildPhase section */
28F22CBB2F56440300A0FC59 /* Sources */ = {
isa = PBXSourcesBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
28F22CC82F56440300A0FC59 /* Sources */ = {
isa = PBXSourcesBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
28F22CD22F56440300A0FC59 /* Sources */ = {
isa = PBXSourcesBuildPhase;
buildActionMask = 2147483647;
files = (
);
runOnlyForDeploymentPostprocessing = 0;
};
/* End PBXSourcesBuildPhase section */
/* Begin PBXTargetDependency section */
28F22CCE2F56440300A0FC59 /* PBXTargetDependency */ = {
isa = PBXTargetDependency;
target = 28F22CBE2F56440300A0FC59 /* leanring-buddy */;
targetProxy = 28F22CCD2F56440300A0FC59 /* PBXContainerItemProxy */;
};
28F22CD82F56440300A0FC59 /* PBXTargetDependency */ = {
isa = PBXTargetDependency;
target = 28F22CBE2F56440300A0FC59 /* leanring-buddy */;
targetProxy = 28F22CD72F56440300A0FC59 /* PBXContainerItemProxy */;
};
/* End PBXTargetDependency section */
/* Begin XCBuildConfiguration section */
28F22CDE2F56440300A0FC59 /* Debug */ = {
isa = XCBuildConfiguration;
buildSettings = {
ALWAYS_SEARCH_USER_PATHS = NO;
ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS = YES;
CLANG_ANALYZER_NONNULL = YES;
CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE;
CLANG_CXX_LANGUAGE_STANDARD = "gnu++20";
CLANG_ENABLE_MODULES = YES;
CLANG_ENABLE_OBJC_ARC = YES;
CLANG_ENABLE_OBJC_WEAK = YES;
CLANG_WARN_BLOCK_CAPTURE_AUTORELEASING = YES;
CLANG_WARN_BOOL_CONVERSION = YES;
CLANG_WARN_COMMA = YES;
CLANG_WARN_CONSTANT_CONVERSION = YES;
CLANG_WARN_DEPRECATED_OBJC_IMPLEMENTATIONS = YES;
CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR;
CLANG_WARN_DOCUMENTATION_COMMENTS = YES;
CLANG_WARN_EMPTY_BODY = YES;
CLANG_WARN_ENUM_CONVERSION = YES;
CLANG_WARN_INFINITE_RECURSION = YES;
CLANG_WARN_INT_CONVERSION = YES;
CLANG_WARN_NON_LITERAL_NULL_CONVERSION = YES;
CLANG_WARN_OBJC_IMPLICIT_RETAIN_SELF = YES;
CLANG_WARN_OBJC_LITERAL_CONVERSION = YES;
CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR;
CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER = YES;
CLANG_WARN_RANGE_LOOP_ANALYSIS = YES;
CLANG_WARN_STRICT_PROTOTYPES = YES;
CLANG_WARN_SUSPICIOUS_MOVE = YES;
CLANG_WARN_UNGUARDED_AVAILABILITY = YES_AGGRESSIVE;
CLANG_WARN_UNREACHABLE_CODE = YES;
CLANG_WARN__DUPLICATE_METHOD_MATCH = YES;
COPY_PHASE_STRIP = NO;
DEBUG_INFORMATION_FORMAT = dwarf;
DEVELOPMENT_TEAM = 6D7X9GGZAW;
ENABLE_STRICT_OBJC_MSGSEND = YES;
ENABLE_TESTABILITY = YES;
ENABLE_USER_SCRIPT_SANDBOXING = YES;
GCC_C_LANGUAGE_STANDARD = gnu17;
GCC_DYNAMIC_NO_PIC = NO;
GCC_NO_COMMON_BLOCKS = YES;
GCC_OPTIMIZATION_LEVEL = 0;
GCC_PREPROCESSOR_DEFINITIONS = (
"DEBUG=1",
"$(inherited)",
);
GCC_WARN_64_TO_32_BIT_CONVERSION = YES;
GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR;
GCC_WARN_UNDECLARED_SELECTOR = YES;
GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE;
GCC_WARN_UNUSED_FUNCTION = YES;
GCC_WARN_UNUSED_VARIABLE = YES;
LOCALIZATION_PREFERS_STRING_CATALOGS = YES;
MACOSX_DEPLOYMENT_TARGET = 14.2;
MTL_ENABLE_DEBUG_INFO = INCLUDE_SOURCE;
MTL_FAST_MATH = YES;
ONLY_ACTIVE_ARCH = YES;
SDKROOT = macosx;
SWIFT_ACTIVE_COMPILATION_CONDITIONS = "DEBUG $(inherited)";
SWIFT_OPTIMIZATION_LEVEL = "-Onone";
};
name = Debug;
};
28F22CDF2F56440300A0FC59 /* Release */ = {
isa = XCBuildConfiguration;
buildSettings = {
ALWAYS_SEARCH_USER_PATHS = NO;
ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS = YES;
CLANG_ANALYZER_NONNULL = YES;
CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE;
CLANG_CXX_LANGUAGE_STANDARD = "gnu++20";
CLANG_ENABLE_MODULES = YES;
CLANG_ENABLE_OBJC_ARC = YES;
CLANG_ENABLE_OBJC_WEAK = YES;
CLANG_WARN_BLOCK_CAPTURE_AUTORELEASING = YES;
CLANG_WARN_BOOL_CONVERSION = YES;
CLANG_WARN_COMMA = YES;
CLANG_WARN_CONSTANT_CONVERSION = YES;
CLANG_WARN_DEPRECATED_OBJC_IMPLEMENTATIONS = YES;
CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR;
CLANG_WARN_DOCUMENTATION_COMMENTS = YES;
CLANG_WARN_EMPTY_BODY = YES;
CLANG_WARN_ENUM_CONVERSION = YES;
CLANG_WARN_INFINITE_RECURSION = YES;
CLANG_WARN_INT_CONVERSION = YES;
CLANG_WARN_NON_LITERAL_NULL_CONVERSION = YES;
CLANG_WARN_OBJC_IMPLICIT_RETAIN_SELF = YES;
CLANG_WARN_OBJC_LITERAL_CONVERSION = YES;
CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR;
CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER = YES;
CLANG_WARN_RANGE_LOOP_ANALYSIS = YES;
CLANG_WARN_STRICT_PROTOTYPES = YES;
CLANG_WARN_SUSPICIOUS_MOVE = YES;
CLANG_WARN_UNGUARDED_AVAILABILITY = YES_AGGRESSIVE;
CLANG_WARN_UNREACHABLE_CODE = YES;
CLANG_WARN__DUPLICATE_METHOD_MATCH = YES;
COPY_PHASE_STRIP = NO;
DEBUG_INFORMATION_FORMAT = "dwarf-with-dsym";
DEVELOPMENT_TEAM = 6D7X9GGZAW;
ENABLE_NS_ASSERTIONS = NO;
ENABLE_STRICT_OBJC_MSGSEND = YES;
ENABLE_USER_SCRIPT_SANDBOXING = YES;
GCC_C_LANGUAGE_STANDARD = gnu17;
GCC_NO_COMMON_BLOCKS = YES;
GCC_WARN_64_TO_32_BIT_CONVERSION = YES;
GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR;
GCC_WARN_UNDECLARED_SELECTOR = YES;
GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE;
GCC_WARN_UNUSED_FUNCTION = YES;
GCC_WARN_UNUSED_VARIABLE = YES;
LOCALIZATION_PREFERS_STRING_CATALOGS = YES;
MACOSX_DEPLOYMENT_TARGET = 14.2;
MTL_ENABLE_DEBUG_INFO = NO;
MTL_FAST_MATH = YES;
SDKROOT = macosx;
SWIFT_COMPILATION_MODE = wholemodule;
};
name = Release;
};
28F22CE12F56440300A0FC59 /* Debug */ = {
isa = XCBuildConfiguration;
buildSettings = {
ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
ASSETCATALOG_COMPILER_GLOBAL_ACCENT_COLOR_NAME = AccentColor;
CODE_SIGN_ENTITLEMENTS = "leanring-buddy/leanring-buddy.entitlements";
CODE_SIGN_STYLE = Automatic;
COMBINE_HIDPI_IMAGES = YES;
CURRENT_PROJECT_VERSION = 1;
DEVELOPMENT_TEAM = 2UDAY4J48G;
ENABLE_APP_SANDBOX = NO;
ENABLE_HARDENED_RUNTIME = YES;
ENABLE_OUTGOING_NETWORK_CONNECTIONS = YES;
ENABLE_PREVIEWS = YES;
ENABLE_USER_SELECTED_FILES = readonly;
GENERATE_INFOPLIST_FILE = YES;
INFOPLIST_FILE = "leanring-buddy/Info.plist";
INFOPLIST_KEY_CFBundleDisplayName = Clicky;
INFOPLIST_KEY_CFBundleName = Clicky;
INFOPLIST_KEY_NSHumanReadableCopyright = "";
INFOPLIST_KEY_NSScreenCaptureUsageDescription = "Clicky needs screen recording access to see your screen and help you.";
LD_RUNPATH_SEARCH_PATHS = (
"$(inherited)",
"@executable_path/../Frameworks",
);
MARKETING_VERSION = 1.0;
PRODUCT_BUNDLE_IDENTIFIER = "com.yourcompany.leanring-buddy";
PRODUCT_NAME = Clicky;
REGISTER_APP_GROUPS = YES;
STRING_CATALOG_GENERATE_SYMBOLS = YES;
SWIFT_APPROACHABLE_CONCURRENCY = YES;
SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor;
SWIFT_EMIT_LOC_STRINGS = YES;
SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES;
SWIFT_VERSION = 5.0;
};
name = Debug;
};
28F22CE22F56440300A0FC59 /* Release */ = {
isa = XCBuildConfiguration;
buildSettings = {
ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
ASSETCATALOG_COMPILER_GLOBAL_ACCENT_COLOR_NAME = AccentColor;
CODE_SIGN_ENTITLEMENTS = "leanring-buddy/leanring-buddy.entitlements";
CODE_SIGN_STYLE = Automatic;
COMBINE_HIDPI_IMAGES = YES;
CURRENT_PROJECT_VERSION = 1;
DEVELOPMENT_TEAM = 2UDAY4J48G;
ENABLE_APP_SANDBOX = NO;
ENABLE_HARDENED_RUNTIME = YES;
ENABLE_OUTGOING_NETWORK_CONNECTIONS = YES;
ENABLE_PREVIEWS = YES;
ENABLE_USER_SELECTED_FILES = readonly;
GENERATE_INFOPLIST_FILE = YES;
INFOPLIST_FILE = "leanring-buddy/Info.plist";
INFOPLIST_KEY_CFBundleDisplayName = Clicky;
INFOPLIST_KEY_CFBundleName = Clicky;
INFOPLIST_KEY_NSHumanReadableCopyright = "";
INFOPLIST_KEY_NSScreenCaptureUsageDescription = "Clicky needs screen recording access to see your screen and help you.";
LD_RUNPATH_SEARCH_PATHS = (
"$(inherited)",
"@executable_path/../Frameworks",
);
MARKETING_VERSION = 1.0;
PRODUCT_BUNDLE_IDENTIFIER = "com.yourcompany.leanring-buddy";
PRODUCT_NAME = Clicky;
REGISTER_APP_GROUPS = YES;
STRING_CATALOG_GENERATE_SYMBOLS = YES;
SWIFT_APPROACHABLE_CONCURRENCY = YES;
SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor;
SWIFT_EMIT_LOC_STRINGS = YES;
SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES;
SWIFT_VERSION = 5.0;
};
name = Release;
};
28F22CE42F56440300A0FC59 /* Debug */ = {
isa = XCBuildConfiguration;
buildSettings = {
BUNDLE_LOADER = "$(TEST_HOST)";
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
DEVELOPMENT_TEAM = 6D7X9GGZAW;
GENERATE_INFOPLIST_FILE = YES;
MACOSX_DEPLOYMENT_TARGET = 14.2;
MARKETING_VERSION = 1.0;
PRODUCT_BUNDLE_IDENTIFIER = "com.yourcompany.leanring-buddyTests";
PRODUCT_NAME = "$(TARGET_NAME)";
STRING_CATALOG_GENERATE_SYMBOLS = NO;
SWIFT_APPROACHABLE_CONCURRENCY = YES;
SWIFT_EMIT_LOC_STRINGS = NO;
SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES;
SWIFT_VERSION = 5.0;
TEST_HOST = "$(BUILT_PRODUCTS_DIR)/leanring-buddy.app/$(BUNDLE_EXECUTABLE_FOLDER_PATH)/leanring-buddy";
};
name = Debug;
};
28F22CE52F56440300A0FC59 /* Release */ = {
isa = XCBuildConfiguration;
buildSettings = {
BUNDLE_LOADER = "$(TEST_HOST)";
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
DEVELOPMENT_TEAM = 6D7X9GGZAW;
GENERATE_INFOPLIST_FILE = YES;
MACOSX_DEPLOYMENT_TARGET = 14.2;
MARKETING_VERSION = 1.0;
PRODUCT_BUNDLE_IDENTIFIER = "com.yourcompany.leanring-buddyTests";
PRODUCT_NAME = "$(TARGET_NAME)";
STRING_CATALOG_GENERATE_SYMBOLS = NO;
SWIFT_APPROACHABLE_CONCURRENCY = YES;
SWIFT_EMIT_LOC_STRINGS = NO;
SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES;
SWIFT_VERSION = 5.0;
TEST_HOST = "$(BUILT_PRODUCTS_DIR)/leanring-buddy.app/$(BUNDLE_EXECUTABLE_FOLDER_PATH)/leanring-buddy";
};
name = Release;
};
28F22CE72F56440300A0FC59 /* Debug */ = {
isa = XCBuildConfiguration;
buildSettings = {
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
DEVELOPMENT_TEAM = 6D7X9GGZAW;
GENERATE_INFOPLIST_FILE = YES;
MARKETING_VERSION = 1.0;
PRODUCT_BUNDLE_IDENTIFIER = "com.yourcompany.leanring-buddyUITests";
PRODUCT_NAME = "$(TARGET_NAME)";
STRING_CATALOG_GENERATE_SYMBOLS = NO;
SWIFT_APPROACHABLE_CONCURRENCY = YES;
SWIFT_EMIT_LOC_STRINGS = NO;
SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES;
SWIFT_VERSION = 5.0;
TEST_TARGET_NAME = "leanring-buddy";
};
name = Debug;
};
28F22CE82F56440300A0FC59 /* Release */ = {
isa = XCBuildConfiguration;
buildSettings = {
CODE_SIGN_STYLE = Automatic;
CURRENT_PROJECT_VERSION = 1;
DEVELOPMENT_TEAM = 6D7X9GGZAW;
GENERATE_INFOPLIST_FILE = YES;
MARKETING_VERSION = 1.0;
PRODUCT_BUNDLE_IDENTIFIER = "com.yourcompany.leanring-buddyUITests";
PRODUCT_NAME = "$(TARGET_NAME)";
STRING_CATALOG_GENERATE_SYMBOLS = NO;
SWIFT_APPROACHABLE_CONCURRENCY = YES;
SWIFT_EMIT_LOC_STRINGS = NO;
SWIFT_UPCOMING_FEATURE_MEMBER_IMPORT_VISIBILITY = YES;
SWIFT_VERSION = 5.0;
TEST_TARGET_NAME = "leanring-buddy";
};
name = Release;
};
/* End XCBuildConfiguration section */
/* Begin XCConfigurationList section */
28F22CBA2F56440300A0FC59 /* Build configuration list for PBXProject "leanring-buddy" */ = {
isa = XCConfigurationList;
buildConfigurations = (
28F22CDE2F56440300A0FC59 /* Debug */,
28F22CDF2F56440300A0FC59 /* Release */,
);
defaultConfigurationIsVisible = 0;
defaultConfigurationName = Release;
};
28F22CE02F56440300A0FC59 /* Build configuration list for PBXNativeTarget "leanring-buddy" */ = {
isa = XCConfigurationList;
buildConfigurations = (
28F22CE12F56440300A0FC59 /* Debug */,
28F22CE22F56440300A0FC59 /* Release */,
);
defaultConfigurationIsVisible = 0;
defaultConfigurationName = Release;
};
28F22CE32F56440300A0FC59 /* Build configuration list for PBXNativeTarget "leanring-buddyTests" */ = {
isa = XCConfigurationList;
buildConfigurations = (
28F22CE42F56440300A0FC59 /* Debug */,
28F22CE52F56440300A0FC59 /* Release */,
);
defaultConfigurationIsVisible = 0;
defaultConfigurationName = Release;
};
28F22CE62F56440300A0FC59 /* Build configuration list for PBXNativeTarget "leanring-buddyUITests" */ = {
isa = XCConfigurationList;
buildConfigurations = (
28F22CE72F56440300A0FC59 /* Debug */,
28F22CE82F56440300A0FC59 /* Release */,
);
defaultConfigurationIsVisible = 0;
defaultConfigurationName = Release;
};
/* End XCConfigurationList section */
/* Begin XCRemoteSwiftPackageReference section */
AA00BB012F6500010039DA55 /* XCRemoteSwiftPackageReference "Sparkle" */ = {
isa = XCRemoteSwiftPackageReference;
repositoryURL = "https://github.com/sparkle-project/Sparkle";
requirement = {
kind = upToNextMajorVersion;
minimumVersion = 2.9.0;
};
};
AA00BB042F6500040039DA55 /* XCRemoteSwiftPackageReference "posthog-ios" */ = {
isa = XCRemoteSwiftPackageReference;
repositoryURL = "https://github.com/PostHog/posthog-ios.git";
requirement = {
kind = upToNextMajorVersion;
minimumVersion = 3.0.0;
};
};
/* End XCRemoteSwiftPackageReference section */
/* Begin XCSwiftPackageProductDependency section */
AA00BB022F6500020039DA55 /* Sparkle */ = {
isa = XCSwiftPackageProductDependency;
package = AA00BB012F6500010039DA55 /* XCRemoteSwiftPackageReference "Sparkle" */;
productName = Sparkle;
};
AA00BB052F6500050039DA55 /* PostHog */ = {
isa = XCSwiftPackageProductDependency;
package = AA00BB042F6500040039DA55 /* XCRemoteSwiftPackageReference "posthog-ios" */;
productName = PostHog;
};
/* End XCSwiftPackageProductDependency section */
};
rootObject = 28F22CB72F56440300A0FC59 /* Project object */;
}
@@ -0,0 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<Workspace
version = "1.0">
<FileRef
location = "self:">
</FileRef>
</Workspace>
@@ -0,0 +1,33 @@
{
"originHash" : "3c6fb67fefedcfcd00708e24ca8088151f21dccfc0ade32ea80c406646277e89",
"pins" : [
{
"identity" : "plcrashreporter",
"kind" : "remoteSourceControl",
"location" : "https://github.com/microsoft/plcrashreporter.git",
"state" : {
"revision" : "0254f941c646b1ed17b243654723d0f071e990d0",
"version" : "1.12.2"
}
},
{
"identity" : "posthog-ios",
"kind" : "remoteSourceControl",
"location" : "https://github.com/PostHog/posthog-ios.git",
"state" : {
"revision" : "09da1be6a614325a6a464c6d2017a9ac858d1b5a",
"version" : "3.47.0"
}
},
{
"identity" : "sparkle",
"kind" : "remoteSourceControl",
"location" : "https://github.com/sparkle-project/Sparkle",
"state" : {
"revision" : "21d8df80440b1ca3b65fa82e40782f1e5a9e6ba2",
"version" : "2.9.0"
}
}
],
"version" : 3
}
@@ -0,0 +1,14 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>SchemeUserState</key>
<dict>
<key>leanring-buddy.xcscheme_^#shared#^_</key>
<dict>
<key>orderHint</key>
<integer>0</integer>
</dict>
</dict>
</dict>
</plist>
@@ -0,0 +1,14 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>SchemeUserState</key>
<dict>
<key>leanring-buddy.xcscheme_^#shared#^_</key>
<dict>
<key>orderHint</key>
<integer>0</integer>
</dict>
</dict>
</dict>
</plist>
+28
View File
@@ -0,0 +1,28 @@
# AGENTS.md - leanring-buddy (Main App Target)
## Source Files
### FloatingSessionButton.swift
- `FloatingSessionButtonManager``@MainActor` class managing the `NSPanel` lifecycle
- `showFloatingButton()` — Creates/shows the panel in top-right of primary screen
- `hideFloatingButton()` — Hides panel (keeps it alive for quick re-show)
- `destroyFloatingButton()` — Removes panel permanently (session ended)
- `onFloatingButtonClicked` — Callback closure, set by ContentView to bring main window to front
- `floatingButtonPanel` — Exposed `NSPanel` reference for screenshot exclusion
- `FloatingButtonView` — Private SwiftUI view with gradient circle, scale+glow hover animation, pointer cursor
### ContentView.swift
- Receives `FloatingSessionButtonManager` via `@EnvironmentObject`
- `isMainWindowCurrentlyFocused` — Tracks main window focus state
- `configureFloatingButtonManager()` — Wires up the click callback
- `startObservingMainWindowFocusChanges()` — Sets up `NSWindow` notification observers
- `updateFloatingButtonVisibility()` — Core logic: show if running + not focused, hide otherwise
- `bringMainWindowToFront()` — Activates app and orders main window front
### ScreenshotManager.swift
- `floatingButtonWindowToExcludeFromCaptures``NSWindow?` reference set by ContentView
- `captureScreen()` — Matches the floating window to an `SCWindow` and excludes it from capture filter
### leanring_buddyApp.swift
- Owns `FloatingSessionButtonManager` as `@StateObject`
- Injects it into ContentView via `.environmentObject()`
@@ -0,0 +1,28 @@
//
// AppBundleConfiguration.swift
// leanring-buddy
//
// Shared helper for reading runtime configuration from the built app bundle.
//
import Foundation
enum AppBundleConfiguration {
static func stringValue(forKey key: String) -> String? {
if let value = Bundle.main.object(forInfoDictionaryKey: key) as? String {
let trimmedValue = value.trimmingCharacters(in: .whitespacesAndNewlines)
if !trimmedValue.isEmpty {
return trimmedValue
}
}
guard let resourceInfoPath = Bundle.main.path(forResource: "Info", ofType: "plist"),
let resourceInfo = NSDictionary(contentsOfFile: resourceInfoPath),
let value = resourceInfo[key] as? String else {
return nil
}
let trimmedValue = value.trimmingCharacters(in: .whitespacesAndNewlines)
return trimmedValue.isEmpty ? nil : trimmedValue
}
}
@@ -0,0 +1,147 @@
//
// AppleSpeechTranscriptionProvider.swift
// leanring-buddy
//
// Local fallback transcription provider backed by Apple's Speech framework.
//
import AVFoundation
import Foundation
import Speech
struct AppleSpeechTranscriptionProviderError: LocalizedError {
let message: String
var errorDescription: String? {
message
}
}
final class AppleSpeechTranscriptionProvider: BuddyTranscriptionProvider {
let displayName = "Apple Speech"
let requiresSpeechRecognitionPermission = true
let isConfigured = true
let unavailableExplanation: String? = nil
func startStreamingSession(
keyterms: [String],
onTranscriptUpdate: @escaping (String) -> Void,
onFinalTranscriptReady: @escaping (String) -> Void,
onError: @escaping (Error) -> Void
) async throws -> any BuddyStreamingTranscriptionSession {
guard let speechRecognizer = Self.makeBestAvailableSpeechRecognizer() else {
throw AppleSpeechTranscriptionProviderError(message: "dictation is not available on this mac.")
}
return try AppleSpeechTranscriptionSession(
speechRecognizer: speechRecognizer,
onTranscriptUpdate: onTranscriptUpdate,
onFinalTranscriptReady: onFinalTranscriptReady,
onError: onError
)
}
private static func makeBestAvailableSpeechRecognizer() -> SFSpeechRecognizer? {
let preferredLocales = [
Locale.autoupdatingCurrent,
Locale(identifier: "en-US")
]
for preferredLocale in preferredLocales {
if let speechRecognizer = SFSpeechRecognizer(locale: preferredLocale) {
return speechRecognizer
}
}
return SFSpeechRecognizer()
}
}
private final class AppleSpeechTranscriptionSession: NSObject, BuddyStreamingTranscriptionSession {
let finalTranscriptFallbackDelaySeconds: TimeInterval = 1.8
private let recognitionRequest: SFSpeechAudioBufferRecognitionRequest
private var recognitionTask: SFSpeechRecognitionTask?
private let onTranscriptUpdate: (String) -> Void
private let onFinalTranscriptReady: (String) -> Void
private let onError: (Error) -> Void
private var latestRecognizedText = ""
private var hasRequestedFinalTranscript = false
private var hasDeliveredFinalTranscript = false
init(
speechRecognizer: SFSpeechRecognizer,
onTranscriptUpdate: @escaping (String) -> Void,
onFinalTranscriptReady: @escaping (String) -> Void,
onError: @escaping (Error) -> Void
) throws {
self.recognitionRequest = SFSpeechAudioBufferRecognitionRequest()
self.onTranscriptUpdate = onTranscriptUpdate
self.onFinalTranscriptReady = onFinalTranscriptReady
self.onError = onError
super.init()
recognitionRequest.shouldReportPartialResults = true
recognitionRequest.taskHint = .dictation
recognitionRequest.addsPunctuation = true
if speechRecognizer.supportsOnDeviceRecognition {
recognitionRequest.requiresOnDeviceRecognition = true
}
recognitionTask = speechRecognizer.recognitionTask(with: recognitionRequest) { [weak self] result, error in
self?.handleRecognitionEvent(result: result, error: error)
}
}
func appendAudioBuffer(_ audioBuffer: AVAudioPCMBuffer) {
guard !hasRequestedFinalTranscript else { return }
recognitionRequest.append(audioBuffer)
}
func requestFinalTranscript() {
guard !hasRequestedFinalTranscript else { return }
hasRequestedFinalTranscript = true
recognitionRequest.endAudio()
}
func cancel() {
recognitionTask?.cancel()
recognitionTask = nil
}
private func handleRecognitionEvent(
result: SFSpeechRecognitionResult?,
error: Error?
) {
if let result {
latestRecognizedText = result.bestTranscription.formattedString
onTranscriptUpdate(latestRecognizedText)
if result.isFinal {
deliverFinalTranscriptIfNeeded(latestRecognizedText)
return
}
}
guard let error else { return }
if hasRequestedFinalTranscript && !latestRecognizedText.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty {
deliverFinalTranscriptIfNeeded(latestRecognizedText)
} else {
onError(error)
}
}
private func deliverFinalTranscriptIfNeeded(_ transcriptText: String) {
guard !hasDeliveredFinalTranscript else { return }
hasDeliveredFinalTranscript = true
onFinalTranscriptReady(transcriptText)
}
deinit {
cancel()
}
}
@@ -0,0 +1,478 @@
//
// AssemblyAIStreamingTranscriptionProvider.swift
// leanring-buddy
//
// Streaming AI transcription provider backed by AssemblyAI's websocket API.
//
import AVFoundation
import Foundation
struct AssemblyAIStreamingTranscriptionProviderError: LocalizedError {
let message: String
var errorDescription: String? {
message
}
}
final class AssemblyAIStreamingTranscriptionProvider: BuddyTranscriptionProvider {
/// URL for the Cloudflare Worker endpoint that returns a short-lived
/// AssemblyAI streaming token. The real API key never leaves the server.
private static let tokenProxyURL = "https://your-worker-name.your-subdomain.workers.dev/transcribe-token"
let displayName = "AssemblyAI"
let requiresSpeechRecognitionPermission = false
var isConfigured: Bool { true }
var unavailableExplanation: String? { nil }
/// Single long-lived URLSession shared across all streaming sessions.
/// Creating and invalidating a URLSession per session corrupts the OS
/// connection pool and causes "Socket is not connected" errors after
/// a few rapid reconnections to the same host.
private let sharedWebSocketURLSession = URLSession(configuration: .default)
func startStreamingSession(
keyterms: [String],
onTranscriptUpdate: @escaping (String) -> Void,
onFinalTranscriptReady: @escaping (String) -> Void,
onError: @escaping (Error) -> Void
) async throws -> any BuddyStreamingTranscriptionSession {
// Fetch a fresh temporary token from the proxy before each session
let temporaryToken = try await fetchTemporaryToken()
print("🎙️ AssemblyAI: fetched temporary token (\(temporaryToken.prefix(20))...)")
let session = AssemblyAIStreamingTranscriptionSession(
apiKey: nil,
temporaryToken: temporaryToken,
urlSession: sharedWebSocketURLSession,
keyterms: keyterms,
onTranscriptUpdate: onTranscriptUpdate,
onFinalTranscriptReady: onFinalTranscriptReady,
onError: onError
)
try await session.open()
return session
}
/// Calls the Cloudflare Worker to get a short-lived AssemblyAI token.
private func fetchTemporaryToken() async throws -> String {
var request = URLRequest(url: URL(string: Self.tokenProxyURL)!)
request.httpMethod = "POST"
let (data, response) = try await URLSession.shared.data(for: request)
guard let httpResponse = response as? HTTPURLResponse,
(200...299).contains(httpResponse.statusCode) else {
let statusCode = (response as? HTTPURLResponse)?.statusCode ?? -1
let body = String(data: data, encoding: .utf8) ?? "unknown"
throw AssemblyAIStreamingTranscriptionProviderError(
message: "Failed to fetch AssemblyAI token (HTTP \(statusCode)): \(body)"
)
}
guard let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let token = json["token"] as? String else {
throw AssemblyAIStreamingTranscriptionProviderError(
message: "Invalid token response from proxy."
)
}
return token
}
}
private final class AssemblyAIStreamingTranscriptionSession: NSObject, BuddyStreamingTranscriptionSession {
private struct MessageEnvelope: Decodable {
let type: String
}
private struct TurnMessage: Decodable {
let type: String
let transcript: String?
let turn_order: Int?
let end_of_turn: Bool?
let turn_is_formatted: Bool?
}
private struct ErrorMessage: Decodable {
let type: String
let error: String?
let message: String?
}
private struct StoredTurnTranscript {
var transcriptText: String
var isFormatted: Bool
}
private static let websocketBaseURLString = "wss://streaming.assemblyai.com/v3/ws"
private static let targetSampleRate = 16_000.0
private static let explicitFinalTranscriptGracePeriodSeconds = 1.4
let finalTranscriptFallbackDelaySeconds: TimeInterval = 2.8
private let apiKey: String?
private let temporaryToken: String?
private let keyterms: [String]
private let onTranscriptUpdate: (String) -> Void
private let onFinalTranscriptReady: (String) -> Void
private let onError: (Error) -> Void
private let stateQueue = DispatchQueue(label: "com.learningbuddy.assemblyai.state")
private let sendQueue = DispatchQueue(label: "com.learningbuddy.assemblyai.send")
private let audioPCM16Converter = BuddyPCM16AudioConverter(targetSampleRate: targetSampleRate)
private let urlSession: URLSession
private var webSocketTask: URLSessionWebSocketTask?
private var readyContinuation: CheckedContinuation<Void, Error>?
private var hasResolvedReadyContinuation = false
private var hasDeliveredFinalTranscript = false
private var isAwaitingExplicitFinalTranscript = false
private var latestTranscriptText = ""
private var activeTurnOrder: Int?
private var activeTurnTranscriptText = ""
private var storedTurnTranscriptsByOrder: [Int: StoredTurnTranscript] = [:]
private var explicitFinalTranscriptDeadlineWorkItem: DispatchWorkItem?
init(
apiKey: String?,
temporaryToken: String?,
urlSession: URLSession,
keyterms: [String],
onTranscriptUpdate: @escaping (String) -> Void,
onFinalTranscriptReady: @escaping (String) -> Void,
onError: @escaping (Error) -> Void
) {
self.apiKey = apiKey
self.temporaryToken = temporaryToken
self.urlSession = urlSession
self.keyterms = keyterms
self.onTranscriptUpdate = onTranscriptUpdate
self.onFinalTranscriptReady = onFinalTranscriptReady
self.onError = onError
}
func open() async throws {
let websocketURL = try Self.makeWebsocketURL(
temporaryToken: temporaryToken,
keyterms: keyterms
)
var websocketRequest = URLRequest(url: websocketURL)
if let apiKey {
websocketRequest.setValue(apiKey, forHTTPHeaderField: "Authorization")
}
let webSocketTask = urlSession.webSocketTask(with: websocketRequest)
self.webSocketTask = webSocketTask
webSocketTask.resume()
receiveNextMessage()
try await withCheckedThrowingContinuation { continuation in
stateQueue.async {
self.readyContinuation = continuation
}
}
}
func appendAudioBuffer(_ audioBuffer: AVAudioPCMBuffer) {
guard let audioPCM16Data = audioPCM16Converter.convertToPCM16Data(from: audioBuffer),
!audioPCM16Data.isEmpty else {
return
}
sendQueue.async { [weak self] in
guard let self, let webSocketTask = self.webSocketTask else { return }
webSocketTask.send(.data(audioPCM16Data)) { [weak self] error in
if let error {
self?.failSession(with: error)
}
}
}
}
func requestFinalTranscript() {
stateQueue.async {
guard !self.hasDeliveredFinalTranscript else { return }
self.isAwaitingExplicitFinalTranscript = true
self.scheduleExplicitFinalTranscriptDeadline()
}
sendJSONMessage(["type": "ForceEndpoint"])
}
func cancel() {
stateQueue.async {
self.explicitFinalTranscriptDeadlineWorkItem?.cancel()
self.explicitFinalTranscriptDeadlineWorkItem = nil
}
sendJSONMessage(["type": "Terminate"])
webSocketTask?.cancel(with: .goingAway, reason: nil)
}
private func receiveNextMessage() {
webSocketTask?.receive { [weak self] result in
guard let self else { return }
switch result {
case .success(let message):
switch message {
case .string(let text):
self.handleIncomingTextMessage(text)
case .data(let data):
if let text = String(data: data, encoding: .utf8) {
self.handleIncomingTextMessage(text)
}
@unknown default:
break
}
self.receiveNextMessage()
case .failure(let error):
self.failSession(with: error)
}
}
}
private func handleIncomingTextMessage(_ text: String) {
guard let messageData = text.data(using: .utf8) else { return }
do {
let envelope = try JSONDecoder().decode(MessageEnvelope.self, from: messageData)
switch envelope.type.lowercased() {
case "begin":
resolveReadyContinuationIfNeeded(with: .success(()))
case "turn":
let turnMessage = try JSONDecoder().decode(TurnMessage.self, from: messageData)
handleTurnMessage(turnMessage)
case "termination":
resolveReadyContinuationIfNeeded(with: .success(()))
stateQueue.async {
if self.isAwaitingExplicitFinalTranscript && !self.hasDeliveredFinalTranscript {
self.deliverFinalTranscriptIfNeeded(self.bestAvailableTranscriptText())
}
}
case "error":
let errorMessage = try JSONDecoder().decode(ErrorMessage.self, from: messageData)
let messageText = errorMessage.error ?? errorMessage.message ?? "AssemblyAI returned an error."
failSession(with: AssemblyAIStreamingTranscriptionProviderError(message: messageText))
default:
break
}
} catch {
failSession(with: error)
}
}
private func handleTurnMessage(_ turnMessage: TurnMessage) {
let transcriptText = turnMessage.transcript?
.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
stateQueue.async {
let turnOrder = turnMessage.turn_order
?? self.activeTurnOrder
?? ((self.storedTurnTranscriptsByOrder.keys.max() ?? -1) + 1)
if turnMessage.end_of_turn == true || turnMessage.turn_is_formatted == true {
self.activeTurnOrder = nil
self.activeTurnTranscriptText = ""
self.storeTurnTranscript(
transcriptText,
forTurnOrder: turnOrder,
isFormatted: turnMessage.turn_is_formatted == true
)
} else {
self.activeTurnOrder = turnOrder
self.activeTurnTranscriptText = transcriptText
}
let fullTranscriptText = self.composeFullTranscript()
self.latestTranscriptText = fullTranscriptText
if !fullTranscriptText.isEmpty {
self.onTranscriptUpdate(fullTranscriptText)
}
guard self.isAwaitingExplicitFinalTranscript else { return }
if turnMessage.end_of_turn == true || turnMessage.turn_is_formatted == true {
self.explicitFinalTranscriptDeadlineWorkItem?.cancel()
self.explicitFinalTranscriptDeadlineWorkItem = nil
self.deliverFinalTranscriptIfNeeded(self.bestAvailableTranscriptText())
}
}
}
private func storeTurnTranscript(
_ transcriptText: String,
forTurnOrder turnOrder: Int,
isFormatted: Bool
) {
guard !transcriptText.isEmpty else { return }
if let existingTurnTranscript = storedTurnTranscriptsByOrder[turnOrder] {
if existingTurnTranscript.isFormatted && !isFormatted {
return
}
}
storedTurnTranscriptsByOrder[turnOrder] = StoredTurnTranscript(
transcriptText: transcriptText,
isFormatted: isFormatted
)
}
private func composeFullTranscript() -> String {
let committedTranscriptSegments = storedTurnTranscriptsByOrder
.sorted(by: { $0.key < $1.key })
.map(\.value.transcriptText)
.filter { !$0.isEmpty }
var transcriptSegments = committedTranscriptSegments
let trimmedActiveTurnTranscriptText = activeTurnTranscriptText
.trimmingCharacters(in: .whitespacesAndNewlines)
if !trimmedActiveTurnTranscriptText.isEmpty {
transcriptSegments.append(trimmedActiveTurnTranscriptText)
}
return transcriptSegments.joined(separator: " ")
}
private func scheduleExplicitFinalTranscriptDeadline() {
explicitFinalTranscriptDeadlineWorkItem?.cancel()
let deadlineWorkItem = DispatchWorkItem { [weak self] in
self?.stateQueue.async {
guard let self else { return }
self.deliverFinalTranscriptIfNeeded(self.bestAvailableTranscriptText())
}
}
explicitFinalTranscriptDeadlineWorkItem = deadlineWorkItem
DispatchQueue.main.asyncAfter(
deadline: .now() + Self.explicitFinalTranscriptGracePeriodSeconds,
execute: deadlineWorkItem
)
}
private func deliverFinalTranscriptIfNeeded(_ transcriptText: String) {
guard !hasDeliveredFinalTranscript else { return }
hasDeliveredFinalTranscript = true
explicitFinalTranscriptDeadlineWorkItem?.cancel()
explicitFinalTranscriptDeadlineWorkItem = nil
onFinalTranscriptReady(transcriptText)
sendJSONMessage(["type": "Terminate"])
}
private func sendJSONMessage(_ payload: [String: Any]) {
guard let jsonData = try? JSONSerialization.data(withJSONObject: payload),
let jsonString = String(data: jsonData, encoding: .utf8) else {
return
}
sendQueue.async { [weak self] in
guard let self, let webSocketTask = self.webSocketTask else { return }
webSocketTask.send(.string(jsonString)) { [weak self] error in
if let error {
self?.failSession(with: error)
}
}
}
}
private func failSession(with error: Error) {
resolveReadyContinuationIfNeeded(with: .failure(error))
stateQueue.async {
let latestTranscriptText = self.bestAvailableTranscriptText()
if self.isAwaitingExplicitFinalTranscript
&& !self.hasDeliveredFinalTranscript
&& !latestTranscriptText.isEmpty {
print("[AssemblyAI] ⚠️ WebSocket error during active session, delivering partial transcript as fallback: \(error.localizedDescription)")
self.deliverFinalTranscriptIfNeeded(latestTranscriptText)
return
}
print("[AssemblyAI] ❌ Session failed with error: \(error.localizedDescription)")
self.onError(error)
}
}
private func bestAvailableTranscriptText() -> String {
let composedTranscriptText = composeFullTranscript()
.trimmingCharacters(in: .whitespacesAndNewlines)
if !composedTranscriptText.isEmpty {
return composedTranscriptText
}
return latestTranscriptText.trimmingCharacters(in: .whitespacesAndNewlines)
}
private func resolveReadyContinuationIfNeeded(with result: Result<Void, Error>) {
stateQueue.async {
guard !self.hasResolvedReadyContinuation else { return }
self.hasResolvedReadyContinuation = true
switch result {
case .success:
self.readyContinuation?.resume()
case .failure(let error):
self.readyContinuation?.resume(throwing: error)
}
self.readyContinuation = nil
}
}
private static func makeWebsocketURL(
temporaryToken: String?,
keyterms: [String]
) throws -> URL {
guard var websocketURLComponents = URLComponents(string: websocketBaseURLString) else {
throw AssemblyAIStreamingTranscriptionProviderError(
message: "AssemblyAI websocket URL is invalid."
)
}
var queryItems = [
URLQueryItem(name: "sample_rate", value: "16000"),
URLQueryItem(name: "encoding", value: "pcm_s16le"),
URLQueryItem(name: "format_turns", value: "true"),
URLQueryItem(name: "speech_model", value: "u3-rt-pro")
]
let normalizedKeyterms = keyterms
.map { $0.trimmingCharacters(in: .whitespacesAndNewlines) }
.filter { !$0.isEmpty }
if !normalizedKeyterms.isEmpty,
let keytermsData = try? JSONSerialization.data(withJSONObject: normalizedKeyterms),
let keytermsJSONString = String(data: keytermsData, encoding: .utf8) {
queryItems.append(URLQueryItem(name: "keyterms_prompt", value: keytermsJSONString))
}
if let temporaryToken {
queryItems.append(URLQueryItem(name: "token", value: temporaryToken))
}
websocketURLComponents.queryItems = queryItems
guard let websocketURL = websocketURLComponents.url else {
throw AssemblyAIStreamingTranscriptionProviderError(
message: "AssemblyAI websocket URL could not be created."
)
}
return websocketURL
}
}
@@ -0,0 +1,11 @@
{
"colors" : [
{
"idiom" : "universal"
}
],
"info" : {
"author" : "xcode",
"version" : 1
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 189 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 8.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 525 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 68 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.8 KiB

@@ -0,0 +1,74 @@
{
"images" : [
{
"filename" : "1024-mac.png",
"idiom" : "ios-marketing",
"scale" : "1x",
"size" : "1024x1024"
},
{
"filename" : "16-mac.png",
"idiom" : "mac",
"scale" : "1x",
"size" : "16x16"
},
{
"filename" : "32-mac.png",
"idiom" : "mac",
"scale" : "2x",
"size" : "16x16"
},
{
"filename" : "32-mac.png",
"idiom" : "mac",
"scale" : "1x",
"size" : "32x32"
},
{
"filename" : "64-mac.png",
"idiom" : "mac",
"scale" : "2x",
"size" : "32x32"
},
{
"filename" : "128-mac.png",
"idiom" : "mac",
"scale" : "1x",
"size" : "128x128"
},
{
"filename" : "256-mac.png",
"idiom" : "mac",
"scale" : "2x",
"size" : "128x128"
},
{
"filename" : "256-mac.png",
"idiom" : "mac",
"scale" : "1x",
"size" : "256x256"
},
{
"filename" : "512-mac.png",
"idiom" : "mac",
"scale" : "2x",
"size" : "256x256"
},
{
"filename" : "512-mac.png",
"idiom" : "mac",
"scale" : "1x",
"size" : "512x512"
},
{
"filename" : "1024-mac.png",
"idiom" : "mac",
"scale" : "2x",
"size" : "512x512"
}
],
"info" : {
"author" : "xcode",
"version" : 1
}
}
@@ -0,0 +1,6 @@
{
"info" : {
"author" : "xcode",
"version" : 1
}
}
@@ -0,0 +1,21 @@
{
"images" : [
{
"filename" : "Image.png",
"idiom" : "universal",
"scale" : "1x"
},
{
"idiom" : "universal",
"scale" : "2x"
},
{
"idiom" : "universal",
"scale" : "3x"
}
],
"info" : {
"author" : "xcode",
"version" : 1
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 39 KiB

@@ -0,0 +1,21 @@
{
"images" : [
{
"filename" : "add-a-project-in-codex.png",
"idiom" : "universal",
"scale" : "1x"
},
{
"idiom" : "universal",
"scale" : "2x"
},
{
"idiom" : "universal",
"scale" : "3x"
}
],
"info" : {
"author" : "xcode",
"version" : 1
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

@@ -0,0 +1,21 @@
{
"images" : [
{
"filename" : "codex-app-screenshot.jpg",
"idiom" : "universal",
"scale" : "1x"
},
{
"idiom" : "universal",
"scale" : "2x"
},
{
"idiom" : "universal",
"scale" : "3x"
}
],
"info" : {
"author" : "xcode",
"version" : 1
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 59 KiB

@@ -0,0 +1,21 @@
{
"images" : [
{
"filename" : "codex-home-screen.png",
"idiom" : "universal",
"scale" : "1x"
},
{
"idiom" : "universal",
"scale" : "2x"
},
{
"idiom" : "universal",
"scale" : "3x"
}
],
"info" : {
"author" : "xcode",
"version" : 1
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 322 KiB

@@ -0,0 +1,21 @@
{
"images" : [
{
"filename" : "codex-permissions.png",
"idiom" : "universal",
"scale" : "1x"
},
{
"idiom" : "universal",
"scale" : "2x"
},
{
"idiom" : "universal",
"scale" : "3x"
}
],
"info" : {
"author" : "xcode",
"version" : 1
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 161 KiB

@@ -0,0 +1,25 @@
{
"images" : [
{
"filename" : "discord-logo.svg",
"idiom" : "universal",
"scale" : "1x"
},
{
"idiom" : "universal",
"scale" : "2x"
},
{
"idiom" : "universal",
"scale" : "3x"
}
],
"info" : {
"author" : "xcode",
"version" : 1
},
"properties" : {
"preserves-vector-representation" : true,
"template-rendering-intent" : "template"
}
}
@@ -0,0 +1,3 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" width="24" height="24" fill="currentColor">
<path d="M20.317 4.37a19.791 19.791 0 0 0-4.885-1.515.074.074 0 0 0-.079.037c-.21.375-.444.864-.608 1.25a18.27 18.27 0 0 0-5.487 0 12.64 12.64 0 0 0-.617-1.25.077.077 0 0 0-.079-.037A19.736 19.736 0 0 0 3.677 4.37a.07.07 0 0 0-.032.027C.533 9.046-.32 13.58.099 18.057a.082.082 0 0 0 .031.057 19.9 19.9 0 0 0 5.993 3.03.078.078 0 0 0 .084-.028c.462-.63.874-1.295 1.226-1.994a.076.076 0 0 0-.041-.106 13.107 13.107 0 0 1-1.872-.892.077.077 0 0 1-.008-.128 10.2 10.2 0 0 0 .372-.292.074.074 0 0 1 .077-.01c3.928 1.793 8.18 1.793 12.062 0a.074.074 0 0 1 .078.01c.12.098.246.198.373.292a.077.077 0 0 1-.006.127 12.299 12.299 0 0 1-1.873.892.077.077 0 0 0-.041.107c.36.698.772 1.362 1.225 1.993a.076.076 0 0 0 .084.028 19.839 19.839 0 0 0 6.002-3.03.077.077 0 0 0 .032-.054c.5-5.177-.838-9.674-3.549-13.66a.061.061 0 0 0-.031-.03zM8.02 15.33c-1.183 0-2.157-1.085-2.157-2.419 0-1.333.956-2.419 2.157-2.419 1.21 0 2.176 1.096 2.157 2.42 0 1.333-.956 2.418-2.157 2.418zm7.975 0c-1.183 0-2.157-1.085-2.157-2.419 0-1.333.955-2.419 2.157-2.419 1.21 0 2.176 1.096 2.157 2.42 0 1.333-.946 2.418-2.157 2.418z"/>
</svg>

After

Width:  |  Height:  |  Size: 1.2 KiB

@@ -0,0 +1,21 @@
{
"images" : [
{
"filename" : "git-tools-prompt.png",
"idiom" : "universal",
"scale" : "1x"
},
{
"idiom" : "universal",
"scale" : "2x"
},
{
"idiom" : "universal",
"scale" : "3x"
}
],
"info" : {
"author" : "xcode",
"version" : 1
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 35 KiB

@@ -0,0 +1,16 @@
{
"images" : [
{
"filename" : "google-logo.svg",
"idiom" : "universal"
}
],
"info" : {
"author" : "xcode",
"version" : 1
},
"properties" : {
"preserves-vector-representation" : true,
"template-rendering-intent" : "original"
}
}
@@ -0,0 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 48 48" width="48" height="48">
<path fill="#EA4335" d="M24 9.5c3.54 0 6.71 1.22 9.21 3.6l6.85-6.85C35.9 2.38 30.47 0 24 0 14.62 0 6.51 5.38 2.56 13.22l7.98 6.19C12.43 13.72 17.74 9.5 24 9.5z"/>
<path fill="#4285F4" d="M46.98 24.55c0-1.57-.15-3.09-.38-4.55H24v9.02h12.94c-.58 2.96-2.26 5.48-4.78 7.18l7.73 6c4.51-4.18 7.09-10.36 7.09-17.65z"/>
<path fill="#FBBC05" d="M10.53 28.59c-.48-1.45-.76-2.99-.76-4.59s.27-3.14.76-4.59l-7.98-6.19C.92 16.46 0 20.12 0 24c0 3.88.92 7.54 2.56 10.78l7.97-6.19z"/>
<path fill="#34A853" d="M24 48c6.48 0 11.93-2.13 15.89-5.81l-7.73-6c-2.15 1.45-4.92 2.3-8.16 2.3-6.26 0-11.57-4.22-13.47-9.91l-7.98 6.19C6.51 42.62 14.62 48 24 48z"/>
<path fill="none" d="M0 0h48v48H0z"/>
</svg>

After

Width:  |  Height:  |  Size: 813 B

@@ -0,0 +1,21 @@
{
"images" : [
{
"filename" : "inside-the-makesomething-project-folder.png",
"idiom" : "universal",
"scale" : "1x"
},
{
"idiom" : "universal",
"scale" : "2x"
},
{
"idiom" : "universal",
"scale" : "3x"
}
],
"info" : {
"author" : "xcode",
"version" : 1
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 541 KiB

@@ -0,0 +1,21 @@
{
"images" : [
{
"filename" : "makesomething-project-folder-in-downloads.png",
"idiom" : "universal",
"scale" : "1x"
},
{
"idiom" : "universal",
"scale" : "2x"
},
{
"idiom" : "universal",
"scale" : "3x"
}
],
"info" : {
"author" : "xcode",
"version" : 1
}
}
@@ -0,0 +1,12 @@
{
"images" : [
{
"filename" : "steve.jpg",
"idiom" : "universal"
}
],
"info" : {
"author" : "xcode",
"version" : 1
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 121 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 121 KiB

@@ -0,0 +1,108 @@
//
// BuddyAudioConversionSupport.swift
// leanring-buddy
//
// Shared audio conversion helpers for voice transcription providers.
//
import AVFoundation
import Foundation
final class BuddyPCM16AudioConverter {
private let targetAudioFormat: AVAudioFormat
private var audioConverter: AVAudioConverter?
private var currentInputFormatDescription: String?
init(targetSampleRate: Double) {
self.targetAudioFormat = AVAudioFormat(
commonFormat: .pcmFormatInt16,
sampleRate: targetSampleRate,
channels: 1,
interleaved: true
)!
}
func convertToPCM16Data(from audioBuffer: AVAudioPCMBuffer) -> Data? {
let inputFormatDescription = audioBuffer.format.settings.description
if currentInputFormatDescription != inputFormatDescription {
audioConverter = AVAudioConverter(from: audioBuffer.format, to: targetAudioFormat)
currentInputFormatDescription = inputFormatDescription
}
guard let audioConverter else { return nil }
let sampleRateRatio = targetAudioFormat.sampleRate / audioBuffer.format.sampleRate
let outputFrameCapacity = AVAudioFrameCount(
(Double(audioBuffer.frameLength) * sampleRateRatio).rounded(.up) + 32
)
guard let outputBuffer = AVAudioPCMBuffer(
pcmFormat: targetAudioFormat,
frameCapacity: outputFrameCapacity
) else {
return nil
}
var hasProvidedSourceBuffer = false
var conversionError: NSError?
let conversionStatus = audioConverter.convert(to: outputBuffer, error: &conversionError) { _, outStatus in
if hasProvidedSourceBuffer {
outStatus.pointee = .noDataNow
return nil
}
hasProvidedSourceBuffer = true
outStatus.pointee = .haveData
return audioBuffer
}
guard conversionStatus != .error else { return nil }
guard let pcmDataPointer = outputBuffer.audioBufferList.pointee.mBuffers.mData else { return nil }
let bytesPerFrame = Int(targetAudioFormat.streamDescription.pointee.mBytesPerFrame)
let byteCount = Int(outputBuffer.frameLength) * bytesPerFrame
guard byteCount > 0 else { return nil }
return Data(bytes: pcmDataPointer, count: byteCount)
}
}
enum BuddyWAVFileBuilder {
static func buildWAVData(
fromPCM16MonoAudio pcm16AudioData: Data,
sampleRate: Int,
channelCount: Int = 1,
bitsPerSample: Int = 16
) -> Data {
let byteRate = sampleRate * channelCount * bitsPerSample / 8
let blockAlign = channelCount * bitsPerSample / 8
let dataChunkSize = UInt32(pcm16AudioData.count)
let fileSize = UInt32(36) + dataChunkSize
var wavData = Data()
wavData.append("RIFF".data(using: .ascii)!)
wavData.append(littleEndianData(from: fileSize))
wavData.append("WAVE".data(using: .ascii)!)
wavData.append("fmt ".data(using: .ascii)!)
wavData.append(littleEndianData(from: UInt32(16)))
wavData.append(littleEndianData(from: UInt16(1)))
wavData.append(littleEndianData(from: UInt16(channelCount)))
wavData.append(littleEndianData(from: UInt32(sampleRate)))
wavData.append(littleEndianData(from: UInt32(byteRate)))
wavData.append(littleEndianData(from: UInt16(blockAlign)))
wavData.append(littleEndianData(from: UInt16(bitsPerSample)))
wavData.append("data".data(using: .ascii)!)
wavData.append(littleEndianData(from: dataChunkSize))
wavData.append(pcm16AudioData)
return wavData
}
private static func littleEndianData<T: FixedWidthInteger>(from value: T) -> Data {
var littleEndianValue = value.littleEndian
return Data(bytes: &littleEndianValue, count: MemoryLayout<T>.size)
}
}
+866
View File
@@ -0,0 +1,866 @@
//
// BuddyDictationManager.swift
// leanring-buddy
//
// Shared push-to-talk dictation manager for the help chat and brainstorm buddy.
// Captures microphone audio with AVAudioEngine, routes it into the active
// transcription provider, and hands the final draft back to the active input bar.
//
import AppKit
import AVFoundation
import Combine
import Foundation
import Speech
enum BuddyPushToTalkShortcut {
enum ShortcutOption {
case shiftFunction
case controlOption
case shiftControl
case controlOptionSpace
case shiftControlSpace
var displayText: String {
switch self {
case .shiftFunction:
return "shift + fn"
case .controlOption:
return "ctrl + option"
case .shiftControl:
return "shift + control"
case .controlOptionSpace:
return "ctrl + option + space"
case .shiftControlSpace:
return "shift + control + space"
}
}
var keyCapsuleLabels: [String] {
switch self {
case .shiftFunction:
return ["shift", "fn"]
case .controlOption:
return ["ctrl", "option"]
case .shiftControl:
return ["shift", "control"]
case .controlOptionSpace:
return ["ctrl", "option", "space"]
case .shiftControlSpace:
return ["shift", "control", "space"]
}
}
fileprivate var modifierOnlyFlags: NSEvent.ModifierFlags? {
switch self {
case .shiftFunction:
return [.shift, .function]
case .controlOption:
return [.control, .option]
case .shiftControl:
return [.shift, .control]
case .controlOptionSpace, .shiftControlSpace:
return nil
}
}
fileprivate var spaceShortcutModifierFlags: NSEvent.ModifierFlags? {
switch self {
case .shiftFunction:
return nil
case .controlOption:
return nil
case .shiftControl:
return nil
case .controlOptionSpace:
return [.control, .option]
case .shiftControlSpace:
return [.shift, .control]
}
}
}
enum ShortcutTransition {
case none
case pressed
case released
}
private enum ShortcutEventType {
case flagsChanged
case keyDown
case keyUp
}
static let currentShortcutOption: ShortcutOption = .controlOption
static let pushToTalkKeyCode: UInt16 = 49 // Space
static let pushToTalkDisplayText = currentShortcutOption.displayText
static let pushToTalkTooltipText = "push to talk (\(pushToTalkDisplayText))"
static func shortcutTransition(
for event: NSEvent,
wasShortcutPreviouslyPressed: Bool
) -> ShortcutTransition {
guard let shortcutEventType = shortcutEventType(for: event.type) else { return .none }
return shortcutTransition(
for: shortcutEventType,
keyCode: event.keyCode,
modifierFlags: event.modifierFlags.intersection(.deviceIndependentFlagsMask),
wasShortcutPreviouslyPressed: wasShortcutPreviouslyPressed
)
}
static func shortcutTransition(
for eventType: CGEventType,
keyCode: UInt16,
modifierFlagsRawValue: UInt64,
wasShortcutPreviouslyPressed: Bool
) -> ShortcutTransition {
guard let shortcutEventType = shortcutEventType(for: eventType) else { return .none }
return shortcutTransition(
for: shortcutEventType,
keyCode: keyCode,
modifierFlags: NSEvent.ModifierFlags(rawValue: UInt(modifierFlagsRawValue))
.intersection(.deviceIndependentFlagsMask),
wasShortcutPreviouslyPressed: wasShortcutPreviouslyPressed
)
}
private static func shortcutEventType(for eventType: NSEvent.EventType) -> ShortcutEventType? {
switch eventType {
case .flagsChanged:
return .flagsChanged
case .keyDown:
return .keyDown
case .keyUp:
return .keyUp
default:
return nil
}
}
private static func shortcutEventType(for eventType: CGEventType) -> ShortcutEventType? {
switch eventType {
case .flagsChanged:
return .flagsChanged
case .keyDown:
return .keyDown
case .keyUp:
return .keyUp
default:
return nil
}
}
private static func shortcutTransition(
for shortcutEventType: ShortcutEventType,
keyCode: UInt16,
modifierFlags: NSEvent.ModifierFlags,
wasShortcutPreviouslyPressed: Bool
) -> ShortcutTransition {
if let modifierOnlyFlags = currentShortcutOption.modifierOnlyFlags {
guard shortcutEventType == .flagsChanged else { return .none }
let isShortcutCurrentlyPressed = modifierFlags.contains(modifierOnlyFlags)
if isShortcutCurrentlyPressed && !wasShortcutPreviouslyPressed {
return .pressed
}
if !isShortcutCurrentlyPressed && wasShortcutPreviouslyPressed {
return .released
}
return .none
}
guard let pushToTalkModifierFlags = currentShortcutOption.spaceShortcutModifierFlags else {
return .none
}
let matchesModifierFlags = modifierFlags.isSuperset(of: pushToTalkModifierFlags)
if shortcutEventType == .keyDown
&& keyCode == pushToTalkKeyCode
&& matchesModifierFlags
&& !wasShortcutPreviouslyPressed {
return .pressed
}
if shortcutEventType == .keyUp
&& keyCode == pushToTalkKeyCode
&& wasShortcutPreviouslyPressed {
return .released
}
return .none
}
}
enum BuddyDictationPermissionProblem {
case microphoneAccessDenied
case speechRecognitionDenied
}
private enum BuddyDictationStartSource {
case microphoneButton
case keyboardShortcut
}
private struct BuddyDictationDraftCallbacks {
let updateDraftText: (String) -> Void
let submitDraftText: (String) -> Void
}
@MainActor
final class BuddyDictationManager: NSObject, ObservableObject {
private static let defaultFinalTranscriptFallbackDelaySeconds: TimeInterval = 2.4
private static let recordedAudioPowerHistoryLength = 44
private static let recordedAudioPowerHistoryBaselineLevel: CGFloat = 0.02
private static let recordedAudioPowerHistorySampleIntervalSeconds: TimeInterval = 0.07
@Published private(set) var isRecordingFromMicrophoneButton = false
@Published private(set) var isRecordingFromKeyboardShortcut = false
@Published private(set) var isKeyboardShortcutSessionActiveOrFinalizing = false
@Published private(set) var isFinalizingTranscript = false
@Published private(set) var isPreparingToRecord = false
@Published private(set) var currentAudioPowerLevel: CGFloat = 0
@Published private(set) var recordedAudioPowerHistory = Array(
repeating: BuddyDictationManager.recordedAudioPowerHistoryBaselineLevel,
count: BuddyDictationManager.recordedAudioPowerHistoryLength
)
@Published private(set) var microphoneButtonRecordingStartedAt: Date?
@Published private(set) var transcriptionProviderDisplayName = ""
@Published var lastErrorMessage: String?
@Published private(set) var currentPermissionProblem: BuddyDictationPermissionProblem?
var isDictationInProgress: Bool {
isPreparingToRecord || isRecordingFromMicrophoneButton || isRecordingFromKeyboardShortcut || isFinalizingTranscript
}
var isActivelyRecordingAudio: Bool {
isRecordingFromMicrophoneButton || isRecordingFromKeyboardShortcut
}
var isMicrophoneButtonActivelyRecordingAudio: Bool {
isRecordingFromMicrophoneButton
}
var isMicrophoneButtonSessionBusy: Bool {
activeStartSource == .microphoneButton
&& (isPreparingToRecord || isRecordingFromMicrophoneButton || isFinalizingTranscript)
}
var needsInitialPermissionPrompt: Bool {
if transcriptionProvider.requiresSpeechRecognitionPermission {
return AVCaptureDevice.authorizationStatus(for: .audio) == .notDetermined
|| SFSpeechRecognizer.authorizationStatus() == .notDetermined
}
return AVCaptureDevice.authorizationStatus(for: .audio) == .notDetermined
}
private let transcriptionProvider: any BuddyTranscriptionProvider
private let audioEngine = AVAudioEngine()
private var activeTranscriptionSession: (any BuddyStreamingTranscriptionSession)?
private var activeStartSource: BuddyDictationStartSource?
private var draftCallbacks: BuddyDictationDraftCallbacks?
private var draftTextBeforeCurrentDictation = ""
private var latestRecognizedText = ""
private var shouldAutomaticallySubmitFinalDraft = false
private var hasFinishedCurrentDictationSession = false
private var finalizeFallbackWorkItem: DispatchWorkItem?
private var pendingStartRequestIdentifier = UUID()
private var contextualKeyterms: [String] = []
private var lastRecordedAudioPowerSampleDate = Date.distantPast
private var activePermissionRequestTask: Task<Bool, Never>?
/// Timestamp of the last completed permission request, used to debounce
/// rapid follow-up requests that arrive before macOS updates its cache.
private var lastPermissionRequestCompletedAt: Date?
override init() {
let transcriptionProvider = BuddyTranscriptionProviderFactory.makeDefaultProvider()
self.transcriptionProvider = transcriptionProvider
self.transcriptionProviderDisplayName = transcriptionProvider.displayName
super.init()
}
func updateContextualKeyterms(_ contextualKeyterms: [String]) {
self.contextualKeyterms = contextualKeyterms
}
func startPersistentDictationFromMicrophoneButton(
currentDraftText: String,
updateDraftText: @escaping (String) -> Void,
submitDraftText: @escaping (String) -> Void
) async {
await startPushToTalk(
startSource: .microphoneButton,
currentDraftText: currentDraftText,
updateDraftText: updateDraftText,
submitDraftText: submitDraftText,
shouldAutomaticallySubmitFinalDraftOnStop: false
)
}
func startPushToTalkFromKeyboardShortcut(
currentDraftText: String,
updateDraftText: @escaping (String) -> Void,
submitDraftText: @escaping (String) -> Void
) async {
await startPushToTalk(
startSource: .keyboardShortcut,
currentDraftText: currentDraftText,
updateDraftText: updateDraftText,
submitDraftText: submitDraftText,
shouldAutomaticallySubmitFinalDraftOnStop: currentDraftText
.trimmingCharacters(in: .whitespacesAndNewlines)
.isEmpty
)
}
func stopPersistentDictationFromMicrophoneButton() {
stopPushToTalk(expectedStartSource: .microphoneButton)
}
func stopPushToTalkFromKeyboardShortcut() {
stopPushToTalk(expectedStartSource: .keyboardShortcut)
}
func cancelCurrentDictation(preserveDraftText: Bool = true) {
pendingStartRequestIdentifier = UUID()
guard isDictationInProgress else { return }
finalizeFallbackWorkItem?.cancel()
finalizeFallbackWorkItem = nil
if preserveDraftText {
let currentDraftText = composeDraftText(withTranscribedText: latestRecognizedText)
draftCallbacks?.updateDraftText(currentDraftText)
}
audioEngine.stop()
audioEngine.inputNode.removeTap(onBus: 0)
activeTranscriptionSession?.cancel()
resetSessionState()
}
func requestInitialPushToTalkPermissionsIfNeeded() async {
guard needsInitialPermissionPrompt else { return }
guard !isDictationInProgress else { return }
lastErrorMessage = nil
currentPermissionProblem = nil
isPreparingToRecord = true
NSApplication.shared.activate(ignoringOtherApps: true)
do {
try await Task.sleep(for: .milliseconds(200))
} catch {
// If the task is cancelled while we are waiting for macOS to bring
// the app forward, we can safely continue into the permission check.
}
let hasPermissions = await requestMicrophoneAndSpeechPermissionsWithoutDuplicatePrompts()
isPreparingToRecord = false
if hasPermissions {
lastErrorMessage = nil
}
}
private func startPushToTalk(
startSource: BuddyDictationStartSource,
currentDraftText: String,
updateDraftText: @escaping (String) -> Void,
submitDraftText: @escaping (String) -> Void,
shouldAutomaticallySubmitFinalDraftOnStop: Bool
) async {
guard !isDictationInProgress else { return }
print("🎙️ BuddyDictationManager: start requested (\(startSource))")
if needsInitialPermissionPrompt {
print("🎙️ BuddyDictationManager: requesting initial permissions")
NSApplication.shared.activate(ignoringOtherApps: true)
do {
try await Task.sleep(for: .milliseconds(200))
} catch {
// If the task is cancelled while the app is being activated,
// we can safely continue into the permission request.
}
}
let startRequestIdentifier = UUID()
pendingStartRequestIdentifier = startRequestIdentifier
lastErrorMessage = nil
currentPermissionProblem = nil
isPreparingToRecord = true
guard await requestMicrophoneAndSpeechPermissionsWithoutDuplicatePrompts() else {
print("🎙️ BuddyDictationManager: permissions missing or denied")
isPreparingToRecord = false
return
}
guard !Task.isCancelled else {
print("🎙️ BuddyDictationManager: start cancelled (shortcut released during permission check)")
isPreparingToRecord = false
return
}
guard pendingStartRequestIdentifier == startRequestIdentifier else {
print("🎙️ BuddyDictationManager: start request superseded")
isPreparingToRecord = false
return
}
draftTextBeforeCurrentDictation = currentDraftText
latestRecognizedText = ""
draftCallbacks = BuddyDictationDraftCallbacks(
updateDraftText: updateDraftText,
submitDraftText: submitDraftText
)
activeStartSource = startSource
shouldAutomaticallySubmitFinalDraft = shouldAutomaticallySubmitFinalDraftOnStop
hasFinishedCurrentDictationSession = false
isFinalizingTranscript = false
isRecordingFromMicrophoneButton = startSource == .microphoneButton
isRecordingFromKeyboardShortcut = startSource == .keyboardShortcut
isKeyboardShortcutSessionActiveOrFinalizing = startSource == .keyboardShortcut
currentAudioPowerLevel = 0
recordedAudioPowerHistory = Array(
repeating: Self.recordedAudioPowerHistoryBaselineLevel,
count: Self.recordedAudioPowerHistoryLength
)
microphoneButtonRecordingStartedAt = nil
lastRecordedAudioPowerSampleDate = .distantPast
guard !Task.isCancelled else {
print("🎙️ BuddyDictationManager: start cancelled (shortcut released before recording began)")
resetSessionState()
return
}
do {
try await startRecognitionSession()
guard !Task.isCancelled else {
print("🎙️ BuddyDictationManager: start cancelled (shortcut released during session start)")
audioEngine.stop()
audioEngine.inputNode.removeTap(onBus: 0)
activeTranscriptionSession?.cancel()
resetSessionState()
return
}
if startSource == .microphoneButton {
microphoneButtonRecordingStartedAt = Date()
}
isPreparingToRecord = false
print("🎙️ BuddyDictationManager: recognition session started")
} catch {
isPreparingToRecord = false
lastErrorMessage = userFacingErrorMessage(
from: error,
fallback: "couldn't start voice input. try again."
)
print("❌ BuddyDictationManager: failed to start recognition session (\(transcriptionProvider.displayName)): \(error)")
resetSessionState()
}
}
private func stopPushToTalk(expectedStartSource: BuddyDictationStartSource) {
pendingStartRequestIdentifier = UUID()
guard activeStartSource == expectedStartSource else {
isPreparingToRecord = false
return
}
guard !isFinalizingTranscript else { return }
print("🎙️ BuddyDictationManager: stop requested (\(expectedStartSource))")
isRecordingFromMicrophoneButton = false
isRecordingFromKeyboardShortcut = false
isFinalizingTranscript = true
let finalTranscriptFallbackDelaySeconds = activeTranscriptionSession?.finalTranscriptFallbackDelaySeconds
?? Self.defaultFinalTranscriptFallbackDelaySeconds
audioEngine.stop()
audioEngine.inputNode.removeTap(onBus: 0)
activeTranscriptionSession?.requestFinalTranscript()
finalizeFallbackWorkItem?.cancel()
let shouldSubmitFinalDraftWhenFallbackTriggers = shouldAutomaticallySubmitFinalDraft
let fallbackWorkItem = DispatchWorkItem { [weak self] in
Task { @MainActor in
self?.finishCurrentDictationSessionIfNeeded(
shouldSubmitFinalDraft: shouldSubmitFinalDraftWhenFallbackTriggers
)
}
}
finalizeFallbackWorkItem = fallbackWorkItem
DispatchQueue.main.asyncAfter(
deadline: .now() + finalTranscriptFallbackDelaySeconds,
execute: fallbackWorkItem
)
}
private func startRecognitionSession() async throws {
activeTranscriptionSession?.cancel()
activeTranscriptionSession = nil
print("🎙️ BuddyDictationManager: opening transcription provider \(transcriptionProvider.displayName)")
let activeTranscriptionSession = try await transcriptionProvider.startStreamingSession(
keyterms: buildTranscriptionKeyterms(),
onTranscriptUpdate: { [weak self] transcriptText in
Task { @MainActor in
self?.latestRecognizedText = transcriptText
}
},
onFinalTranscriptReady: { [weak self] transcriptText in
Task { @MainActor in
guard let self else { return }
self.latestRecognizedText = transcriptText
if self.isFinalizingTranscript {
self.finishCurrentDictationSessionIfNeeded(
shouldSubmitFinalDraft: self.shouldAutomaticallySubmitFinalDraft
)
}
}
},
onError: { [weak self] error in
Task { @MainActor in
self?.handleRecognitionError(error)
}
}
)
self.activeTranscriptionSession = activeTranscriptionSession
print("🎙️ BuddyDictationManager: provider ready, starting audio engine")
let inputNode = audioEngine.inputNode
let inputFormat = inputNode.outputFormat(forBus: 0)
inputNode.removeTap(onBus: 0)
inputNode.installTap(onBus: 0, bufferSize: 1024, format: inputFormat) { [weak self] buffer, _ in
self?.activeTranscriptionSession?.appendAudioBuffer(buffer)
self?.updateAudioPowerLevel(from: buffer)
}
audioEngine.prepare()
try audioEngine.start()
}
private func handleRecognitionError(_ error: Error) {
if hasFinishedCurrentDictationSession {
return
}
if isFinalizingTranscript && !latestRecognizedText.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty {
finishCurrentDictationSessionIfNeeded(
shouldSubmitFinalDraft: shouldAutomaticallySubmitFinalDraft
)
} else {
print("❌ Buddy dictation error (\(transcriptionProvider.displayName)): \(error)")
lastErrorMessage = userFacingErrorMessage(
from: error,
fallback: "couldn't transcribe that. try again."
)
cancelCurrentDictation(preserveDraftText: false)
}
}
private func finishCurrentDictationSessionIfNeeded(shouldSubmitFinalDraft: Bool) {
guard !hasFinishedCurrentDictationSession else { return }
hasFinishedCurrentDictationSession = true
finalizeFallbackWorkItem?.cancel()
finalizeFallbackWorkItem = nil
let finalDraftText = composeDraftText(withTranscribedText: latestRecognizedText)
let finalTranscriptText = latestRecognizedText.trimmingCharacters(in: .whitespacesAndNewlines)
let currentDraftCallbacks = draftCallbacks
if !shouldSubmitFinalDraft && !finalDraftText.isEmpty {
currentDraftCallbacks?.updateDraftText(finalDraftText)
}
audioEngine.stop()
audioEngine.inputNode.removeTap(onBus: 0)
activeTranscriptionSession?.cancel()
resetSessionState()
guard shouldSubmitFinalDraft else { return }
guard !finalTranscriptText.isEmpty else { return }
currentDraftCallbacks?.submitDraftText(finalDraftText)
}
private func composeDraftText(withTranscribedText transcribedText: String) -> String {
let trimmedTranscriptText = transcribedText.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmedTranscriptText.isEmpty else {
return draftTextBeforeCurrentDictation
}
let trimmedExistingDraftText = draftTextBeforeCurrentDictation
.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmedExistingDraftText.isEmpty else {
return trimmedTranscriptText
}
if draftTextBeforeCurrentDictation.hasSuffix(" ") || draftTextBeforeCurrentDictation.hasSuffix("\n") {
return draftTextBeforeCurrentDictation + trimmedTranscriptText
}
return draftTextBeforeCurrentDictation + " " + trimmedTranscriptText
}
private func resetSessionState() {
pendingStartRequestIdentifier = UUID()
activeTranscriptionSession = nil
draftCallbacks = nil
activeStartSource = nil
draftTextBeforeCurrentDictation = ""
latestRecognizedText = ""
shouldAutomaticallySubmitFinalDraft = false
hasFinishedCurrentDictationSession = false
isPreparingToRecord = false
isRecordingFromMicrophoneButton = false
isRecordingFromKeyboardShortcut = false
isKeyboardShortcutSessionActiveOrFinalizing = false
isFinalizingTranscript = false
currentAudioPowerLevel = 0
recordedAudioPowerHistory = Array(
repeating: Self.recordedAudioPowerHistoryBaselineLevel,
count: Self.recordedAudioPowerHistoryLength
)
microphoneButtonRecordingStartedAt = nil
lastRecordedAudioPowerSampleDate = .distantPast
}
private func buildTranscriptionKeyterms() -> [String] {
let baseKeyterms = [
"makesomething",
"Learning Buddy",
"Codex",
"Claude",
"Anthropic",
"OpenAI",
"SwiftUI",
"Xcode",
"Vercel",
"Next.js",
"localhost"
]
let combinedKeyterms = baseKeyterms + contextualKeyterms
var uniqueNormalizedKeyterms = Set<String>()
var orderedKeyterms: [String] = []
for keyterm in combinedKeyterms {
let trimmedKeyterm = keyterm.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmedKeyterm.isEmpty else { continue }
let normalizedKeyterm = trimmedKeyterm.lowercased()
if uniqueNormalizedKeyterms.contains(normalizedKeyterm) {
continue
}
uniqueNormalizedKeyterms.insert(normalizedKeyterm)
orderedKeyterms.append(trimmedKeyterm)
}
return orderedKeyterms
}
private func updateAudioPowerLevel(from audioBuffer: AVAudioPCMBuffer) {
guard let channelData = audioBuffer.floatChannelData else { return }
let channelSamples = channelData[0]
let frameCount = Int(audioBuffer.frameLength)
guard frameCount > 0 else { return }
var summedSquares: Float = 0
for sampleIndex in 0..<frameCount {
let sample = channelSamples[sampleIndex]
summedSquares += sample * sample
}
let rootMeanSquare = sqrt(summedSquares / Float(frameCount))
let boostedLevel = min(max(rootMeanSquare * 10.2, 0), 1)
DispatchQueue.main.async { [weak self] in
guard let self else { return }
let smoothedAudioPowerLevel = max(
CGFloat(boostedLevel),
self.currentAudioPowerLevel * 0.72
)
self.currentAudioPowerLevel = smoothedAudioPowerLevel
let now = Date()
if now.timeIntervalSince(self.lastRecordedAudioPowerSampleDate)
>= Self.recordedAudioPowerHistorySampleIntervalSeconds {
self.lastRecordedAudioPowerSampleDate = now
self.appendRecordedAudioPowerSample(
max(CGFloat(boostedLevel), Self.recordedAudioPowerHistoryBaselineLevel)
)
}
}
}
private func appendRecordedAudioPowerSample(_ audioPowerSample: CGFloat) {
var updatedRecordedAudioPowerHistory = recordedAudioPowerHistory
updatedRecordedAudioPowerHistory.append(audioPowerSample)
if updatedRecordedAudioPowerHistory.count > Self.recordedAudioPowerHistoryLength {
updatedRecordedAudioPowerHistory.removeFirst(
updatedRecordedAudioPowerHistory.count - Self.recordedAudioPowerHistoryLength
)
}
recordedAudioPowerHistory = updatedRecordedAudioPowerHistory
}
private func requestMicrophoneAndSpeechPermissionsIfNeeded() async -> Bool {
let hasMicrophonePermission = await requestMicrophonePermissionIfNeeded()
guard hasMicrophonePermission else {
lastErrorMessage = "microphone permission is required for push to talk."
return false
}
guard transcriptionProvider.requiresSpeechRecognitionPermission else {
return true
}
let hasSpeechRecognitionPermission = await requestSpeechRecognitionPermissionIfNeeded()
guard hasSpeechRecognitionPermission else {
lastErrorMessage = "speech recognition permission is required for push to talk."
return false
}
return true
}
/// macOS can show the microphone/speech sheet again if we accidentally fan out
/// multiple permission requests before the first one finishes. We keep exactly
/// one in-flight request task so rapid repeat presses all await the same result.
///
/// After the task completes, we skip re-requesting for a short cooldown period
/// so macOS has time to update its authorization cache. This prevents the
/// permission dialog from popping up again on rapid follow-up presses.
private func requestMicrophoneAndSpeechPermissionsWithoutDuplicatePrompts() async -> Bool {
// If a permission request is already in-flight, reuse it.
if let activePermissionRequestTask {
return await activePermissionRequestTask.value
}
// If we just finished a permission request very recently, skip re-requesting.
// macOS can briefly report .notDetermined even after the user tapped Allow,
// so we trust the cached result for a short window.
if let lastPermissionRequestCompletedAt,
Date().timeIntervalSince(lastPermissionRequestCompletedAt) < 1.0 {
return AVCaptureDevice.authorizationStatus(for: .audio) != .denied
&& AVCaptureDevice.authorizationStatus(for: .audio) != .restricted
}
let permissionRequestTask = Task { @MainActor in
await self.requestMicrophoneAndSpeechPermissionsIfNeeded()
}
activePermissionRequestTask = permissionRequestTask
let hasPermissions = await permissionRequestTask.value
activePermissionRequestTask = nil
lastPermissionRequestCompletedAt = Date()
return hasPermissions
}
private func requestMicrophonePermissionIfNeeded() async -> Bool {
switch AVCaptureDevice.authorizationStatus(for: .audio) {
case .authorized:
currentPermissionProblem = nil
return true
case .notDetermined:
let isGranted = await withCheckedContinuation { continuation in
AVCaptureDevice.requestAccess(for: .audio) { isGranted in
continuation.resume(returning: isGranted)
}
}
currentPermissionProblem = isGranted ? nil : .microphoneAccessDenied
return isGranted
case .denied, .restricted:
currentPermissionProblem = .microphoneAccessDenied
return false
@unknown default:
currentPermissionProblem = .microphoneAccessDenied
return false
}
}
private func requestSpeechRecognitionPermissionIfNeeded() async -> Bool {
switch SFSpeechRecognizer.authorizationStatus() {
case .authorized:
currentPermissionProblem = nil
return true
case .notDetermined:
let isGranted = await withCheckedContinuation { continuation in
SFSpeechRecognizer.requestAuthorization { authorizationStatus in
continuation.resume(returning: authorizationStatus == .authorized)
}
}
currentPermissionProblem = isGranted ? nil : .speechRecognitionDenied
return isGranted
case .denied, .restricted:
currentPermissionProblem = .speechRecognitionDenied
return false
@unknown default:
currentPermissionProblem = .speechRecognitionDenied
return false
}
}
func openRelevantPrivacySettings() {
let settingsURLString: String
switch currentPermissionProblem {
case .microphoneAccessDenied:
settingsURLString = "x-apple.systempreferences:com.apple.preference.security?Privacy_Microphone"
case .speechRecognitionDenied:
settingsURLString = "x-apple.systempreferences:com.apple.preference.security?Privacy_SpeechRecognition"
case nil:
settingsURLString = "x-apple.systempreferences:com.apple.preference.security"
}
guard let settingsURL = URL(string: settingsURLString) else { return }
NSWorkspace.shared.open(settingsURL)
}
private func userFacingErrorMessage(from error: Error, fallback: String) -> String {
if let localizedError = error as? LocalizedError,
let errorDescription = localizedError.errorDescription?
.trimmingCharacters(in: .whitespacesAndNewlines),
!errorDescription.isEmpty {
return errorDescription
}
let errorDescription = error.localizedDescription.trimmingCharacters(in: .whitespacesAndNewlines)
if !errorDescription.isEmpty,
errorDescription != "The operation couldnt be completed." {
return errorDescription
}
return fallback
}
}
@@ -0,0 +1,100 @@
//
// BuddyTranscriptionProvider.swift
// leanring-buddy
//
// Shared protocol surface for voice transcription backends.
//
import AVFoundation
import Foundation
protocol BuddyStreamingTranscriptionSession: AnyObject {
var finalTranscriptFallbackDelaySeconds: TimeInterval { get }
func appendAudioBuffer(_ audioBuffer: AVAudioPCMBuffer)
func requestFinalTranscript()
func cancel()
}
protocol BuddyTranscriptionProvider {
var displayName: String { get }
var requiresSpeechRecognitionPermission: Bool { get }
var isConfigured: Bool { get }
var unavailableExplanation: String? { get }
func startStreamingSession(
keyterms: [String],
onTranscriptUpdate: @escaping (String) -> Void,
onFinalTranscriptReady: @escaping (String) -> Void,
onError: @escaping (Error) -> Void
) async throws -> any BuddyStreamingTranscriptionSession
}
enum BuddyTranscriptionProviderFactory {
private enum PreferredProvider: String {
case assemblyAI = "assemblyai"
case openAI = "openai"
case appleSpeech = "apple"
}
static func makeDefaultProvider() -> any BuddyTranscriptionProvider {
let provider = resolveProvider()
print("🎙️ Transcription: using \(provider.displayName)")
return provider
}
private static func resolveProvider() -> any BuddyTranscriptionProvider {
let preferredProviderRawValue = AppBundleConfiguration
.stringValue(forKey: "VoiceTranscriptionProvider")?
.lowercased()
let preferredProvider = preferredProviderRawValue.flatMap(PreferredProvider.init(rawValue:))
let assemblyAIProvider = AssemblyAIStreamingTranscriptionProvider()
let openAIProvider = OpenAIAudioTranscriptionProvider()
if preferredProvider == .appleSpeech {
return AppleSpeechTranscriptionProvider()
}
if preferredProvider == .assemblyAI {
if assemblyAIProvider.isConfigured {
return assemblyAIProvider
}
print("⚠️ Transcription: AssemblyAI preferred but not configured, falling back")
if openAIProvider.isConfigured {
print("⚠️ Transcription: using OpenAI as fallback")
return openAIProvider
}
print("⚠️ Transcription: using Apple Speech as fallback")
return AppleSpeechTranscriptionProvider()
}
if preferredProvider == .openAI {
if openAIProvider.isConfigured {
return openAIProvider
}
print("⚠️ Transcription: OpenAI preferred but not configured, falling back")
if assemblyAIProvider.isConfigured {
print("⚠️ Transcription: using AssemblyAI as fallback")
return assemblyAIProvider
}
print("⚠️ Transcription: using Apple Speech as fallback")
return AppleSpeechTranscriptionProvider()
}
if assemblyAIProvider.isConfigured {
return assemblyAIProvider
}
if openAIProvider.isConfigured {
return openAIProvider
}
return AppleSpeechTranscriptionProvider()
}
}
+291
View File
@@ -0,0 +1,291 @@
//
// ClaudeAPI.swift
// Claude API Implementation with streaming support
//
import Foundation
/// Claude API helper with streaming for progressive text display.
class ClaudeAPI {
private static let tlsWarmupLock = NSLock()
private static var hasStartedTLSWarmup = false
private let apiURL: URL
var model: String
private let session: URLSession
init(proxyURL: String, model: String = "claude-sonnet-4-6") {
self.apiURL = URL(string: proxyURL)!
self.model = model
// Use .default instead of .ephemeral so TLS session tickets are cached.
// Ephemeral sessions do a full TLS handshake on every request, which causes
// transient -1200 (errSSLPeerHandshakeFail) errors with large image payloads.
// Disable URL/cookie caching to avoid storing responses or credentials on disk.
let config = URLSessionConfiguration.default
config.timeoutIntervalForRequest = 120
config.timeoutIntervalForResource = 300
config.waitsForConnectivity = true
config.urlCache = nil
config.httpCookieStorage = nil
self.session = URLSession(configuration: config)
// Fire a lightweight HEAD request in the background to pre-establish the TLS
// connection. This caches the TLS session ticket so the first real API call
// (which carries a large image payload) doesn't need a cold TLS handshake.
warmUpTLSConnectionIfNeeded()
}
private func makeAPIRequest() -> URLRequest {
var request = URLRequest(url: apiURL)
request.httpMethod = "POST"
request.timeoutInterval = 120
request.setValue("application/json", forHTTPHeaderField: "Content-Type")
return request
}
/// Detects the MIME type of image data by inspecting the first bytes.
/// Screen captures from ScreenCaptureKit are JPEG, but pasted images from the
/// clipboard are PNG. The API rejects requests where the declared media_type
/// doesn't match the actual image format.
private func detectImageMediaType(for imageData: Data) -> String {
// PNG files start with the 8-byte signature: 89 50 4E 47 0D 0A 1A 0A
if imageData.count >= 4 {
let pngSignature: [UInt8] = [0x89, 0x50, 0x4E, 0x47]
let firstFourBytes = [UInt8](imageData.prefix(4))
if firstFourBytes == pngSignature {
return "image/png"
}
}
// Default to JPEG screen captures use JPEG compression
return "image/jpeg"
}
/// Sends a no-op HEAD request to the API host to establish and cache a TLS session.
/// Failures are silently ignored this is purely an optimization.
private func warmUpTLSConnectionIfNeeded() {
Self.tlsWarmupLock.lock()
let shouldStartTLSWarmup = !Self.hasStartedTLSWarmup
if shouldStartTLSWarmup {
Self.hasStartedTLSWarmup = true
}
Self.tlsWarmupLock.unlock()
guard shouldStartTLSWarmup else { return }
guard var warmupURLComponents = URLComponents(url: apiURL, resolvingAgainstBaseURL: false) else {
return
}
// The TLS session ticket is host-scoped, so warming the root host is enough.
// Hitting the host instead of `/v1/messages` avoids extra endpoint-specific noise.
warmupURLComponents.path = "/"
warmupURLComponents.query = nil
warmupURLComponents.fragment = nil
guard let warmupURL = warmupURLComponents.url else {
return
}
var warmupRequest = URLRequest(url: warmupURL)
warmupRequest.httpMethod = "HEAD"
warmupRequest.timeoutInterval = 10
session.dataTask(with: warmupRequest) { _, _, _ in
// Response doesn't matter the TLS handshake is the goal
}.resume()
}
/// Send a vision request to Claude with streaming.
/// Calls `onTextChunk` on the main actor each time new text arrives so the UI updates progressively.
/// Returns the full accumulated text and total duration when the stream completes.
func analyzeImageStreaming(
images: [(data: Data, label: String)],
systemPrompt: String,
conversationHistory: [(userPlaceholder: String, assistantResponse: String)] = [],
userPrompt: String,
onTextChunk: @MainActor @Sendable (String) -> Void
) async throws -> (text: String, duration: TimeInterval) {
let startTime = Date()
var request = makeAPIRequest()
// Build messages array
var messages: [[String: Any]] = []
for (userPlaceholder, assistantResponse) in conversationHistory {
messages.append(["role": "user", "content": userPlaceholder])
messages.append(["role": "assistant", "content": assistantResponse])
}
// Build current message with all labeled images + prompt
var contentBlocks: [[String: Any]] = []
for image in images {
contentBlocks.append([
"type": "image",
"source": [
"type": "base64",
"media_type": detectImageMediaType(for: image.data),
"data": image.data.base64EncodedString()
]
])
contentBlocks.append([
"type": "text",
"text": image.label
])
}
contentBlocks.append([
"type": "text",
"text": userPrompt
])
messages.append(["role": "user", "content": contentBlocks])
let body: [String: Any] = [
"model": model,
"max_tokens": 1024,
"stream": true,
"system": systemPrompt,
"messages": messages
]
let bodyData = try JSONSerialization.data(withJSONObject: body)
request.httpBody = bodyData
let payloadMB = Double(bodyData.count) / 1_048_576.0
print("🌐 Claude streaming request: \(String(format: "%.1f", payloadMB))MB, \(images.count) image(s)")
// Use bytes streaming for SSE (Server-Sent Events)
let (byteStream, response) = try await session.bytes(for: request)
guard let httpResponse = response as? HTTPURLResponse else {
throw NSError(
domain: "ClaudeAPI",
code: -1,
userInfo: [NSLocalizedDescriptionKey: "Invalid HTTP response"]
)
}
// If non-2xx status, read the full body as error text
guard (200...299).contains(httpResponse.statusCode) else {
var errorBodyChunks: [String] = []
for try await line in byteStream.lines {
errorBodyChunks.append(line)
}
let errorBody = errorBodyChunks.joined(separator: "\n")
throw NSError(
domain: "ClaudeAPI",
code: httpResponse.statusCode,
userInfo: [NSLocalizedDescriptionKey: "API Error (\(httpResponse.statusCode)): \(errorBody)"]
)
}
// Parse SSE stream each event is "data: {json}\n\n"
var accumulatedResponseText = ""
for try await line in byteStream.lines {
// SSE lines look like: "data: {...}"
guard line.hasPrefix("data: ") else { continue }
let jsonString = String(line.dropFirst(6)) // Drop "data: " prefix
// End of stream marker
guard jsonString != "[DONE]" else { break }
guard let jsonData = jsonString.data(using: .utf8),
let eventPayload = try? JSONSerialization.jsonObject(with: jsonData) as? [String: Any],
let eventType = eventPayload["type"] as? String else {
continue
}
// We care about content_block_delta events that contain text chunks
if eventType == "content_block_delta",
let delta = eventPayload["delta"] as? [String: Any],
let deltaType = delta["type"] as? String,
deltaType == "text_delta",
let textChunk = delta["text"] as? String {
accumulatedResponseText += textChunk
// Send the accumulated text so far to the UI for progressive rendering
let currentAccumulatedText = accumulatedResponseText
await onTextChunk(currentAccumulatedText)
}
}
let duration = Date().timeIntervalSince(startTime)
return (text: accumulatedResponseText, duration: duration)
}
/// Non-streaming fallback for validation requests where we don't need progressive display.
func analyzeImage(
images: [(data: Data, label: String)],
systemPrompt: String,
conversationHistory: [(userPlaceholder: String, assistantResponse: String)] = [],
userPrompt: String
) async throws -> (text: String, duration: TimeInterval) {
let startTime = Date()
var request = makeAPIRequest()
var messages: [[String: Any]] = []
for (userPlaceholder, assistantResponse) in conversationHistory {
messages.append(["role": "user", "content": userPlaceholder])
messages.append(["role": "assistant", "content": assistantResponse])
}
// Build current message with all labeled images + prompt
var contentBlocks: [[String: Any]] = []
for image in images {
contentBlocks.append([
"type": "image",
"source": [
"type": "base64",
"media_type": detectImageMediaType(for: image.data),
"data": image.data.base64EncodedString()
]
])
contentBlocks.append([
"type": "text",
"text": image.label
])
}
contentBlocks.append([
"type": "text",
"text": userPrompt
])
messages.append(["role": "user", "content": contentBlocks])
let body: [String: Any] = [
"model": model,
"max_tokens": 256,
"system": systemPrompt,
"messages": messages
]
let bodyData = try JSONSerialization.data(withJSONObject: body)
request.httpBody = bodyData
let payloadMB = Double(bodyData.count) / 1_048_576.0
print("🌐 Claude request: \(String(format: "%.1f", payloadMB))MB, \(images.count) image(s)")
let (data, response) = try await session.data(for: request)
guard let httpResponse = response as? HTTPURLResponse,
(200...299).contains(httpResponse.statusCode) else {
let responseString = String(data: data, encoding: .utf8) ?? "Unknown error"
throw NSError(
domain: "ClaudeAPI",
code: (response as? HTTPURLResponse)?.statusCode ?? -1,
userInfo: [NSLocalizedDescriptionKey: "API Error: \(responseString)"]
)
}
let json = try JSONSerialization.jsonObject(with: data) as? [String: Any]
guard let content = json?["content"] as? [[String: Any]],
let textBlock = content.first(where: { ($0["type"] as? String) == "text" }),
let text = textBlock["text"] as? String else {
throw NSError(
domain: "ClaudeAPI",
code: -1,
userInfo: [NSLocalizedDescriptionKey: "Invalid response format"]
)
}
let duration = Date().timeIntervalSince(startTime)
return (text: text, duration: duration)
}
}
+121
View File
@@ -0,0 +1,121 @@
//
// ClickyAnalytics.swift
// leanring-buddy
//
// Centralized PostHog analytics wrapper. All event names and properties
// are defined here so instrumentation is consistent and easy to audit.
//
import Foundation
import PostHog
enum ClickyAnalytics {
// MARK: - Setup
static func configure() {
let config = PostHogConfig(
apiKey: "phc_xcQPygmhTMzzYh8wNW92CCwoXmnzqyChAixh8zgpqC3C",
host: "https://us.i.posthog.com"
)
PostHogSDK.shared.setup(config)
}
// MARK: - App Lifecycle
/// Fired once on every app launch in applicationDidFinishLaunching.
static func trackAppOpened() {
let version = Bundle.main.infoDictionary?["CFBundleShortVersionString"] as? String ?? "unknown"
PostHogSDK.shared.capture("app_opened", properties: [
"app_version": version
])
}
// MARK: - Onboarding
/// User clicked the Start button to begin onboarding for the first time.
static func trackOnboardingStarted() {
PostHogSDK.shared.capture("onboarding_started")
}
/// User clicked "Watch Onboarding Again" from the panel footer.
static func trackOnboardingReplayed() {
PostHogSDK.shared.capture("onboarding_replayed")
}
/// The onboarding video finished playing to the end.
static func trackOnboardingVideoCompleted() {
PostHogSDK.shared.capture("onboarding_video_completed")
}
/// The 40s onboarding demo interaction where Clicky points at something.
static func trackOnboardingDemoTriggered() {
PostHogSDK.shared.capture("onboarding_demo_triggered")
}
// MARK: - Permissions
/// All three permissions (accessibility, screen recording, mic) are granted.
static func trackAllPermissionsGranted() {
PostHogSDK.shared.capture("all_permissions_granted")
}
/// A single permission was granted. Called when polling detects a change.
static func trackPermissionGranted(permission: String) {
PostHogSDK.shared.capture("permission_granted", properties: [
"permission": permission
])
}
// MARK: - Voice Interaction
/// User pressed the push-to-talk shortcut (control+option) to start talking.
static func trackPushToTalkStarted() {
PostHogSDK.shared.capture("push_to_talk_started")
}
/// User released the shortcut transcript is being finalized.
static func trackPushToTalkReleased() {
PostHogSDK.shared.capture("push_to_talk_released")
}
/// Transcription completed and the user's message is being sent to the AI.
static func trackUserMessageSent(transcript: String) {
PostHogSDK.shared.capture("user_message_sent", properties: [
"transcript": transcript,
"character_count": transcript.count
])
}
/// Claude responded and the response is being spoken via TTS.
static func trackAIResponseReceived(response: String) {
PostHogSDK.shared.capture("ai_response_received", properties: [
"response": response,
"character_count": response.count
])
}
/// Claude's response included a [POINT:x,y:label] coordinate tag,
/// so the buddy is flying to point at a UI element.
static func trackElementPointed(elementLabel: String?) {
PostHogSDK.shared.capture("element_pointed", properties: [
"element_label": elementLabel ?? "unknown"
])
}
// MARK: - Errors
/// An error occurred during the AI response pipeline.
static func trackResponseError(error: String) {
PostHogSDK.shared.capture("response_error", properties: [
"error": error
])
}
/// An error occurred during TTS playback.
static func trackTTSError(error: String) {
PostHogSDK.shared.capture("tts_error", properties: [
"error": error
])
}
}
File diff suppressed because it is too large Load Diff
+761
View File
@@ -0,0 +1,761 @@
//
// CompanionPanelView.swift
// leanring-buddy
//
// The SwiftUI content hosted inside the menu bar panel. Shows the companion
// voice status, push-to-talk shortcut, and quick settings. Designed to feel
// like Loom's recording panel dark, rounded, minimal, and special.
//
import AVFoundation
import SwiftUI
struct CompanionPanelView: View {
@ObservedObject var companionManager: CompanionManager
@State private var emailInput: String = ""
var body: some View {
VStack(alignment: .leading, spacing: 0) {
panelHeader
Divider()
.background(DS.Colors.borderSubtle)
.padding(.horizontal, 16)
permissionsCopySection
.padding(.top, 16)
.padding(.horizontal, 16)
if companionManager.hasCompletedOnboarding && companionManager.allPermissionsGranted {
Spacer()
.frame(height: 12)
modelPickerRow
.padding(.horizontal, 16)
}
if !companionManager.allPermissionsGranted {
Spacer()
.frame(height: 16)
settingsSection
.padding(.horizontal, 16)
}
if !companionManager.hasCompletedOnboarding && companionManager.allPermissionsGranted {
Spacer()
.frame(height: 16)
startButton
.padding(.horizontal, 16)
}
// Show Clicky toggle hidden for now
// if companionManager.hasCompletedOnboarding && companionManager.allPermissionsGranted {
// Spacer()
// .frame(height: 16)
//
// showClickyCursorToggleRow
// .padding(.horizontal, 16)
// }
if companionManager.hasCompletedOnboarding && companionManager.allPermissionsGranted {
Spacer()
.frame(height: 16)
dmFarzaButton
.padding(.horizontal, 16)
}
Spacer()
.frame(height: 12)
Divider()
.background(DS.Colors.borderSubtle)
.padding(.horizontal, 16)
footerSection
.padding(.horizontal, 16)
.padding(.vertical, 12)
}
.frame(width: 320)
.background(panelBackground)
}
// MARK: - Header
private var panelHeader: some View {
HStack {
HStack(spacing: 8) {
// Animated status dot
Circle()
.fill(statusDotColor)
.frame(width: 8, height: 8)
.shadow(color: statusDotColor.opacity(0.6), radius: 4)
Text("Clicky")
.font(.system(size: 14, weight: .semibold))
.foregroundColor(DS.Colors.textPrimary)
}
Spacer()
Text(statusText)
.font(.system(size: 12, weight: .medium))
.foregroundColor(DS.Colors.textTertiary)
Button(action: {
NotificationCenter.default.post(name: .clickyDismissPanel, object: nil)
}) {
Image(systemName: "xmark")
.font(.system(size: 10, weight: .semibold))
.foregroundColor(DS.Colors.textTertiary)
.frame(width: 20, height: 20)
.background(
Circle()
.fill(Color.white.opacity(0.08))
)
}
.buttonStyle(.plain)
.pointerCursor()
}
.padding(.horizontal, 16)
.padding(.vertical, 14)
}
// MARK: - Permissions Copy
@ViewBuilder
private var permissionsCopySection: some View {
if companionManager.hasCompletedOnboarding && companionManager.allPermissionsGranted {
Text("Hold Control+Option to talk.")
.font(.system(size: 12, weight: .medium))
.foregroundColor(DS.Colors.textSecondary)
.frame(maxWidth: .infinity, alignment: .leading)
} else if companionManager.allPermissionsGranted && !companionManager.hasSubmittedEmail {
VStack(alignment: .leading, spacing: 4) {
Text("Drop your email to get started.")
.font(.system(size: 12, weight: .medium))
.foregroundColor(DS.Colors.textSecondary)
Text("If I keep building this, I'll keep you in the loop.")
.font(.system(size: 11))
.foregroundColor(DS.Colors.textTertiary)
}
.frame(maxWidth: .infinity, alignment: .leading)
} else if companionManager.allPermissionsGranted {
Text("You're all set. Hit Start to meet Clicky.")
.font(.system(size: 12, weight: .medium))
.foregroundColor(DS.Colors.textSecondary)
.frame(maxWidth: .infinity, alignment: .leading)
} else if companionManager.hasCompletedOnboarding {
// Permissions were revoked after onboarding tell user to re-grant
VStack(alignment: .leading, spacing: 6) {
Text("Permissions needed")
.font(.system(size: 12, weight: .bold))
.foregroundColor(DS.Colors.textSecondary)
Text("Some permissions were revoked. Grant all four below to keep using Clicky.")
.font(.system(size: 11))
.foregroundColor(DS.Colors.textTertiary)
.fixedSize(horizontal: false, vertical: true)
}
.frame(maxWidth: .infinity, alignment: .leading)
} else {
VStack(alignment: .leading, spacing: 6) {
Text("Hi, I'm Farza. This is Clicky.")
.font(.system(size: 12, weight: .bold))
.foregroundColor(DS.Colors.textSecondary)
Text("A side project I made for fun to help me learn stuff as I use my computer.")
.font(.system(size: 11))
.foregroundColor(DS.Colors.textTertiary)
.fixedSize(horizontal: false, vertical: true)
Text("Nothing runs in the background. Clicky will only take a screenshot when you press the hot key. So, you can give that permission in peace. If you are still sus, eh, I can't do much there champ.")
.font(.system(size: 11))
.foregroundColor(Color(red: 0.9, green: 0.4, blue: 0.4))
.fixedSize(horizontal: false, vertical: true)
}
.frame(maxWidth: .infinity, alignment: .leading)
}
}
// MARK: - Email + Start Button
@ViewBuilder
private var startButton: some View {
if !companionManager.hasCompletedOnboarding && companionManager.allPermissionsGranted {
if !companionManager.hasSubmittedEmail {
VStack(spacing: 8) {
TextField("Enter your email", text: $emailInput)
.textFieldStyle(.plain)
.font(.system(size: 13))
.foregroundColor(DS.Colors.textPrimary)
.padding(.horizontal, 12)
.padding(.vertical, 8)
.background(
RoundedRectangle(cornerRadius: DS.CornerRadius.medium, style: .continuous)
.fill(Color.white.opacity(0.08))
)
.overlay(
RoundedRectangle(cornerRadius: DS.CornerRadius.medium, style: .continuous)
.stroke(DS.Colors.borderSubtle, lineWidth: 0.5)
)
Button(action: {
companionManager.submitEmail(emailInput)
}) {
Text("Submit")
.font(.system(size: 14, weight: .semibold))
.foregroundColor(DS.Colors.textOnAccent)
.frame(maxWidth: .infinity)
.padding(.vertical, 10)
.background(
RoundedRectangle(cornerRadius: DS.CornerRadius.large, style: .continuous)
.fill(emailInput.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty
? DS.Colors.accent.opacity(0.4)
: DS.Colors.accent)
)
}
.buttonStyle(.plain)
.pointerCursor()
.disabled(emailInput.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty)
}
} else {
Button(action: {
companionManager.triggerOnboarding()
}) {
Text("Start")
.font(.system(size: 14, weight: .semibold))
.foregroundColor(DS.Colors.textOnAccent)
.frame(maxWidth: .infinity)
.padding(.vertical, 10)
.background(
RoundedRectangle(cornerRadius: DS.CornerRadius.large, style: .continuous)
.fill(DS.Colors.accent)
)
}
.buttonStyle(.plain)
.pointerCursor()
}
}
}
// MARK: - Permissions
private var settingsSection: some View {
VStack(spacing: 2) {
Text("PERMISSIONS")
.font(.system(size: 10, weight: .semibold, design: .rounded))
.foregroundColor(DS.Colors.textTertiary)
.frame(maxWidth: .infinity, alignment: .leading)
.padding(.bottom, 6)
microphonePermissionRow
accessibilityPermissionRow
screenRecordingPermissionRow
if companionManager.hasScreenRecordingPermission {
screenContentPermissionRow
}
}
}
private var accessibilityPermissionRow: some View {
let isGranted = companionManager.hasAccessibilityPermission
return HStack {
HStack(spacing: 8) {
Image(systemName: "hand.raised")
.font(.system(size: 12, weight: .medium))
.foregroundColor(isGranted ? DS.Colors.textTertiary : DS.Colors.warning)
.frame(width: 16)
Text("Accessibility")
.font(.system(size: 13, weight: .medium))
.foregroundColor(DS.Colors.textSecondary)
}
Spacer()
if isGranted {
HStack(spacing: 4) {
Circle()
.fill(DS.Colors.success)
.frame(width: 6, height: 6)
Text("Granted")
.font(.system(size: 11, weight: .medium))
.foregroundColor(DS.Colors.success)
}
} else {
HStack(spacing: 6) {
Button(action: {
// Triggers the system accessibility prompt (AXIsProcessTrustedWithOptions)
// on first attempt, then opens System Settings on subsequent attempts.
WindowPositionManager.requestAccessibilityPermission()
}) {
Text("Grant")
.font(.system(size: 11, weight: .semibold))
.foregroundColor(DS.Colors.textOnAccent)
.padding(.horizontal, 10)
.padding(.vertical, 4)
.background(
Capsule()
.fill(DS.Colors.accent)
)
}
.buttonStyle(.plain)
.pointerCursor()
Button(action: {
// Reveals the app in Finder so the user can drag it into
// the Accessibility list if it doesn't appear automatically
// (common with unsigned dev builds).
WindowPositionManager.revealAppInFinder()
WindowPositionManager.openAccessibilitySettings()
}) {
Text("Find App")
.font(.system(size: 11, weight: .semibold))
.foregroundColor(DS.Colors.textSecondary)
.padding(.horizontal, 10)
.padding(.vertical, 4)
.background(
Capsule()
.stroke(DS.Colors.borderSubtle, lineWidth: 0.8)
)
}
.buttonStyle(.plain)
.pointerCursor()
}
}
}
.padding(.vertical, 6)
}
private var screenRecordingPermissionRow: some View {
let isGranted = companionManager.hasScreenRecordingPermission
return HStack {
HStack(spacing: 8) {
Image(systemName: "rectangle.dashed.badge.record")
.font(.system(size: 12, weight: .medium))
.foregroundColor(isGranted ? DS.Colors.textTertiary : DS.Colors.warning)
.frame(width: 16)
VStack(alignment: .leading, spacing: 1) {
Text("Screen Recording")
.font(.system(size: 13, weight: .medium))
.foregroundColor(DS.Colors.textSecondary)
Text(isGranted
? "Only takes a screenshot when you use the hotkey"
: "Quit and reopen after granting")
.font(.system(size: 10))
.foregroundColor(DS.Colors.textTertiary)
}
}
Spacer()
if isGranted {
HStack(spacing: 4) {
Circle()
.fill(DS.Colors.success)
.frame(width: 6, height: 6)
Text("Granted")
.font(.system(size: 11, weight: .medium))
.foregroundColor(DS.Colors.success)
}
} else {
Button(action: {
// Triggers the native macOS screen recording prompt on first
// attempt (auto-adds app to the list), then opens System Settings
// on subsequent attempts.
WindowPositionManager.requestScreenRecordingPermission()
}) {
Text("Grant")
.font(.system(size: 11, weight: .semibold))
.foregroundColor(DS.Colors.textOnAccent)
.padding(.horizontal, 10)
.padding(.vertical, 4)
.background(
Capsule()
.fill(DS.Colors.accent)
)
}
.buttonStyle(.plain)
.pointerCursor()
}
}
.padding(.vertical, 6)
}
private var screenContentPermissionRow: some View {
let isGranted = companionManager.hasScreenContentPermission
return HStack {
HStack(spacing: 8) {
Image(systemName: "eye")
.font(.system(size: 12, weight: .medium))
.foregroundColor(isGranted ? DS.Colors.textTertiary : DS.Colors.warning)
.frame(width: 16)
Text("Screen Content")
.font(.system(size: 13, weight: .medium))
.foregroundColor(DS.Colors.textSecondary)
}
Spacer()
if isGranted {
HStack(spacing: 4) {
Circle()
.fill(DS.Colors.success)
.frame(width: 6, height: 6)
Text("Granted")
.font(.system(size: 11, weight: .medium))
.foregroundColor(DS.Colors.success)
}
} else {
Button(action: {
companionManager.requestScreenContentPermission()
}) {
Text("Grant")
.font(.system(size: 11, weight: .semibold))
.foregroundColor(DS.Colors.textOnAccent)
.padding(.horizontal, 10)
.padding(.vertical, 4)
.background(
Capsule()
.fill(DS.Colors.accent)
)
}
.buttonStyle(.plain)
.pointerCursor()
}
}
.padding(.vertical, 6)
}
private var microphonePermissionRow: some View {
let isGranted = companionManager.hasMicrophonePermission
return HStack {
HStack(spacing: 8) {
Image(systemName: "mic")
.font(.system(size: 12, weight: .medium))
.foregroundColor(isGranted ? DS.Colors.textTertiary : DS.Colors.warning)
.frame(width: 16)
Text("Microphone")
.font(.system(size: 13, weight: .medium))
.foregroundColor(DS.Colors.textSecondary)
}
Spacer()
if isGranted {
HStack(spacing: 4) {
Circle()
.fill(DS.Colors.success)
.frame(width: 6, height: 6)
Text("Granted")
.font(.system(size: 11, weight: .medium))
.foregroundColor(DS.Colors.success)
}
} else {
Button(action: {
// Triggers the native macOS microphone permission dialog on
// first attempt. If already denied, opens System Settings.
let status = AVCaptureDevice.authorizationStatus(for: .audio)
if status == .notDetermined {
AVCaptureDevice.requestAccess(for: .audio) { _ in }
} else {
if let url = URL(string: "x-apple.systempreferences:com.apple.preference.security?Privacy_Microphone") {
NSWorkspace.shared.open(url)
}
}
}) {
Text("Grant")
.font(.system(size: 11, weight: .semibold))
.foregroundColor(DS.Colors.textOnAccent)
.padding(.horizontal, 10)
.padding(.vertical, 4)
.background(
Capsule()
.fill(DS.Colors.accent)
)
}
.buttonStyle(.plain)
.pointerCursor()
}
}
.padding(.vertical, 6)
}
private func permissionRow(
label: String,
iconName: String,
isGranted: Bool,
settingsURL: String
) -> some View {
HStack {
HStack(spacing: 8) {
Image(systemName: iconName)
.font(.system(size: 12, weight: .medium))
.foregroundColor(isGranted ? DS.Colors.textTertiary : DS.Colors.warning)
.frame(width: 16)
Text(label)
.font(.system(size: 13, weight: .medium))
.foregroundColor(DS.Colors.textSecondary)
}
Spacer()
if isGranted {
HStack(spacing: 4) {
Circle()
.fill(DS.Colors.success)
.frame(width: 6, height: 6)
Text("Granted")
.font(.system(size: 11, weight: .medium))
.foregroundColor(DS.Colors.success)
}
} else {
Button(action: {
if let url = URL(string: settingsURL) {
NSWorkspace.shared.open(url)
}
}) {
Text("Grant")
.font(.system(size: 11, weight: .semibold))
.foregroundColor(DS.Colors.textOnAccent)
.padding(.horizontal, 10)
.padding(.vertical, 4)
.background(
Capsule()
.fill(DS.Colors.accent)
)
}
.buttonStyle(.plain)
.pointerCursor()
}
}
.padding(.vertical, 6)
}
// MARK: - Show Clicky Cursor Toggle
private var showClickyCursorToggleRow: some View {
HStack {
HStack(spacing: 8) {
Image(systemName: "cursorarrow")
.font(.system(size: 12, weight: .medium))
.foregroundColor(DS.Colors.textTertiary)
.frame(width: 16)
Text("Show Clicky")
.font(.system(size: 13, weight: .medium))
.foregroundColor(DS.Colors.textSecondary)
}
Spacer()
Toggle("", isOn: Binding(
get: { companionManager.isClickyCursorEnabled },
set: { companionManager.setClickyCursorEnabled($0) }
))
.toggleStyle(.switch)
.labelsHidden()
.tint(DS.Colors.accent)
.scaleEffect(0.8)
}
.padding(.vertical, 4)
}
private var speechToTextProviderRow: some View {
HStack {
HStack(spacing: 8) {
Image(systemName: "mic.badge.waveform")
.font(.system(size: 12, weight: .medium))
.foregroundColor(DS.Colors.textTertiary)
.frame(width: 16)
Text("Speech to Text")
.font(.system(size: 13, weight: .medium))
.foregroundColor(DS.Colors.textSecondary)
}
Spacer()
Text(companionManager.buddyDictationManager.transcriptionProviderDisplayName)
.font(.system(size: 11, weight: .medium))
.foregroundColor(DS.Colors.textTertiary)
}
.padding(.vertical, 4)
}
// MARK: - Model Picker
private var modelPickerRow: some View {
HStack {
Text("Model")
.font(.system(size: 13, weight: .medium))
.foregroundColor(DS.Colors.textSecondary)
Spacer()
HStack(spacing: 0) {
modelOptionButton(label: "Sonnet", modelID: "claude-sonnet-4-6")
modelOptionButton(label: "Opus", modelID: "claude-opus-4-6")
}
.background(
RoundedRectangle(cornerRadius: 6, style: .continuous)
.fill(Color.white.opacity(0.06))
)
.overlay(
RoundedRectangle(cornerRadius: 6, style: .continuous)
.stroke(DS.Colors.borderSubtle, lineWidth: 0.5)
)
}
.padding(.vertical, 4)
}
private func modelOptionButton(label: String, modelID: String) -> some View {
let isSelected = companionManager.selectedModel == modelID
return Button(action: {
companionManager.setSelectedModel(modelID)
}) {
Text(label)
.font(.system(size: 11, weight: .medium))
.foregroundColor(isSelected ? DS.Colors.textPrimary : DS.Colors.textTertiary)
.padding(.horizontal, 10)
.padding(.vertical, 5)
.background(
RoundedRectangle(cornerRadius: 5, style: .continuous)
.fill(isSelected ? Color.white.opacity(0.1) : Color.clear)
)
}
.buttonStyle(.plain)
.pointerCursor()
}
// MARK: - DM Farza Button
private var dmFarzaButton: some View {
Button(action: {
if let url = URL(string: "https://x.com/farzatv") {
NSWorkspace.shared.open(url)
}
}) {
HStack(spacing: 8) {
Image(systemName: "bubble.left.fill")
.font(.system(size: 12, weight: .medium))
VStack(alignment: .leading, spacing: 2) {
Text("Got feedback? DM me")
.font(.system(size: 12, weight: .semibold))
Text("Bugs, ideas, anything — I read every message.")
.font(.system(size: 10))
.foregroundColor(DS.Colors.textTertiary)
}
}
.foregroundColor(DS.Colors.textSecondary)
.frame(maxWidth: .infinity, alignment: .leading)
.padding(.horizontal, 12)
.padding(.vertical, 10)
.background(
RoundedRectangle(cornerRadius: DS.CornerRadius.medium, style: .continuous)
.fill(Color.white.opacity(0.06))
)
.overlay(
RoundedRectangle(cornerRadius: DS.CornerRadius.medium, style: .continuous)
.stroke(DS.Colors.borderSubtle, lineWidth: 0.5)
)
}
.buttonStyle(.plain)
.pointerCursor()
}
// MARK: - Footer
private var footerSection: some View {
HStack {
Button(action: {
NSApp.terminate(nil)
}) {
HStack(spacing: 6) {
Image(systemName: "power")
.font(.system(size: 11, weight: .medium))
Text("Quit Clicky")
.font(.system(size: 12, weight: .medium))
}
.foregroundColor(DS.Colors.textTertiary)
}
.buttonStyle(.plain)
.pointerCursor()
if companionManager.hasCompletedOnboarding {
Spacer()
Button(action: {
companionManager.replayOnboarding()
}) {
HStack(spacing: 6) {
Image(systemName: "play.circle")
.font(.system(size: 11, weight: .medium))
Text("Watch Onboarding Again")
.font(.system(size: 12, weight: .medium))
}
.foregroundColor(DS.Colors.textTertiary)
}
.buttonStyle(.plain)
.pointerCursor()
}
}
}
// MARK: - Visual Helpers
private var panelBackground: some View {
RoundedRectangle(cornerRadius: 12, style: .continuous)
.fill(DS.Colors.background)
.shadow(color: Color.black.opacity(0.5), radius: 20, x: 0, y: 10)
.shadow(color: Color.black.opacity(0.3), radius: 4, x: 0, y: 2)
}
private var statusDotColor: Color {
if !companionManager.isOverlayVisible {
return DS.Colors.textTertiary
}
switch companionManager.voiceState {
case .idle:
return DS.Colors.success
case .listening:
return DS.Colors.blue400
case .processing, .responding:
return DS.Colors.blue400
}
}
private var statusText: String {
if !companionManager.hasCompletedOnboarding || !companionManager.allPermissionsGranted {
return "Setup"
}
if !companionManager.isOverlayVisible {
return "Ready"
}
switch companionManager.voiceState {
case .idle:
return "Active"
case .listening:
return "Listening"
case .processing:
return "Processing"
case .responding:
return "Responding"
}
}
}
@@ -0,0 +1,217 @@
//
// CompanionResponseOverlay.swift
// leanring-buddy
//
// Cursor-following overlay that displays streaming AI response text.
// Uses a non-activating NSPanel so it floats above all apps without
// stealing focus, and repositions itself near the mouse cursor each frame.
//
import AppKit
import Combine
import SwiftUI
// MARK: - View Model
@MainActor
final class CompanionResponseOverlayViewModel: ObservableObject {
@Published var streamingResponseText: String = ""
@Published var isShowingResponse: Bool = false
}
// MARK: - Overlay Manager
@MainActor
final class CompanionResponseOverlayManager {
private let overlayViewModel = CompanionResponseOverlayViewModel()
private var overlayPanel: NSPanel?
private var cursorTrackingTimer: Timer?
private var autoHideWorkItem: DispatchWorkItem?
/// The horizontal offset from the cursor to the left edge of the overlay panel.
private let cursorOffsetX: CGFloat = 22
/// The vertical offset from the cursor downward to the top edge of the overlay panel.
private let cursorOffsetY: CGFloat = 6
/// Maximum width of the overlay panel.
private let overlayMaxWidth: CGFloat = 340
func showOverlayAndBeginStreaming() {
autoHideWorkItem?.cancel()
autoHideWorkItem = nil
overlayViewModel.streamingResponseText = ""
overlayViewModel.isShowingResponse = true
createOverlayPanelIfNeeded()
startCursorTracking()
overlayPanel?.alphaValue = 1
overlayPanel?.orderFrontRegardless()
}
func updateStreamingText(_ accumulatedText: String) {
overlayViewModel.streamingResponseText = accumulatedText
resizePanelToFitContent()
}
func finishStreaming() {
// Keep the response visible for a few seconds after streaming ends,
// then fade out so the user has time to read the last chunk.
let hideWork = DispatchWorkItem { [weak self] in
self?.fadeOutAndHide()
}
autoHideWorkItem = hideWork
DispatchQueue.main.asyncAfter(deadline: .now() + 6, execute: hideWork)
}
func hideOverlay() {
autoHideWorkItem?.cancel()
autoHideWorkItem = nil
stopCursorTracking()
overlayViewModel.isShowingResponse = false
overlayViewModel.streamingResponseText = ""
overlayPanel?.orderOut(nil)
}
// MARK: - Private
private func createOverlayPanelIfNeeded() {
if overlayPanel != nil { return }
let initialFrame = NSRect(x: 0, y: 0, width: overlayMaxWidth, height: 40)
let responseOverlayPanel = NSPanel(
contentRect: initialFrame,
styleMask: [.borderless, .nonactivatingPanel],
backing: .buffered,
defer: false
)
responseOverlayPanel.level = .statusBar
responseOverlayPanel.isOpaque = false
responseOverlayPanel.backgroundColor = .clear
responseOverlayPanel.hasShadow = false
responseOverlayPanel.ignoresMouseEvents = true
responseOverlayPanel.hidesOnDeactivate = false
responseOverlayPanel.collectionBehavior = [.canJoinAllSpaces, .fullScreenAuxiliary, .stationary]
responseOverlayPanel.isExcludedFromWindowsMenu = true
let hostingView = NSHostingView(
rootView: CompanionResponseOverlayView(viewModel: overlayViewModel)
.frame(maxWidth: overlayMaxWidth)
)
hostingView.frame = initialFrame
responseOverlayPanel.contentView = hostingView
overlayPanel = responseOverlayPanel
}
private func startCursorTracking() {
// 60fps cursor tracking so the panel stays glued to the mouse
cursorTrackingTimer = Timer.scheduledTimer(withTimeInterval: 1.0 / 60.0, repeats: true) { [weak self] _ in
Task { @MainActor [weak self] in
self?.repositionPanelNearCursor()
}
}
}
private func stopCursorTracking() {
cursorTrackingTimer?.invalidate()
cursorTrackingTimer = nil
}
private func repositionPanelNearCursor() {
guard let overlayPanel else { return }
let mouseLocation = NSEvent.mouseLocation
let panelSize = overlayPanel.frame.size
// Position the panel to the right of and slightly below the cursor.
// In macOS screen coordinates, Y increases upward, so "below" means
// subtracting from the cursor Y.
var panelOriginX = mouseLocation.x + cursorOffsetX
var panelOriginY = mouseLocation.y - cursorOffsetY - panelSize.height
// Clamp to the visible frame of the screen containing the cursor
// so the panel never goes off-screen.
if let currentScreen = screenContainingPoint(mouseLocation) {
let visibleFrame = currentScreen.visibleFrame
// If the panel would go off the right edge, flip it to the left of the cursor
if panelOriginX + panelSize.width > visibleFrame.maxX {
panelOriginX = mouseLocation.x - cursorOffsetX - panelSize.width
}
// If the panel would go below the bottom edge, push it above the cursor
if panelOriginY < visibleFrame.minY {
panelOriginY = mouseLocation.y + cursorOffsetY
}
// Final clamp
panelOriginX = max(visibleFrame.minX, min(panelOriginX, visibleFrame.maxX - panelSize.width))
panelOriginY = max(visibleFrame.minY, min(panelOriginY, visibleFrame.maxY - panelSize.height))
}
overlayPanel.setFrameOrigin(CGPoint(x: panelOriginX, y: panelOriginY))
}
private func resizePanelToFitContent() {
guard let overlayPanel, let contentView = overlayPanel.contentView else { return }
let fittingSize = contentView.fittingSize
let newWidth = min(fittingSize.width, overlayMaxWidth)
let newHeight = fittingSize.height
// Keep the panel origin relative to the cursor (the timer handles that),
// but update the frame size so the content fits.
var frame = overlayPanel.frame
let heightDelta = newHeight - frame.height
frame.size = CGSize(width: newWidth, height: newHeight)
// Adjust origin Y so the panel grows upward (toward the cursor), not downward
frame.origin.y -= heightDelta
overlayPanel.setFrame(frame, display: true)
contentView.frame = NSRect(origin: .zero, size: frame.size)
}
private func fadeOutAndHide() {
guard let overlayPanel else { return }
NSAnimationContext.runAnimationGroup({ context in
context.duration = 0.4
overlayPanel.animator().alphaValue = 0
}, completionHandler: { [weak self] in
Task { @MainActor in
self?.hideOverlay()
}
})
}
private func screenContainingPoint(_ point: CGPoint) -> NSScreen? {
NSScreen.screens.first { $0.frame.contains(point) }
}
}
// MARK: - SwiftUI View
private struct CompanionResponseOverlayView: View {
@ObservedObject var viewModel: CompanionResponseOverlayViewModel
var body: some View {
if viewModel.isShowingResponse {
Text(viewModel.streamingResponseText.isEmpty ? "..." : viewModel.streamingResponseText)
.font(.system(size: 13, weight: .regular))
.foregroundColor(DS.Colors.textPrimary)
.lineSpacing(3)
.fixedSize(horizontal: false, vertical: true)
.frame(maxWidth: 300, alignment: .leading)
.padding(.horizontal, 14)
.padding(.vertical, 10)
.background(
RoundedRectangle(cornerRadius: 10, style: .continuous)
.fill(DS.Colors.surface1.opacity(0.95))
.overlay(
RoundedRectangle(cornerRadius: 10, style: .continuous)
.stroke(DS.Colors.borderSubtle.opacity(0.5), lineWidth: 0.8)
)
.shadow(color: Color.black.opacity(0.35), radius: 16, x: 0, y: 8)
)
}
}
}
@@ -0,0 +1,132 @@
//
// CompanionScreenCaptureUtility.swift
// leanring-buddy
//
// Standalone screenshot capture for the companion voice flow.
// Decoupled from the legacy ScreenshotManager so the companion mode
// can capture screenshots independently without session state.
//
import AppKit
import ScreenCaptureKit
struct CompanionScreenCapture {
let imageData: Data
let label: String
let isCursorScreen: Bool
let displayWidthInPoints: Int
let displayHeightInPoints: Int
let displayFrame: CGRect
let screenshotWidthInPixels: Int
let screenshotHeightInPixels: Int
}
@MainActor
enum CompanionScreenCaptureUtility {
/// Captures all connected displays as JPEG data, labeling each with
/// whether the user's cursor is on that screen. This gives the AI
/// full context across multiple monitors.
static func captureAllScreensAsJPEG() async throws -> [CompanionScreenCapture] {
let content = try await SCShareableContent.excludingDesktopWindows(false, onScreenWindowsOnly: true)
guard !content.displays.isEmpty else {
throw NSError(domain: "CompanionScreenCapture", code: -1,
userInfo: [NSLocalizedDescriptionKey: "No display available for capture"])
}
let mouseLocation = NSEvent.mouseLocation
// Exclude all windows belonging to this app so the AI sees
// only the user's content, not our overlays or panels.
let ownBundleIdentifier = Bundle.main.bundleIdentifier
let ownAppWindows = content.windows.filter { window in
window.owningApplication?.bundleIdentifier == ownBundleIdentifier
}
// Build a lookup from display ID to NSScreen so we can use AppKit-coordinate
// frames instead of CG-coordinate frames. NSEvent.mouseLocation and NSScreen.frame
// both use AppKit coordinates (bottom-left origin), while SCDisplay.frame uses
// Core Graphics coordinates (top-left origin). On multi-display setups, the Y
// origins differ for secondary displays, which breaks cursor-contains checks
// and downstream coordinate conversions.
var nsScreenByDisplayID: [CGDirectDisplayID: NSScreen] = [:]
for screen in NSScreen.screens {
if let screenNumber = screen.deviceDescription[NSDeviceDescriptionKey("NSScreenNumber")] as? CGDirectDisplayID {
nsScreenByDisplayID[screenNumber] = screen
}
}
// Sort displays so the cursor screen is always first
let sortedDisplays = content.displays.sorted { displayA, displayB in
let frameA = nsScreenByDisplayID[displayA.displayID]?.frame ?? displayA.frame
let frameB = nsScreenByDisplayID[displayB.displayID]?.frame ?? displayB.frame
let aContainsCursor = frameA.contains(mouseLocation)
let bContainsCursor = frameB.contains(mouseLocation)
if aContainsCursor != bContainsCursor { return aContainsCursor }
return false
}
var capturedScreens: [CompanionScreenCapture] = []
for (displayIndex, display) in sortedDisplays.enumerated() {
// Use NSScreen.frame (AppKit coordinates, bottom-left origin) so
// displayFrame is in the same coordinate system as NSEvent.mouseLocation
// and the overlay window's screenFrame in BlueCursorView.
let displayFrame = nsScreenByDisplayID[display.displayID]?.frame
?? CGRect(x: display.frame.origin.x, y: display.frame.origin.y,
width: CGFloat(display.width), height: CGFloat(display.height))
let isCursorScreen = displayFrame.contains(mouseLocation)
let filter = SCContentFilter(display: display, excludingWindows: ownAppWindows)
let configuration = SCStreamConfiguration()
let maxDimension = 1280
let aspectRatio = CGFloat(display.width) / CGFloat(display.height)
if display.width >= display.height {
configuration.width = maxDimension
configuration.height = Int(CGFloat(maxDimension) / aspectRatio)
} else {
configuration.height = maxDimension
configuration.width = Int(CGFloat(maxDimension) * aspectRatio)
}
let cgImage = try await SCScreenshotManager.captureImage(
contentFilter: filter,
configuration: configuration
)
guard let jpegData = NSBitmapImageRep(cgImage: cgImage)
.representation(using: .jpeg, properties: [.compressionFactor: 0.8]) else {
continue
}
let screenLabel: String
if sortedDisplays.count == 1 {
screenLabel = "user's screen (cursor is here)"
} else if isCursorScreen {
screenLabel = "screen \(displayIndex + 1) of \(sortedDisplays.count) — cursor is on this screen (primary focus)"
} else {
screenLabel = "screen \(displayIndex + 1) of \(sortedDisplays.count) — secondary screen"
}
capturedScreens.append(CompanionScreenCapture(
imageData: jpegData,
label: screenLabel,
isCursorScreen: isCursorScreen,
displayWidthInPoints: Int(displayFrame.width),
displayHeightInPoints: Int(displayFrame.height),
displayFrame: displayFrame,
screenshotWidthInPixels: configuration.width,
screenshotHeightInPixels: configuration.height
))
}
guard !capturedScreens.isEmpty else {
throw NSError(domain: "CompanionScreenCapture", code: -2,
userInfo: [NSLocalizedDescriptionKey: "Failed to capture any screen"])
}
return capturedScreens
}
}
+880
View File
@@ -0,0 +1,880 @@
//
// DesignSystem.swift
// leanring-buddy
//
// Centralized design system using a blue accent palette on dark surfaces,
// with a unified button style system. All colors, button styles, and
// interaction states are defined here as the single source of truth.
//
import SwiftUI
import AppKit
// MARK: - Design System Namespace
/// The top-level namespace for all design system tokens.
/// Usage: `DS.Colors.background`, `DS.Colors.accent`, etc.
enum DS {
// MARK: - Color Tokens
enum Colors {
// Backgrounds
// Layered surfaces from deepest to most elevated.
// Higher surfaces are lighter, creating a sense of depth.
/// The deepest background used for the main app window fill.
static let background = Color(hex: "#101211")
/// First elevation layer used for cards, sidebar, top bar backgrounds.
static let surface1 = Color(hex: "#171918")
/// Second elevation layer used for input fields, elevated cards, chat bubbles.
static let surface2 = Color(hex: "#202221")
/// Third elevation layer used for hover backgrounds on interactive elements.
static let surface3 = Color(hex: "#272A29")
/// Fourth elevation layer used for active/pressed states on interactive elements.
static let surface4 = Color(hex: "#2E3130")
// Borders
/// Subtle border used for card outlines, dividers, input field borders.
static let borderSubtle = Color(hex: "#373B39")
/// Strong border used for focused inputs, hovered card outlines.
static let borderStrong = Color(hex: "#444947")
// Text
/// Primary text main body text, titles, headings.
static let textPrimary = Color(hex: "#ECEEED")
/// Secondary text descriptions, hints, muted labels.
static let textSecondary = Color(hex: "#ADB5B2")
/// Tertiary text very muted, used for section labels, timestamps, disabled text.
static let textTertiary = Color(hex: "#6B736F")
/// Text used on top of the accent fill (#2563eb blue), like the primary button label.
/// White on #2563eb achieves ~5.1:1 contrast WCAG AA compliant.
/// White on #1d4ed8 hover achieves ~6.5:1 also WCAG AA compliant.
static let textOnAccent: Color = .white
// Tailwind Blue Scale
// Full Tailwind CSS v4 blue palette for consistent blue usage.
//
// Usage guide:
// 50100 Very subtle tinted backgrounds (selected rows, hover fills on dark surfaces)
// 200300 Light text/icons on dark backgrounds, disabled states
// 400 Bright accent text, links, icons, chat user bubbles
// 500 Mid-tone fills, badges, secondary buttons
// 600 Primary action fills (buttons, toggles) main accent
// 700 Hover/pressed state for primary actions
// 800900 Deep backgrounds, dark overlays, header bars
// 950 Deepest blue near-black tinted backgrounds
static let blue50 = Color(hex: "#eff6ff")
static let blue100 = Color(hex: "#dbeafe")
static let blue200 = Color(hex: "#bfdbfe")
static let blue300 = Color(hex: "#93c5fd")
static let blue400 = Color(hex: "#60a5fa")
static let blue500 = Color(hex: "#3b82f6")
static let blue600 = Color(hex: "#2563eb")
static let blue700 = Color(hex: "#1d4ed8")
static let blue800 = Color(hex: "#1e40af")
static let blue900 = Color(hex: "#1e3a8a")
static let blue950 = Color(hex: "#172554")
// Accent (derived from blue scale)
// The primary fill is Blue 600; hover darkens to Blue 700.
/// Accent fill used for solid button backgrounds.
/// #2563eb ~5.1:1 contrast with white text (WCAG AA).
static let accent = blue600
/// Accent hover slightly darker blue for hover state.
/// #1d4ed8 ~6.5:1 contrast with white text (WCAG AA+).
static let accentHover = blue700
/// Accent text bright blue used for accent-colored text and icons
/// on dark backgrounds (links, active nav items, highlighted labels).
static let accentText = blue400
/// Very subtle accent tint used for selected item backgrounds (e.g. current step
/// in the sidebar). Low opacity so it doesn't overpower.
static let accentSubtle = blue500.opacity(0.10)
// Semantic Colors
/// Destructive/error actions delete buttons, error messages, close button hover.
static let destructive = Color(hex: "#E5484D") // Radix Red 9
/// Destructive hover state.
static let destructiveHover = Color(hex: "#F2555A") // Radix Red 10
/// Destructive used for text on dark backgrounds (brighter for readability).
static let destructiveText = Color(hex: "#FF6369") // Radix Red 11
/// Success checkmarks, granted status, completion indicators.
/// Independent green so success states are visually distinct from the blue accent.
static let success = Color(hex: "#34D399") // Tailwind Emerald 400
/// Warning caution messages, manual verification failure explanations.
static let warning = Color(hex: "#FFB224") // Radix Amber 9
/// Warning text brighter variant for text on dark backgrounds.
static let warningText = Color(hex: "#F1A10D") // Radix Amber 11
/// Info/feature highlight used for prompt card headers, code highlights.
/// Lighter than accentText so informational elements are visually distinct
/// from interactive accent-colored elements.
static let info = Color(hex: "#70B8FF") // Radix Blue 9
/// Inline code text color slightly brighter blue for monospace code snippets.
static let codeText = Color(hex: "#9DC2FF") // Radix Blue 11 variant
// Overlay Cursor
/// The blue cursor/bubble color used in OverlayWindow.
/// Kept distinct from the accent since it serves a different purpose
/// (screen overlay vs in-app UI).
static let overlayCursorBlue = Color(hex: "#3380FF")
// Floating Button Gradient
/// The floating session button gradient colors (unchanged from original
/// this gradient is intentionally distinct from the rest of the palette
/// to make the floating button stand out as a "jewel" on the desktop).
static let floatingGradientPurple = Color(hex: "#8F46EB")
static let floatingGradientPink = Color(hex: "#E84D9E")
static let floatingGradientOrange = Color(hex: "#FF8C33")
// Help Chat
/// User message bubble background in the help chat.
/// Blue 800 deep blue that's clearly distinct from the dark surface
/// while keeping white text highly readable (~9:1 contrast).
static let helpChatUserBubble = blue800
/// Slightly lighter variant for hover/pressed states on user bubbles.
static let helpChatUserBubbleHover = blue700
/// Footer/backdrop behind the floating help chat.
/// Slightly lighter than the main window background so the chat zone reads
/// as a distinct docked surface even before the pill input is visible.
static let helpChatBackdrop = Color(hex: "#212121")
// Disabled State
// Following Material Design 3's disabled pattern:
// Container: onSurface at 12% opacity
// Content: onSurface at 38% opacity
/// Disabled button/container background.
static var disabledBackground: Color {
textPrimary.opacity(0.12)
}
/// Disabled text/icon color.
static var disabledText: Color {
textPrimary.opacity(0.38)
}
}
// MARK: - Spacing (for reference, not enforced)
enum Spacing {
static let xs: CGFloat = 4
static let sm: CGFloat = 8
static let md: CGFloat = 12
static let lg: CGFloat = 16
static let xl: CGFloat = 20
static let xxl: CGFloat = 24
static let xxxl: CGFloat = 32
}
// MARK: - Corner Radii
enum CornerRadius {
/// Small elements like tags, badges.
static let small: CGFloat = 6
/// Buttons, input fields, small cards.
static let medium: CGFloat = 8
/// Cards, dialogs, chat bubbles.
static let large: CGFloat = 10
/// Large panels, permission cards.
static let extraLarge: CGFloat = 12
/// Pill-shaped buttons (the continue button).
static let pill: CGFloat = .infinity
}
// MARK: - Animation Durations
enum Animation {
/// Quick state changes hover in/out, press feedback.
static let fast: Double = 0.15
/// Standard transitions content reveal, button state changes.
static let normal: Double = 0.25
/// Slower, more dramatic fade-ins, celebration screen elements.
static let slow: Double = 0.4
}
// MARK: - State Layer Opacities
// Based on Material Design 3's state layer system.
// A "state layer" overlays the button's content color at these opacities.
enum StateLayer {
/// Hover: subtle highlight to indicate interactivity.
static let hover: Double = 0.08
/// Focus: keyboard navigation indicator (slightly stronger than hover).
static let focus: Double = 0.12
/// Pressed: active press feedback (same strength as focus).
static let pressed: Double = 0.12
/// Dragged: strongest overlay (rarely used).
static let dragged: Double = 0.16
}
}
// MARK: - Button Styles
/// Primary button the main call-to-action per screen.
/// Accent-colored background with white text. One per view maximum.
/// Used for: "start"/"resume", "let's go", "continue", "verify completion".
struct DSPrimaryButtonStyle: ButtonStyle {
var isFullWidth: Bool = true
@State private var isHovered = false
// Separate state for the scale expansion so it animates on a slower,
// more gradual timeline (0.6s) than the background color snap (0.15s).
@State private var isHoverScaleExpanded = false
// Whether the hover glow shadow is active. Builds up gradually (0.6s)
// on hover entry, fades out faster (0.3s) on exit.
@State private var isHoverGlowActive = false
// Continuously toggles while hovered to drive a gentle breathing pulse
// in the glow shadow. Creates a living, organic feel like the button
// is softly glowing, not just statically lit.
@State private var isGlowBreathingIn = false
func makeBody(configuration: Configuration) -> some View {
configuration.label
.font(.system(size: 16, weight: .medium))
.foregroundColor(DS.Colors.textOnAccent)
.frame(maxWidth: isFullWidth ? .infinity : nil)
.padding(.vertical, 14)
.padding(.horizontal, isFullWidth ? 0 : 20)
.background(
Capsule()
.fill(buttonBackgroundColor(isPressed: configuration.isPressed))
)
// Hover glow builds up gradually, then gently breathes while hovered.
// The breathing oscillates opacity and radius on a slow 2.5s loop,
// creating a candle-flame-like "alive" quality rather than a static highlight.
.shadow(
color: DS.Colors.accent.opacity(
isHoverGlowActive ? (isGlowBreathingIn ? 0.32 : 0.18) : 0
),
radius: isHoverGlowActive ? (isGlowBreathingIn ? 16 : 10) : 0
)
// Hover: gradually expand to 1.03. Press: snap down to 0.97.
.scaleEffect(configuration.isPressed ? 0.97 : (isHoverScaleExpanded ? 1.03 : 1.0))
.animation(.easeOut(duration: 0.1), value: configuration.isPressed)
.onHover { hovering in
// Background color fast snap so the button feels responsive
withAnimation(.easeOut(duration: 0.15)) {
isHovered = hovering
}
// Scale slow, gradual expansion (like the button is swelling)
withAnimation(.easeInOut(duration: hovering ? 0.6 : 0.3)) {
isHoverScaleExpanded = hovering
}
// Glow builds up gradually on entry, fades faster on exit
withAnimation(.easeInOut(duration: hovering ? 0.6 : 0.3)) {
isHoverGlowActive = hovering
}
// Breathing glow loop gentle pulse while hovered.
// The 2.5s cycle keeps it feeling organic, not mechanical.
if hovering {
withAnimation(
.easeInOut(duration: 2.5)
.repeatForever(autoreverses: true)
) {
isGlowBreathingIn = true
}
} else {
// Override the repeating animation with a finite one to stop cleanly
withAnimation(.easeOut(duration: 0.3)) {
isGlowBreathingIn = false
}
}
if hovering { NSCursor.pointingHand.push() } else { NSCursor.pop() }
}
}
private func buttonBackgroundColor(isPressed: Bool) -> Color {
if isPressed {
// Pressed: brighten slightly beyond hover
return DS.Colors.accentHover.blendedWithWhite(fraction: DS.StateLayer.pressed)
} else if isHovered {
return DS.Colors.accentHover
} else {
return DS.Colors.accent
}
}
}
/// Secondary button supporting actions, less visual weight than primary.
/// Surface-colored background with primary text. Used for: action buttons
/// (download, open link), embedded element buttons.
struct DSSecondaryButtonStyle: ButtonStyle {
var isFullWidth: Bool = true
@State private var isHovered = false
func makeBody(configuration: Configuration) -> some View {
configuration.label
.font(.system(size: 16, weight: .medium))
.foregroundColor(DS.Colors.textPrimary)
.frame(maxWidth: isFullWidth ? .infinity : nil)
.padding(.vertical, 12)
.padding(.horizontal, isFullWidth ? 0 : 16)
.background(
Capsule()
.fill(buttonBackgroundColor(isPressed: configuration.isPressed))
)
.scaleEffect(configuration.isPressed ? 0.97 : 1.0)
.animation(.easeOut(duration: DS.Animation.fast), value: configuration.isPressed)
.animation(.easeOut(duration: DS.Animation.fast), value: isHovered)
.onHover { hovering in
isHovered = hovering
if hovering { NSCursor.pointingHand.push() } else { NSCursor.pop() }
}
}
private func buttonBackgroundColor(isPressed: Bool) -> Color {
if isPressed {
return DS.Colors.surface4
} else if isHovered {
return DS.Colors.surface3
} else {
return DS.Colors.surface2
}
}
}
/// Tertiary/ghost button low-emphasis actions with subtle hover background.
/// Transparent at rest, shows surface fill on hover. Used for: navigation
/// links, sidebar items, medium-low emphasis actions.
struct DSTertiaryButtonStyle: ButtonStyle {
@State private var isHovered = false
func makeBody(configuration: Configuration) -> some View {
configuration.label
.font(.system(size: 16, weight: .medium))
.foregroundColor(
configuration.isPressed
? DS.Colors.accentHover
: isHovered
? DS.Colors.accentText
: DS.Colors.textSecondary
)
.padding(.vertical, 8)
.padding(.horizontal, 12)
.background(
Capsule()
.fill(buttonBackgroundColor(isPressed: configuration.isPressed))
)
.scaleEffect(configuration.isPressed ? 0.97 : 1.0)
.animation(.easeOut(duration: DS.Animation.fast), value: configuration.isPressed)
.animation(.easeOut(duration: DS.Animation.fast), value: isHovered)
.onHover { hovering in
isHovered = hovering
if hovering { NSCursor.pointingHand.push() } else { NSCursor.pop() }
}
}
private func buttonBackgroundColor(isPressed: Bool) -> Color {
if isPressed {
return DS.Colors.surface3
} else if isHovered {
return DS.Colors.surface2
} else {
return Color.clear
}
}
}
/// Text button the lowest-emphasis button style. No background on any
/// state, not even hover. Only the text color changes. Used for: "restart",
/// "skip", "cancel", and other truly minimal inline actions where a
/// background would add too much visual weight.
struct DSTextButtonStyle: ButtonStyle {
var fontSize: CGFloat = 14
@State private var isHovered = false
func makeBody(configuration: Configuration) -> some View {
configuration.label
.font(.system(size: fontSize, weight: .medium))
.foregroundColor(
configuration.isPressed
? DS.Colors.textPrimary
: isHovered
? DS.Colors.textPrimary
: DS.Colors.textTertiary
)
.animation(.easeOut(duration: DS.Animation.fast), value: configuration.isPressed)
.animation(.easeOut(duration: DS.Animation.fast), value: isHovered)
.onHover { hovering in
isHovered = hovering
if hovering { NSCursor.pointingHand.push() } else { NSCursor.pop() }
}
}
}
/// Outlined button medium emphasis, used where a border helps define
/// the button's bounds. Used for: display selector, copy prompt.
struct DSOutlinedButtonStyle: ButtonStyle {
var isFullWidth: Bool = true
@State private var isHovered = false
func makeBody(configuration: Configuration) -> some View {
configuration.label
.font(.system(size: 16, weight: .medium))
.foregroundColor(DS.Colors.textPrimary)
.frame(maxWidth: isFullWidth ? .infinity : nil)
.padding(.vertical, 12)
.padding(.horizontal, isFullWidth ? 0 : 16)
.background(
Capsule()
.fill(buttonBackgroundColor(isPressed: configuration.isPressed))
)
.overlay(
Capsule()
.stroke(
borderColor(isPressed: configuration.isPressed),
lineWidth: 1
)
)
.scaleEffect(configuration.isPressed ? 0.97 : 1.0)
.animation(.easeOut(duration: DS.Animation.fast), value: configuration.isPressed)
.animation(.easeOut(duration: DS.Animation.fast), value: isHovered)
.onHover { hovering in
isHovered = hovering
if hovering { NSCursor.pointingHand.push() } else { NSCursor.pop() }
}
}
private func buttonBackgroundColor(isPressed: Bool) -> Color {
if isPressed {
return DS.Colors.surface3
} else if isHovered {
return DS.Colors.surface2
} else {
return DS.Colors.surface1
}
}
private func borderColor(isPressed: Bool) -> Color {
if isPressed || isHovered {
return DS.Colors.borderStrong
} else {
return DS.Colors.borderSubtle
}
}
}
/// Destructive button for dangerous/irreversible actions (close session, delete).
/// Red-tinted background that intensifies on hover and press.
struct DSDestructiveButtonStyle: ButtonStyle {
@State private var isHovered = false
func makeBody(configuration: Configuration) -> some View {
configuration.label
.font(.system(size: 16, weight: .medium))
.foregroundColor(
isHovered || configuration.isPressed
? .white
: DS.Colors.destructiveText
)
.padding(.vertical, 10)
.padding(.horizontal, 16)
.background(
Capsule()
.fill(buttonBackgroundColor(isPressed: configuration.isPressed))
)
.overlay(
Capsule()
.stroke(
borderColor(isPressed: configuration.isPressed),
lineWidth: 1
)
)
.scaleEffect(configuration.isPressed ? 0.97 : 1.0)
.animation(.easeOut(duration: DS.Animation.fast), value: configuration.isPressed)
.animation(.easeOut(duration: DS.Animation.fast), value: isHovered)
.onHover { hovering in
isHovered = hovering
if hovering { NSCursor.pointingHand.push() } else { NSCursor.pop() }
}
}
private func buttonBackgroundColor(isPressed: Bool) -> Color {
if isPressed {
return DS.Colors.destructive.opacity(0.40)
} else if isHovered {
return DS.Colors.destructive.opacity(0.30)
} else {
return DS.Colors.destructive.opacity(0.10)
}
}
private func borderColor(isPressed: Bool) -> Color {
if isPressed || isHovered {
return DS.Colors.destructive.opacity(0.40)
} else {
return DS.Colors.destructive.opacity(0.15)
}
}
}
/// Icon-only button compact circular button for utility actions.
/// Used for: close button (x), send message, small toolbar actions.
struct DSIconButtonStyle: ButtonStyle {
var size: CGFloat = 28
var isDestructiveOnHover: Bool = false
var tooltipText: String? = nil
/// Controls horizontal alignment of the tooltip relative to the button.
/// Use `.leading` for buttons near the left edge of the window (tooltip extends right),
/// `.trailing` for buttons near the right edge (tooltip extends left),
/// and `.center` for buttons in the middle.
var tooltipAlignment: Alignment = .center
@State private var isHovered = false
@State private var isTooltipVisible = false
@State private var tooltipShowWorkItem: DispatchWorkItem? = nil
func makeBody(configuration: Configuration) -> some View {
configuration.label
.font(.system(size: size * 0.43, weight: .semibold))
.foregroundColor(iconColor(isPressed: configuration.isPressed))
.frame(width: size, height: size)
.background(
Circle()
.fill(circleBackgroundColor(isPressed: configuration.isPressed))
)
.overlay(
Circle()
.stroke(circleBorderColor(isPressed: configuration.isPressed), lineWidth: 1)
)
.scaleEffect(configuration.isPressed ? 0.93 : 1.0)
.animation(.easeOut(duration: DS.Animation.fast), value: configuration.isPressed)
.animation(.easeOut(duration: DS.Animation.fast), value: isHovered)
.contentShape(Circle())
// Cursor change via AppKit cursor rects more reliable than NSCursor.push/pop
// because cursor rects are managed at the window level and don't conflict
// with SwiftUI's internal cursor handling.
.overlay(PointerCursorView())
.onHover { hovering in
isHovered = hovering
// Show the tooltip after a delay (like native tooltips), hide immediately
tooltipShowWorkItem?.cancel()
if hovering {
let workItem = DispatchWorkItem {
withAnimation(.easeOut(duration: 0.15)) {
isTooltipVisible = true
}
}
tooltipShowWorkItem = workItem
DispatchQueue.main.asyncAfter(deadline: .now() + 0.6, execute: workItem)
} else {
withAnimation(.easeOut(duration: 0.1)) {
isTooltipVisible = false
}
}
}
// Custom styled tooltip positioned above the button with enough gap
// to not overlap the button. Horizontally aligned based on tooltipAlignment
// so tooltips near window edges don't clip outside the visible area.
// Uses .allowsHitTesting(false) so the tooltip doesn't interfere
// with the button's hover state.
.overlay(
Group {
if isTooltipVisible, let text = tooltipText, !text.isEmpty {
Text(text)
.font(.system(size: 11, weight: .medium))
.foregroundColor(DS.Colors.textSecondary)
.padding(.horizontal, 10)
.padding(.vertical, 5)
.background(
RoundedRectangle(cornerRadius: 6)
.fill(DS.Colors.surface3.opacity(0.85))
)
.overlay(
ZStack {
RoundedRectangle(cornerRadius: 6)
.stroke(Color.white.opacity(0.20), lineWidth: 0.8)
RoundedRectangle(cornerRadius: 6)
.trim(from: 0, to: 0.5)
.stroke(
LinearGradient(
colors: [
Color.white.opacity(0.10),
Color.white.opacity(0.02)
],
startPoint: .top,
endPoint: .bottom
),
lineWidth: 0.8
)
}
)
.shadow(color: Color.black.opacity(0.42), radius: 14, x: 0, y: 8)
.shadow(color: Color.black.opacity(0.26), radius: 4, x: 0, y: 2)
.fixedSize()
.offset(y: -(size / 2 + 20))
.allowsHitTesting(false)
.transition(.opacity)
}
},
alignment: tooltipAlignment
)
}
private func iconColor(isPressed: Bool) -> Color {
if isDestructiveOnHover && (isHovered || isPressed) {
return .white
}
if isPressed {
return DS.Colors.textPrimary
} else if isHovered {
return DS.Colors.textPrimary
} else {
return DS.Colors.textSecondary
}
}
private func circleBackgroundColor(isPressed: Bool) -> Color {
if isDestructiveOnHover {
if isPressed {
return DS.Colors.destructive.opacity(0.40)
} else if isHovered {
return DS.Colors.destructive.opacity(0.30)
} else {
return DS.Colors.surface2
}
}
if isPressed {
return DS.Colors.surface4
} else if isHovered {
return DS.Colors.surface3
} else {
return DS.Colors.surface2
}
}
private func circleBorderColor(isPressed: Bool) -> Color {
if isDestructiveOnHover && (isHovered || isPressed) {
return DS.Colors.destructive.opacity(0.30)
}
if isPressed || isHovered {
return DS.Colors.borderStrong
} else {
return DS.Colors.borderSubtle.opacity(0.5)
}
}
}
// MARK: - Convenience View Extensions
extension View {
/// Applies the primary button style (accent-colored CTA).
func dsPrimaryButtonStyle(isFullWidth: Bool = true) -> some View {
self.buttonStyle(DSPrimaryButtonStyle(isFullWidth: isFullWidth))
}
/// Applies the secondary button style (surface-colored supporting action).
func dsSecondaryButtonStyle(isFullWidth: Bool = true) -> some View {
self.buttonStyle(DSSecondaryButtonStyle(isFullWidth: isFullWidth))
}
/// Applies the tertiary/ghost button style (subtle hover background).
func dsTertiaryButtonStyle() -> some View {
self.buttonStyle(DSTertiaryButtonStyle())
}
/// Applies the text-only button style (no background ever, just color change).
func dsTextButtonStyle(fontSize: CGFloat = 14) -> some View {
self.buttonStyle(DSTextButtonStyle(fontSize: fontSize))
}
/// Applies the outlined button style (bordered, medium emphasis).
func dsOutlinedButtonStyle(isFullWidth: Bool = true) -> some View {
self.buttonStyle(DSOutlinedButtonStyle(isFullWidth: isFullWidth))
}
/// Applies the destructive button style (red-tinted danger action).
func dsDestructiveButtonStyle() -> some View {
self.buttonStyle(DSDestructiveButtonStyle())
}
/// Applies the icon-only button style (compact circle).
/// `tooltipAlignment` controls where the tooltip sits horizontally relative to the button:
/// `.leading` for left-edge buttons, `.trailing` for right-edge buttons, `.center` for middle.
func dsIconButtonStyle(size: CGFloat = 28, isDestructiveOnHover: Bool = false, tooltip: String? = nil, tooltipAlignment: Alignment = .center) -> some View {
self.buttonStyle(DSIconButtonStyle(size: size, isDestructiveOnHover: isDestructiveOnHover, tooltipText: tooltip, tooltipAlignment: tooltipAlignment))
}
/// Attaches the shared pointing-hand cursor treatment used across interactive controls.
/// Disabled controls can opt out so they keep the default arrow cursor.
func pointerCursor(isEnabled: Bool = true) -> some View {
self.overlay {
if isEnabled {
PointerCursorView()
}
}
}
}
// MARK: - Buddy Composer Visual Style
enum BuddyComposerVisualStyle {
static let waveformLeadingColor = Color(hex: "#F3FBFF")
static let waveformTrailingColor = Color(hex: "#8FD2FF")
static let waveformGlowColor = Color(hex: "#AEE3FF")
}
// MARK: - Pointer Cursor (AppKit Bridge)
/// Uses AppKit's cursor rect system to reliably show a pointing hand cursor.
/// More reliable than NSCursor.push()/pop() inside SwiftUI's .onHover because
/// cursor rects are managed at the window level and don't conflict with
/// SwiftUI's internal cursor handling.
private class PointerCursorNSView: NSView {
override func resetCursorRects() {
super.resetCursorRects()
addCursorRect(bounds, cursor: .pointingHand)
}
override func hitTest(_ point: NSPoint) -> NSView? {
return nil
}
}
private struct PointerCursorView: NSViewRepresentable {
func makeNSView(context: Context) -> NSView {
return PointerCursorNSView()
}
func updateNSView(_ nsView: NSView, context: Context) {
// Invalidate cursor rects when the view updates (e.g., resizes)
// so AppKit recalculates the cursor area.
nsView.window?.invalidateCursorRects(for: nsView)
}
}
// MARK: - I-Beam Cursor (AppKit Bridge)
/// Uses AppKit's cursor rect system to reliably show an I-beam (text selection) cursor.
/// Same approach as PointerCursorView cursor rects are managed at the window level
/// and don't conflict with SwiftUI's internal cursor handling.
/// Unlike NSCursor.push()/pop() in .onHover, this avoids cursor stack imbalance
/// when the mouse moves quickly between views.
private class IBeamCursorNSView: NSView {
override func resetCursorRects() {
super.resetCursorRects()
addCursorRect(bounds, cursor: .iBeam)
}
/// Pass through all mouse events so the TextField underneath still receives
/// focus, clicks, and text selection. Cursor rects are registered with the
/// window (via resetCursorRects) and work independently of hit testing.
override func hitTest(_ point: NSPoint) -> NSView? {
return nil
}
}
struct IBeamCursorView: NSViewRepresentable {
func makeNSView(context: Context) -> NSView {
return IBeamCursorNSView()
}
func updateNSView(_ nsView: NSView, context: Context) {
// Invalidate cursor rects when the view updates (e.g., resizes)
// so AppKit recalculates the cursor area.
nsView.window?.invalidateCursorRects(for: nsView)
}
}
// MARK: - Native Tooltip
/// Uses AppKit's `NSView.toolTip` to show a tooltip on hover.
/// SwiftUI's `.help()` conflicts with `.onHover` tracking areas, so
/// this bridges directly to AppKit's tooltip system which works independently.
private struct NativeTooltipView: NSViewRepresentable {
let tooltip: String
func makeNSView(context: Context) -> NSView {
let view = NSView()
view.toolTip = tooltip
return view
}
func updateNSView(_ nsView: NSView, context: Context) {
nsView.toolTip = tooltip
}
}
extension View {
/// Attaches a native macOS tooltip that works even alongside `.onHover`.
func nativeTooltip(_ text: String?) -> some View {
if let text = text, !text.isEmpty {
return AnyView(self.overlay(NativeTooltipView(tooltip: text)))
} else {
return AnyView(self)
}
}
}
// MARK: - Color Utilities
extension Color {
/// Create a Color from a hex string like "#FF5733" or "FF5733".
init(hex: String) {
let hexSanitized = hex.trimmingCharacters(in: .whitespacesAndNewlines)
.replacingOccurrences(of: "#", with: "")
var rgbValue: UInt64 = 0
Scanner(string: hexSanitized).scanHexInt64(&rgbValue)
let red = Double((rgbValue & 0xFF0000) >> 16) / 255.0
let green = Double((rgbValue & 0x00FF00) >> 8) / 255.0
let blue = Double(rgbValue & 0x0000FF) / 255.0
self.init(red: red, green: green, blue: blue)
}
/// Returns a lighter version of this color by blending toward white.
/// `fraction` is 0.0 (no change) to 1.0 (pure white).
func blendedWithWhite(fraction: Double) -> Color {
// Convert to NSColor to access RGB components for blending
guard let nsColor = NSColor(self).usingColorSpace(.sRGB) else { return self }
let red = nsColor.redComponent + (1.0 - nsColor.redComponent) * fraction
let green = nsColor.greenComponent + (1.0 - nsColor.greenComponent) * fraction
let blue = nsColor.blueComponent + (1.0 - nsColor.blueComponent) * fraction
return Color(red: red, green: green, blue: blue)
}
}
@@ -0,0 +1,335 @@
//
// ElementLocationDetector.swift
// leanring-buddy
//
// Uses Claude's Computer Use API to identify the screen location of UI elements
// in screenshots. When a user asks about a visible element (e.g., "click the
// blue button"), this detects the element's coordinates so the buddy can
// animate to it and point at it.
//
import AppKit
import Foundation
/// Detects the screen location of UI elements in screenshots using Claude's Computer Use API.
/// The Computer Use tool definition activates Claude's specialized pixel-counting training,
/// which is significantly more accurate than regular vision API coordinate extraction.
///
/// **Aspect ratio matching**: Instead of always resizing to 1024x768 (4:3), we pick the
/// Anthropic-recommended resolution closest to the display's actual aspect ratio. Most
/// Macs are 16:10 1280x800. This avoids distorting the image Claude sees, which
/// significantly improves X-axis coordinate accuracy.
class ElementLocationDetector {
private let apiKey: String
private let apiURL: URL
private let model: String
private let session: URLSession
/// Anthropic-recommended resolutions for Computer Use, paired with their aspect ratios.
/// We pick the one closest to the actual display aspect ratio to avoid distortion.
/// Higher resolutions get downsampled by the API and degrade precision, so these
/// are intentionally small.
private static let supportedComputerUseResolutions: [(width: Int, height: Int, aspectRatio: Double)] = [
(1024, 768, 1024.0 / 768.0), // 4:3 = 1.333 (legacy displays)
(1280, 800, 1280.0 / 800.0), // 16:10 = 1.600 (MacBook Air, MacBook Pro, most Macs)
(1366, 768, 1366.0 / 768.0) // ~16:9 = 1.779 (external monitors, ultrawide fallback)
]
init(apiKey: String, model: String = "claude-sonnet-4-6") {
self.apiKey = apiKey
self.apiURL = URL(string: "https://api.anthropic.com/v1/messages")!
self.model = model
let config = URLSessionConfiguration.default
config.timeoutIntervalForRequest = 15
config.timeoutIntervalForResource = 20
config.waitsForConnectivity = false
config.urlCache = nil
config.httpCookieStorage = nil
self.session = URLSession(configuration: config)
}
/// Detects the screen location of a UI element the user is asking about.
///
/// - Parameters:
/// - screenshotData: JPEG or PNG screenshot data from ScreenCaptureKit
/// - userQuestion: The user's voice transcript (e.g., "How do I add a project?")
/// - displayWidthInPoints: The captured display's width in screen points
/// - displayHeightInPoints: The captured display's height in screen points
///
/// - Returns: A `CGPoint` in display-local macOS coordinates (bottom-left origin) if an
/// element was identified, or `nil` if no element was found or detection failed.
func detectElementLocation(
screenshotData: Data,
userQuestion: String,
displayWidthInPoints: Int,
displayHeightInPoints: Int
) async -> CGPoint? {
// Pick the Computer Use resolution that best matches this display's aspect ratio.
// This avoids stretching the screenshot (e.g., squishing a 16:10 Mac display
// into 4:3), which would distort the image Claude sees and degrade X-axis accuracy.
let computerUseResolution = bestComputerUseResolution(
forDisplayWidth: displayWidthInPoints,
displayHeight: displayHeightInPoints
)
print("🎯 ElementLocationDetector: display is \(displayWidthInPoints)x\(displayHeightInPoints) " +
"(ratio \(String(format: "%.3f", Double(displayWidthInPoints) / Double(displayHeightInPoints)))), " +
"using Computer Use resolution \(computerUseResolution.width)x\(computerUseResolution.height)")
// Resize the screenshot to the chosen Computer Use resolution
guard let resizedScreenshotData = resizeScreenshotForComputerUse(
originalImageData: screenshotData,
targetWidth: computerUseResolution.width,
targetHeight: computerUseResolution.height
) else {
print("⚠️ ElementLocationDetector: failed to resize screenshot")
return nil
}
// Make the Computer Use API call with the matching resolution declared
guard let computerUseCoordinate = await callComputerUseAPI(
resizedScreenshotData: resizedScreenshotData,
userQuestion: userQuestion,
declaredDisplayWidth: computerUseResolution.width,
declaredDisplayHeight: computerUseResolution.height
) else {
return nil
}
// Clamp coordinates to the valid range Claude occasionally returns
// values slightly outside the declared display dimensions, which would
// map to off-screen positions after scaling.
let clampedX = max(0, min(computerUseCoordinate.x, CGFloat(computerUseResolution.width)))
let clampedY = max(0, min(computerUseCoordinate.y, CGFloat(computerUseResolution.height)))
// Scale coordinates from the Computer Use resolution back to actual display point dimensions
let scaledX = (clampedX / CGFloat(computerUseResolution.width)) * CGFloat(displayWidthInPoints)
let scaledYTopLeftOrigin = (clampedY / CGFloat(computerUseResolution.height)) * CGFloat(displayHeightInPoints)
// Convert from top-left origin (Computer Use / CoreGraphics) to bottom-left origin (AppKit)
let scaledYBottomLeftOrigin = CGFloat(displayHeightInPoints) - scaledYTopLeftOrigin
print("🎯 ElementLocationDetector: mapped (\(Int(clampedX)), \(Int(clampedY))) in " +
"\(computerUseResolution.width)x\(computerUseResolution.height)" +
"(\(Int(scaledX)), \(Int(scaledYBottomLeftOrigin))) in " +
"\(displayWidthInPoints)x\(displayHeightInPoints) display-local AppKit coords")
return CGPoint(x: scaledX, y: scaledYBottomLeftOrigin)
}
// MARK: - Private Helpers
/// Picks the Anthropic-recommended Computer Use resolution whose aspect ratio
/// is closest to the actual display, minimizing image distortion.
private func bestComputerUseResolution(
forDisplayWidth displayWidth: Int,
displayHeight: Int
) -> (width: Int, height: Int) {
let displayAspectRatio = Double(displayWidth) / Double(max(1, displayHeight))
var bestWidth = 1280
var bestHeight = 800
var smallestAspectRatioDifference = Double.greatestFiniteMagnitude
for resolution in Self.supportedComputerUseResolutions {
let difference = abs(displayAspectRatio - resolution.aspectRatio)
if difference < smallestAspectRatioDifference {
smallestAspectRatioDifference = difference
bestWidth = resolution.width
bestHeight = resolution.height
}
}
return (width: bestWidth, height: bestHeight)
}
/// Calls the Claude Computer Use API with a resized screenshot and user question.
/// Returns the raw coordinate from Claude's response in the declared resolution space, or nil.
private func callComputerUseAPI(
resizedScreenshotData: Data,
userQuestion: String,
declaredDisplayWidth: Int,
declaredDisplayHeight: Int
) async -> CGPoint? {
var request = URLRequest(url: apiURL)
request.httpMethod = "POST"
request.timeoutInterval = 15
request.setValue(apiKey, forHTTPHeaderField: "x-api-key")
request.setValue("2023-06-01", forHTTPHeaderField: "anthropic-version")
request.setValue("application/json", forHTTPHeaderField: "Content-Type")
// The beta header activates Computer Use capabilities and the specialized
// pixel-counting training that makes coordinate detection accurate.
request.setValue("computer-use-2025-11-24", forHTTPHeaderField: "anthropic-beta")
// Detect image media type (PNG vs JPEG)
let mediaType = detectImageMediaType(for: resizedScreenshotData)
let base64Screenshot = resizedScreenshotData.base64EncodedString()
let userPrompt = """
The user asked this question while looking at their screen: "\(userQuestion)"
Look at the screenshot. If there is a specific UI element (button, link, menu item, text field, icon, etc.) that the user should interact with or is asking about, click on that element.
If the question is purely conceptual (e.g., "what does HTML mean?") and there's no specific element to point to, just respond with text saying "no specific element".
"""
let body: [String: Any] = [
"model": model,
"max_tokens": 256,
"tools": [
[
"type": "computer_20251124",
"name": "computer",
"display_width_px": declaredDisplayWidth,
"display_height_px": declaredDisplayHeight
]
],
"messages": [
[
"role": "user",
"content": [
[
"type": "image",
"source": [
"type": "base64",
"media_type": mediaType,
"data": base64Screenshot
]
],
[
"type": "text",
"text": userPrompt
]
]
]
]
]
do {
let bodyData = try JSONSerialization.data(withJSONObject: body)
request.httpBody = bodyData
let payloadMB = Double(bodyData.count) / 1_048_576.0
print("🎯 ElementLocationDetector: sending \(String(format: "%.1f", payloadMB))MB request " +
"(declared \(declaredDisplayWidth)x\(declaredDisplayHeight))")
let (data, response) = try await session.data(for: request)
guard let httpResponse = response as? HTTPURLResponse,
(200...299).contains(httpResponse.statusCode) else {
let statusCode = (response as? HTTPURLResponse)?.statusCode ?? -1
let errorBody = String(data: data, encoding: .utf8) ?? "unknown"
print("⚠️ ElementLocationDetector: API error \(statusCode): \(errorBody.prefix(200))")
return nil
}
return parseCoordinateFromResponse(data: data)
} catch {
print("⚠️ ElementLocationDetector: request failed: \(error.localizedDescription)")
return nil
}
}
/// Parses the Computer Use API response to extract click coordinates.
/// Claude returns a `tool_use` content block with `{"action": "left_click", "coordinate": [x, y]}`.
/// If Claude returns text instead (no element found), returns nil.
private func parseCoordinateFromResponse(data: Data) -> CGPoint? {
guard let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let contentBlocks = json["content"] as? [[String: Any]] else {
print("⚠️ ElementLocationDetector: could not parse response JSON")
return nil
}
// Look for a tool_use content block (Claude's Computer Use response format)
for block in contentBlocks {
guard let blockType = block["type"] as? String,
blockType == "tool_use",
let input = block["input"] as? [String: Any],
let coordinate = input["coordinate"] as? [NSNumber],
coordinate.count == 2 else {
continue
}
let x = CGFloat(coordinate[0].doubleValue)
let y = CGFloat(coordinate[1].doubleValue)
print("🎯 ElementLocationDetector: raw coordinate (\(Int(x)), \(Int(y)))")
return CGPoint(x: x, y: y)
}
// No tool_use block found Claude responded with text (no element to point at)
print("🎯 ElementLocationDetector: no specific element detected (conceptual question)")
return nil
}
/// Resizes screenshot data to the specified Computer Use resolution.
/// The target resolution should match the display's aspect ratio to avoid
/// distortion that degrades coordinate accuracy.
///
/// **Critical Retina fix**: Uses `NSBitmapImageRep` directly instead of
/// `NSImage.lockFocus()`. On Retina displays (2x backing scale), lockFocus
/// creates a bitmap at 2× the declared size (e.g., 2560×1600 for a 1280×800
/// NSImage). This means the JPEG sent to Claude would be 2× larger than the
/// resolution declared in the Computer Use tool definition, causing Claude's
/// pixel-counting to return coordinates in the wrong scale.
private func resizeScreenshotForComputerUse(
originalImageData: Data,
targetWidth: Int,
targetHeight: Int
) -> Data? {
guard let originalImage = NSImage(data: originalImageData) else { return nil }
// Create a bitmap representation with exact pixel dimensions.
// This bypasses NSImage's Retina-aware coordinate system which would
// otherwise double the actual pixel count on 2x displays.
guard let bitmapRep = NSBitmapImageRep(
bitmapDataPlanes: nil,
pixelsWide: targetWidth,
pixelsHigh: targetHeight,
bitsPerSample: 8,
samplesPerPixel: 4,
hasAlpha: true,
isPlanar: false,
colorSpaceName: .deviceRGB,
bytesPerRow: 0,
bitsPerPixel: 0
) else {
return nil
}
// Set the point size to match pixel dimensions (1:1, no Retina scaling).
bitmapRep.size = NSSize(width: targetWidth, height: targetHeight)
// Draw the original image into the exact-pixel-dimension bitmap
NSGraphicsContext.saveGraphicsState()
let graphicsContext = NSGraphicsContext(bitmapImageRep: bitmapRep)
NSGraphicsContext.current = graphicsContext
graphicsContext?.imageInterpolation = .high
originalImage.draw(
in: NSRect(x: 0, y: 0, width: targetWidth, height: targetHeight),
from: NSRect(origin: .zero, size: originalImage.size),
operation: .copy,
fraction: 1.0
)
NSGraphicsContext.restoreGraphicsState()
guard let jpegData = bitmapRep.representation(using: .jpeg, properties: [.compressionFactor: 0.85]) else {
return nil
}
return jpegData
}
/// Detects MIME type by inspecting the first bytes of image data.
private func detectImageMediaType(for imageData: Data) -> String {
if imageData.count >= 4 {
let pngSignature: [UInt8] = [0x89, 0x50, 0x4E, 0x47]
let firstFourBytes = [UInt8](imageData.prefix(4))
if firstFourBytes == pngSignature {
return "image/png"
}
}
return "image/jpeg"
}
}
+81
View File
@@ -0,0 +1,81 @@
//
// ElevenLabsTTSClient.swift
// leanring-buddy
//
// Streams text-to-speech audio from ElevenLabs and plays it back
// through the system audio output. Uses the streaming endpoint so
// playback begins before the full audio has been generated.
//
import AVFoundation
import Foundation
@MainActor
final class ElevenLabsTTSClient {
private let proxyURL: URL
private let session: URLSession
/// The audio player for the current TTS playback. Kept alive so the
/// audio finishes playing even if the caller doesn't hold a reference.
private var audioPlayer: AVAudioPlayer?
init(proxyURL: String) {
self.proxyURL = URL(string: proxyURL)!
let configuration = URLSessionConfiguration.default
configuration.timeoutIntervalForRequest = 30
configuration.timeoutIntervalForResource = 60
self.session = URLSession(configuration: configuration)
}
/// Sends `text` to ElevenLabs TTS and plays the resulting audio.
/// Throws on network or decoding errors. Cancellation-safe.
func speakText(_ text: String) async throws {
var request = URLRequest(url: proxyURL)
request.httpMethod = "POST"
request.setValue("application/json", forHTTPHeaderField: "Content-Type")
request.setValue("audio/mpeg", forHTTPHeaderField: "Accept")
let body: [String: Any] = [
"text": text,
"model_id": "eleven_flash_v2_5",
"voice_settings": [
"stability": 0.5,
"similarity_boost": 0.75
]
]
request.httpBody = try JSONSerialization.data(withJSONObject: body)
let (data, response) = try await session.data(for: request)
guard let httpResponse = response as? HTTPURLResponse else {
throw NSError(domain: "ElevenLabsTTS", code: -1,
userInfo: [NSLocalizedDescriptionKey: "Invalid response"])
}
guard (200...299).contains(httpResponse.statusCode) else {
let errorBody = String(data: data, encoding: .utf8) ?? "Unknown error"
throw NSError(domain: "ElevenLabsTTS", code: httpResponse.statusCode,
userInfo: [NSLocalizedDescriptionKey: "TTS API error (\(httpResponse.statusCode)): \(errorBody)"])
}
try Task.checkCancellation()
let player = try AVAudioPlayer(data: data)
self.audioPlayer = player
player.play()
print("🔊 ElevenLabs TTS: playing \(data.count / 1024)KB audio")
}
/// Whether TTS audio is currently playing back.
var isPlaying: Bool {
audioPlayer?.isPlaying ?? false
}
/// Stops any in-progress playback immediately.
func stopPlayback() {
audioPlayer?.stop()
audioPlayer = nil
}
}
@@ -0,0 +1,132 @@
//
// GlobalPushToTalkShortcutMonitor.swift
// leanring-buddy
//
// Captures push-to-talk keyboard shortcuts while makesomething is running in the
// background. Uses a listen-only CGEvent tap so modifier-only shortcuts like
// ctrl + option behave more like a real system-wide voice tool.
//
import AppKit
import Combine
import CoreGraphics
import Foundation
final class GlobalPushToTalkShortcutMonitor: ObservableObject {
let shortcutTransitionPublisher = PassthroughSubject<BuddyPushToTalkShortcut.ShortcutTransition, Never>()
private var globalEventTap: CFMachPort?
private var globalEventTapRunLoopSource: CFRunLoopSource?
/// Mutated exclusively from the CGEvent tap callback, which runs on
/// `CFRunLoopGetMain()` and therefore always executes on the main thread.
/// Published so the overlay can hide immediately on key release without
/// waiting for the async dictation state pipeline to catch up.
@Published private(set) var isShortcutCurrentlyPressed = false
deinit {
stop()
}
func start() {
// If the event tap is already running, don't restart it.
// Restarting resets isShortcutCurrentlyPressed, which would kill
// the waveform overlay mid-press when the permission poller calls
// refreshAllPermissions start() every few seconds.
guard globalEventTap == nil else { return }
let monitoredEventTypes: [CGEventType] = [.flagsChanged, .keyDown, .keyUp]
let eventMask = monitoredEventTypes.reduce(CGEventMask(0)) { currentMask, eventType in
currentMask | (CGEventMask(1) << eventType.rawValue)
}
let eventTapCallback: CGEventTapCallBack = { _, eventType, event, userInfo in
guard let userInfo else {
return Unmanaged.passUnretained(event)
}
let globalPushToTalkShortcutMonitor = Unmanaged<GlobalPushToTalkShortcutMonitor>
.fromOpaque(userInfo)
.takeUnretainedValue()
return globalPushToTalkShortcutMonitor.handleGlobalEventTap(
eventType: eventType,
event: event
)
}
guard let globalEventTap = CGEvent.tapCreate(
tap: .cgSessionEventTap,
place: .headInsertEventTap,
options: .listenOnly,
eventsOfInterest: eventMask,
callback: eventTapCallback,
userInfo: Unmanaged.passUnretained(self).toOpaque()
) else {
print("⚠️ Global push-to-talk: couldn't create CGEvent tap")
return
}
guard let globalEventTapRunLoopSource = CFMachPortCreateRunLoopSource(
kCFAllocatorDefault,
globalEventTap,
0
) else {
CFMachPortInvalidate(globalEventTap)
print("⚠️ Global push-to-talk: couldn't create event tap run loop source")
return
}
self.globalEventTap = globalEventTap
self.globalEventTapRunLoopSource = globalEventTapRunLoopSource
CFRunLoopAddSource(CFRunLoopGetMain(), globalEventTapRunLoopSource, .commonModes)
CGEvent.tapEnable(tap: globalEventTap, enable: true)
}
func stop() {
isShortcutCurrentlyPressed = false
if let globalEventTapRunLoopSource {
CFRunLoopRemoveSource(CFRunLoopGetMain(), globalEventTapRunLoopSource, .commonModes)
self.globalEventTapRunLoopSource = nil
}
if let globalEventTap {
CFMachPortInvalidate(globalEventTap)
self.globalEventTap = nil
}
}
private func handleGlobalEventTap(
eventType: CGEventType,
event: CGEvent
) -> Unmanaged<CGEvent>? {
if eventType == .tapDisabledByTimeout || eventType == .tapDisabledByUserInput {
if let globalEventTap {
CGEvent.tapEnable(tap: globalEventTap, enable: true)
}
return Unmanaged.passUnretained(event)
}
let eventKeyCode = UInt16(event.getIntegerValueField(.keyboardEventKeycode))
let shortcutTransition = BuddyPushToTalkShortcut.shortcutTransition(
for: eventType,
keyCode: eventKeyCode,
modifierFlagsRawValue: event.flags.rawValue,
wasShortcutPreviouslyPressed: isShortcutCurrentlyPressed
)
switch shortcutTransition {
case .none:
break
case .pressed:
isShortcutCurrentlyPressed = true
shortcutTransitionPublisher.send(.pressed)
case .released:
isShortcutCurrentlyPressed = false
shortcutTransitionPublisher.send(.released)
}
return Unmanaged.passUnretained(event)
}
}
+20
View File
@@ -0,0 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>LSUIElement</key>
<true/>
<key>SUFeedURL</key>
<string>https://raw.githubusercontent.com/julianjear/makesomething-mac-app/main/appcast.xml</string>
<key>SUPublicEDKey</key>
<string>/l3d2rw5ZZFRU3AadP/w2Zf8FHfhA6bKv16BQOV5OSk=</string>
<key>VoiceTranscriptionProvider</key>
<string>assemblyai</string>
<key>NSMicrophoneUsageDescription</key>
<string>Clicky uses your microphone so you can talk to it</string>
<key>NSScreenCaptureUsageDescription</key>
<string>Clicky needs screen recording access to see your screen and help you.</string>
<key>NSSpeechRecognitionUsageDescription</key>
<string>Clicky uses speech recognition to transcribe your voice when you talk to it</string>
</dict>
</plist>
+243
View File
@@ -0,0 +1,243 @@
//
// MenuBarPanelManager.swift
// leanring-buddy
//
// Manages the NSStatusItem (menu bar icon) and a custom borderless NSPanel
// that drops down below it when clicked. The panel hosts a SwiftUI view
// (CompanionPanelView) via NSHostingView. Uses the same NSPanel pattern as
// FloatingSessionButton and GlobalPushToTalkOverlay for consistency.
//
// The panel is non-activating so it does not steal focus from the user's
// current app, and auto-dismisses when the user clicks outside.
//
import AppKit
import SwiftUI
extension Notification.Name {
static let clickyDismissPanel = Notification.Name("clickyDismissPanel")
}
/// Custom NSPanel subclass that can become the key window even with
/// .nonactivatingPanel style, allowing text fields to receive focus.
private class KeyablePanel: NSPanel {
override var canBecomeKey: Bool { true }
}
@MainActor
final class MenuBarPanelManager: NSObject {
private var statusItem: NSStatusItem?
private var panel: NSPanel?
private var clickOutsideMonitor: Any?
private var dismissPanelObserver: NSObjectProtocol?
private let companionManager: CompanionManager
private let panelWidth: CGFloat = 320
private let panelHeight: CGFloat = 380
init(companionManager: CompanionManager) {
self.companionManager = companionManager
super.init()
createStatusItem()
dismissPanelObserver = NotificationCenter.default.addObserver(
forName: .clickyDismissPanel,
object: nil,
queue: .main
) { [weak self] _ in
self?.hidePanel()
}
}
deinit {
if let monitor = clickOutsideMonitor {
NSEvent.removeMonitor(monitor)
}
if let observer = dismissPanelObserver {
NotificationCenter.default.removeObserver(observer)
}
}
// MARK: - Status Item
private func createStatusItem() {
statusItem = NSStatusBar.system.statusItem(withLength: NSStatusItem.squareLength)
guard let button = statusItem?.button else { return }
button.image = makeClickyMenuBarIcon()
button.image?.isTemplate = true
button.action = #selector(statusItemClicked)
button.target = self
}
/// Draws the clicky triangle as a menu bar icon. Uses the same shape
/// and rotation as the in-app cursor so the menu bar icon matches.
private func makeClickyMenuBarIcon() -> NSImage {
let iconSize: CGFloat = 18
let image = NSImage(size: NSSize(width: iconSize, height: iconSize))
image.lockFocus()
let triangleSize = iconSize * 0.7
let cx = iconSize * 0.50
let cy = iconSize * 0.50
let height = triangleSize * sqrt(3.0) / 2.0
let top = CGPoint(x: cx, y: cy + height / 1.5)
let bottomLeft = CGPoint(x: cx - triangleSize / 2, y: cy - height / 3)
let bottomRight = CGPoint(x: cx + triangleSize / 2, y: cy - height / 3)
let angle = 35.0 * .pi / 180.0
func rotate(_ point: CGPoint) -> CGPoint {
let dx = point.x - cx, dy = point.y - cy
let cosA = CGFloat(cos(angle)), sinA = CGFloat(sin(angle))
return CGPoint(x: cx + cosA * dx - sinA * dy, y: cy + sinA * dx + cosA * dy)
}
let path = NSBezierPath()
path.move(to: rotate(top))
path.line(to: rotate(bottomLeft))
path.line(to: rotate(bottomRight))
path.close()
NSColor.black.setFill()
path.fill()
image.unlockFocus()
return image
}
/// Opens the panel automatically on app launch so the user sees
/// permissions and the start button right away.
func showPanelOnLaunch() {
// Small delay so the status item has time to appear in the menu bar
DispatchQueue.main.asyncAfter(deadline: .now() + 0.3) {
self.showPanel()
}
}
@objc private func statusItemClicked() {
if let panel, panel.isVisible {
hidePanel()
} else {
showPanel()
}
}
// MARK: - Panel Lifecycle
private func showPanel() {
if panel == nil {
createPanel()
}
positionPanelBelowStatusItem()
panel?.makeKeyAndOrderFront(nil)
panel?.orderFrontRegardless()
installClickOutsideMonitor()
}
private func hidePanel() {
panel?.orderOut(nil)
removeClickOutsideMonitor()
}
private func createPanel() {
let companionPanelView = CompanionPanelView(companionManager: companionManager)
.frame(width: panelWidth)
let hostingView = NSHostingView(rootView: companionPanelView)
hostingView.frame = NSRect(x: 0, y: 0, width: panelWidth, height: panelHeight)
hostingView.wantsLayer = true
hostingView.layer?.backgroundColor = .clear
let menuBarPanel = KeyablePanel(
contentRect: NSRect(x: 0, y: 0, width: panelWidth, height: panelHeight),
styleMask: [.borderless, .nonactivatingPanel],
backing: .buffered,
defer: false
)
menuBarPanel.isFloatingPanel = true
menuBarPanel.level = .floating
menuBarPanel.isOpaque = false
menuBarPanel.backgroundColor = .clear
menuBarPanel.hasShadow = false
menuBarPanel.hidesOnDeactivate = false
menuBarPanel.isExcludedFromWindowsMenu = true
menuBarPanel.collectionBehavior = [.canJoinAllSpaces, .fullScreenAuxiliary]
menuBarPanel.isMovableByWindowBackground = false
menuBarPanel.titleVisibility = .hidden
menuBarPanel.titlebarAppearsTransparent = true
menuBarPanel.contentView = hostingView
panel = menuBarPanel
}
private func positionPanelBelowStatusItem() {
guard let panel else { return }
guard let buttonWindow = statusItem?.button?.window else { return }
let statusItemFrame = buttonWindow.frame
let gapBelowMenuBar: CGFloat = 4
// Calculate the panel's content height from the hosting view's fitting size
// so the panel snugly wraps the SwiftUI content instead of using a fixed height.
let fittingSize = panel.contentView?.fittingSize ?? CGSize(width: panelWidth, height: panelHeight)
let actualPanelHeight = fittingSize.height
// Horizontally center the panel beneath the status item icon
let panelOriginX = statusItemFrame.midX - (panelWidth / 2)
let panelOriginY = statusItemFrame.minY - actualPanelHeight - gapBelowMenuBar
panel.setFrame(
NSRect(x: panelOriginX, y: panelOriginY, width: panelWidth, height: actualPanelHeight),
display: true
)
}
// MARK: - Click Outside Dismissal
/// Installs a global event monitor that hides the panel when the user clicks
/// anywhere outside it the same transient dismissal behavior as NSPopover.
/// Uses a short delay so that system permission dialogs (triggered by Grant
/// buttons in the panel) don't immediately dismiss the panel when they appear.
private func installClickOutsideMonitor() {
removeClickOutsideMonitor()
clickOutsideMonitor = NSEvent.addGlobalMonitorForEvents(
matching: [.leftMouseDown, .rightMouseDown]
) { [weak self] event in
guard let self, let panel = self.panel else { return }
// Check if the click is inside the status item button if so, the
// statusItemClicked handler will toggle the panel, so don't also hide.
let clickLocation = NSEvent.mouseLocation
if panel.frame.contains(clickLocation) {
return
}
// Delay dismissal slightly to avoid closing the panel when
// a system permission dialog appears (e.g. microphone access).
DispatchQueue.main.asyncAfter(deadline: .now() + 0.3) {
guard panel.isVisible else { return }
// If permissions aren't all granted yet, a system dialog
// may have focus don't dismiss during onboarding.
if !self.companionManager.allPermissionsGranted && !NSApp.isActive {
return
}
self.hidePanel()
}
}
}
private func removeClickOutsideMonitor() {
if let monitor = clickOutsideMonitor {
NSEvent.removeMonitor(monitor)
clickOutsideMonitor = nil
}
}
}
+142
View File
@@ -0,0 +1,142 @@
//
// OpenAIAPI.swift
// OpenAI API Implementation
//
import Foundation
/// OpenAI API helper for vision analysis
class OpenAIAPI {
private let apiKey: String
private let apiURL: URL
private let model: String
private let session: URLSession
init(apiKey: String, model: String = "gpt-5.2-2025-12-11") {
self.apiKey = apiKey
self.apiURL = URL(string: "https://api.openai.com/v1/chat/completions")!
self.model = model
// Use .default instead of .ephemeral so TLS session tickets are cached.
// Ephemeral sessions do a full TLS handshake on every request, which causes
// transient -1200 (errSSLPeerHandshakeFail) errors with large image payloads.
// Disable URL/cookie caching to avoid storing responses or credentials on disk.
let config = URLSessionConfiguration.default
config.timeoutIntervalForRequest = 120
config.timeoutIntervalForResource = 300
config.waitsForConnectivity = true
config.urlCache = nil
config.httpCookieStorage = nil
self.session = URLSession(configuration: config)
// Fire a lightweight HEAD request in the background to pre-establish the TLS
// connection. This caches the TLS session ticket so the first real API call
// (which carries a large image payload) doesn't need a cold TLS handshake.
warmUpTLSConnection()
}
/// Sends a no-op HEAD request to the API host to establish and cache a TLS session.
/// Failures are silently ignored this is purely an optimization.
private func warmUpTLSConnection() {
var warmupRequest = URLRequest(url: apiURL)
warmupRequest.httpMethod = "HEAD"
warmupRequest.timeoutInterval = 10
session.dataTask(with: warmupRequest) { _, _, _ in
// Response doesn't matter the TLS handshake is the goal
}.resume()
}
/// Send a vision request to OpenAI with one or more labeled images.
func analyzeImage(
images: [(data: Data, label: String)],
systemPrompt: String,
conversationHistory: [(userPlaceholder: String, assistantResponse: String)] = [],
userPrompt: String
) async throws -> (text: String, duration: TimeInterval) {
let startTime = Date()
// Build request
var request = URLRequest(url: apiURL)
request.httpMethod = "POST"
request.timeoutInterval = 120
request.setValue("Bearer \(apiKey)", forHTTPHeaderField: "Authorization")
request.setValue("application/json", forHTTPHeaderField: "Content-Type")
// Build messages array
var messages: [[String: Any]] = []
// Add system message first
messages.append([
"role": "system",
"content": systemPrompt
])
// Add conversation history
for (userPlaceholder, assistantResponse) in conversationHistory {
messages.append(["role": "user", "content": userPlaceholder])
messages.append(["role": "assistant", "content": assistantResponse])
}
// Build current message with all labeled images + prompt
var contentBlocks: [[String: Any]] = []
for image in images {
contentBlocks.append([
"type": "text",
"text": image.label
])
contentBlocks.append([
"type": "image_url",
"image_url": [
"url": "data:image/jpeg;base64,\(image.data.base64EncodedString())"
]
])
}
contentBlocks.append([
"type": "text",
"text": userPrompt
])
messages.append(["role": "user", "content": contentBlocks])
// Build request body
let body: [String: Any] = [
"model": model,
// `max_tokens` is deprecated/incompatible for some newer OpenAI models.
"max_completion_tokens": 600,
"messages": messages
]
let bodyData = try JSONSerialization.data(withJSONObject: body)
request.httpBody = bodyData
let payloadMB = Double(bodyData.count) / 1_048_576.0
print("🌐 OpenAI request: \(String(format: "%.1f", payloadMB))MB, \(images.count) image(s)")
// Send request
let (data, response) = try await session.data(for: request)
guard let httpResponse = response as? HTTPURLResponse,
(200...299).contains(httpResponse.statusCode) else {
let responseString = String(data: data, encoding: .utf8) ?? "Unknown error"
throw NSError(
domain: "OpenAIAPI",
code: (response as? HTTPURLResponse)?.statusCode ?? -1,
userInfo: [NSLocalizedDescriptionKey: "API Error: \(responseString)"]
)
}
// Parse response
let json = try JSONSerialization.jsonObject(with: data) as? [String: Any]
guard let choices = json?["choices"] as? [[String: Any]],
let firstChoice = choices.first,
let message = firstChoice["message"] as? [String: Any],
let text = message["content"] as? String else {
throw NSError(
domain: "OpenAIAPI",
code: -1,
userInfo: [NSLocalizedDescriptionKey: "Invalid response format"]
)
}
let duration = Date().timeIntervalSince(startTime)
return (text: text, duration: duration)
}
}
@@ -0,0 +1,317 @@
//
// OpenAIAudioTranscriptionProvider.swift
// leanring-buddy
//
// AI transcription provider backed by OpenAI's audio transcription API.
//
import AVFoundation
import Foundation
struct OpenAIAudioTranscriptionProviderError: LocalizedError {
let message: String
var errorDescription: String? {
message
}
}
final class OpenAIAudioTranscriptionProvider: BuddyTranscriptionProvider {
private let apiKey = AppBundleConfiguration.stringValue(forKey: "OpenAIAPIKey")
private let modelName = AppBundleConfiguration.stringValue(forKey: "OpenAITranscriptionModel")
?? "gpt-4o-transcribe"
let displayName = "OpenAI"
let requiresSpeechRecognitionPermission = false
var isConfigured: Bool {
apiKey != nil
}
var unavailableExplanation: String? {
guard !isConfigured else { return nil }
return "OpenAI transcription is not configured. Add OpenAIAPIKey to Info.plist."
}
func startStreamingSession(
keyterms: [String],
onTranscriptUpdate: @escaping (String) -> Void,
onFinalTranscriptReady: @escaping (String) -> Void,
onError: @escaping (Error) -> Void
) async throws -> any BuddyStreamingTranscriptionSession {
guard let apiKey else {
throw OpenAIAudioTranscriptionProviderError(
message: unavailableExplanation ?? "OpenAI transcription is not configured."
)
}
return OpenAIAudioTranscriptionSession(
apiKey: apiKey,
modelName: modelName,
keyterms: keyterms,
onTranscriptUpdate: onTranscriptUpdate,
onFinalTranscriptReady: onFinalTranscriptReady,
onError: onError
)
}
}
private final class OpenAIAudioTranscriptionSession: BuddyStreamingTranscriptionSession {
let finalTranscriptFallbackDelaySeconds: TimeInterval = 8.0
private struct TranscriptionResponse: Decodable {
let text: String
}
private static let transcriptionURL = URL(string: "https://api.openai.com/v1/audio/transcriptions")!
private static let targetSampleRate = 16_000
private let apiKey: String
private let modelName: String
private let keyterms: [String]
private let onTranscriptUpdate: (String) -> Void
private let onFinalTranscriptReady: (String) -> Void
private let onError: (Error) -> Void
private let stateQueue = DispatchQueue(label: "com.learningbuddy.openai.transcription")
private let audioPCM16Converter = BuddyPCM16AudioConverter(
targetSampleRate: Double(targetSampleRate)
)
private let urlSession: URLSession
private var bufferedPCM16AudioData = Data()
private var hasRequestedFinalTranscript = false
private var hasDeliveredFinalTranscript = false
private var isCancelled = false
private var transcriptionUploadTask: Task<Void, Never>?
init(
apiKey: String,
modelName: String,
keyterms: [String],
onTranscriptUpdate: @escaping (String) -> Void,
onFinalTranscriptReady: @escaping (String) -> Void,
onError: @escaping (Error) -> Void
) {
self.apiKey = apiKey
self.modelName = modelName
self.keyterms = keyterms
self.onTranscriptUpdate = onTranscriptUpdate
self.onFinalTranscriptReady = onFinalTranscriptReady
self.onError = onError
let urlSessionConfiguration = URLSessionConfiguration.default
urlSessionConfiguration.timeoutIntervalForRequest = 45
urlSessionConfiguration.timeoutIntervalForResource = 90
urlSessionConfiguration.waitsForConnectivity = true
self.urlSession = URLSession(configuration: urlSessionConfiguration)
}
func appendAudioBuffer(_ audioBuffer: AVAudioPCMBuffer) {
guard let audioPCM16Data = audioPCM16Converter.convertToPCM16Data(from: audioBuffer),
!audioPCM16Data.isEmpty else {
return
}
stateQueue.async {
guard !self.hasRequestedFinalTranscript, !self.isCancelled else { return }
self.bufferedPCM16AudioData.append(audioPCM16Data)
}
}
func requestFinalTranscript() {
stateQueue.async {
guard !self.hasRequestedFinalTranscript, !self.isCancelled else { return }
self.hasRequestedFinalTranscript = true
let bufferedPCM16AudioData = self.bufferedPCM16AudioData
self.transcriptionUploadTask = Task { [weak self] in
await self?.transcribeBufferedAudio(bufferedPCM16AudioData)
}
}
}
func cancel() {
stateQueue.async {
self.isCancelled = true
self.bufferedPCM16AudioData.removeAll(keepingCapacity: false)
}
transcriptionUploadTask?.cancel()
urlSession.invalidateAndCancel()
}
private func transcribeBufferedAudio(_ bufferedPCM16AudioData: Data) async {
guard !Task.isCancelled else { return }
let trimmedAudioDataIsEmpty = stateQueue.sync {
isCancelled || bufferedPCM16AudioData.isEmpty
}
if trimmedAudioDataIsEmpty {
deliverFinalTranscript("")
return
}
let wavAudioData = BuddyWAVFileBuilder.buildWAVData(
fromPCM16MonoAudio: bufferedPCM16AudioData,
sampleRate: Self.targetSampleRate
)
do {
let transcriptText = try await requestTranscription(for: wavAudioData)
guard !stateQueue.sync(execute: { isCancelled }) else { return }
if !transcriptText.isEmpty {
onTranscriptUpdate(transcriptText)
}
deliverFinalTranscript(transcriptText)
} catch {
guard !stateQueue.sync(execute: { isCancelled }) else { return }
print("[OpenAI Transcription] ❌ Upload failed (audio size: \(wavAudioData.count) bytes): \(error.localizedDescription)")
onError(error)
}
}
private func requestTranscription(for wavAudioData: Data) async throws -> String {
let multipartBoundary = "Boundary-\(UUID().uuidString)"
var request = URLRequest(url: Self.transcriptionURL)
request.httpMethod = "POST"
request.setValue("Bearer \(apiKey)", forHTTPHeaderField: "Authorization")
request.setValue("multipart/form-data; boundary=\(multipartBoundary)", forHTTPHeaderField: "Content-Type")
let requestBodyData = makeMultipartRequestBody(
boundary: multipartBoundary,
wavAudioData: wavAudioData
)
request.httpBody = requestBodyData
let (responseData, response) = try await urlSession.data(for: request)
guard let httpResponse = response as? HTTPURLResponse else {
throw OpenAIAudioTranscriptionProviderError(
message: "OpenAI transcription returned an invalid response."
)
}
guard (200...299).contains(httpResponse.statusCode) else {
let responseText = String(data: responseData, encoding: .utf8) ?? "Unknown error"
throw OpenAIAudioTranscriptionProviderError(
message: "OpenAI transcription failed: \(responseText)"
)
}
if let transcriptionResponse = try? JSONDecoder().decode(
TranscriptionResponse.self,
from: responseData
) {
return transcriptionResponse.text.trimmingCharacters(in: .whitespacesAndNewlines)
}
let responseText = String(data: responseData, encoding: .utf8)?
.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
if !responseText.isEmpty {
return responseText
}
throw OpenAIAudioTranscriptionProviderError(
message: "OpenAI transcription returned an empty transcript."
)
}
private func makeMultipartRequestBody(
boundary: String,
wavAudioData: Data
) -> Data {
var requestBodyData = Data()
requestBodyData.appendMultipartFormField(
named: "model",
value: modelName,
usingBoundary: boundary
)
requestBodyData.appendMultipartFormField(
named: "language",
value: "en",
usingBoundary: boundary
)
requestBodyData.appendMultipartFormField(
named: "response_format",
value: "json",
usingBoundary: boundary
)
if let contextualPrompt = transcriptionPromptText() {
requestBodyData.appendMultipartFormField(
named: "prompt",
value: contextualPrompt,
usingBoundary: boundary
)
}
requestBodyData.appendMultipartFileField(
named: "file",
filename: "voice-input.wav",
mimeType: "audio/wav",
fileData: wavAudioData,
usingBoundary: boundary
)
requestBodyData.appendString("--\(boundary)--\r\n")
return requestBodyData
}
private func transcriptionPromptText() -> String? {
let normalizedKeyterms = keyterms
.map { $0.trimmingCharacters(in: .whitespacesAndNewlines) }
.filter { !$0.isEmpty }
guard !normalizedKeyterms.isEmpty else { return nil }
return """
This is a short push-to-talk transcript for a coding and product app. Expect product names, technical terms, and app-specific vocabulary such as: \(normalizedKeyterms.joined(separator: ", ")).
"""
}
private func deliverFinalTranscript(_ transcriptText: String) {
guard !hasDeliveredFinalTranscript else { return }
hasDeliveredFinalTranscript = true
onFinalTranscriptReady(transcriptText)
}
deinit {
cancel()
}
}
private extension Data {
mutating func appendString(_ string: String) {
append(string.data(using: .utf8)!)
}
mutating func appendMultipartFormField(
named fieldName: String,
value: String,
usingBoundary boundary: String
) {
appendString("--\(boundary)\r\n")
appendString("Content-Disposition: form-data; name=\"\(fieldName)\"\r\n\r\n")
appendString("\(value)\r\n")
}
mutating func appendMultipartFileField(
named fieldName: String,
filename: String,
mimeType: String,
fileData: Data,
usingBoundary boundary: String
) {
appendString("--\(boundary)\r\n")
appendString("Content-Disposition: form-data; name=\"\(fieldName)\"; filename=\"\(filename)\"\r\n")
appendString("Content-Type: \(mimeType)\r\n\r\n")
append(fileData)
appendString("\r\n")
}
}
+881
View File
@@ -0,0 +1,881 @@
//
// OverlayWindow.swift
// leanring-buddy
//
// System-wide transparent overlay window for blue glowing cursor.
// One OverlayWindow is created per screen so the cursor buddy
// seamlessly follows the cursor across multiple monitors.
//
import AppKit
import AVFoundation
import SwiftUI
class OverlayWindow: NSWindow {
init(screen: NSScreen) {
// Create window covering entire screen
super.init(
contentRect: screen.frame,
styleMask: .borderless,
backing: .buffered,
defer: false
)
// Make window transparent and non-interactive
self.isOpaque = false
self.backgroundColor = .clear
self.level = .screenSaver // Always on top, above submenus and popups
self.ignoresMouseEvents = true // Click-through
self.collectionBehavior = [.canJoinAllSpaces, .stationary, .fullScreenAuxiliary]
self.isReleasedWhenClosed = false
self.hasShadow = false
// Important: Allow the window to appear even when app is not active
self.hidesOnDeactivate = false
// Cover the entire screen
self.setFrame(screen.frame, display: true)
// Make sure it's on the right screen
if let screenForWindow = NSScreen.screens.first(where: { $0.frame == screen.frame }) {
self.setFrameOrigin(screenForWindow.frame.origin)
}
}
// Prevent window from becoming key (no focus stealing)
override var canBecomeKey: Bool {
return false
}
override var canBecomeMain: Bool {
return false
}
}
// Cursor-like triangle shape (equilateral)
struct Triangle: Shape {
func path(in rect: CGRect) -> Path {
var path = Path()
let size = min(rect.width, rect.height)
let height = size * sqrt(3.0) / 2.0
// Top vertex
path.move(to: CGPoint(x: rect.midX, y: rect.midY - height / 1.5))
// Bottom left vertex
path.addLine(to: CGPoint(x: rect.midX - size / 2, y: rect.midY + height / 3))
// Bottom right vertex
path.addLine(to: CGPoint(x: rect.midX + size / 2, y: rect.midY + height / 3))
path.closeSubpath()
return path
}
}
// PreferenceKey for tracking bubble size
struct SizePreferenceKey: PreferenceKey {
static var defaultValue: CGSize = .zero
static func reduce(value: inout CGSize, nextValue: () -> CGSize) {
value = nextValue()
}
}
struct NavigationBubbleSizePreferenceKey: PreferenceKey {
static var defaultValue: CGSize = .zero
static func reduce(value: inout CGSize, nextValue: () -> CGSize) {
value = nextValue()
}
}
/// The buddy's behavioral mode. Controls whether it follows the cursor,
/// is flying toward a detected UI element, or is pointing at an element.
enum BuddyNavigationMode {
/// Default buddy follows the mouse cursor with spring animation
case followingCursor
/// Buddy is animating toward a detected UI element location
case navigatingToTarget
/// Buddy has arrived at the target and is pointing at it with a speech bubble
case pointingAtTarget
}
// SwiftUI view for the blue glowing cursor pointer.
// Each screen gets its own BlueCursorView. The view checks whether
// the cursor is currently on THIS screen and only shows the buddy
// triangle when it is. During voice interaction, the triangle is
// replaced by a waveform (listening), spinner (processing), or
// streaming text bubble (responding).
struct BlueCursorView: View {
let screenFrame: CGRect
let isFirstAppearance: Bool
@ObservedObject var companionManager: CompanionManager
@State private var cursorPosition: CGPoint
@State private var isCursorOnThisScreen: Bool
init(screenFrame: CGRect, isFirstAppearance: Bool, companionManager: CompanionManager) {
self.screenFrame = screenFrame
self.isFirstAppearance = isFirstAppearance
self.companionManager = companionManager
// Seed the cursor position from the current mouse location so the
// buddy doesn't flash at (0,0) before onAppear fires.
let mouseLocation = NSEvent.mouseLocation
let localX = mouseLocation.x - screenFrame.origin.x
let localY = screenFrame.height - (mouseLocation.y - screenFrame.origin.y)
_cursorPosition = State(initialValue: CGPoint(x: localX + 35, y: localY + 25))
_isCursorOnThisScreen = State(initialValue: screenFrame.contains(mouseLocation))
}
@State private var timer: Timer?
@State private var welcomeText: String = ""
@State private var showWelcome: Bool = true
@State private var bubbleSize: CGSize = .zero
@State private var bubbleOpacity: Double = 1.0
@State private var cursorOpacity: Double = 0.0
// MARK: - Buddy Navigation State
/// The buddy's current behavioral mode (following cursor, navigating, or pointing).
@State private var buddyNavigationMode: BuddyNavigationMode = .followingCursor
/// The rotation angle of the triangle in degrees. Default is -35° (cursor-like).
/// Changes to face the direction of travel when navigating to a target.
@State private var triangleRotationDegrees: Double = -35.0
/// Speech bubble text shown when pointing at a detected element.
@State private var navigationBubbleText: String = ""
@State private var navigationBubbleOpacity: Double = 0.0
@State private var navigationBubbleSize: CGSize = .zero
/// The cursor position at the moment navigation started, used to detect
/// if the user moves the cursor enough to cancel the navigation.
@State private var cursorPositionWhenNavigationStarted: CGPoint = .zero
/// Timer driving the frame-by-frame bezier arc flight animation.
/// Invalidated when the flight completes, is canceled, or the view disappears.
@State private var navigationAnimationTimer: Timer?
/// Scale factor applied to the buddy triangle during flight. Grows to ~1.3x
/// at the midpoint of the arc and shrinks back to 1.0x on landing, creating
/// an energetic "swooping" feel.
@State private var buddyFlightScale: CGFloat = 1.0
/// Scale factor for the navigation speech bubble's pop-in entrance.
/// Starts at 0.5 and springs to 1.0 when the first character appears.
@State private var navigationBubbleScale: CGFloat = 1.0
/// True when the buddy is flying BACK to the cursor after pointing.
/// Only during the return flight can cursor movement cancel the animation.
@State private var isReturningToCursor: Bool = false
// MARK: - Onboarding Video Layout
private let onboardingVideoPlayerWidth: CGFloat = 330
private let onboardingVideoPlayerHeight: CGFloat = 186
private let fullWelcomeMessage = "hey! i'm clicky"
private let navigationPointerPhrases = [
"right here!",
"this one!",
"over here!",
"click this!",
"here it is!",
"found it!"
]
var body: some View {
ZStack {
// Nearly transparent background (helps with compositing)
Color.black.opacity(0.001)
// Welcome speech bubble (first launch only)
if isCursorOnThisScreen && showWelcome && !welcomeText.isEmpty {
Text(welcomeText)
.font(.system(size: 11, weight: .medium))
.foregroundColor(.white)
.padding(.horizontal, 8)
.padding(.vertical, 4)
.background(
RoundedRectangle(cornerRadius: 6, style: .continuous)
.fill(DS.Colors.overlayCursorBlue)
.shadow(color: DS.Colors.overlayCursorBlue.opacity(0.5), radius: 6, x: 0, y: 0)
)
.fixedSize()
.overlay(
GeometryReader { geo in
Color.clear
.preference(key: SizePreferenceKey.self, value: geo.size)
}
)
.opacity(bubbleOpacity)
.position(x: cursorPosition.x + 10 + (bubbleSize.width / 2), y: cursorPosition.y + 18)
.animation(.spring(response: 0.2, dampingFraction: 0.6, blendDuration: 0), value: cursorPosition)
.animation(.easeOut(duration: 0.5), value: bubbleOpacity)
.onPreferenceChange(SizePreferenceKey.self) { newSize in
bubbleSize = newSize
}
}
// Onboarding video always in the view tree so opacity animation works
// reliably. When no player exists or opacity is 0, nothing is visible.
// allowsHitTesting(false) prevents it from intercepting clicks.
OnboardingVideoPlayerView(player: companionManager.onboardingVideoPlayer)
.frame(width: onboardingVideoPlayerWidth, height: onboardingVideoPlayerHeight)
.clipShape(RoundedRectangle(cornerRadius: 10, style: .continuous))
.shadow(color: Color.black.opacity(0.4 * companionManager.onboardingVideoOpacity), radius: 12, x: 0, y: 6)
.opacity(isCursorOnThisScreen ? companionManager.onboardingVideoOpacity : 0)
.position(
x: cursorPosition.x + 10 + (onboardingVideoPlayerWidth / 2),
y: cursorPosition.y + 18 + (onboardingVideoPlayerHeight / 2)
)
.animation(.spring(response: 0.2, dampingFraction: 0.6, blendDuration: 0), value: cursorPosition)
.animation(.easeInOut(duration: 2.0), value: companionManager.onboardingVideoOpacity)
.allowsHitTesting(false)
// Onboarding prompt "press control + option and say hi" streamed after video ends
if isCursorOnThisScreen && companionManager.showOnboardingPrompt && !companionManager.onboardingPromptText.isEmpty {
Text(companionManager.onboardingPromptText)
.font(.system(size: 11, weight: .medium))
.foregroundColor(.white)
.padding(.horizontal, 8)
.padding(.vertical, 4)
.background(
RoundedRectangle(cornerRadius: 6, style: .continuous)
.fill(DS.Colors.overlayCursorBlue)
.shadow(color: DS.Colors.overlayCursorBlue.opacity(0.5), radius: 6, x: 0, y: 0)
)
.fixedSize()
.overlay(
GeometryReader { geo in
Color.clear
.preference(key: SizePreferenceKey.self, value: geo.size)
}
)
.opacity(companionManager.onboardingPromptOpacity)
.position(x: cursorPosition.x + 10 + (bubbleSize.width / 2), y: cursorPosition.y + 18)
.animation(.spring(response: 0.2, dampingFraction: 0.6, blendDuration: 0), value: cursorPosition)
.animation(.easeOut(duration: 0.4), value: companionManager.onboardingPromptOpacity)
.onPreferenceChange(SizePreferenceKey.self) { newSize in
bubbleSize = newSize
}
}
// Navigation pointer bubble shown when buddy arrives at a detected element.
// Pops in with a scale-bounce (0.5x 1.0x spring) and a bright initial
// glow that settles, creating a "materializing" effect.
if buddyNavigationMode == .pointingAtTarget && !navigationBubbleText.isEmpty {
Text(navigationBubbleText)
.font(.system(size: 11, weight: .medium))
.foregroundColor(.white)
.padding(.horizontal, 8)
.padding(.vertical, 4)
.background(
RoundedRectangle(cornerRadius: 6, style: .continuous)
.fill(DS.Colors.overlayCursorBlue)
.shadow(
color: DS.Colors.overlayCursorBlue.opacity(0.5 + (1.0 - navigationBubbleScale) * 1.0),
radius: 6 + (1.0 - navigationBubbleScale) * 16,
x: 0, y: 0
)
)
.fixedSize()
.overlay(
GeometryReader { geo in
Color.clear
.preference(key: NavigationBubbleSizePreferenceKey.self, value: geo.size)
}
)
.scaleEffect(navigationBubbleScale)
.opacity(navigationBubbleOpacity)
.position(x: cursorPosition.x + 10 + (navigationBubbleSize.width / 2), y: cursorPosition.y + 18)
.animation(.spring(response: 0.2, dampingFraction: 0.6, blendDuration: 0), value: cursorPosition)
.animation(.spring(response: 0.4, dampingFraction: 0.6), value: navigationBubbleScale)
.animation(.easeOut(duration: 0.5), value: navigationBubbleOpacity)
.onPreferenceChange(NavigationBubbleSizePreferenceKey.self) { newSize in
navigationBubbleSize = newSize
}
}
// Blue triangle cursor shown when idle or while TTS is playing (responding).
// All three states (triangle, waveform, spinner) stay in the view tree
// permanently and cross-fade via opacity so SwiftUI doesn't remove/re-insert
// them (which caused a visible cursor "pop").
//
// During cursor following: fast spring animation for snappy tracking.
// During navigation: NO implicit animation the frame-by-frame bezier
// timer controls position directly at 60fps for a smooth arc flight.
Triangle()
.fill(DS.Colors.overlayCursorBlue)
.frame(width: 16, height: 16)
.rotationEffect(.degrees(triangleRotationDegrees))
.shadow(color: DS.Colors.overlayCursorBlue, radius: 8 + (buddyFlightScale - 1.0) * 20, x: 0, y: 0)
.scaleEffect(buddyFlightScale)
.opacity(buddyIsVisibleOnThisScreen && (companionManager.voiceState == .idle || companionManager.voiceState == .responding) ? cursorOpacity : 0)
.position(cursorPosition)
.animation(
buddyNavigationMode == .followingCursor
? .spring(response: 0.2, dampingFraction: 0.6, blendDuration: 0)
: nil,
value: cursorPosition
)
.animation(.easeIn(duration: 0.25), value: companionManager.voiceState)
.animation(
buddyNavigationMode == .navigatingToTarget ? nil : .easeInOut(duration: 0.3),
value: triangleRotationDegrees
)
// Blue waveform replaces the triangle while listening
BlueCursorWaveformView(audioPowerLevel: companionManager.currentAudioPowerLevel)
.opacity(buddyIsVisibleOnThisScreen && companionManager.voiceState == .listening ? cursorOpacity : 0)
.position(cursorPosition)
.animation(.spring(response: 0.2, dampingFraction: 0.6, blendDuration: 0), value: cursorPosition)
.animation(.easeIn(duration: 0.15), value: companionManager.voiceState)
// Blue spinner shown while the AI is processing (transcription + Claude + waiting for TTS)
BlueCursorSpinnerView()
.opacity(buddyIsVisibleOnThisScreen && companionManager.voiceState == .processing ? cursorOpacity : 0)
.position(cursorPosition)
.animation(.spring(response: 0.2, dampingFraction: 0.6, blendDuration: 0), value: cursorPosition)
.animation(.easeIn(duration: 0.15), value: companionManager.voiceState)
}
.frame(width: screenFrame.width, height: screenFrame.height)
.ignoresSafeArea()
.onAppear {
// Set initial cursor position immediately before starting animation
let mouseLocation = NSEvent.mouseLocation
isCursorOnThisScreen = screenFrame.contains(mouseLocation)
let swiftUIPosition = convertScreenPointToSwiftUICoordinates(mouseLocation)
self.cursorPosition = CGPoint(x: swiftUIPosition.x + 35, y: swiftUIPosition.y + 25)
startTrackingCursor()
// Only show welcome message on first appearance (app start)
// and only if the cursor starts on this screen
if isFirstAppearance && isCursorOnThisScreen {
withAnimation(.easeIn(duration: 2.0)) {
self.cursorOpacity = 1.0
}
DispatchQueue.main.asyncAfter(deadline: .now() + 2.0) {
self.bubbleOpacity = 0.0
startWelcomeAnimation()
}
} else {
self.cursorOpacity = 1.0
}
}
.onDisappear {
timer?.invalidate()
navigationAnimationTimer?.invalidate()
companionManager.tearDownOnboardingVideo()
}
.onChange(of: companionManager.detectedElementScreenLocation) { newLocation in
// When a UI element location is detected, navigate the buddy to
// that position so it points at the element.
guard let screenLocation = newLocation,
let displayFrame = companionManager.detectedElementDisplayFrame else {
return
}
// Only navigate if the target is on THIS screen
guard screenFrame.contains(CGPoint(x: displayFrame.midX, y: displayFrame.midY))
|| displayFrame == screenFrame else {
return
}
startNavigatingToElement(screenLocation: screenLocation)
}
}
/// Whether the buddy triangle should be visible on this screen.
/// True when cursor is on this screen during normal following, or
/// when navigating/pointing at a target on this screen. When another
/// screen is navigating (detectedElementScreenLocation is set but this
/// screen isn't the one animating), hide the cursor so only one buddy
/// is ever visible at a time.
private var buddyIsVisibleOnThisScreen: Bool {
switch buddyNavigationMode {
case .followingCursor:
// If another screen's BlueCursorView is navigating to an element,
// hide the cursor on this screen to prevent a duplicate buddy
if companionManager.detectedElementScreenLocation != nil {
return false
}
return isCursorOnThisScreen
case .navigatingToTarget, .pointingAtTarget:
return true
}
}
// MARK: - Cursor Tracking
private func startTrackingCursor() {
timer = Timer.scheduledTimer(withTimeInterval: 0.016, repeats: true) { _ in
let mouseLocation = NSEvent.mouseLocation
self.isCursorOnThisScreen = self.screenFrame.contains(mouseLocation)
// During forward flight or pointing, the buddy is NOT interrupted by
// mouse movement it completes its full animation and return flight.
// Only during the RETURN flight do we allow cursor movement to cancel
// (so the buddy snaps to following if the user moves while it's flying back).
if self.buddyNavigationMode == .navigatingToTarget && self.isReturningToCursor {
let currentMouseInSwiftUI = self.convertScreenPointToSwiftUICoordinates(mouseLocation)
let distanceFromNavigationStart = hypot(
currentMouseInSwiftUI.x - self.cursorPositionWhenNavigationStarted.x,
currentMouseInSwiftUI.y - self.cursorPositionWhenNavigationStarted.y
)
if distanceFromNavigationStart > 100 {
cancelNavigationAndResumeFollowing()
}
return
}
// During forward navigation or pointing, just skip cursor tracking
if self.buddyNavigationMode != .followingCursor {
return
}
// Normal cursor following
let swiftUIPosition = self.convertScreenPointToSwiftUICoordinates(mouseLocation)
let buddyX = swiftUIPosition.x + 35
let buddyY = swiftUIPosition.y + 25
self.cursorPosition = CGPoint(x: buddyX, y: buddyY)
}
}
/// Converts a macOS screen point (AppKit, bottom-left origin) to SwiftUI
/// coordinates (top-left origin) relative to this screen's overlay window.
private func convertScreenPointToSwiftUICoordinates(_ screenPoint: CGPoint) -> CGPoint {
let x = screenPoint.x - screenFrame.origin.x
let y = (screenFrame.origin.y + screenFrame.height) - screenPoint.y
return CGPoint(x: x, y: y)
}
// MARK: - Element Navigation
/// Starts animating the buddy toward a detected UI element location.
private func startNavigatingToElement(screenLocation: CGPoint) {
// Don't interrupt welcome animation
guard !showWelcome || welcomeText.isEmpty else { return }
// Convert the AppKit screen location to SwiftUI coordinates for this screen
let targetInSwiftUI = convertScreenPointToSwiftUICoordinates(screenLocation)
// Offset the target so the buddy sits beside the element rather than
// directly on top of it 8px to the right, 12px below.
let offsetTarget = CGPoint(
x: targetInSwiftUI.x + 8,
y: targetInSwiftUI.y + 12
)
// Clamp target to screen bounds with padding
let clampedTarget = CGPoint(
x: max(20, min(offsetTarget.x, screenFrame.width - 20)),
y: max(20, min(offsetTarget.y, screenFrame.height - 20))
)
// Record the current cursor position so we can detect if the user
// moves the mouse enough to cancel the return flight
let mouseLocation = NSEvent.mouseLocation
cursorPositionWhenNavigationStarted = convertScreenPointToSwiftUICoordinates(mouseLocation)
// Enter navigation mode stop cursor following
buddyNavigationMode = .navigatingToTarget
isReturningToCursor = false
animateBezierFlightArc(to: clampedTarget) {
guard self.buddyNavigationMode == .navigatingToTarget else { return }
self.startPointingAtElement()
}
}
/// Animates the buddy along a quadratic bezier arc from its current position
/// to the specified destination. The triangle rotates to face its direction
/// of travel (tangent to the curve) each frame, scales up at the midpoint
/// for a "swooping" feel, and the glow intensifies during flight.
private func animateBezierFlightArc(
to destination: CGPoint,
onComplete: @escaping () -> Void
) {
navigationAnimationTimer?.invalidate()
let startPosition = cursorPosition
let endPosition = destination
let deltaX = endPosition.x - startPosition.x
let deltaY = endPosition.y - startPosition.y
let distance = hypot(deltaX, deltaY)
// Flight duration scales with distance: short hops are quick, long
// flights are more dramatic. Clamped to 0.6s1.4s.
let flightDurationSeconds = min(max(distance / 800.0, 0.6), 1.4)
let frameInterval: Double = 1.0 / 60.0
let totalFrames = Int(flightDurationSeconds / frameInterval)
var currentFrame = 0
// Control point for the quadratic bezier arc. Offset the midpoint
// upward (negative Y in SwiftUI) so the buddy flies in a parabolic arc.
let midPoint = CGPoint(
x: (startPosition.x + endPosition.x) / 2.0,
y: (startPosition.y + endPosition.y) / 2.0
)
let arcHeight = min(distance * 0.2, 80.0)
let controlPoint = CGPoint(x: midPoint.x, y: midPoint.y - arcHeight)
navigationAnimationTimer = Timer.scheduledTimer(withTimeInterval: frameInterval, repeats: true) { _ in
currentFrame += 1
if currentFrame > totalFrames {
self.navigationAnimationTimer?.invalidate()
self.navigationAnimationTimer = nil
self.cursorPosition = endPosition
self.buddyFlightScale = 1.0
onComplete()
return
}
// Linear progress 01 over the flight duration
let linearProgress = Double(currentFrame) / Double(totalFrames)
// Smoothstep easeInOut: 3t² - 2t³ (Hermite interpolation)
let t = linearProgress * linearProgress * (3.0 - 2.0 * linearProgress)
// Quadratic bezier: B(t) = (1-t)²·P0 + 2(1-t)t·P1 + t²·P2
let oneMinusT = 1.0 - t
let bezierX = oneMinusT * oneMinusT * startPosition.x
+ 2.0 * oneMinusT * t * controlPoint.x
+ t * t * endPosition.x
let bezierY = oneMinusT * oneMinusT * startPosition.y
+ 2.0 * oneMinusT * t * controlPoint.y
+ t * t * endPosition.y
self.cursorPosition = CGPoint(x: bezierX, y: bezierY)
// Rotation: face the direction of travel by computing the tangent
// to the bezier curve. B'(t) = 2(1-t)(P1-P0) + 2t(P2-P1)
let tangentX = 2.0 * oneMinusT * (controlPoint.x - startPosition.x)
+ 2.0 * t * (endPosition.x - controlPoint.x)
let tangentY = 2.0 * oneMinusT * (controlPoint.y - startPosition.y)
+ 2.0 * t * (endPosition.y - controlPoint.y)
// +90° offset because the triangle's "tip" points up at 0° rotation,
// and atan2 returns 0° for rightward movement
self.triangleRotationDegrees = atan2(tangentY, tangentX) * (180.0 / .pi) + 90.0
// Scale pulse: sin curve peaks at midpoint of the flight.
// Buddy grows to ~1.3x at the apex, then shrinks back to 1.0x on landing.
let scalePulse = sin(linearProgress * .pi)
self.buddyFlightScale = 1.0 + scalePulse * 0.3
}
}
/// Transitions to pointing mode shows a speech bubble with a bouncy
/// scale-in entrance and variable-speed character streaming.
private func startPointingAtElement() {
buddyNavigationMode = .pointingAtTarget
// Rotate back to default pointer angle now that we've arrived
triangleRotationDegrees = -35.0
// Reset navigation bubble state start small for the scale-bounce entrance
navigationBubbleText = ""
navigationBubbleOpacity = 1.0
navigationBubbleSize = .zero
navigationBubbleScale = 0.5
// Use custom bubble text from the companion manager (e.g. onboarding demo)
// if available, otherwise fall back to a random pointer phrase
let pointerPhrase = companionManager.detectedElementBubbleText
?? navigationPointerPhrases.randomElement()
?? "right here!"
streamNavigationBubbleCharacter(phrase: pointerPhrase, characterIndex: 0) {
// All characters streamed hold for 3 seconds, then fly back
DispatchQueue.main.asyncAfter(deadline: .now() + 3.0) {
guard self.buddyNavigationMode == .pointingAtTarget else { return }
self.navigationBubbleOpacity = 0.0
DispatchQueue.main.asyncAfter(deadline: .now() + 0.5) {
guard self.buddyNavigationMode == .pointingAtTarget else { return }
self.startFlyingBackToCursor()
}
}
}
}
/// Streams the navigation bubble text one character at a time with variable
/// delays (3060ms) for a natural "speaking" rhythm.
private func streamNavigationBubbleCharacter(
phrase: String,
characterIndex: Int,
onComplete: @escaping () -> Void
) {
guard buddyNavigationMode == .pointingAtTarget else { return }
guard characterIndex < phrase.count else {
onComplete()
return
}
let charIndex = phrase.index(phrase.startIndex, offsetBy: characterIndex)
navigationBubbleText.append(phrase[charIndex])
// On the first character, trigger the scale-bounce entrance
if characterIndex == 0 {
navigationBubbleScale = 1.0
}
let characterDelay = Double.random(in: 0.03...0.06)
DispatchQueue.main.asyncAfter(deadline: .now() + characterDelay) {
self.streamNavigationBubbleCharacter(
phrase: phrase,
characterIndex: characterIndex + 1,
onComplete: onComplete
)
}
}
/// Flies the buddy back to the current cursor position after pointing is done.
private func startFlyingBackToCursor() {
let mouseLocation = NSEvent.mouseLocation
let cursorInSwiftUI = convertScreenPointToSwiftUICoordinates(mouseLocation)
let cursorWithTrackingOffset = CGPoint(x: cursorInSwiftUI.x + 35, y: cursorInSwiftUI.y + 25)
cursorPositionWhenNavigationStarted = cursorInSwiftUI
buddyNavigationMode = .navigatingToTarget
isReturningToCursor = true
animateBezierFlightArc(to: cursorWithTrackingOffset) {
self.finishNavigationAndResumeFollowing()
}
}
/// Cancels an in-progress navigation because the user moved the cursor.
private func cancelNavigationAndResumeFollowing() {
navigationAnimationTimer?.invalidate()
navigationAnimationTimer = nil
navigationBubbleText = ""
navigationBubbleOpacity = 0.0
navigationBubbleScale = 1.0
buddyFlightScale = 1.0
finishNavigationAndResumeFollowing()
}
/// Returns the buddy to normal cursor-following mode after navigation completes.
private func finishNavigationAndResumeFollowing() {
navigationAnimationTimer?.invalidate()
navigationAnimationTimer = nil
buddyNavigationMode = .followingCursor
isReturningToCursor = false
triangleRotationDegrees = -35.0
buddyFlightScale = 1.0
navigationBubbleText = ""
navigationBubbleOpacity = 0.0
navigationBubbleScale = 1.0
companionManager.clearDetectedElementLocation()
}
// MARK: - Welcome Animation
private func startWelcomeAnimation() {
withAnimation(.easeIn(duration: 0.4)) {
self.bubbleOpacity = 1.0
}
var currentIndex = 0
Timer.scheduledTimer(withTimeInterval: 0.03, repeats: true) { timer in
guard currentIndex < self.fullWelcomeMessage.count else {
timer.invalidate()
// Hold the text for 2 seconds, then fade it out
DispatchQueue.main.asyncAfter(deadline: .now() + 2.0) {
self.bubbleOpacity = 0.0
}
DispatchQueue.main.asyncAfter(deadline: .now() + 2.5) {
self.showWelcome = false
// Start the onboarding video right after the welcome text disappears
self.companionManager.setupOnboardingVideo()
}
return
}
let index = self.fullWelcomeMessage.index(self.fullWelcomeMessage.startIndex, offsetBy: currentIndex)
self.welcomeText.append(self.fullWelcomeMessage[index])
currentIndex += 1
}
}
}
// MARK: - Blue Cursor Waveform
/// A small blue waveform that replaces the triangle cursor while
/// the user is holding the push-to-talk shortcut and speaking.
private struct BlueCursorWaveformView: View {
let audioPowerLevel: CGFloat
private let barCount = 5
private let listeningBarProfile: [CGFloat] = [0.4, 0.7, 1.0, 0.7, 0.4]
var body: some View {
TimelineView(.animation(minimumInterval: 1.0 / 36.0)) { timelineContext in
HStack(alignment: .center, spacing: 2) {
ForEach(0..<barCount, id: \.self) { barIndex in
RoundedRectangle(cornerRadius: 1.5, style: .continuous)
.fill(DS.Colors.overlayCursorBlue)
.frame(
width: 2,
height: barHeight(
for: barIndex,
timelineDate: timelineContext.date
)
)
}
}
.shadow(color: DS.Colors.overlayCursorBlue.opacity(0.6), radius: 6, x: 0, y: 0)
.animation(.linear(duration: 0.08), value: audioPowerLevel)
}
}
private func barHeight(for barIndex: Int, timelineDate: Date) -> CGFloat {
let animationPhase = CGFloat(timelineDate.timeIntervalSinceReferenceDate * 3.6) + CGFloat(barIndex) * 0.35
let normalizedAudioPowerLevel = max(audioPowerLevel - 0.008, 0)
let easedAudioPowerLevel = pow(min(normalizedAudioPowerLevel * 2.85, 1), 0.76)
let reactiveHeight = easedAudioPowerLevel * 10 * listeningBarProfile[barIndex]
let idlePulse = (sin(animationPhase) + 1) / 2 * 1.5
return 3 + reactiveHeight + idlePulse
}
}
// MARK: - Blue Cursor Spinner
/// A small blue spinning indicator that replaces the triangle cursor
/// while the AI is processing a voice input.
private struct BlueCursorSpinnerView: View {
@State private var isSpinning = false
var body: some View {
Circle()
.trim(from: 0.15, to: 0.85)
.stroke(
AngularGradient(
colors: [
DS.Colors.overlayCursorBlue.opacity(0.0),
DS.Colors.overlayCursorBlue
],
center: .center
),
style: StrokeStyle(lineWidth: 2.5, lineCap: .round)
)
.frame(width: 14, height: 14)
.rotationEffect(.degrees(isSpinning ? 360 : 0))
.shadow(color: DS.Colors.overlayCursorBlue.opacity(0.6), radius: 6, x: 0, y: 0)
.onAppear {
withAnimation(.linear(duration: 0.8).repeatForever(autoreverses: false)) {
isSpinning = true
}
}
}
}
// Manager for overlay windows creates one per screen so the cursor
// buddy seamlessly follows the cursor across multiple monitors.
@MainActor
class OverlayWindowManager {
private var overlayWindows: [OverlayWindow] = []
var hasShownOverlayBefore = false
func showOverlay(onScreens screens: [NSScreen], companionManager: CompanionManager) {
// Hide any existing overlays
hideOverlay()
// Track if this is the first time showing overlay (welcome message)
let isFirstAppearance = !hasShownOverlayBefore
hasShownOverlayBefore = true
// Create one overlay window per screen
for screen in screens {
let window = OverlayWindow(screen: screen)
let contentView = BlueCursorView(
screenFrame: screen.frame,
isFirstAppearance: isFirstAppearance,
companionManager: companionManager
)
let hostingView = NSHostingView(rootView: contentView)
hostingView.frame = screen.frame
window.contentView = hostingView
overlayWindows.append(window)
window.orderFrontRegardless()
}
}
func hideOverlay() {
for window in overlayWindows {
window.orderOut(nil)
window.contentView = nil
}
overlayWindows.removeAll()
}
/// Fades out overlay windows over `duration` seconds, then removes them.
func fadeOutAndHideOverlay(duration: TimeInterval = 0.4) {
let windowsToFade = overlayWindows
overlayWindows.removeAll()
NSAnimationContext.runAnimationGroup({ context in
context.duration = duration
context.timingFunction = CAMediaTimingFunction(name: .easeIn)
for window in windowsToFade {
window.animator().alphaValue = 0
}
}, completionHandler: {
for window in windowsToFade {
window.orderOut(nil)
window.contentView = nil
}
})
}
func isShowingOverlay() -> Bool {
return !overlayWindows.isEmpty
}
}
// MARK: - Onboarding Video Player
/// NSViewRepresentable wrapping an AVPlayerLayer so HLS video plays
/// inside SwiftUI. Uses a custom NSView subclass to keep the player
/// layer sized to the view's bounds automatically.
private struct OnboardingVideoPlayerView: NSViewRepresentable {
let player: AVPlayer?
func makeNSView(context: Context) -> AVPlayerNSView {
let view = AVPlayerNSView()
view.player = player
return view
}
func updateNSView(_ nsView: AVPlayerNSView, context: Context) {
nsView.player = player
}
}
private class AVPlayerNSView: NSView {
var player: AVPlayer? {
didSet { playerLayer.player = player }
}
private let playerLayer = AVPlayerLayer()
override init(frame: NSRect) {
super.init(frame: frame)
wantsLayer = true
playerLayer.videoGravity = .resizeAspectFill
layer?.addSublayer(playerLayer)
}
required init?(coder: NSCoder) { fatalError() }
override func layout() {
super.layout()
playerLayer.frame = bounds
}
}
+262
View File
@@ -0,0 +1,262 @@
//
// WindowPositionManager.swift
// leanring-buddy
//
// Manages positioning the app window on the right edge of the screen
// and shrinking overlapping windows from other apps via the Accessibility API.
//
import AppKit
import ApplicationServices
import ScreenCaptureKit
enum PermissionRequestPresentationDestination: Equatable {
case alreadyGranted
case systemPrompt
case systemSettings
}
@MainActor
class WindowPositionManager {
private static var hasAttemptedAccessibilitySystemPromptDuringCurrentLaunch = false
private static var hasAttemptedScreenRecordingSystemPromptDuringCurrentLaunch = false
private static let hasPreviouslyConfirmedScreenRecordingPermissionUserDefaultsKey = "com.learningbuddy.hasPreviouslyConfirmedScreenRecordingPermission"
/// Returns true when the Mac currently has more than one connected display.
/// Uses AppKit's screen list, which is available without ScreenCaptureKit's
/// shareable-content permission prompt.
static func currentMacHasMultipleDisplays() -> Bool {
NSScreen.screens.count > 1
}
// MARK: - Accessibility Permission
/// Returns true if the app has Accessibility permission.
static func hasAccessibilityPermission() -> Bool {
AXIsProcessTrusted()
}
/// Presents exactly one permission path per tap: the system prompt on the first
/// attempt, then System Settings on later attempts after macOS has already shown
/// its one-time alert.
@discardableResult
static func requestAccessibilityPermission() -> PermissionRequestPresentationDestination {
let presentationDestination = permissionRequestPresentationDestination(
hasPermissionNow: hasAccessibilityPermission(),
hasAttemptedSystemPrompt: hasAttemptedAccessibilitySystemPromptDuringCurrentLaunch
)
switch presentationDestination {
case .alreadyGranted:
return .alreadyGranted
case .systemPrompt:
hasAttemptedAccessibilitySystemPromptDuringCurrentLaunch = true
let options = [kAXTrustedCheckOptionPrompt.takeUnretainedValue(): true] as CFDictionary
_ = AXIsProcessTrustedWithOptions(options)
case .systemSettings:
openAccessibilitySettings()
}
return presentationDestination
}
/// Opens System Settings to the Accessibility pane.
static func openAccessibilitySettings() {
guard let url = URL(string: "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility") else { return }
NSWorkspace.shared.open(url)
}
/// Reveals the running app bundle in Finder so the user can drag it into
/// the Accessibility list if it doesn't appear automatically.
static func revealAppInFinder() {
guard let appURL = Bundle.main.bundleURL as URL? else { return }
NSWorkspace.shared.activateFileViewerSelecting([appURL])
}
// MARK: - Screen Recording Permission
/// Returns true if Screen Recording permission is granted.
static func hasScreenRecordingPermission() -> Bool {
let hasScreenRecordingPermissionNow = CGPreflightScreenCaptureAccess()
if hasScreenRecordingPermissionNow {
UserDefaults.standard.set(true, forKey: hasPreviouslyConfirmedScreenRecordingPermissionUserDefaultsKey)
}
return hasScreenRecordingPermissionNow
}
/// Returns true when the app should proceed with session launch without showing
/// the permission gate again. This intentionally falls back to the last known
/// granted state because CGPreflightScreenCaptureAccess() can sometimes return a
/// false negative even though the user has already approved the app.
static func shouldTreatScreenRecordingPermissionAsGrantedForSessionLaunch() -> Bool {
shouldTreatScreenRecordingPermissionAsGrantedForSessionLaunch(
hasScreenRecordingPermissionNow: hasScreenRecordingPermission(),
hasPreviouslyConfirmedScreenRecordingPermission: UserDefaults.standard.bool(forKey: hasPreviouslyConfirmedScreenRecordingPermissionUserDefaultsKey)
)
}
static func shouldTreatScreenRecordingPermissionAsGrantedForSessionLaunch(
hasScreenRecordingPermissionNow: Bool,
hasPreviouslyConfirmedScreenRecordingPermission: Bool
) -> Bool {
hasScreenRecordingPermissionNow || hasPreviouslyConfirmedScreenRecordingPermission
}
static func clearPreviouslyConfirmedScreenRecordingPermission() {
UserDefaults.standard.removeObject(forKey: hasPreviouslyConfirmedScreenRecordingPermissionUserDefaultsKey)
}
/// Prompts the system dialog for Screen Recording permission.
/// Uses the system prompt once, then opens System Settings on later attempts so
/// the user never gets the prompt and the Settings pane at the same time.
@discardableResult
static func requestScreenRecordingPermission() -> PermissionRequestPresentationDestination {
let presentationDestination = permissionRequestPresentationDestination(
hasPermissionNow: hasScreenRecordingPermission(),
hasAttemptedSystemPrompt: hasAttemptedScreenRecordingSystemPromptDuringCurrentLaunch
)
switch presentationDestination {
case .alreadyGranted:
return .alreadyGranted
case .systemPrompt:
hasAttemptedScreenRecordingSystemPromptDuringCurrentLaunch = true
_ = CGRequestScreenCaptureAccess()
case .systemSettings:
openScreenRecordingSettings()
}
return presentationDestination
}
/// Opens System Settings to the Screen Recording pane.
static func openScreenRecordingSettings() {
guard let url = URL(string: "x-apple.systempreferences:com.apple.preference.security?Privacy_ScreenCapture") else { return }
NSWorkspace.shared.open(url)
}
static func permissionRequestPresentationDestination(
hasPermissionNow: Bool,
hasAttemptedSystemPrompt: Bool
) -> PermissionRequestPresentationDestination {
if hasPermissionNow {
return .alreadyGranted
}
if hasAttemptedSystemPrompt {
return .systemSettings
}
return .systemPrompt
}
// MARK: - Window Positioning
/// Positions the app's main window pinned to the right edge of the screen
/// that contains the given display ID, vertically centered.
static func pinMainWindowToRight(onDisplayID displayID: CGDirectDisplayID?) {
guard let mainWindow = NSApp.windows.first(where: { !($0 is NSPanel) }) else { return }
// Find the NSScreen matching the selected display, or fall back to the screen
// the window is currently on, or finally the main screen.
let targetScreen: NSScreen
if let displayID,
let matchingScreen = NSScreen.screens.first(where: { $0.displayID == displayID }) {
targetScreen = matchingScreen
} else if let currentScreen = mainWindow.screen {
targetScreen = currentScreen
} else if let mainScreen = NSScreen.main {
targetScreen = mainScreen
} else {
return
}
let visibleFrame = targetScreen.visibleFrame
let windowSize = mainWindow.frame.size
let x = visibleFrame.maxX - windowSize.width
let y = visibleFrame.minY + (visibleFrame.height - windowSize.height) / 2.0
mainWindow.setFrameOrigin(NSPoint(x: x, y: y))
}
// MARK: - Shrink Overlapping Windows
/// Checks if the frontmost (non-self) app's focused window overlaps our app window
/// on the same monitor and, if so, shrinks it so it no longer overlaps.
/// Only operates if both windows are on the same screen as `targetDisplayID`.
static func shrinkOverlappingFocusedWindow(targetDisplayID: CGDirectDisplayID?) {
guard hasAccessibilityPermission() else { return }
guard let mainWindow = NSApp.windows.first(where: { !($0 is NSPanel) }) else { return }
guard let mainScreen = mainWindow.screen else { return }
// Only operate if the main window is on the target display
if let targetDisplayID, mainScreen.displayID != targetDisplayID {
return
}
// Get the frontmost application that isn't us
guard let frontApp = NSWorkspace.shared.frontmostApplication,
frontApp.processIdentifier != ProcessInfo.processInfo.processIdentifier else {
return
}
let appElement = AXUIElementCreateApplication(frontApp.processIdentifier)
// Get the focused window of the front app
var focusedWindowValue: AnyObject?
let focusedResult = AXUIElementCopyAttributeValue(appElement, kAXFocusedWindowAttribute as CFString, &focusedWindowValue)
guard focusedResult == .success, let focusedWindow = focusedWindowValue else { return }
// Get position and size of the focused window
var positionValue: AnyObject?
var sizeValue: AnyObject?
guard AXUIElementCopyAttributeValue(focusedWindow as! AXUIElement, kAXPositionAttribute as CFString, &positionValue) == .success,
AXUIElementCopyAttributeValue(focusedWindow as! AXUIElement, kAXSizeAttribute as CFString, &sizeValue) == .success else {
return
}
var otherPosition = CGPoint.zero
var otherSize = CGSize.zero
guard AXValueGetValue(positionValue as! AXValue, .cgPoint, &otherPosition),
AXValueGetValue(sizeValue as! AXValue, .cgSize, &otherSize) else {
return
}
// The other window's frame in screen coordinates (top-left origin from AX API).
// Convert to check if it's on the same screen as our window.
let otherRight = otherPosition.x + otherSize.width
let ourLeft = mainWindow.frame.origin.x
// Check that the other window is on the same screen by verifying its origin
// falls within the target screen's bounds.
let screenFrame = mainScreen.frame
let otherCenterX = otherPosition.x + otherSize.width / 2
// AX uses top-left origin, NSScreen uses bottom-left. Convert AX Y to NSScreen Y.
let otherNSScreenY = screenFrame.maxY - otherPosition.y - otherSize.height
let otherCenterY = otherNSScreenY + otherSize.height / 2
let otherCenter = NSPoint(x: otherCenterX, y: otherCenterY)
guard screenFrame.contains(otherCenter) else { return }
// If the other window's right edge extends past our window's left edge, shrink it.
if otherRight > ourLeft {
let newWidth = ourLeft - otherPosition.x
guard newWidth > 200 else { return } // Don't shrink too small
var newSize = CGSize(width: newWidth, height: otherSize.height)
guard let newSizeValue = AXValueCreate(.cgSize, &newSize) else { return }
AXUIElementSetAttributeValue(focusedWindow as! AXUIElement, kAXSizeAttribute as CFString, newSizeValue)
}
}
}
// MARK: - NSScreen Extension
extension NSScreen {
/// The CGDirectDisplayID for this screen.
var displayID: CGDirectDisplayID {
let key = NSDeviceDescriptionKey("NSScreenNumber")
return deviceDescription[key] as? CGDirectDisplayID ?? 0
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,18 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.app-sandbox</key>
<false/>
<key>com.apple.security.network.client</key>
<true/>
<key>com.apple.security.device.camera</key>
<true/>
<key>com.apple.security.device.audio-input</key>
<true/>
<key>com.apple.security.temporary-exception.mach-lookup.global-name</key>
<array>
<string>com.apple.screencapturekit.picker</string>
</array>
</dict>
</plist>
+89
View File
@@ -0,0 +1,89 @@
//
// leanring_buddyApp.swift
// leanring-buddy
//
// Menu bar-only companion app. No dock icon, no main window just an
// always-available status item in the macOS menu bar. Clicking the icon
// opens a floating panel with companion voice controls.
//
import ServiceManagement
import SwiftUI
import Sparkle
@main
struct leanring_buddyApp: App {
@NSApplicationDelegateAdaptor(CompanionAppDelegate.self) var appDelegate
var body: some Scene {
// The app lives entirely in the menu bar panel managed by the AppDelegate.
// This empty Settings scene satisfies SwiftUI's requirement for at least
// one scene but is never shown (LSUIElement=true removes the app menu).
Settings {
EmptyView()
}
}
}
/// Manages the companion lifecycle: creates the menu bar panel and starts
/// the companion voice pipeline on launch.
@MainActor
final class CompanionAppDelegate: NSObject, NSApplicationDelegate {
private var menuBarPanelManager: MenuBarPanelManager?
private let companionManager = CompanionManager()
private var sparkleUpdaterController: SPUStandardUpdaterController?
func applicationDidFinishLaunching(_ notification: Notification) {
print("🎯 Clicky: Starting...")
print("🎯 Clicky: Version \(Bundle.main.infoDictionary?["CFBundleShortVersionString"] as? String ?? "unknown")")
UserDefaults.standard.register(defaults: ["NSInitialToolTipDelay": 0])
ClickyAnalytics.configure()
ClickyAnalytics.trackAppOpened()
menuBarPanelManager = MenuBarPanelManager(companionManager: companionManager)
companionManager.start()
// Auto-open the panel if the user still needs to do something:
// either they haven't onboarded yet, or permissions were revoked.
if !companionManager.hasCompletedOnboarding || !companionManager.allPermissionsGranted {
menuBarPanelManager?.showPanelOnLaunch()
}
registerAsLoginItemIfNeeded()
// startSparkleUpdater()
}
func applicationWillTerminate(_ notification: Notification) {
companionManager.stop()
}
/// Registers the app as a login item so it launches automatically on
/// startup. Uses SMAppService which shows the app in System Settings >
/// General > Login Items, letting the user toggle it off if they want.
private func registerAsLoginItemIfNeeded() {
let loginItemService = SMAppService.mainApp
if loginItemService.status != .enabled {
do {
try loginItemService.register()
print("🎯 Clicky: Registered as login item")
} catch {
print("⚠️ Clicky: Failed to register as login item: \(error)")
}
}
}
private func startSparkleUpdater() {
let updaterController = SPUStandardUpdaterController(
startingUpdater: false,
updaterDelegate: nil,
userDriverDelegate: nil
)
self.sparkleUpdaterController = updaterController
do {
try updaterController.updater.start()
} catch {
print("⚠️ Clicky: Sparkle updater failed to start: \(error)")
}
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 121 KiB

@@ -0,0 +1,40 @@
//
// leanring_buddyTests.swift
// leanring-buddyTests
//
// Created by thorfinn on 3/2/26.
//
import Testing
@testable import leanring_buddy
struct leanring_buddyTests {
@Test func firstPermissionRequestUsesSystemPromptOnly() async throws {
let presentationDestination = WindowPositionManager.permissionRequestPresentationDestination(
hasPermissionNow: false,
hasAttemptedSystemPrompt: false
)
#expect(presentationDestination == .systemPrompt)
}
@Test func repeatedPermissionRequestOpensSystemSettings() async throws {
let presentationDestination = WindowPositionManager.permissionRequestPresentationDestination(
hasPermissionNow: false,
hasAttemptedSystemPrompt: true
)
#expect(presentationDestination == .systemSettings)
}
@Test func knownGrantedScreenRecordingPermissionSkipsTheGate() async throws {
let shouldTreatPermissionAsGranted = WindowPositionManager.shouldTreatScreenRecordingPermissionAsGrantedForSessionLaunch(
hasScreenRecordingPermissionNow: false,
hasPreviouslyConfirmedScreenRecordingPermission: true
)
#expect(shouldTreatPermissionAsGranted)
}
}
@@ -0,0 +1,41 @@
//
// leanring_buddyUITests.swift
// leanring-buddyUITests
//
// Created by thorfinn on 3/2/26.
//
import XCTest
final class leanring_buddyUITests: XCTestCase {
override func setUpWithError() throws {
// Put setup code here. This method is called before the invocation of each test method in the class.
// In UI tests it is usually best to stop immediately when a failure occurs.
continueAfterFailure = false
// In UI tests its important to set the initial state - such as interface orientation - required for your tests before they run. The setUp method is a good place to do this.
}
override func tearDownWithError() throws {
// Put teardown code here. This method is called after the invocation of each test method in the class.
}
@MainActor
func testExample() throws {
// UI tests must launch the application that they test.
let app = XCUIApplication()
app.launch()
// Use XCTAssert and related functions to verify your tests produce the correct results.
}
@MainActor
func testLaunchPerformance() throws {
// This measures how long it takes to launch your application.
measure(metrics: [XCTApplicationLaunchMetric()]) {
XCUIApplication().launch()
}
}
}
@@ -0,0 +1,33 @@
//
// leanring_buddyUITestsLaunchTests.swift
// leanring-buddyUITests
//
// Created by thorfinn on 3/2/26.
//
import XCTest
final class leanring_buddyUITestsLaunchTests: XCTestCase {
override class var runsForEachTargetApplicationUIConfiguration: Bool {
true
}
override func setUpWithError() throws {
continueAfterFailure = false
}
@MainActor
func testLaunch() throws {
let app = XCUIApplication()
app.launch()
// Insert steps here to perform after app launch but before taking a screenshot,
// such as logging into a test account or navigating somewhere in the app
let attachment = XCTAttachment(screenshot: app.screenshot())
attachment.name = "Launch Screen"
attachment.lifetime = .keepAlways
add(attachment)
}
}
+62
View File
@@ -0,0 +1,62 @@
# Release Scripts
## `release.sh` — Ship a new version of makesomething
Automates the full release pipeline: build → sign → DMG → notarize → Sparkle appcast → GitHub Release.
### Quick start
```bash
# Auto-bumps version and build number from the latest GitHub Release
./scripts/release.sh
```
The script checks GitHub for the latest release (e.g. `v1.5`, build 6) and automatically bumps to `v1.6`, build 7. You'll see a confirmation prompt before anything runs.
### Override version or build
```bash
# Set a specific marketing version (auto-bumps build)
./scripts/release.sh 2.0
# Set both marketing version and build number
./scripts/release.sh 2.0 10
```
### Safety
- **Duplicate detection**: If the tag already exists on GitHub, the script exits with an error and suggests what to do.
- **Confirmation prompt**: Shows the version, build, and previous release before proceeding. Press `y` to continue.
### What it does
1. Fetches the latest release from GitHub to determine version + build
2. Archives the app via `xcodebuild`
3. Exports a signed `.app` with Developer ID
4. Creates a DMG with the drag-to-Applications background
5. Notarizes the DMG with Apple (Gatekeeper compliance)
6. Signs the DMG with the Sparkle EdDSA key
7. Generates `appcast.xml` for Sparkle auto-updates
8. Creates a GitHub Release with the DMG attached
9. Pushes the updated `appcast.xml` to the releases repo
### One-time setup (prerequisites)
1. **Xcode** with your Developer ID signing certificate
2. **Homebrew tools**:
```bash
brew install create-dmg gh
```
3. **GitHub CLI auth**:
```bash
gh auth login
```
4. **Apple notarization credentials** (stored in Keychain):
```bash
xcrun notarytool store-credentials "AC_PASSWORD" \
--apple-id YOUR_APPLE_ID \
--team-id YOUR_TEAM_ID
```
You'll be prompted for an app-specific password (generate one at [appleid.apple.com](https://appleid.apple.com)).
5. **Sparkle EdDSA key** — already generated and stored in Keychain (done during initial Sparkle setup)
6. **Build the project in Xcode at least once** so SPM downloads Sparkle and the Sparkle CLI tools are available
+276
View File
@@ -0,0 +1,276 @@
#!/bin/bash
set -euo pipefail
# Add Homebrew to PATH so create-dmg and gh are available in non-interactive shells
export PATH="/opt/homebrew/bin:$PATH"
# =============================================================================
# release.sh — Automates the full release pipeline for makesomething
#
# What it does (in order):
# 1. Auto-detects version + build from the latest GitHub Release
# 2. Archives the app via xcodebuild
# 3. Exports a signed + notarized .app
# 4. Wraps it in a DMG with the drag-to-Applications background
# 5. Notarizes the DMG with Apple (so Gatekeeper won't block it)
# 6. Signs the DMG with your Sparkle EdDSA key
# 7. Generates/updates appcast.xml automatically
# 8. Creates a GitHub Release with the DMG attached
# 9. Pushes the updated appcast.xml to the releases repo (makesomething-mac-app)
#
# Usage:
# ./scripts/release.sh Auto-bumps: 1.5 → 1.6, build 6 → 7
# ./scripts/release.sh 2.0 Sets marketing version to 2.0, auto-bumps build
# ./scripts/release.sh 2.0 10 Sets both marketing version and build number
#
# Prerequisites (one-time setup):
# - Xcode with your Developer ID signing certificate
# - `brew install create-dmg gh`
# - `gh auth login` (GitHub CLI authenticated)
# - Sparkle EdDSA key in your Keychain (already generated)
# - `xcrun notarytool store-credentials "AC_PASSWORD"` (Apple notarization credentials)
# =============================================================================
# ── Configuration ────────────────────────────────────────────────────────────
SCHEME="leanring-buddy"
APP_NAME="makesomething"
PROJECT_DIR="$(cd "$(dirname "$0")/.." && pwd)"
BUILD_DIR="${PROJECT_DIR}/build"
ARCHIVE_PATH="${BUILD_DIR}/${APP_NAME}.xcarchive"
EXPORT_DIR="${BUILD_DIR}/export"
DMG_OUTPUT_DIR="${BUILD_DIR}/dmg"
RELEASES_DIR="${PROJECT_DIR}/releases" # where generate_appcast reads DMGs from
DMG_BACKGROUND="${PROJECT_DIR}/dmg-background.png"
GITHUB_REPO="julianjear/makesomething-mac-app"
# Sparkle tools (auto-discovered from Xcode's SPM cache)
SPARKLE_BIN=$(find ~/Library/Developer/Xcode/DerivedData/leanring-buddy*/SourcePackages/artifacts/sparkle/Sparkle/bin -maxdepth 0 2>/dev/null | head -1)
if [ -z "$SPARKLE_BIN" ]; then
echo "❌ Sparkle tools not found. Build the project in Xcode first so SPM downloads Sparkle."
exit 1
fi
# ── Auto-detect version from latest GitHub Release ──────────────────────────
# Fetches the latest release tag (e.g. "v1.5") and build number from GitHub.
# If no arguments are provided, bumps the minor version by 0.1 and build by 1.
# You can override either or both by passing arguments.
echo "🔍 Checking latest release on GitHub..."
LATEST_TAG=$(gh release view --repo "${GITHUB_REPO}" --json tagName --jq '.tagName' 2>/dev/null || echo "")
if [ -n "$LATEST_TAG" ]; then
# Strip the "v" prefix to get the version number (e.g. "v1.5" → "1.5")
LATEST_VERSION="${LATEST_TAG#v}"
# Get the build number from the latest release's app bundle inside the DMG.
# We download just the release metadata (not the DMG) and parse the body/notes,
# but the simplest reliable approach is to track it from the GitHub release title
# or from a known incrementing sequence. We use the GitHub API to get asset info
# and derive the build number from the release list count.
LATEST_BUILD=$(gh release list --repo "${GITHUB_REPO}" --json tagName --jq 'length' 2>/dev/null || echo "0")
echo " Latest release: ${LATEST_TAG} (build ${LATEST_BUILD})"
else
LATEST_VERSION="0.0"
LATEST_BUILD=0
echo " No previous releases found — starting from scratch"
fi
# Determine the next marketing version: bump minor by 0.1
# e.g. "1.5" → "1.6", "2.9" → "3.0" (carries over)
if [ $# -ge 1 ]; then
MARKETING_VERSION="$1"
else
MAJOR=$(echo "$LATEST_VERSION" | cut -d. -f1)
MINOR=$(echo "$LATEST_VERSION" | cut -d. -f2)
NEXT_MINOR=$((MINOR + 1))
if [ "$NEXT_MINOR" -ge 10 ]; then
MAJOR=$((MAJOR + 1))
NEXT_MINOR=0
fi
MARKETING_VERSION="${MAJOR}.${NEXT_MINOR}"
fi
# Determine the next build number: always increment by 1
if [ $# -ge 2 ]; then
BUILD_NUMBER="$2"
else
BUILD_NUMBER=$((LATEST_BUILD + 1))
fi
DMG_FILENAME="${APP_NAME}.dmg"
TAG="v${MARKETING_VERSION}"
# ── Safety checks ────────────────────────────────────────────────────────────
# Check if this tag already exists on GitHub to prevent accidental duplicates
if gh release view "${TAG}" --repo "${GITHUB_REPO}" &>/dev/null; then
echo ""
echo "❌ Release ${TAG} already exists on GitHub!"
echo " https://github.com/${GITHUB_REPO}/releases/tag/${TAG}"
echo ""
echo " To release a new version, either:"
echo " • Run without arguments to auto-bump: ./scripts/release.sh"
echo " • Specify a higher version: ./scripts/release.sh $(echo "${MARKETING_VERSION} + 0.1" | bc)"
echo " • Delete the existing release first: gh release delete ${TAG} --repo ${GITHUB_REPO} --yes"
exit 1
fi
echo ""
echo "🚀 Releasing ${APP_NAME} v${MARKETING_VERSION} (build ${BUILD_NUMBER})"
echo " Previous: ${LATEST_TAG:-none}"
echo ""
# Confirm with the user before proceeding
read -p " Proceed? (y/N) " -n 1 -r
echo ""
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo " Aborted."
exit 0
fi
echo ""
# ── Step 1: Clean build directory ────────────────────────────────────────────
echo "🧹 Cleaning build directory and stale DMGs..."
rm -rf "${BUILD_DIR}"
# Remove any leftover temp DMGs from create-dmg (rw.*.dmg) and the previous
# same-named DMG so create-dmg and generate_appcast don't choke on duplicates.
rm -f "${RELEASES_DIR}"/rw.*.dmg "${RELEASES_DIR}/${DMG_FILENAME}"
mkdir -p "${BUILD_DIR}" "${EXPORT_DIR}" "${DMG_OUTPUT_DIR}" "${RELEASES_DIR}"
# ── Step 2: Archive ──────────────────────────────────────────────────────────
echo "📦 Archiving..."
xcodebuild archive \
-scheme "${SCHEME}" \
-archivePath "${ARCHIVE_PATH}" \
MARKETING_VERSION="${MARKETING_VERSION}" \
CURRENT_PROJECT_VERSION="${BUILD_NUMBER}" \
2>&1 | tail -5
echo "✅ Archive created"
# ── Step 3: Export (signed + notarized) ──────────────────────────────────────
# Create an export options plist for Developer ID distribution.
# This tells xcodebuild to sign with your Developer ID certificate
# and submit to Apple for notarization automatically.
EXPORT_OPTIONS="${BUILD_DIR}/ExportOptions.plist"
cat > "${EXPORT_OPTIONS}" << 'PLIST'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>method</key>
<string>developer-id</string>
<key>destination</key>
<string>export</string>
</dict>
</plist>
PLIST
echo "📤 Exporting (signing + notarizing — this may take a few minutes)..."
xcodebuild -exportArchive \
-archivePath "${ARCHIVE_PATH}" \
-exportPath "${EXPORT_DIR}" \
-exportOptionsPlist "${EXPORT_OPTIONS}" \
2>&1 | tail -5
echo "✅ Export complete (signed + notarized)"
# ── Step 4: Create DMG ──────────────────────────────────────────────────────
DMG_PATH="${RELEASES_DIR}/${DMG_FILENAME}"
echo "💿 Creating DMG..."
create-dmg \
--volname "${APP_NAME}" \
--window-pos 200 120 \
--window-size 660 400 \
--icon-size 100 \
--icon "${APP_NAME}.app" 160 195 \
--app-drop-link 500 195 \
--background "${DMG_BACKGROUND}" \
"${DMG_PATH}" \
"${EXPORT_DIR}/${APP_NAME}.app" \
2>&1 | tail -3
echo "✅ DMG created: ${DMG_PATH}"
# ── Step 5: Notarize the DMG ─────────────────────────────────────────────────
# The .app inside the DMG is already signed with Developer ID, but the DMG
# itself needs to be submitted to Apple for notarization so Gatekeeper
# allows users to open it without the "Apple could not verify" warning.
# Requires stored credentials: xcrun notarytool store-credentials "AC_PASSWORD"
echo "🔏 Notarizing DMG with Apple (this may take a few minutes)..."
xcrun notarytool submit "${DMG_PATH}" \
--keychain-profile "AC_PASSWORD" \
--wait
echo "📎 Stapling notarization ticket to DMG..."
xcrun stapler staple "${DMG_PATH}"
echo "✅ DMG notarized and stapled"
# ── Step 6: Sign DMG with Sparkle EdDSA key ─────────────────────────────────
echo "🔐 Signing DMG with Sparkle EdDSA key..."
"${SPARKLE_BIN}/sign_update" "${DMG_PATH}"
# ── Step 7: Generate / update appcast.xml ────────────────────────────────────
# generate_appcast reads all DMGs in the releases/ directory, extracts version
# info from the app bundle inside each DMG, signs with your EdDSA key, and
# produces appcast.xml. The --download-url-prefix tells it where users will
# actually download the DMG from (GitHub Releases).
echo "📡 Generating appcast.xml..."
"${SPARKLE_BIN}/generate_appcast" \
--download-url-prefix "https://github.com/${GITHUB_REPO}/releases/download/${TAG}/" \
-o "${PROJECT_DIR}/appcast.xml" \
"${RELEASES_DIR}"
echo "✅ appcast.xml updated"
# ── Step 8: Create GitHub Release ────────────────────────────────────────────
# Create the release first so the DMG download URL is live before we push the
# appcast that references it.
echo "🏷️ Creating GitHub Release ${TAG}..."
gh release create "${TAG}" "${DMG_PATH}" \
--repo "${GITHUB_REPO}" \
--title "v${MARKETING_VERSION}" \
--notes "makesomething v${MARKETING_VERSION}" \
--latest
# ── Step 9: Push appcast.xml to the releases repo ───────────────────────────
# The appcast lives in makesomething-mac-app (the releases repo), not in the
# source code repo. We clone it to a temp dir, copy the new appcast, and push.
echo "📝 Pushing appcast.xml to ${GITHUB_REPO}..."
RELEASES_REPO_DIR=$(mktemp -d)
git clone --depth 1 "https://github.com/${GITHUB_REPO}.git" "${RELEASES_REPO_DIR}" 2>&1 | tail -2
cp "${PROJECT_DIR}/appcast.xml" "${RELEASES_REPO_DIR}/appcast.xml"
cd "${RELEASES_REPO_DIR}"
git add appcast.xml
git commit -m "Update appcast.xml for v${MARKETING_VERSION}" || echo " (no changes to commit)"
git push || echo " (push failed — you may need to push manually)"
cd "${PROJECT_DIR}"
rm -rf "${RELEASES_REPO_DIR}"
echo ""
echo "═══════════════════════════════════════════════════════════════"
echo "✅ Release v${MARKETING_VERSION} (build ${BUILD_NUMBER}) complete!"
echo ""
echo " DMG: ${DMG_PATH}"
echo " Appcast: ${PROJECT_DIR}/appcast.xml"
echo " Release: https://github.com/${GITHUB_REPO}/releases/tag/${TAG}"
echo ""
echo " Download URL (always latest):"
echo " https://github.com/${GITHUB_REPO}/releases/latest/download/${DMG_FILENAME}"
echo "═══════════════════════════════════════════════════════════════"
+1617
View File
File diff suppressed because it is too large Load Diff
+11
View File
@@ -0,0 +1,11 @@
{
"name": "clicky-proxy",
"private": true,
"scripts": {
"dev": "wrangler dev",
"deploy": "wrangler deploy"
},
"devDependencies": {
"wrangler": "^3.0.0"
}
}
+141
View File
@@ -0,0 +1,141 @@
/**
* Clicky Proxy Worker
*
* Proxies requests to Claude and ElevenLabs APIs so the app never
* ships with raw API keys. Keys are stored as Cloudflare secrets.
*
* Routes:
* POST /chat → Anthropic Messages API (streaming)
* POST /tts → ElevenLabs TTS API
*/
interface Env {
ANTHROPIC_API_KEY: string;
ELEVENLABS_API_KEY: string;
ELEVENLABS_VOICE_ID: string;
ASSEMBLYAI_API_KEY: string;
}
export default {
async fetch(request: Request, env: Env): Promise<Response> {
const url = new URL(request.url);
if (request.method !== "POST") {
return new Response("Method not allowed", { status: 405 });
}
try {
if (url.pathname === "/chat") {
return await handleChat(request, env);
}
if (url.pathname === "/tts") {
return await handleTTS(request, env);
}
if (url.pathname === "/transcribe-token") {
return await handleTranscribeToken(env);
}
} catch (error) {
console.error(`[${url.pathname}] Unhandled error:`, error);
return new Response(
JSON.stringify({ error: String(error) }),
{ status: 500, headers: { "content-type": "application/json" } }
);
}
return new Response("Not found", { status: 404 });
},
};
async function handleChat(request: Request, env: Env): Promise<Response> {
const body = await request.text();
const response = await fetch("https://api.anthropic.com/v1/messages", {
method: "POST",
headers: {
"x-api-key": env.ANTHROPIC_API_KEY,
"anthropic-version": "2023-06-01",
"content-type": "application/json",
},
body,
});
if (!response.ok) {
const errorBody = await response.text();
console.error(`[/chat] Anthropic API error ${response.status}: ${errorBody}`);
return new Response(errorBody, {
status: response.status,
headers: { "content-type": "application/json" },
});
}
return new Response(response.body, {
status: response.status,
headers: {
"content-type": response.headers.get("content-type") || "text/event-stream",
"cache-control": "no-cache",
},
});
}
async function handleTranscribeToken(env: Env): Promise<Response> {
const response = await fetch(
"https://streaming.assemblyai.com/v3/token?expires_in_seconds=480",
{
method: "GET",
headers: {
authorization: env.ASSEMBLYAI_API_KEY,
},
}
);
if (!response.ok) {
const errorBody = await response.text();
console.error(`[/transcribe-token] AssemblyAI token error ${response.status}: ${errorBody}`);
return new Response(errorBody, {
status: response.status,
headers: { "content-type": "application/json" },
});
}
const data = await response.text();
return new Response(data, {
status: 200,
headers: { "content-type": "application/json" },
});
}
async function handleTTS(request: Request, env: Env): Promise<Response> {
const body = await request.text();
const voiceId = env.ELEVENLABS_VOICE_ID;
const response = await fetch(
`https://api.elevenlabs.io/v1/text-to-speech/${voiceId}`,
{
method: "POST",
headers: {
"xi-api-key": env.ELEVENLABS_API_KEY,
"content-type": "application/json",
accept: "audio/mpeg",
},
body,
}
);
if (!response.ok) {
const errorBody = await response.text();
console.error(`[/tts] ElevenLabs API error ${response.status}: ${errorBody}`);
return new Response(errorBody, {
status: response.status,
headers: { "content-type": "application/json" },
});
}
return new Response(response.body, {
status: response.status,
headers: {
"content-type": response.headers.get("content-type") || "audio/mpeg",
},
});
}
+6
View File
@@ -0,0 +1,6 @@
name = "clicky-proxy"
main = "src/index.ts"
compatibility_date = "2024-01-01"
[vars]
ELEVENLABS_VOICE_ID = "kPzsL2i3teMYv0FxEYQ6"