elizaos--eliza/plugins/plugin-facewear/docs/smartglasses.md

@elizaos/plugin-facewear

Even Realities G1/G2 smartglasses integration for ElizaOS.

Implemented Surfaces

• View Manager page at /apps/smartglasses for whole-headset pairing, diagnostics, platform setup guidance, report export, and bridge-backed Wi-Fi scan/configuration/setup prompts when a native bridge exposes those APIs. The page uses a native Even/Mentra bridge when present for iOS/Android-style hosts and falls back to direct Web Bluetooth for desktop browsers.
• G1 UART protocol constants for Nordic UART service:
  • service 6E400001-B5A3-F393-E0A9-E50E24DCCA9E
  • TX 6E400002-B5A3-F393-E0A9-E50E24DCCA9E
  • RX 6E400003-B5A3-F393-E0A9-E50E24DCCA9E
• Display text streaming with G1 0x4E packets, five-line page wrapping using the hardware-derived G1 glyph width profile, official 191-byte payload chunking, Even AI in-progress/completion status (0x31/0x40), and direct Text Show status (0x71).
• RSVP word-group display using the same wrapped G1 display path for rapid reading-style presentation.
• Microphone open/close with G1 0x0E packets and right-lens mic routing.
• Direct G1 0xF1 mic data is exposed as raw LC3 audio (audioData, audioEncoding: "lc3") with packet sequence tracking and gap counts. A native/platform LC3 decoder can be injected with setAudioDecoder or setSmartglassesAudioDecoderForRuntime to convert those chunks into PCM16 for onAudio callbacks. EvenHub/G2 bridge audioEvent.audioPcm data and MentraOS mic_pcm events are exposed as 16 kHz PCM16 and converted to float samples automatically; MentraOS mic_lc3 events are preserved as LC3.
• MentraOS/G2 local_transcription and transcription:* bridge events are normalized into SMARTGLASSES_TRANSCRIPT runtime events and onTranscript callbacks, so host STT output from the glasses microphone reaches Eliza.
• Side tap handling:
  • single tap enables mic
  • double tap disables mic
  • long press / Even AI activation (0xF5 0x17) enables mic
  • Even AI recording stop (0xF5 0x18) disables mic
• EvenHub simulator click/double-click events are normalized to the same tap labels, so simulator input exercises the same microphone toggle path.
• EvenHub/G2 scroll-up and scroll-down events are normalized and routed to the same manual page controls as direct G1 page-up/page-down.
• Incoming hardware event parsing for interaction, physical, battery, device, init, display-result, notification, dashboard, serial-number response, heartbeat, mic response, mic data, response, and error categories.
• Runtime events:
  • SMARTGLASSES_EVENT
  • SMARTGLASSES_AUDIO
  • SMARTGLASSES_TRANSCRIPT
• Managed heartbeat loop for G1 connection maintenance, matching the upstream SDK pattern of periodic 0x25 packets.
• G1 controls:
  • clear display
  • start AI, connection-ready init (0x4D 0x01 left / 0xF4 0x01 right by default, official EvenDemoApp iOS same-init 0x4D 0x01 to both lenses, or EvenDemoApp Android 0xF4 0x01 to both lenses), exit to dashboard, native function exit (0x18), serial request/response (0x34), app whitelist (0x04), raw packet writes, one-shot heartbeat, and managed heartbeat start/stop
  • manual-mode page up/down (0xF5 0x01, left for page-up and right for page-down)
  • silent mode
  • brightness
  • dashboard show/hide, legacy position, and official height/depth positioning
  • dashboard layout, calendar item, time/weather, and setup payload packets from the Fahrplan G1 dashboard model
  • navigation init, direction text, primary/secondary image, poller, and end packets
  • translation setup/start/language/original/translated text overlay packets
  • head-up angle
  • wear detection
  • notes add/delete
  • QuickNote voice-note list parsing plus voice-note audio fetch/delete subcommands
  • notifications, including notification ID packet headers
  • 1-bit BMP generation/transfer framing with CRC for native 576x136 image payloads
• Dual-lens writes are serialized left-then-right for commands that target both lenses, matching the native G1 guidance. Microphone activation remains right-lens only.
• Transports:
  • EvenBridgeTransport for EvenHub/G2-style bridge hosts, including simulator-style sendStartUpPage, onEvenHubEvent, and audioControl, plus MentraOS Bluetooth SDK displayText, clearDisplay, setMicState, mic_pcm, mic_lc3, and local transcription events
  • WebBluetoothG1Transport for direct browser BLE
  • NobleG1Transport for optional Node/Bun BLE through @abandonware/noble
  • MockSmartglassesTransport for tests and examples

Direct G1 BLE does not expose a verified Wi-Fi provisioning command in the reviewed upstreams. Wi-Fi scan/configuration is therefore available only through native/bridge APIs such as Mentra/ASG bridge surfaces.

Eliza Actions

• SMARTGLASSES_DISPLAY_TEXT
• SMARTGLASSES_MICROPHONE
• SMARTGLASSES_CONTROL
• SMARTGLASSES_STATUS

SMARTGLASSES_DISPLAY_TEXT accepts plain text or JSON:

{ "text": "Show this on the glasses", "mode": "text" }

Use mode: "ai" for the default Even AI streaming/completion display path or mode: "text" for the direct Text Show path documented by EvenDemoApp. JSON input can also include pageHoldMs between pages and completionDelayMs before sending the final completion frame:

{ "text": "Paced multi-page display", "mode": "ai", "pageHoldMs": 1200, "completionDelayMs": 250 }

SMARTGLASSES_CONTROL accepts JSON such as:

{ "op": "brightness", "level": 10, "auto": true }

Supported op values: connect, disconnect, clear, exit_dashboard, exit_function, start_ai, connection_ready, page_up, page_down, rsvp_text, heartbeat, heartbeat_start, heartbeat_stop, battery_status, raw, get_serial, app_whitelist, g1_setup, silent_mode, brightness, wifi_scan, wifi_status, wifi_configure, wifi_setup, dashboard, dashboard_position, dashboard_layout, dashboard_calendar, dashboard_time_weather, headup_angle, wear_detection, navigation_start, navigation_directions, navigation_primary_image, navigation_secondary_image, navigation_poller, navigation_end, translate_setup, translate_start, translate_languages, translate_original, translate_translated, note_add, note_delete, voice_note_list, voice_note_fetch, voice_note_delete, voice_note_delete_all, notification, bmp_image.

connect pairs or reconnects the whole headset through the configured transport and sends connection-ready init packets by default. Aliases include pair, pair_headset, and connect_headset. Pass init: false to reconnect without sending init packets:

{ "op": "connect", "init": false }

disconnect closes the active headset transport. Aliases include unpair and disconnect_headset.

Bridge-backed Wi-Fi actions are available only when the active transport exposes native phone/headset setup APIs:

{ "op": "wifi_scan" }

{ "op": "wifi_configure", "ssid": "Home Wi-Fi", "password": "secret" }

Mentra-style SDK hosts can expose a native setup prompt instead of direct credential APIs:

{ "op": "wifi_setup", "reason": "Eliza needs headset Wi-Fi" }

heartbeat_start accepts optional intervalMs and immediate fields. By default it sends a heartbeat immediately and then every 8 seconds until heartbeat_stop, service disconnect, or service stop.

rsvp_text accepts text, optional wordsPerGroup, wpm, paddingChar, mode, and skipDelay. It mirrors the RSVP display pattern from the Python examples while still using the plugin's normal display packet encoder:

{ "op": "rsvp_text", "text": "Rapid serial visual presentation", "wordsPerGroup": 2, "wpm": 250 }

raw accepts data, bytes, hex, or base64 plus optional side/target (left, right, or both). This keeps low-level SDK commands such as native connection-ready packets reachable while connection_ready covers the common left/right initialization pair directly. Pass initMode: "official" to send the EvenDemoApp iOS same-init form (0x4D 0x01) to both lenses, or initMode: "android-f4" to send the EvenDemoApp Android form (0xF4 0x01) to both lenses:

{ "op": "connection_ready", "initMode": "official" }

bmp_image accepts prebuilt BMP bytes through hex, base64, or data. It can also generate a native 1-bit BMP from grayscale pixels:

{ "op": "bmp_image", "pixels": [0, 255, 255, 0], "width": 2, "height": 2 }

Dashboard/navigation/translation operations expose the packet families used by Fahrplan in addition to plain text display:

{ "op": "dashboard_position", "height": 3, "depth": 7 }

{ "op": "dashboard_calendar", "name": "Standup", "time": "13:30-14:30", "location": "Lab" }

{ "op": "navigation_directions", "totalDuration": "4 min", "totalDistance": "1 km", "direction": "Main St", "distance": "200 m", "speed": "30", "directionTurn": 3 }

Navigation image operations accept image as an array, hex, or base64 bit plane. overlay uses the same formats and defaults to an all-zero overlay when omitted, which keeps secondary navigation image action payloads compact enough for Eliza JSON extraction:

{ "op": "navigation_secondary_image", "image": "<base64 bit plane>" }

{ "op": "translate_translated", "text": "bonjour", "syncId": 3 }

Configuration

• SMARTGLASSES_TRANSPORT: auto (default), even-bridge, web-bluetooth, or noble.
• SMARTGLASSES_SCAN_TIMEOUT_MS: optional Noble BLE scan timeout in milliseconds.
• SMARTGLASSES_AUTO_INIT: send left/right initialization packets after Eliza-managed startup. Defaults to true.
• SMARTGLASSES_INIT_MODE: lens-specific (default) sends 0x4D 0x01 to the left lens and 0xF4 0x01 to the right lens; official sends 0x4D 0x01 to both lenses as in EvenDemoApp iOS; android-f4 sends 0xF4 0x01 to both lenses as in EvenDemoApp Android.

Microphone Audio

EvenHub/G2 bridge hosts and the EvenHub simulator deliver audioEvent.audioPcm as signed 16 kHz PCM16, which the service converts to Float32Array samples for onAudio. MentraOS Bluetooth SDK bridge hosts can also deliver mic_pcm events with signed 16 kHz PCM16, or mic_lc3 events that flow through the same LC3 decoder hook as direct G1 packets.

Direct G1 hardware sends 0xF1 packets containing LC3 frames. The upstream Android/iOS examples decode those frames with platform-native LC3 code, so this package preserves raw LC3 by default and exposes a decoder hook for host apps that have a native or WASM LC3 decoder:

service.setAudioDecoder(async (audio, context) => {
  if (context.encoding !== "lc3") return null;
  return decodeLc3ToPcm16(audio);
});

For plugin-managed startup, register the same hook before the service starts:

setSmartglassesAudioDecoderForRuntime(decodeLc3ToPcm16);

Upstream Evidence

The implementation was derived from the ignored research checkouts in research/even-realities/. The reviewed sources and resulting implementation choices are:

See docs/smartglasses-upstream-audit.md for the full source-to-implementation audit with local file references and test coverage.

Source	Relevant findings applied here
fabioglimb/even-toolkit	EvenHub/G2 bridge audio uses rawBridge.audioControl or callEvenApp("audioControl", { isOpen }); PCM arrives as event.audioEvent.audioPcm at 16 kHz.
BxNxM/even-dev	Simulator apps use waitForEvenAppBridge, onEvenHubEvent, sendStartUpPage, createStartUpPageContainer, rebuildPageContainer, click/double-click event codes, and the @evenrealities/evenhub-simulator automation/audio model.
emingenc/even_glasses	G1 command models for 0x4E display, RSVP word-group display, 0x0E mic control, 0x25 heartbeat, settings, notes, notifications, BMP transfer, and incoming notification handlers for init/display/notification events; BLE discovery identifies lenses by _L_ and _R_.
binarythinktank/eveng1_python_sdk	Nordic UART UUIDs, heartbeat packet [0x25, 0x06, 0x00, seq, 0x04, seq], managed heartbeat loop for connection health, display packet header shape, mic response semantics, LC3 mic chunks, and broad BLE scan followed by left/right name matching.
meyskens/fahrplan	Confirms practical G1 app surfaces: notification mirroring, dashboard/checklist style display, dashboard calendar/time-weather/setup packet families, navigation mode packets, translation overlay packets, QuickNote/voice-note 0x21 metadata, 0x1E voice-note audio fetch/delete subcommands, and live transcription using the glasses microphone; Linux BLE caveats informed the explicit hardware smoke paths.
nickustinov/weather-even-g2	G2 app event handling treats captured listEvent, textEvent, and sysEvent payloads as click/scroll sources and handles missing zero-value list indices from protobuf.
jappyjan/even-realities	EvenHub SDK wrapper patterns for startup/rebuild page containers and event listener registration.
emingenc/g1_flutter_blue_plus	Flutter BLE connection flow scans broadly, matches _L_/_R_, then discovers the Nordic UART service and TX/RX characteristics.
nickustinov/tesla-even-g2	G2 bridge event normalization for nested eventType, event_type, Event_Type, click/double-click/scroll codes, and captured-event fallback to click.
galfaroth/awesome-even-realities-g1	Cross-check index for the G1 ecosystem and the Python wrapper sources used above.
even-realities/EvenDemoApp	Official demo evidence for mic on the right lens, 0xF1 LC3 mic packet shape, text streaming, notification flows, native function exit, serial request, app whitelist packetization, connection-ready same-init packets (0x4D 0x01 to both lenses), and BMP data/end/CRC framing.
MentraOS/MentraOS	Confirmed official-style dashboard height/depth position packets (0x26 0x08 0x00 seq 0x02 0x01 height depth) and G1 battery status requests/responses.
Mentra-Community/MentraOS	Architecture context for smartglasses apps as BLE-to-phone/cloud pipelines, native bridge display APIs (displayText, clearDisplay), setMicState, mic_pcm/mic_lc3 event streams, local_transcription/transcription:* events, G1 glyph-width display profile for pixel-aware wrapping, display bandwidth limits, and hardware-dependent verification requirements.

The implementation intentionally separates direct G1 BLE transports from the EvenHub/G2 bridge transport because the upstream projects expose those as different runtime surfaces.

Verification

For the full software proof from the repository root:

bun run audit:smartglasses-software

That command runs the Facewear plugin lint/typecheck/test/app-registration gates, then the example software gate, repairs Facewear lockfile churn, and reruns the consolidation, Even research self-test, Even research audit, and completion self-test guards.

bun run --cwd plugins/plugin-facewear lint
bun run --cwd plugins/plugin-facewear typecheck
bun run --cwd plugins/plugin-facewear test
bun run --cwd plugins/plugin-facewear build
bun run --cwd packages/examples/smartglasses test
bun run --cwd packages/examples/smartglasses start
bun run --cwd packages/examples/smartglasses smoke:package
bun run --cwd packages/examples/smartglasses smoke:simulator
bun run --cwd packages/examples/smartglasses typecheck
bun run --cwd packages/examples/smartglasses verify:software
bun run --cwd packages/examples/smartglasses hardware:status-latest

Hardware Smoke

The browser hardware smoke example exercises the direct G1 BLE path:

bun run --cwd packages/examples/smartglasses dev:hardware

bun run --cwd packages/examples/smartglasses dev:hardware

Open http://127.0.0.1:5178/hardware-smoke.html in a Web Bluetooth-capable browser. Use Connect Headset for the two-step whole-headset flow; the same button prompts for the left lens and then the right lens because browsers require a user gesture for each Bluetooth picker. The page sends connection-ready/init, serial request, display, mic, brightness, dashboard, head-up angle, and wear-detection commands, then logs a structured evidence checklist for single-tap mic enable, double-tap mic disable, LC3 audio notifications, serial responses, packet writes, and final service status.

The View Manager page at /apps/smartglasses exposes the same setup and diagnostic intent inside Eliza. Its copied/downloaded report includes whole-headset scanDiagnosis, physicalBlocker, setup hint, next action, observed serial number, outbound G1 write evidence, and microphone audio metadata including sample rate, encoding, sequence when available, side, and byte count. The View Manager checklist treats serial request and observed serial response as separate evidence so a request packet alone does not satisfy the serial proof.

The EvenHub simulator can exercise the G2 bridge surface without physical hardware:

bun run --cwd packages/examples/smartglasses dev:simulator
bun run --cwd packages/examples/smartglasses simulator

bun run --cwd packages/examples/smartglasses dev:simulator
bun run --cwd packages/examples/smartglasses simulator

An automated simulator smoke harness is also available:

bun run --cwd packages/examples/smartglasses smoke:simulator

bun run --cwd packages/examples/smartglasses smoke:simulator

The simulator currently publishes 16 kHz signed PCM audio events through the EvenHub bridge. It is useful for bridge display, input, audio-control, and automation checks, but it is not a replacement for physical G1/G2 validation.

To exercise simulator microphone delivery, pass an EvenHub simulator audio input ID. The automated smoke will forward it as --aid, click to enable the bridge microphone, and wait for an audioEvent.audioPcm event:

SMARTGLASSES_SIMULATOR_AUDIO_DEVICE="coreaudio:BuiltInMicrophoneDevice" bun run --cwd packages/examples/smartglasses smoke:simulator

The Node/Bun hardware smoke exercises the Noble transport:

bun run --cwd packages/examples/smartglasses hardware:noble

It scans for left/right lenses, connects to the UART service, sends connection-ready/init, serial request, display, and settings packets, disables the right microphone, waits for a wearing physical state, then requires a single tap to enable microphone capture, speech audio, and a double tap to disable capture during the smoke window. The glasses must be out of the charging cradle and worn for tap and microphone evidence. SMARTGLASSES_SCAN_TIMEOUT_MS, SMARTGLASSES_HOLD_MS, SMARTGLASSES_WEARING_TIMEOUT_MS=30000, SMARTGLASSES_INIT_MODE=official|lens-specific|android-f4, and SMARTGLASSES_DIRECT_MIC_MS=15000 can tune scan, microphone wait time, wearing wait, init variant, and direct mic diagnostics. Set SMARTGLASSES_REPORT_PATH=./smartglasses-hardware-report.json to write a structured evidence artifact covering packet writes, serial state, headset physical/battery/device state, side-tap mic state, audio chunks, and final service status.

For the final auditable hardware proof run, use the bundled latest-report helpers. They write /tmp/smartglasses-hardware-report-latest.json, print a setup/status summary even when the smoke fails, and then run the strict validator:

bun run --cwd packages/examples/smartglasses hardware:prove:bleak

bun run --cwd packages/examples/smartglasses hardware:prove:bleak

bun run --cwd packages/examples/smartglasses hardware:prove:noble

bun run --cwd packages/examples/smartglasses hardware:prove:noble

If the short proof run connects both lenses but never observes worn state, run the watch proof wrapper. It waits longer while preserving the same latest-report, status, and freshness-validation flow:

If the status summary reports physicalBlocker: "headset_not_found", the scan did not discover either lens. Remove both lenses from the charging base, keep them near this device, and rerun the watch proof. Failed Bleak reports include discoveredDevices, discoveredDeviceCount, and discoveredG1DeviceCount to show whether CoreBluetooth saw nearby BLE devices but no Even/G1 lenses. If Noble reports physicalBlocker: "transport_unavailable" on macOS, the native Noble binding is not usable for this runtime; use the Bleak/CoreBluetooth proof or rebuild @abandonware/noble for the current Node/Bun ABI.

bun run --cwd packages/examples/smartglasses hardware:prove:bleak:watch

bun run --cwd packages/examples/smartglasses hardware:prove:bleak:watch

bun run --cwd packages/examples/smartglasses hardware:prove:noble:watch

bun run --cwd packages/examples/smartglasses hardware:prove:noble:watch

Inspect and validate that latest artifact independently before treating physical hardware support as proven. The latest validator requires the /tmp report to be fresh within ten minutes; use hardware:validate-report <path> for historical artifacts:

bun run --cwd packages/examples/smartglasses hardware:status-latest
bun run --cwd packages/examples/smartglasses hardware:validate-latest

bun run --cwd packages/examples/smartglasses hardware:status-latest
bun run --cwd packages/examples/smartglasses hardware:validate-latest

A passing report must show real lens connection, connection-ready/init writes, display packet writes, serial request and observed serial response, settings writes, observed tap events, single-tap microphone enable, double-tap microphone disable, non-empty microphone audio, and final connected service status with serial and audio counters. Simulator and mock tests are useful regression coverage, but they do not satisfy this physical hardware gate. The validator reports headsetInCradle and wearingStateNotObserved separately when the setup state explains missing tap or microphone evidence.

To validate a custom report path instead of the latest helper output:

bun run --cwd packages/examples/smartglasses hardware:validate-report ./smartglasses-hardware-report.json