426e9eeabd
Voice Workbench / headless workbench (mocked backends) (push) Has been cancelled
Voice Workbench / real acoustic lane (nightly, provisioned only) (push) Has been cancelled
ci / test (push) Has been cancelled
ci / lint-and-format (push) Has been cancelled
ci / build (push) Has been cancelled
ci / dev-startup (push) Has been cancelled
gitleaks / gitleaks (push) Has been cancelled
Markdown Links / Relative Markdown Links (push) Has been cancelled
Quality (Extended) / Homepage Build (PR smoke) (push) Has been cancelled
Quality (Extended) / Comment-only diff guard (push) Has been cancelled
Quality (Extended) / Format + Type Safety Ratchet (push) Has been cancelled
Quality (Extended) / Develop Gate (secret scan + UI determinism) (push) Has been cancelled
Quality (Extended) / Develop Gate (lint) (push) Has been cancelled
Chat shell gestures / Chat shell gesture + parity e2e (push) Has been cancelled
Cloud Gateway Discord / Test (push) Has been cancelled
Benchmark Bridge Tests / benchmark (bunx @biomejs/biome check packages/lifeops-bench/src, benchmark-lint) (push) Has been cancelled
Benchmark Bridge Tests / benchmark (bunx vitest run --config packages/lifeops-bench/vitest.config.ts --root packages/lifeops-bench --passWithNoTests, benchmark-tests) (push) Has been cancelled
Build Agent Image / build-and-push (push) Has been cancelled
Dev Smoke / bun run dev onboarding chat (push) Has been cancelled
Dev Smoke / Vite HMR dependency-level smoke (push) Has been cancelled
Electrobun Submodule Guard / electrobun gitlink is fetchable (push) Has been cancelled
Publish @elizaos/example-code / check_npm (push) Has been cancelled
Publish @elizaos/example-code / publish_npm (push) Has been cancelled
Publish @elizaos/plugin-elizacloud / verify_version (push) Has been cancelled
Publish @elizaos/plugin-elizacloud / publish_npm (push) Has been cancelled
Sandbox Live Smoke / Sandbox live smoke (push) Has been cancelled
Snap Build & Test / Build Snap (amd64) (push) Has been cancelled
Snap Build & Test / Build Snap (arm64) (push) Has been cancelled
Test Packaging / elizaos CLI global-install smoke (node + bun) (push) Has been cancelled
Cloud Gateway Webhook / Test (push) Has been cancelled
Cloud Tests / lint-and-types (push) Has been cancelled
Cloud Tests / unit-tests (push) Has been cancelled
Cloud Tests / integration-tests (push) Has been cancelled
Cloud Tests / e2e-tests (push) Has been cancelled
CodeQL Advanced / Analyze (javascript-typescript) (push) Has been cancelled
Deploy Apps Worker (Product 2) / Determine environment (push) Has been cancelled
Deploy Apps Worker (Product 2) / Deploy apps worker to apps-control host (${{ needs.determine-env.outputs.environment }}) (push) Has been cancelled
Deploy Eliza Provisioning Worker / Determine environment (push) Has been cancelled
Deploy Eliza Provisioning Worker / Deploy worker to Hetzner host (${{ needs.determine-env.outputs.environment }} @ ${{ needs.determine-env.outputs.deployment_sha }}) (push) Has been cancelled
Dev Smoke / Classify changed paths (push) Has been cancelled
supply-chain / sbom (push) Has been cancelled
supply-chain / vulnerability-scan (push) Has been cancelled
Build, Push & Deploy to Phala Cloud / build-and-push (push) Has been cancelled
Test Packaging / Validate Packaging Configs (push) Has been cancelled
Test Packaging / Build & Test PyPI Package (push) Has been cancelled
Test Packaging / PyPI on Python ${{ matrix.python }} (push) Has been cancelled
Test Packaging / Pack & Test JS Tarballs (push) Has been cancelled
UI Fixture E2E / ui-fixture-e2e (push) Has been cancelled
UI Fixture E2E / fixture-e2e (push) Has been cancelled
UI Story Gate / story-gate (push) Has been cancelled
vault-ci / test (macos-latest) (push) Has been cancelled
vault-ci / test (ubuntu-latest) (push) Has been cancelled
vault-ci / test (windows-latest) (push) Has been cancelled
vault-ci / app-core wiring tests (push) Has been cancelled
verify-patches / verify patches/CHECKSUMS.sha256 (push) Has been cancelled
Voice Benchmark Smoke / voice-emotion fixture smoke (push) Has been cancelled
Voice Benchmark Smoke / voiceagentbench fixture smoke (push) Has been cancelled
Voice Benchmark Smoke / voicebench-quality unit smoke (push) Has been cancelled
Voice Benchmark Smoke / voicebench TypeScript unit (no audio) (push) Has been cancelled
Voice Benchmark Smoke / voice bench smoke summary (push) Has been cancelled
Windows CI / windows ([bun run --cwd packages/app-core test bun run --cwd packages/elizaos test bun run --cwd packages/cloud/shared test], app-and-cli) (push) Has been cancelled
Windows CI / windows ([bun run --cwd packages/scenario-runner test bun run --cwd packages/vault test bun run --cwd packages/security test bun run --cwd plugins/plugin-coding-tools test], framework-packages) (push) Has been cancelled
Windows CI / windows ([bun run --cwd plugins/plugin-elizacloud test bun run --cwd plugins/plugin-discord test bun run --cwd plugins/plugin-anthropic test bun run --cwd plugins/plugin-openai test bun run --cwd plugins/plugin-app-control test bun run --cwd plugins/pl… (push) Has been cancelled
Windows CI / windows ([node packages/scripts/run-turbo.mjs run build --filter=@elizaos/core --filter=@elizaos/shared --filter=@elizaos/agent --concurrency=4 node packages/scripts/run-bash-linux-only.mjs scripts/verify-riscv64-buildpaths.sh node packages/scripts/run… (push) Has been cancelled
Windows CI / windows ([node packages/scripts/run-turbo.mjs run typecheck --filter=@elizaos/core --filter=@elizaos/shared --filter=@elizaos/cloud-shared --concurrency=4 bun run --cwd packages/core test bun run --cwd packages/shared test], core-runtime, 75) (push) Has been cancelled
1324 lines
44 KiB
TypeScript
1324 lines
44 KiB
TypeScript
/**
|
|
* Direct playback action for URLs, library tracks, Spotify links, and search
|
|
* results.
|
|
*
|
|
* It bridges MusicService, Discord voice state, library lookups, and
|
|
* progressive user feedback before queueing or starting audio.
|
|
*/
|
|
import {
|
|
type Action,
|
|
type ActionExample,
|
|
type ActionResult,
|
|
type HandlerCallback,
|
|
type IAgentRuntime,
|
|
logger,
|
|
type Memory,
|
|
type State,
|
|
} from "@elizaos/core";
|
|
import {
|
|
type BaseGuildVoiceChannel,
|
|
type DiscordPluginServiceLike,
|
|
isDiscordPluginServiceLike,
|
|
type MusicPlayerDiscordVoiceManager,
|
|
} from "../discordVoice";
|
|
import { MusicService } from "../service";
|
|
import { isPlaybackTransportControlOnlyMessage } from "../utils/playbackTransportIntent";
|
|
import { ProgressiveMessage } from "../utils/progressiveMessage";
|
|
import { selectedContextMatches } from "../utils/selectedContextMatches";
|
|
import { mergedOptions, requireMusicConfirmation } from "./confirmation";
|
|
import { MUSIC_PLAYER_ACTION_DOCS } from "./music-player-action-docs";
|
|
|
|
// Local contracts for cross-plugin / native deps that ship without d.ts files.
|
|
// `@elizaos/plugin-music` builds with `dts: false`, and `discord.js`
|
|
// is an optional peer dep we don't want to take a hard type dependency on.
|
|
interface DetectedMusicEntity {
|
|
confidence: number;
|
|
name: string;
|
|
type: "song" | "artist" | "album" | string;
|
|
}
|
|
|
|
interface LibraryTrack {
|
|
url: string;
|
|
title: string;
|
|
duration?: number;
|
|
playCount?: number;
|
|
}
|
|
|
|
interface SpotifyTrackInfo {
|
|
name?: string;
|
|
artists?: { name: string }[];
|
|
}
|
|
|
|
interface SpotifyClientLike {
|
|
isConfigured?: () => boolean;
|
|
getTrack?: (trackId: string) => Promise<SpotifyTrackInfo | null>;
|
|
}
|
|
|
|
interface MusicLibraryLookupService {
|
|
addSong?(song: {
|
|
url: string;
|
|
title: string;
|
|
duration?: number;
|
|
requestedBy?: string;
|
|
}): Promise<unknown>;
|
|
detectEntities?(text: string): Promise<DetectedMusicEntity[] | null>;
|
|
getLastPlayedSong?(): Promise<LibraryTrack | null>;
|
|
getSong?(url: string): Promise<LibraryTrack | null>;
|
|
getSongByUrl?(url: string): Promise<LibraryTrack | null>;
|
|
searchLibrary?(query: string, limit?: number): Promise<LibraryTrack[]>;
|
|
searchYouTube?(
|
|
query: string,
|
|
options?: { limit?: number; includeShorts?: boolean },
|
|
): Promise<
|
|
Array<{
|
|
url: string;
|
|
title: string;
|
|
duration?: number;
|
|
channel?: string;
|
|
views?: number;
|
|
}>
|
|
>;
|
|
spotifyClient?: SpotifyClientLike;
|
|
trackTrackRequest?(
|
|
entityId: string,
|
|
track: { url: string; title: string },
|
|
roomId?: string,
|
|
worldId?: string,
|
|
): Promise<void>;
|
|
}
|
|
|
|
// Discord service name constant
|
|
const DISCORD_SERVICE_NAME = "discord";
|
|
|
|
type PlayAudioOptions = Record<string, unknown> | undefined;
|
|
type ActionResultData = NonNullable<ActionResult["data"]>;
|
|
|
|
function getPlayAudioQuery(message: Memory, options: PlayAudioOptions): string {
|
|
const merged = mergedOptions(options);
|
|
const explicitQuery =
|
|
typeof merged.query === "string" && merged.query.trim().length > 0
|
|
? merged.query.trim()
|
|
: typeof merged.url === "string" && merged.url.trim().length > 0
|
|
? merged.url.trim()
|
|
: "";
|
|
return explicitQuery || message.content.text || "";
|
|
}
|
|
|
|
function failureResult(
|
|
text: string,
|
|
error: string,
|
|
data?: ActionResultData,
|
|
): ActionResult {
|
|
return { success: false, text, error, data };
|
|
}
|
|
|
|
const PLAY_AUDIO_CONTEXTS = ["media", "automation"] as const;
|
|
|
|
/**
|
|
* Extract Spotify track/album/playlist info from URL
|
|
* Returns search query that can be used to find the track on YouTube
|
|
*/
|
|
const extractSpotifyInfo = (
|
|
spotifyUrl: string,
|
|
): { type: "track" | "album" | "playlist"; searchQuery: string } | null => {
|
|
try {
|
|
const url = new URL(spotifyUrl);
|
|
const pathParts = url.pathname.split("/").filter((p) => p);
|
|
|
|
if (pathParts.length < 2) return null;
|
|
|
|
const type = pathParts[0] as "track" | "album" | "playlist";
|
|
const id = pathParts[1];
|
|
|
|
if (!["track", "album", "playlist"].includes(type) || !id) return null;
|
|
|
|
// For Spotify URLs, we'll need to convert to a search query
|
|
// The actual track info will be fetched via Spotify API if available
|
|
return { type, searchQuery: spotifyUrl }; // Return URL as search query for now
|
|
} catch {
|
|
return null;
|
|
}
|
|
};
|
|
|
|
/**
|
|
* Extract supported audio URL from the message text using regex
|
|
* Supports:
|
|
* - YouTube: various formats (youtube.com, youtu.be)
|
|
* - SoundCloud: soundcloud.com URLs
|
|
* - Spotify: open.spotify.com URLs (converted to search queries)
|
|
* - All yt-dlp supported platforms: Twitch, Vimeo, Bandcamp, Mixcloud, TikTok, Twitter/X, Instagram, etc.
|
|
*/
|
|
const extractAudioUrl = (
|
|
messageText: string,
|
|
): {
|
|
url: string | null;
|
|
isSpotify: boolean;
|
|
spotifyInfo?: { type: string; searchQuery: string };
|
|
} => {
|
|
if (!messageText) return { url: null, isSpotify: false };
|
|
|
|
// YouTube URL patterns
|
|
const youtubeRegex =
|
|
/(?:https?:\/\/)?(?:www\.)?(?:youtube\.com\/(?:watch\?v=|embed\/)|youtu\.be\/)([a-zA-Z0-9_-]{11})/;
|
|
const youtubeMatch = messageText.match(youtubeRegex);
|
|
if (youtubeMatch?.[1]) {
|
|
return {
|
|
url: `https://www.youtube.com/watch?v=${youtubeMatch[1]}`,
|
|
isSpotify: false,
|
|
};
|
|
}
|
|
|
|
// Spotify URL patterns (needs special handling - convert to search)
|
|
const spotifyRegex =
|
|
/(?:https?:\/\/)?(?:open\.)?spotify\.com\/(?:track|album|playlist)\/[a-zA-Z0-9]+(?:\?[^\s]*)?/i;
|
|
const spotifyMatch = messageText.match(spotifyRegex);
|
|
if (spotifyMatch) {
|
|
let url = spotifyMatch[0];
|
|
if (!url.startsWith("http://") && !url.startsWith("https://")) {
|
|
url = `https://${url}`;
|
|
}
|
|
const spotifyInfo = extractSpotifyInfo(url);
|
|
return {
|
|
url: null,
|
|
isSpotify: true,
|
|
spotifyInfo: spotifyInfo || undefined,
|
|
};
|
|
}
|
|
|
|
// Generic URL pattern for other platforms
|
|
// yt-dlp supports 1000+ sites, so we're permissive - if it looks like a valid URL,
|
|
// we'll pass it to yt-dlp and let it try to extract it
|
|
const genericUrlRegex =
|
|
/https?:\/\/(?:www\.)?(?:[a-zA-Z0-9-]+\.)+[a-zA-Z]{2,}(?:\/[^\s]*)?/i;
|
|
const genericMatch = messageText.match(genericUrlRegex);
|
|
if (genericMatch) {
|
|
const url = genericMatch[0];
|
|
try {
|
|
const parsedUrl = new URL(url);
|
|
const domain = parsedUrl.hostname.toLowerCase().replace(/^www\./, "");
|
|
|
|
// Exclude known non-media domains (email, file protocols, etc.)
|
|
const excludedDomains = [
|
|
"mailto:",
|
|
"file:",
|
|
"data:",
|
|
"javascript:",
|
|
"gmail.com",
|
|
"outlook.com",
|
|
"yahoo.com", // Email providers
|
|
];
|
|
|
|
// If it's a valid HTTP/HTTPS URL and not excluded, pass it to yt-dlp
|
|
// yt-dlp is very permissive and will attempt extraction from most URLs
|
|
if (parsedUrl.protocol === "http:" || parsedUrl.protocol === "https:") {
|
|
if (!excludedDomains.some((excluded) => domain.includes(excluded))) {
|
|
return { url, isSpotify: false };
|
|
}
|
|
}
|
|
} catch {
|
|
// Invalid URL, ignore
|
|
}
|
|
}
|
|
|
|
return { url: null, isSpotify: false };
|
|
};
|
|
|
|
const escapeRegExp = (value: string): string =>
|
|
value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
|
|
|
const normalizeQuotes = (value: string): string =>
|
|
value.replace(/[“”]/g, '"').replace(/[‘’]/g, "'").replace(/\s+/g, " ").trim();
|
|
|
|
const SEARCH_FILLER_TOKENS = [
|
|
"dj",
|
|
"djynai",
|
|
"ai",
|
|
"please",
|
|
"pls",
|
|
"plz",
|
|
"thanks",
|
|
"thankyou",
|
|
"song",
|
|
"track",
|
|
"music",
|
|
"video",
|
|
"play",
|
|
"queue",
|
|
"add",
|
|
];
|
|
|
|
const stripFillerTokens = (query: string): string => {
|
|
const tokens = query
|
|
.split(/[\s,]+/)
|
|
.map((token) => token.trim())
|
|
.filter(Boolean);
|
|
|
|
const filtered = tokens.filter(
|
|
(token) => !SEARCH_FILLER_TOKENS.includes(token.toLowerCase()),
|
|
);
|
|
|
|
if (filtered.length >= 2) {
|
|
return filtered.join(" ");
|
|
}
|
|
|
|
return tokens.join(" ");
|
|
};
|
|
|
|
const dedupeStrings = (values: string[]): string[] => {
|
|
const seen = new Set<string>();
|
|
const result: string[] = [];
|
|
for (const value of values) {
|
|
const normalized = value.toLowerCase();
|
|
if (normalized.length === 0) {
|
|
continue;
|
|
}
|
|
if (!seen.has(normalized)) {
|
|
seen.add(normalized);
|
|
result.push(value);
|
|
}
|
|
}
|
|
return result;
|
|
};
|
|
|
|
const buildQueryFromEntities = (
|
|
entities: DetectedMusicEntity[],
|
|
fallback: string,
|
|
): string => {
|
|
if (!entities || entities.length === 0) {
|
|
return fallback;
|
|
}
|
|
|
|
const sorted = [...entities].sort((a, b) => b.confidence - a.confidence);
|
|
const songs = sorted.filter((entity) => entity.type === "song");
|
|
const artists = sorted.filter((entity) => entity.type === "artist");
|
|
const albums = sorted.filter((entity) => entity.type === "album");
|
|
|
|
const candidateParts: string[] = [];
|
|
if (songs[0]) {
|
|
candidateParts.push(songs[0].name);
|
|
}
|
|
if (artists[0]) {
|
|
candidateParts.push(artists[0].name);
|
|
} else if (!songs[0] && albums[0]) {
|
|
candidateParts.push(albums[0].name);
|
|
}
|
|
|
|
const combined = dedupeStrings(candidateParts).join(" ").trim();
|
|
if (combined.length >= 3) {
|
|
return combined;
|
|
}
|
|
|
|
const topNames = dedupeStrings(sorted.map((entity) => entity.name))
|
|
.slice(0, 3)
|
|
.join(" ")
|
|
.trim();
|
|
if (topNames.length >= 3) {
|
|
return topNames;
|
|
}
|
|
|
|
return fallback;
|
|
};
|
|
|
|
const enhanceSearchQuery = async (
|
|
runtime: IAgentRuntime,
|
|
baseQuery: string,
|
|
originalText: string,
|
|
): Promise<string> => {
|
|
let refined = normalizeQuotes(baseQuery);
|
|
refined = stripFillerTokens(refined);
|
|
|
|
try {
|
|
const musicLibrary = runtime.getService(
|
|
"musicLibrary",
|
|
) as MusicLibraryLookupService | null;
|
|
if (musicLibrary?.detectEntities) {
|
|
const detectionSource = originalText || baseQuery;
|
|
const entities = await musicLibrary.detectEntities(detectionSource);
|
|
if (entities && entities.length > 0) {
|
|
const entityQuery = buildQueryFromEntities(entities, refined);
|
|
if (entityQuery && entityQuery.length >= 3) {
|
|
refined = entityQuery;
|
|
}
|
|
}
|
|
}
|
|
} catch (error) {
|
|
logger.debug(
|
|
`Music entity detection unavailable for query refinement: ${
|
|
error instanceof Error ? error.message : String(error)
|
|
}`,
|
|
);
|
|
}
|
|
|
|
return refined;
|
|
};
|
|
|
|
/**
|
|
* Check if the message is a pronoun reference (it, that, this, etc.)
|
|
*/
|
|
const isPronounReference = (messageText: string): boolean => {
|
|
if (!messageText) return false;
|
|
|
|
const text = messageText.toLowerCase().trim();
|
|
|
|
// Common pronoun patterns
|
|
const pronounPatterns = [
|
|
/^(?:play|queue|add)\s+(?:it|that|this)(?:\s+song|\s+one|\s+track)?$/i,
|
|
/^(?:it|that|this)$/i,
|
|
/^(?:that\s+song|this\s+song|that\s+one|this\s+one)$/i,
|
|
/^(?:play|queue)\s+(?:the\s+)?(?:last|previous|recent)\s+(?:song|one|track)$/i,
|
|
];
|
|
|
|
return pronounPatterns.some((pattern) => pattern.test(text));
|
|
};
|
|
|
|
/**
|
|
* Extract search query from message text for playing music
|
|
* Handles various natural language patterns
|
|
*/
|
|
const extractSearchQuery = (
|
|
messageText: string,
|
|
botName?: string,
|
|
): string | null => {
|
|
if (!messageText) return null;
|
|
|
|
// Remove common play/queue prefixes
|
|
let query = messageText.trim();
|
|
|
|
// Remove bot name if present ANYWHERE (case insensitive, allow punctuation)
|
|
// Use global flag to remove all occurrences
|
|
if (botName) {
|
|
const escapedName = escapeRegExp(botName);
|
|
// Match bot name with optional @, followed by optional punctuation/whitespace
|
|
const botNamePattern = new RegExp(`@?${escapedName}[\\s,:-]*`, "gi");
|
|
query = query.replace(botNamePattern, "").trim();
|
|
}
|
|
|
|
// Patterns to remove (case insensitive) - order matters!
|
|
// Using ^ anchor for start-of-string patterns
|
|
const prefixPatterns = [
|
|
// Greetings
|
|
/^(?:hey|hi|yo|alright|okay|ok)[,\s]+/i,
|
|
// Politeness
|
|
/^(?:please|pls|plz)[,\s]+/i,
|
|
// Modal verbs / requests
|
|
/^(?:can you|could you|would you|will you|can|could|would|will)[,\s]+/i,
|
|
// Additional conversational phrases
|
|
/^(?:go ahead and|i want to hear|i want|i wanna hear|i wanna|let's hear|let me hear|throw on|cue up|find and)[,\s]+/i,
|
|
// Action verbs with optional articles/words - be more aggressive
|
|
/^(?:play|queue|add|put on|start|begin|load|search for|search|find|get|show me|give me)[,\s]+(?:me\s+)?(?:the\s+)?(?:song|track|music|video|audio)?[,\s]*/i,
|
|
// Clean up any remaining common words at start
|
|
/^(?:me|the|a|an|some)[,\s]+/i,
|
|
];
|
|
|
|
// Apply each pattern multiple times until no more matches
|
|
for (const pattern of prefixPatterns) {
|
|
let prevQuery = "";
|
|
while (prevQuery !== query) {
|
|
prevQuery = query;
|
|
query = query.replace(pattern, "").trim();
|
|
}
|
|
}
|
|
|
|
// Remove any leading/trailing commas or punctuation
|
|
query = query.replace(/^[,\s:-]+|[,\s:-]+$/g, "").trim();
|
|
|
|
// Require minimum 3 characters to avoid ambiguous searches
|
|
if (query.length < 3) {
|
|
return null;
|
|
}
|
|
|
|
return stripFillerTokens(query);
|
|
};
|
|
|
|
/**
|
|
* Get the current voice channel the bot is in, or the user's voice channel
|
|
*/
|
|
const getCurrentVoiceChannel = async (
|
|
discordService: DiscordPluginServiceLike,
|
|
guildId: string,
|
|
userId?: string,
|
|
): Promise<BaseGuildVoiceChannel | null> => {
|
|
if (!discordService.client) return null;
|
|
|
|
try {
|
|
const guild = await discordService.client.guilds.fetch(guildId);
|
|
if (!guild) return null;
|
|
|
|
// First, check if the bot is in a voice channel
|
|
const botMember = guild.members.me;
|
|
if (botMember?.voice?.channel) {
|
|
return botMember.voice.channel as BaseGuildVoiceChannel;
|
|
}
|
|
|
|
// If not, check if the user is in a voice channel
|
|
if (userId) {
|
|
const member = await guild.members.fetch(userId).catch(() => null);
|
|
if (member?.voice?.channel) {
|
|
return member.voice.channel as BaseGuildVoiceChannel;
|
|
}
|
|
}
|
|
|
|
return null;
|
|
} catch (error) {
|
|
logger.error(
|
|
"Error getting current voice channel:",
|
|
error instanceof Error ? error.message : String(error),
|
|
);
|
|
return null;
|
|
}
|
|
};
|
|
|
|
export const playAudio: Action = {
|
|
name: "PLAY_AUDIO",
|
|
contexts: ["media", "automation"],
|
|
contextGate: { anyOf: ["media", "automation"] },
|
|
roleGate: { minRole: "USER" },
|
|
similes: [
|
|
"PLAY_YOUTUBE",
|
|
"PLAY_YOUTUBE_AUDIO",
|
|
"PLAY_VIDEO_AUDIO",
|
|
"PLAY_MUSIC",
|
|
"PLAY_SONG",
|
|
"PLAY_TRACK",
|
|
"START_MUSIC",
|
|
"PLAY_THIS",
|
|
"STREAM_YOUTUBE",
|
|
"PLAY_FROM_YOUTUBE",
|
|
"QUEUE_SONG",
|
|
"ADD_TO_QUEUE",
|
|
],
|
|
description:
|
|
"Start playing a new song: provide a track name, artist, search words, or a media URL. " +
|
|
"Requires confirmed:true before playback or queue changes. " +
|
|
"Never use PLAY_AUDIO for pause, resume, stop, skip, or queue — those go through PLAYBACK_OP " +
|
|
"with op=pause|resume|skip|stop|queue. Do not pass action=pause or similar params to PLAY_AUDIO. " +
|
|
MUSIC_PLAYER_ACTION_DOCS,
|
|
descriptionCompressed:
|
|
"Play new song by name/artist/URL. Not for pause/resume/stop/skip.",
|
|
parameters: [
|
|
{
|
|
name: "query",
|
|
description:
|
|
"Track name, artist, search phrase, or direct media URL to play.",
|
|
required: false,
|
|
schema: { type: "string", minLength: 3 },
|
|
},
|
|
{
|
|
name: "url",
|
|
description:
|
|
"Direct media URL to play. Prefer query for standard song requests.",
|
|
required: false,
|
|
schema: { type: "string", minLength: 8 },
|
|
},
|
|
{
|
|
name: "confirmed",
|
|
description: "Must be true to play or queue the requested audio.",
|
|
required: false,
|
|
schema: { type: "boolean", default: false },
|
|
},
|
|
],
|
|
validate: async (_runtime: IAgentRuntime, message: Memory, state: State) => {
|
|
const text = message.content.text || "";
|
|
if (isPlaybackTransportControlOnlyMessage(text)) {
|
|
return false;
|
|
}
|
|
return selectedContextMatches(state, PLAY_AUDIO_CONTEXTS);
|
|
},
|
|
handler: async (
|
|
runtime: IAgentRuntime,
|
|
message: Memory,
|
|
state: State,
|
|
options: Record<string, unknown> | undefined,
|
|
callback: HandlerCallback,
|
|
): Promise<ActionResult | undefined> => {
|
|
const messageText = getPlayAudioQuery(message, options);
|
|
// Create progressive message helper for status updates
|
|
//
|
|
// Why use ProgressiveMessage: playAudio has a deep pipeline (library check,
|
|
// search, fetch metadata, setup voice, queue). This can take 5-15 seconds.
|
|
// Without feedback, users think the bot is broken. ProgressiveMessage shows
|
|
// status updates that edit the same Discord message, keeping users informed.
|
|
const progress = new ProgressiveMessage(
|
|
callback,
|
|
message.content.source || "discord",
|
|
);
|
|
|
|
const isDiscord = message.content.source === "discord";
|
|
const maybeDiscordService = isDiscord
|
|
? runtime.getService(DISCORD_SERVICE_NAME)
|
|
: null;
|
|
const discordService = isDiscordPluginServiceLike(maybeDiscordService)
|
|
? maybeDiscordService
|
|
: null;
|
|
|
|
// For Discord, we need the Discord service
|
|
if (isDiscord && !discordService?.client) {
|
|
logger.error("Discord service not found or not initialized");
|
|
await progress.fail("Discord service is not available.");
|
|
return failureResult(
|
|
"Discord service is not available.",
|
|
"DISCORD_SERVICE_UNAVAILABLE",
|
|
);
|
|
}
|
|
if (isPlaybackTransportControlOnlyMessage(messageText)) {
|
|
const text =
|
|
"To pause, resume, stop, skip, or queue, use PLAYBACK_OP with op=pause|resume|skip|stop|queue — not PLAY_AUDIO.";
|
|
await callback({
|
|
text,
|
|
source: message.content.source || "discord",
|
|
});
|
|
return failureResult(text, "Transport control is not PLAY_AUDIO", {
|
|
query: messageText,
|
|
});
|
|
}
|
|
|
|
const preview = `Confirmation required before playing or queueing audio for: "${messageText}".`;
|
|
const confirmBlock = await requireMusicConfirmation({
|
|
runtime,
|
|
message,
|
|
actionName: "PLAY_AUDIO",
|
|
pendingKey: `play:${messageText.slice(0, 160)}`,
|
|
preview,
|
|
callback,
|
|
});
|
|
if (confirmBlock) return confirmBlock;
|
|
|
|
// Extract audio URL from message text using regex (YouTube, SoundCloud, Spotify, etc.)
|
|
const urlResult = extractAudioUrl(messageText);
|
|
let audioUrl = urlResult.url;
|
|
let videoTitle = "Unknown Title";
|
|
let videoDuration: number | undefined;
|
|
let searchQuery: string | null = null;
|
|
|
|
try {
|
|
// Initial status update (transient - will be replaced quickly)
|
|
//
|
|
// Why no "important" flag: Library check is usually instant (< 100ms).
|
|
// Showing this on web/CLI would just be noise. On Discord it's fine
|
|
// because it gets edited away immediately.
|
|
progress.update("🔍 Looking up track...");
|
|
const musicLibraryLookup = runtime.getService(
|
|
"musicLibrary",
|
|
) as MusicLibraryLookupService | null;
|
|
|
|
// Lazy-load play-dl for video info
|
|
const play = await import("@vookav2/play-dl").then((m) => m.default || m);
|
|
|
|
// Handle Spotify URLs - convert to YouTube search
|
|
if (urlResult.isSpotify && urlResult.spotifyInfo) {
|
|
try {
|
|
// Try to get track info from Spotify API if available
|
|
const musicLibrary = runtime.getService(
|
|
"musicLibrary",
|
|
) as MusicLibraryLookupService | null;
|
|
let spotifyTrackInfo: SpotifyTrackInfo | null = null;
|
|
|
|
if (musicLibrary?.spotifyClient) {
|
|
const spotifyClient = musicLibrary.spotifyClient;
|
|
if (spotifyClient.isConfigured?.()) {
|
|
const spotifyUrl = urlResult.spotifyInfo.searchQuery;
|
|
if (urlResult.spotifyInfo.type === "track") {
|
|
const trackId = spotifyUrl.match(/track\/([a-zA-Z0-9]+)/)?.[1];
|
|
if (trackId && spotifyClient.getTrack) {
|
|
spotifyTrackInfo = await spotifyClient
|
|
.getTrack(trackId)
|
|
.catch(() => null);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// Build search query from Spotify track info or use URL
|
|
if (spotifyTrackInfo?.name) {
|
|
const artistNames =
|
|
spotifyTrackInfo.artists?.map((a) => a.name).join(" ") || "";
|
|
searchQuery = `${artistNames} ${spotifyTrackInfo.name}`.trim();
|
|
videoTitle = spotifyTrackInfo.name;
|
|
logger.info(
|
|
`Spotify track detected: ${videoTitle} - searching YouTube...`,
|
|
);
|
|
} else {
|
|
// Fallback: extract from URL or use generic search
|
|
searchQuery = `spotify:${urlResult.spotifyInfo.type}`;
|
|
logger.info(
|
|
`Spotify ${urlResult.spotifyInfo.type} detected - will search YouTube for similar content`,
|
|
);
|
|
}
|
|
|
|
// Set audioUrl to null so we trigger YouTube search
|
|
audioUrl = null;
|
|
} catch (error) {
|
|
logger.debug(`Error processing Spotify URL: ${error}`);
|
|
// Fallback to YouTube search
|
|
searchQuery = "spotify track";
|
|
audioUrl = null;
|
|
}
|
|
}
|
|
|
|
// If no URL found, try to search for the content
|
|
if (!audioUrl) {
|
|
// Check if this is a pronoun reference (it, that, this, etc.)
|
|
if (isPronounReference(messageText)) {
|
|
// Try to get the last played song from music library (if available)
|
|
try {
|
|
const musicLibrary = runtime.getService(
|
|
"musicLibrary",
|
|
) as MusicLibraryLookupService | null;
|
|
if (musicLibrary?.getLastPlayedSong) {
|
|
const lastSong = await musicLibrary.getLastPlayedSong();
|
|
if (lastSong) {
|
|
logger.info(
|
|
`Pronoun reference detected, using last played song: ${lastSong.title}`,
|
|
);
|
|
audioUrl = lastSong.url;
|
|
videoTitle = lastSong.title;
|
|
videoDuration = lastSong.duration;
|
|
searchQuery = `(reference: ${lastSong.title})`; // Mark as reference
|
|
}
|
|
}
|
|
} catch (error) {
|
|
logger.debug(
|
|
`Music library not available for pronoun resolution: ${error}`,
|
|
);
|
|
}
|
|
|
|
if (!audioUrl) {
|
|
await progress.fail(
|
|
"I couldn't find what song you're referring to. No songs have been played recently. Please specify the song name or provide a link.",
|
|
);
|
|
return failureResult(
|
|
"I couldn't find what song you're referring to. No songs have been played recently. Please specify the song name or provide a link.",
|
|
"NO_RECENT_TRACK_REFERENCE",
|
|
{ query: messageText },
|
|
);
|
|
}
|
|
} else {
|
|
// Extract bot name from runtime for filtering
|
|
const botName = runtime.character.name;
|
|
searchQuery = extractSearchQuery(messageText, botName);
|
|
|
|
if (!searchQuery) {
|
|
await progress.fail(
|
|
"I couldn't understand what you want me to play. Please provide a link or tell me what song to search for (at least 3 characters).",
|
|
);
|
|
return failureResult(
|
|
"I couldn't understand what you want me to play. Please provide a link or tell me what song to search for (at least 3 characters).",
|
|
"UNPARSEABLE_PLAY_REQUEST",
|
|
{ query: messageText },
|
|
);
|
|
}
|
|
}
|
|
}
|
|
|
|
if (!audioUrl && searchQuery) {
|
|
searchQuery = await enhanceSearchQuery(
|
|
runtime,
|
|
searchQuery,
|
|
messageText,
|
|
);
|
|
}
|
|
|
|
// Track whether we found the song in the local library
|
|
let foundInLibrary = false;
|
|
|
|
// Only search if we don't already have a URL (from pronoun reference)
|
|
if (!audioUrl && searchQuery) {
|
|
// Step 1: Check local music library first
|
|
try {
|
|
const musicLibrary = runtime.getService(
|
|
"musicLibrary",
|
|
) as MusicLibraryLookupService | null;
|
|
if (musicLibrary?.searchLibrary) {
|
|
logger.debug(`Searching local music library for: ${searchQuery}`);
|
|
const libraryResults = await musicLibrary.searchLibrary(
|
|
searchQuery,
|
|
5,
|
|
);
|
|
|
|
if (libraryResults && libraryResults.length > 0) {
|
|
// Found in library! Use the top result (sorted by play count)
|
|
const topLibraryResult = libraryResults[0];
|
|
audioUrl = topLibraryResult.url;
|
|
videoTitle = topLibraryResult.title;
|
|
videoDuration = topLibraryResult.duration;
|
|
foundInLibrary = true;
|
|
|
|
logger.info(
|
|
`Found in library (${topLibraryResult.playCount} plays): ${videoTitle} at ${audioUrl}`,
|
|
);
|
|
} else {
|
|
logger.debug(`No matches in library for: ${searchQuery}`);
|
|
}
|
|
}
|
|
} catch (error) {
|
|
logger.debug(`Music library search unavailable: ${error}`);
|
|
// Continue to YouTube search if library search fails
|
|
}
|
|
|
|
// Step 2: If not found in library, search for track
|
|
if (!audioUrl) {
|
|
// Important update - show on all platforms since search can take a while
|
|
//
|
|
// Why "important: true": YouTube/web search takes 2-10 seconds. Without
|
|
// feedback, users think the bot froze. Even on web/CLI where we can't
|
|
// edit, this is worth showing as a new message so users know we're working.
|
|
progress.update("🔍 Searching for track...", { important: true });
|
|
logger.debug(`Searching for: ${searchQuery}`);
|
|
|
|
try {
|
|
const searchResults = await musicLibraryLookup?.searchYouTube?.(
|
|
searchQuery,
|
|
{ limit: 3 },
|
|
);
|
|
|
|
if (!searchResults || searchResults.length === 0) {
|
|
await progress.fail(
|
|
`I couldn't find any results for "${searchQuery}". Try being more specific or providing a direct link.`,
|
|
);
|
|
return failureResult(
|
|
`I couldn't find any results for "${searchQuery}". Try being more specific or providing a direct link.`,
|
|
"NO_SEARCH_RESULTS",
|
|
{ searchQuery },
|
|
);
|
|
}
|
|
|
|
// Use first result
|
|
const topResult = searchResults[0];
|
|
audioUrl = topResult.url;
|
|
videoTitle = topResult.title;
|
|
videoDuration = topResult.duration;
|
|
|
|
logger.info(`Found on YouTube: ${videoTitle} at ${audioUrl}`);
|
|
} catch (error) {
|
|
logger.error(
|
|
"Error searching YouTube:",
|
|
error instanceof Error ? error.message : String(error),
|
|
);
|
|
const errorMsg =
|
|
error instanceof Error ? error.message : String(error);
|
|
await progress.fail(
|
|
`I had trouble searching for "${searchQuery}". ${
|
|
errorMsg.includes("browseEndpoint") ||
|
|
errorMsg.includes("navigationEndpoint")
|
|
? "Try being more specific with song and artist names."
|
|
: "Please try again or provide a direct link."
|
|
}`,
|
|
);
|
|
return failureResult(
|
|
`I had trouble searching for "${searchQuery}". ${
|
|
errorMsg.includes("browseEndpoint") ||
|
|
errorMsg.includes("navigationEndpoint")
|
|
? "Try being more specific with song and artist names."
|
|
: "Please try again or provide a direct link."
|
|
}`,
|
|
errorMsg,
|
|
{ searchQuery },
|
|
);
|
|
}
|
|
}
|
|
}
|
|
|
|
// For URL-based requests, get video/track info
|
|
if (audioUrl && !videoDuration) {
|
|
// First, check if we have this URL in our music library
|
|
try {
|
|
const musicLibrary = runtime.getService(
|
|
"musicLibrary",
|
|
) as MusicLibraryLookupService | null;
|
|
const getSongByUrl =
|
|
musicLibrary?.getSongByUrl ?? musicLibrary?.getSong;
|
|
if (musicLibrary && getSongByUrl) {
|
|
logger.debug(`Checking library for URL: ${audioUrl}`);
|
|
const libraryTrack = await getSongByUrl.call(
|
|
musicLibrary,
|
|
audioUrl,
|
|
);
|
|
|
|
if (libraryTrack) {
|
|
// Found in library! Use cached info
|
|
videoTitle = libraryTrack.title;
|
|
videoDuration = libraryTrack.duration;
|
|
foundInLibrary = true;
|
|
logger.info(
|
|
`Found URL in library (${libraryTrack.playCount || 0} plays): ${videoTitle}`,
|
|
);
|
|
}
|
|
}
|
|
} catch (error) {
|
|
logger.debug(`Music library lookup by URL unavailable: ${error}`);
|
|
// Continue to fetch video info if library lookup fails
|
|
}
|
|
|
|
// Only fetch video info if we didn't find it in the library
|
|
if (!videoDuration) {
|
|
// Check if it's a YouTube URL (play-dl only supports YouTube)
|
|
const isYouTube =
|
|
audioUrl.includes("youtube.com") || audioUrl.includes("youtu.be");
|
|
|
|
if (isYouTube) {
|
|
// Validate YouTube URL
|
|
if (!play.yt_validate(audioUrl)) {
|
|
await progress.fail("The provided URL is not valid.");
|
|
return failureResult(
|
|
"The provided URL is not valid.",
|
|
"INVALID_URL",
|
|
{
|
|
audioUrl,
|
|
},
|
|
);
|
|
}
|
|
|
|
// Get video info for YouTube URLs
|
|
const videoInfo = await play.video_info(audioUrl).catch((error) => {
|
|
logger.error(
|
|
"Error getting YouTube video info:",
|
|
error instanceof Error ? error.message : String(error),
|
|
);
|
|
return null;
|
|
});
|
|
|
|
if (!videoInfo) {
|
|
await progress.fail(
|
|
"I could not access that content. It may be private, unavailable, or the URL may be invalid.",
|
|
);
|
|
return failureResult(
|
|
"I could not access that content. It may be private, unavailable, or the URL may be invalid.",
|
|
"INACCESSIBLE_MEDIA",
|
|
{ audioUrl },
|
|
);
|
|
}
|
|
|
|
videoTitle = videoInfo.video_details.title || "Unknown Title";
|
|
videoDuration = videoInfo.video_details.durationInSec;
|
|
} else {
|
|
// For SoundCloud and other platforms, yt-dlp will handle it
|
|
// We can try to extract title from URL or use a generic title
|
|
// The actual title will be determined when the track is played
|
|
const urlParts = audioUrl.split("/");
|
|
const lastPart = urlParts[urlParts.length - 1];
|
|
videoTitle = lastPart
|
|
? decodeURIComponent(lastPart.replace(/[-_]/g, " "))
|
|
: "Unknown Title";
|
|
logger.info(`Using URL for non-YouTube platform: ${audioUrl}`);
|
|
}
|
|
}
|
|
}
|
|
|
|
const room = state.data.room || (await runtime.getRoom(message.roomId));
|
|
let currentServerId = room?.serverId;
|
|
|
|
// For non-Discord platforms, create a deterministic server ID
|
|
if (!currentServerId) {
|
|
if (isDiscord) {
|
|
await progress.fail("I could not determine which server you are in.");
|
|
return failureResult(
|
|
"I could not determine which server you are in.",
|
|
"DISCORD_SERVER_UNRESOLVED",
|
|
);
|
|
} else {
|
|
// For web/CLI, use room ID as server ID
|
|
currentServerId = `web-${message.roomId}`;
|
|
}
|
|
} else if (!isDiscord) {
|
|
// Prefix with 'web-' for non-Discord platforms to avoid conflicts with Discord server IDs
|
|
currentServerId = `web-${currentServerId}`;
|
|
}
|
|
// For Discord, use the server ID as-is (no prefix)
|
|
|
|
// Use entityId (UUID) not fromId (Discord snowflake) for requestedBy
|
|
// WHY: fromId in metadata is the raw Discord snowflake ID for security reference
|
|
// entityId is the proper UUID created by createUniqueUuid(runtime, discordId)
|
|
const requestUserId = message.entityId;
|
|
|
|
// Discord-specific voice channel logic
|
|
// For Discord, use the original server ID (without web- prefix)
|
|
const discordServerId = isDiscord
|
|
? room?.serverId || currentServerId
|
|
: null;
|
|
let voiceManager: MusicPlayerDiscordVoiceManager | null = null;
|
|
if (isDiscord && discordService && discordServerId) {
|
|
// Get current voice channel
|
|
let voiceChannel = await getCurrentVoiceChannel(
|
|
discordService,
|
|
discordServerId,
|
|
requestUserId,
|
|
);
|
|
|
|
// If bot is not in a voice channel, try to join the user's voice channel
|
|
if (!voiceChannel && requestUserId) {
|
|
const guild =
|
|
await discordService.client.guilds.fetch(discordServerId);
|
|
const member = await guild.members
|
|
.fetch(requestUserId)
|
|
.catch(() => null);
|
|
if (member?.voice?.channel) {
|
|
voiceChannel = member.voice.channel as BaseGuildVoiceChannel;
|
|
voiceManager = discordService.voiceManager ?? null;
|
|
if (voiceManager) {
|
|
await voiceManager.joinChannel(voiceChannel);
|
|
}
|
|
}
|
|
}
|
|
|
|
if (!voiceChannel) {
|
|
await progress.fail(
|
|
"I'm not in a voice channel, and you don't appear to be in one either. Please join a voice channel first, or ask me to join one.",
|
|
);
|
|
return failureResult(
|
|
"I'm not in a voice channel, and you don't appear to be in one either. Please join a voice channel first, or ask me to join one.",
|
|
"VOICE_CHANNEL_REQUIRED",
|
|
);
|
|
}
|
|
|
|
voiceManager = discordService.voiceManager ?? null;
|
|
if (!voiceManager) {
|
|
await progress.fail(
|
|
"Voice functionality is not available at the moment.",
|
|
);
|
|
return failureResult(
|
|
"Voice functionality is not available at the moment.",
|
|
"VOICE_MANAGER_UNAVAILABLE",
|
|
);
|
|
}
|
|
|
|
// Get voice connection
|
|
let connection = voiceManager.getVoiceConnection(discordServerId);
|
|
if (!connection) {
|
|
// Try to join the voice channel
|
|
await voiceManager.joinChannel(voiceChannel);
|
|
// Wait a bit for connection to establish
|
|
await new Promise((resolve) => setTimeout(resolve, 1000));
|
|
connection = voiceManager.getVoiceConnection(discordServerId);
|
|
if (!connection) {
|
|
await progress.fail(
|
|
"I could not establish a voice connection. Please try again.",
|
|
);
|
|
return failureResult(
|
|
"I could not establish a voice connection. Please try again.",
|
|
"VOICE_CONNECTION_FAILED",
|
|
);
|
|
}
|
|
}
|
|
|
|
// Register music channel (channel 1) if not already registered
|
|
// This ensures the channel has the right configuration for music playback
|
|
voiceManager.emit("registerChannel", {
|
|
channel: 1,
|
|
priority: 50,
|
|
canPause: true,
|
|
interruptible: true,
|
|
volume: 1.0,
|
|
duckVolume: 0.3,
|
|
});
|
|
}
|
|
|
|
// Get or create music service
|
|
let musicService = runtime.getService("music") as MusicService;
|
|
if (!musicService) {
|
|
musicService = new MusicService(runtime);
|
|
// Note: Service registration would typically happen in plugin init
|
|
// For now, we'll use it directly
|
|
}
|
|
|
|
// For Discord, ensure voice manager is set
|
|
if (isDiscord && voiceManager) {
|
|
musicService.setVoiceManager(voiceManager);
|
|
}
|
|
|
|
// Check queue state BEFORE adding track
|
|
// This tells us if the track will start playing immediately or be queued
|
|
const currentTrack = musicService.getCurrentTrack(currentServerId);
|
|
const queueLength = musicService.getQueueList(currentServerId).length;
|
|
const willPlayImmediately = !currentTrack && queueLength === 0;
|
|
|
|
// Build response message BEFORE adding track
|
|
let responseText: string;
|
|
if (willPlayImmediately) {
|
|
// Track will start playing immediately
|
|
if (foundInLibrary) {
|
|
responseText = `📚 Playing from your library: **${videoTitle}**`;
|
|
} else if (searchQuery) {
|
|
responseText = `🔎 Found and now playing: **${videoTitle}**`;
|
|
} else {
|
|
responseText = `Now playing: **${videoTitle}**`;
|
|
}
|
|
} else {
|
|
// Track will be added to queue
|
|
const position = queueLength + 1; // Position after adding
|
|
if (foundInLibrary) {
|
|
responseText = `📚 Added from your library (position ${position}): **${videoTitle}**`;
|
|
} else if (searchQuery) {
|
|
responseText = `🔎 Found and added to queue (position ${position}): **${videoTitle}**`;
|
|
} else {
|
|
responseText = `Added to queue (position ${position}): **${videoTitle}**`;
|
|
}
|
|
}
|
|
|
|
// For non-Discord platforms, provide web streaming URLs
|
|
if (!isDiscord) {
|
|
const baseUrl = process.env.SERVER_URL || "http://localhost:3000";
|
|
const streamUrl = `${baseUrl}/music-player/stream?guildId=${encodeURIComponent(currentServerId)}`;
|
|
const nowPlayingUrl = `${baseUrl}/music-player/now-playing?guildId=${encodeURIComponent(currentServerId)}`;
|
|
responseText += `\n\n🌐 Web streaming: ${streamUrl}`;
|
|
responseText += `\n📊 Track info: ${nowPlayingUrl}`;
|
|
}
|
|
|
|
// Show we're setting up playback (transient - skip on non-editing platforms)
|
|
//
|
|
// Why no "important" flag: Voice setup and queue add are fast (< 200ms).
|
|
// This is just "progressive polish" for Discord - not worth showing on
|
|
// web/CLI where it would create an extra message.
|
|
progress.update("✨ Setting up playback...");
|
|
|
|
// Add track to queue (this may start playback immediately)
|
|
const track = await musicService.addTrack(currentServerId, {
|
|
url: audioUrl as string,
|
|
title: videoTitle,
|
|
duration: videoDuration,
|
|
requestedBy: requestUserId,
|
|
});
|
|
|
|
// Send final message with track info
|
|
logger.info(`[PLAY_AUDIO] Completing with: ${responseText}`);
|
|
await progress.complete(responseText);
|
|
logger.info("[PLAY_AUDIO] Progressive complete sent successfully");
|
|
|
|
// Now do background operations (library tracking, memory creation)
|
|
// These don't need to block the user feedback
|
|
|
|
// Persist an action-log memory so the agent remembers what it played.
|
|
// WHY entityId = agentId: this is the agent's action, not a user
|
|
// message. Using message.entityId (the user) made it appear as a
|
|
// blank user bubble in the web chat because the content has
|
|
// `thought` but no `text`.
|
|
runtime
|
|
.createMemory(
|
|
{
|
|
entityId: runtime.agentId,
|
|
agentId: message.agentId,
|
|
roomId: message.roomId,
|
|
content: {
|
|
source: "action",
|
|
thought: searchQuery
|
|
? `Searched for "${searchQuery}" and playing: ${videoTitle}`
|
|
: `Playing audio: ${videoTitle}`,
|
|
actions: ["PLAY_AUDIO"],
|
|
},
|
|
metadata: {
|
|
type: "custom" as const,
|
|
kind: "PLAY_AUDIO",
|
|
audioUrl: audioUrl,
|
|
videoTitle,
|
|
trackId: track.id,
|
|
...(searchQuery && { searchQuery }),
|
|
},
|
|
},
|
|
"messages",
|
|
)
|
|
.catch((error) => logger.warn(`Failed to create memory: ${error}`));
|
|
|
|
// Track the request in user preferences (background)
|
|
try {
|
|
const musicLibrary = runtime.getService(
|
|
"musicLibrary",
|
|
) as MusicLibraryLookupService | null;
|
|
if (musicLibrary?.trackTrackRequest) {
|
|
musicLibrary
|
|
.trackTrackRequest(
|
|
message.entityId,
|
|
{
|
|
url: audioUrl as string,
|
|
title: videoTitle,
|
|
},
|
|
message.roomId,
|
|
message.worldId,
|
|
)
|
|
.catch((error: Error) =>
|
|
logger.warn(`Failed to track request: ${error}`),
|
|
);
|
|
}
|
|
} catch (error) {
|
|
logger.warn(`Failed to track request: ${error}`);
|
|
}
|
|
|
|
// Save to global music library for future reference (background)
|
|
try {
|
|
const musicLibrary = runtime.getService(
|
|
"musicLibrary",
|
|
) as MusicLibraryLookupService | null;
|
|
logger.info(
|
|
`[PlayAudio] Music library service: ${musicLibrary ? "found" : "NOT FOUND"}`,
|
|
);
|
|
if (musicLibrary?.addSong) {
|
|
logger.info(
|
|
`[PlayAudio] Adding to music library: "${videoTitle}" (${audioUrl})`,
|
|
);
|
|
musicLibrary
|
|
.addSong({
|
|
url: audioUrl as string,
|
|
title: videoTitle,
|
|
duration: videoDuration,
|
|
requestedBy: requestUserId,
|
|
})
|
|
.then(() => {
|
|
logger.info(
|
|
`[PlayAudio] ✅ Song added to music library: "${videoTitle}"`,
|
|
);
|
|
})
|
|
.catch((error: Error) => {
|
|
logger.error(
|
|
`[PlayAudio] ❌ Failed to add song to library: ${error.message}`,
|
|
);
|
|
});
|
|
} else if (musicLibrary) {
|
|
logger.warn(
|
|
`[PlayAudio] Music library service found but addSong method missing. Available methods: ${Object.keys(musicLibrary).join(", ")}`,
|
|
);
|
|
}
|
|
} catch (error) {
|
|
logger.error(`[PlayAudio] Error accessing music library: ${error}`);
|
|
}
|
|
|
|
// Memories already created via callback, return success
|
|
return {
|
|
success: true,
|
|
text: responseText,
|
|
data: {
|
|
query: messageText,
|
|
searchQuery,
|
|
audioUrl,
|
|
videoTitle,
|
|
videoDuration,
|
|
foundInLibrary,
|
|
willPlayImmediately,
|
|
queueLengthBeforeAdd: queueLength,
|
|
trackId: track.id,
|
|
serverId: currentServerId,
|
|
},
|
|
};
|
|
} catch (error) {
|
|
const errorMessage =
|
|
error instanceof Error ? error.message : String(error);
|
|
logger.error("Error playing audio:", errorMessage);
|
|
|
|
// Provide specific error messages based on the error type
|
|
let userMessage =
|
|
"❌ I encountered an error while trying to play that track.";
|
|
|
|
const errorLower = errorMessage.toLowerCase();
|
|
if (
|
|
errorLower.includes("age") &&
|
|
(errorLower.includes("verification") ||
|
|
errorLower.includes("restriction") ||
|
|
errorLower.includes("restricted"))
|
|
) {
|
|
userMessage =
|
|
"⚠️ This content requires age verification or authentication.";
|
|
} else if (
|
|
errorLower.includes("region") ||
|
|
errorLower.includes("not available") ||
|
|
errorLower.includes("unavailable")
|
|
) {
|
|
userMessage =
|
|
"⚠️ This content is not available in your region or has been removed.";
|
|
} else if (
|
|
errorLower.includes("private") ||
|
|
errorLower.includes("unlisted")
|
|
) {
|
|
userMessage =
|
|
"⚠️ This content is private or unlisted and cannot be accessed.";
|
|
} else if (
|
|
errorLower.includes("authentication") ||
|
|
errorLower.includes("cookies") ||
|
|
errorLower.includes("login")
|
|
) {
|
|
userMessage = "⚠️ This content requires authentication.";
|
|
} else if (errorLower.includes("unable to access")) {
|
|
userMessage = "⚠️ Unable to access this track.";
|
|
}
|
|
|
|
await progress.fail(userMessage);
|
|
return failureResult(userMessage, errorMessage, { query: messageText });
|
|
}
|
|
},
|
|
examples: [
|
|
[
|
|
{
|
|
name: "{{name1}}",
|
|
content: {
|
|
text: "Play https://www.youtube.com/watch?v=dQw4w9WgXcQ",
|
|
},
|
|
},
|
|
{
|
|
name: "{{name2}}",
|
|
content: {
|
|
text: "I'll play that YouTube video in the voice channel!",
|
|
actions: ["PLAY_AUDIO"],
|
|
},
|
|
},
|
|
],
|
|
[
|
|
{
|
|
name: "{{name1}}",
|
|
content: {
|
|
text: "Play Surefire by Wilderado",
|
|
},
|
|
},
|
|
{
|
|
name: "{{name2}}",
|
|
content: {
|
|
text: "I'll search for that song and play it!",
|
|
actions: ["PLAY_AUDIO"],
|
|
},
|
|
},
|
|
],
|
|
[
|
|
{
|
|
name: "{{name1}}",
|
|
content: {
|
|
text: "Can you play Bohemian Rhapsody?",
|
|
},
|
|
},
|
|
{
|
|
name: "{{name2}}",
|
|
content: {
|
|
text: "Finding and playing that now!",
|
|
actions: ["PLAY_AUDIO"],
|
|
},
|
|
},
|
|
],
|
|
[
|
|
{
|
|
name: "{{name1}}",
|
|
content: {
|
|
text: "Queue Never Gonna Give You Up",
|
|
},
|
|
},
|
|
{
|
|
name: "{{name2}}",
|
|
content: {
|
|
text: "I'll search for that and add it to the queue!",
|
|
actions: ["PLAY_AUDIO"],
|
|
},
|
|
},
|
|
],
|
|
[
|
|
{
|
|
name: "{{name1}}",
|
|
content: {
|
|
text: "play music from youtube.com/watch?v=dQw4w9WgXcQ",
|
|
},
|
|
},
|
|
{
|
|
name: "{{name2}}",
|
|
content: {
|
|
text: "I'll stream that YouTube video's audio!",
|
|
actions: ["PLAY_AUDIO"],
|
|
},
|
|
},
|
|
],
|
|
] as ActionExample[][],
|
|
} as Action;
|
|
|
|
export default playAudio;
|