Files
dyolink/backend/src/modules/voice/voice.service.ts

353 lines
11 KiB
TypeScript
Raw Normal View History

feat(backend): voice extraction endpoint POST /voice/extract behind JwtAuthGuard + ClinicOrgGuard, plus GET /voice/availability so the frontend can decide whether to render the microphone — it cannot learn that from NEXT_PUBLIC_*, which are baked in at build time. Audio is held in memory for the request only: never written to disk, never a Prisma row. The transcript goes back to the client and is not persisted. What is logged is structured and patient-free — clip length, which fields resolved, unresolved count, vendor cost, outcome — with log lines as the interim sink until this repo has metrics infrastructure. On extraction failure the transcript still travels back in the error details, so the words the clinician already paid for can be salvaged into a note. v1 ships ungated beyond a configured locale profile; the Plan.features design is deferred, not dropped. From review of this commit, four of which were load-bearing: - Express's 100 kb default body limit rejected any recording past ~20 seconds, making the endpoint unusable at its own 2-minute cap. Body parsers are now registered explicitly with a 10 MB limit scoped to the voice route only. Verified empirically: 600 KB reaches /api/voice/extract, while /api/auth/login still 413s. - ThrottlerGuard keys on req.ip, so behind nginx the whole deployment would share one bucket and an abuser rotating IPs would bypass it. VoiceThrottlerGuard keys on the user id instead — with no plan gate, this is the only control on metered vendor spend. - ThrottlerException had no 429 fallback and surfaced as INTERNAL_ERROR; the guard now throws VOICE_RATE_LIMITED directly. - durationMs was optional, so omitting it bypassed VOICE_MAX_RECORDING_MS entirely. It is required. - VOICE_UNSUPPORTED_FORMAT was dead code — the DTO's @IsIn already rejects unknown containers — so it is gone rather than left unreachable. ThrottlerModule is deliberately not bound as a global APP_GUARD: a global ThrottlerGuard rate-limits every route against every named throttler, which would have capped the whole API at the voice limit. All seven remaining VOICE_* codes have errors.* keys in en, fa and nl. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 18:27:14 +03:30
import { HttpStatus, Injectable, Logger } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { LinkStatus } from '@prisma/client';
import { PrismaService } from '../../../prisma/prisma.service';
import { AppException, ErrorCode } from '../../common/errors';
import {
civilDateInZone,
isValidIanaTimeZone,
} from '../../common/zoned-civil-time';
import type { VoiceConfig, VoiceProfile } from '../../configs/configurations';
import { hasEffectivePermission } from '../../common/membership-permissions';
import { normalizeCatalogLocale } from '../catalog/catalog-label.service';
import { ProsthesisCatalogService } from '../prosthesis-catalog/prosthesis-catalog.service';
import { TreatmentCatalogService } from '../treatment-catalog/treatment-catalog.service';
feat(frontend): split Add detail into a segmented control with voice The microphone becomes the second segment of the Add detail button, built like the detail chip's trash affordance in the same file — an overflow-hidden rounded wrapper holding two raw <button>s divided by border-s — rather than two shared Buttons, which each hardcode their own rounding and would fight a segmented control. border-s puts the mic at the logical end: visually right in en/nl, visually left in fa, on the same side as the chip's trash in both directions. The two halves share a wrapper and nothing else. Add keeps its exact behaviour. The control never changes size while recording; the timer and level meter live in a bar between the header row and the chip strip, because the header is sm:justify-between and growing the button would shove the row on every start and stop. The meter exists to prove the microphone is actually hearing something — silence and a dead mic look identical otherwise. Voice reaches the editor as one optional `voice` prop, so its absence *is* the unavailable state and the two cannot disagree. Fixes from review of this commit: - mountedRef was set false on unmount and never re-armed, so under StrictMode the hook was permanently "unmounted" in dev and recording silently never started. - onStart guarded only on `phase`, which does not change until getUserMedia resolves; a second click during the permission prompt orphaned the first MediaStream, leaving the mic indicator lit. - Week start is now per locale. "Next Thursday" is week-relative, and hardcoding Saturday put an en/nl clinician's deadline a week out. - A missing `which` on a weekday intent is read as "this" rather than failing — a bare weekday carries no qualifier, and rejecting it discarded a real deadline. - durationMs is client-reported and so is a claim, not enforcement; the cap is now also checked against the vendor's own usage.seconds. - Blob type falls back to the recorder's actual mimeType before webm, so old Safari's mp4/aac clips are not mislabelled. Two review findings were rejected as incorrect, both re-verified against live sources: google/gemini-3.7-flash does exist on OpenRouter (1M context, $0.375/$1.875 per M), and base64 JSON input_audio is the documented primary path for /audio/transcriptions, with multipart as the OpenAI-compatible alternative. The spec's stale "unverified" note is corrected, and the provider now has unit tests covering the request shape, usage parsing, and that a vendor error body never reaches the thrown message. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 19:47:10 +03:30
import { weekStartForLocale } from './due-date.resolver';
feat(backend): voice extraction endpoint POST /voice/extract behind JwtAuthGuard + ClinicOrgGuard, plus GET /voice/availability so the frontend can decide whether to render the microphone — it cannot learn that from NEXT_PUBLIC_*, which are baked in at build time. Audio is held in memory for the request only: never written to disk, never a Prisma row. The transcript goes back to the client and is not persisted. What is logged is structured and patient-free — clip length, which fields resolved, unresolved count, vendor cost, outcome — with log lines as the interim sink until this repo has metrics infrastructure. On extraction failure the transcript still travels back in the error details, so the words the clinician already paid for can be salvaged into a note. v1 ships ungated beyond a configured locale profile; the Plan.features design is deferred, not dropped. From review of this commit, four of which were load-bearing: - Express's 100 kb default body limit rejected any recording past ~20 seconds, making the endpoint unusable at its own 2-minute cap. Body parsers are now registered explicitly with a 10 MB limit scoped to the voice route only. Verified empirically: 600 KB reaches /api/voice/extract, while /api/auth/login still 413s. - ThrottlerGuard keys on req.ip, so behind nginx the whole deployment would share one bucket and an abuser rotating IPs would bypass it. VoiceThrottlerGuard keys on the user id instead — with no plan gate, this is the only control on metered vendor spend. - ThrottlerException had no 429 fallback and surfaced as INTERNAL_ERROR; the guard now throws VOICE_RATE_LIMITED directly. - durationMs was optional, so omitting it bypassed VOICE_MAX_RECORDING_MS entirely. It is required. - VOICE_UNSUPPORTED_FORMAT was dead code — the DTO's @IsIn already rejects unknown containers — so it is gone rather than left unreachable. ThrottlerModule is deliberately not bound as a global APP_GUARD: a global ThrottlerGuard rate-limits every route against every named throttler, which would have capped the whole API at the voice limit. All seven remaining VOICE_* codes have errors.* keys in en, fa and nl. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 18:27:14 +03:30
import {
resolveVoiceIntent,
type ResolvedExtraction,
} from './extraction.resolver';
import {
OpenRouterAsrProvider,
OpenRouterExtractionProvider,
} from './openrouter.provider';
import {
VoiceProviderError,
type AsrProvider,
type ExtractionCatalog,
type ExtractionProvider,
} from './voice.providers';
import type { ExtractVoiceDto } from './dto/voice.dto';
export type VoiceAvailability = {
enabled: boolean;
locales: string[];
maxRecordingMs: number | null;
};
export type VoiceExtractionResponse = ResolvedExtraction & {
transcript: string;
};
@Injectable()
export class VoiceService {
private readonly logger = new Logger(VoiceService.name);
constructor(
private readonly prisma: PrismaService,
private readonly config: ConfigService,
private readonly treatmentCatalog: TreatmentCatalogService,
private readonly prosthesisCatalog: ProsthesisCatalogService,
) {}
private get voiceConfig(): VoiceConfig {
return this.config.get<VoiceConfig>('voice')!;
}
/**
* What the frontend needs to decide whether to render the microphone at all.
*
* v1 ships ungated beyond a configured locale profile no plan check. The
* Plan.features design is deferred, not dropped.
*/
getAvailability(): VoiceAvailability {
const voice = this.voiceConfig;
const hasKey = Boolean(voice.openRouter.apiKey);
const locales = hasKey ? Object.keys(voice.profiles) : [];
return {
enabled: locales.length > 0,
locales,
maxRecordingMs: voice.maxRecordingMs,
};
}
async extract(
user: { id: string; organizationId?: string },
dto: ExtractVoiceDto,
locale: string,
signal?: AbortSignal,
): Promise<VoiceExtractionResponse> {
const startedAt = Date.now();
const organizationId = this.assertOrganization(user);
await this.assertCanEditTreatment(user.id, organizationId);
const catalogLocale = normalizeCatalogLocale(locale);
const profile = this.resolveProfile(catalogLocale);
this.assertWithinCap(dto.durationMs);
const timeZone = isValidIanaTimeZone(dto.timeZone) ? dto.timeZone : 'UTC';
const todayIso = civilDateInZone(new Date(), timeZone);
const { asr, extraction } = this.buildProviders(profile);
// Stage 1 — audio never touches disk and is not retained beyond this call.
let transcript: string;
let asrCost: number | null = null;
feat(frontend): split Add detail into a segmented control with voice The microphone becomes the second segment of the Add detail button, built like the detail chip's trash affordance in the same file — an overflow-hidden rounded wrapper holding two raw <button>s divided by border-s — rather than two shared Buttons, which each hardcode their own rounding and would fight a segmented control. border-s puts the mic at the logical end: visually right in en/nl, visually left in fa, on the same side as the chip's trash in both directions. The two halves share a wrapper and nothing else. Add keeps its exact behaviour. The control never changes size while recording; the timer and level meter live in a bar between the header row and the chip strip, because the header is sm:justify-between and growing the button would shove the row on every start and stop. The meter exists to prove the microphone is actually hearing something — silence and a dead mic look identical otherwise. Voice reaches the editor as one optional `voice` prop, so its absence *is* the unavailable state and the two cannot disagree. Fixes from review of this commit: - mountedRef was set false on unmount and never re-armed, so under StrictMode the hook was permanently "unmounted" in dev and recording silently never started. - onStart guarded only on `phase`, which does not change until getUserMedia resolves; a second click during the permission prompt orphaned the first MediaStream, leaving the mic indicator lit. - Week start is now per locale. "Next Thursday" is week-relative, and hardcoding Saturday put an en/nl clinician's deadline a week out. - A missing `which` on a weekday intent is read as "this" rather than failing — a bare weekday carries no qualifier, and rejecting it discarded a real deadline. - durationMs is client-reported and so is a claim, not enforcement; the cap is now also checked against the vendor's own usage.seconds. - Blob type falls back to the recorder's actual mimeType before webm, so old Safari's mp4/aac clips are not mislabelled. Two review findings were rejected as incorrect, both re-verified against live sources: google/gemini-3.7-flash does exist on OpenRouter (1M context, $0.375/$1.875 per M), and base64 JSON input_audio is the documented primary path for /audio/transcriptions, with multipart as the OpenAI-compatible alternative. The spec's stale "unverified" note is corrected, and the provider now has unit tests covering the request shape, usage parsing, and that a vendor error body never reaches the thrown message. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 19:47:10 +03:30
let asrSeconds: number | null = null;
feat(backend): voice extraction endpoint POST /voice/extract behind JwtAuthGuard + ClinicOrgGuard, plus GET /voice/availability so the frontend can decide whether to render the microphone — it cannot learn that from NEXT_PUBLIC_*, which are baked in at build time. Audio is held in memory for the request only: never written to disk, never a Prisma row. The transcript goes back to the client and is not persisted. What is logged is structured and patient-free — clip length, which fields resolved, unresolved count, vendor cost, outcome — with log lines as the interim sink until this repo has metrics infrastructure. On extraction failure the transcript still travels back in the error details, so the words the clinician already paid for can be salvaged into a note. v1 ships ungated beyond a configured locale profile; the Plan.features design is deferred, not dropped. From review of this commit, four of which were load-bearing: - Express's 100 kb default body limit rejected any recording past ~20 seconds, making the endpoint unusable at its own 2-minute cap. Body parsers are now registered explicitly with a 10 MB limit scoped to the voice route only. Verified empirically: 600 KB reaches /api/voice/extract, while /api/auth/login still 413s. - ThrottlerGuard keys on req.ip, so behind nginx the whole deployment would share one bucket and an abuser rotating IPs would bypass it. VoiceThrottlerGuard keys on the user id instead — with no plan gate, this is the only control on metered vendor spend. - ThrottlerException had no 429 fallback and surfaced as INTERNAL_ERROR; the guard now throws VOICE_RATE_LIMITED directly. - durationMs was optional, so omitting it bypassed VOICE_MAX_RECORDING_MS entirely. It is required. - VOICE_UNSUPPORTED_FORMAT was dead code — the DTO's @IsIn already rejects unknown containers — so it is gone rather than left unreachable. ThrottlerModule is deliberately not bound as a global APP_GUARD: a global ThrottlerGuard rate-limits every route against every named throttler, which would have capped the whole API at the voice limit. All seven remaining VOICE_* codes have errors.* keys in en, fa and nl. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 18:27:14 +03:30
try {
const result = await asr.transcribe(
{ data: dto.audio, format: dto.format },
catalogLocale,
signal,
);
transcript = result.text;
asrCost = result.usage.costUsd;
feat(frontend): split Add detail into a segmented control with voice The microphone becomes the second segment of the Add detail button, built like the detail chip's trash affordance in the same file — an overflow-hidden rounded wrapper holding two raw <button>s divided by border-s — rather than two shared Buttons, which each hardcode their own rounding and would fight a segmented control. border-s puts the mic at the logical end: visually right in en/nl, visually left in fa, on the same side as the chip's trash in both directions. The two halves share a wrapper and nothing else. Add keeps its exact behaviour. The control never changes size while recording; the timer and level meter live in a bar between the header row and the chip strip, because the header is sm:justify-between and growing the button would shove the row on every start and stop. The meter exists to prove the microphone is actually hearing something — silence and a dead mic look identical otherwise. Voice reaches the editor as one optional `voice` prop, so its absence *is* the unavailable state and the two cannot disagree. Fixes from review of this commit: - mountedRef was set false on unmount and never re-armed, so under StrictMode the hook was permanently "unmounted" in dev and recording silently never started. - onStart guarded only on `phase`, which does not change until getUserMedia resolves; a second click during the permission prompt orphaned the first MediaStream, leaving the mic indicator lit. - Week start is now per locale. "Next Thursday" is week-relative, and hardcoding Saturday put an en/nl clinician's deadline a week out. - A missing `which` on a weekday intent is read as "this" rather than failing — a bare weekday carries no qualifier, and rejecting it discarded a real deadline. - durationMs is client-reported and so is a claim, not enforcement; the cap is now also checked against the vendor's own usage.seconds. - Blob type falls back to the recorder's actual mimeType before webm, so old Safari's mp4/aac clips are not mislabelled. Two review findings were rejected as incorrect, both re-verified against live sources: google/gemini-3.7-flash does exist on OpenRouter (1M context, $0.375/$1.875 per M), and base64 JSON input_audio is the documented primary path for /audio/transcriptions, with multipart as the OpenAI-compatible alternative. The spec's stale "unverified" note is corrected, and the provider now has unit tests covering the request shape, usage parsing, and that a vendor error body never reaches the thrown message. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 19:47:10 +03:30
asrSeconds = result.usage.seconds;
feat(backend): voice extraction endpoint POST /voice/extract behind JwtAuthGuard + ClinicOrgGuard, plus GET /voice/availability so the frontend can decide whether to render the microphone — it cannot learn that from NEXT_PUBLIC_*, which are baked in at build time. Audio is held in memory for the request only: never written to disk, never a Prisma row. The transcript goes back to the client and is not persisted. What is logged is structured and patient-free — clip length, which fields resolved, unresolved count, vendor cost, outcome — with log lines as the interim sink until this repo has metrics infrastructure. On extraction failure the transcript still travels back in the error details, so the words the clinician already paid for can be salvaged into a note. v1 ships ungated beyond a configured locale profile; the Plan.features design is deferred, not dropped. From review of this commit, four of which were load-bearing: - Express's 100 kb default body limit rejected any recording past ~20 seconds, making the endpoint unusable at its own 2-minute cap. Body parsers are now registered explicitly with a 10 MB limit scoped to the voice route only. Verified empirically: 600 KB reaches /api/voice/extract, while /api/auth/login still 413s. - ThrottlerGuard keys on req.ip, so behind nginx the whole deployment would share one bucket and an abuser rotating IPs would bypass it. VoiceThrottlerGuard keys on the user id instead — with no plan gate, this is the only control on metered vendor spend. - ThrottlerException had no 429 fallback and surfaced as INTERNAL_ERROR; the guard now throws VOICE_RATE_LIMITED directly. - durationMs was optional, so omitting it bypassed VOICE_MAX_RECORDING_MS entirely. It is required. - VOICE_UNSUPPORTED_FORMAT was dead code — the DTO's @IsIn already rejects unknown containers — so it is gone rather than left unreachable. ThrottlerModule is deliberately not bound as a global APP_GUARD: a global ThrottlerGuard rate-limits every route against every named throttler, which would have capped the whole API at the voice limit. All seven remaining VOICE_* codes have errors.* keys in en, fa and nl. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 18:27:14 +03:30
} catch (error) {
throw this.toAppException(error, 'asr');
}
feat(frontend): split Add detail into a segmented control with voice The microphone becomes the second segment of the Add detail button, built like the detail chip's trash affordance in the same file — an overflow-hidden rounded wrapper holding two raw <button>s divided by border-s — rather than two shared Buttons, which each hardcode their own rounding and would fight a segmented control. border-s puts the mic at the logical end: visually right in en/nl, visually left in fa, on the same side as the chip's trash in both directions. The two halves share a wrapper and nothing else. Add keeps its exact behaviour. The control never changes size while recording; the timer and level meter live in a bar between the header row and the chip strip, because the header is sm:justify-between and growing the button would shove the row on every start and stop. The meter exists to prove the microphone is actually hearing something — silence and a dead mic look identical otherwise. Voice reaches the editor as one optional `voice` prop, so its absence *is* the unavailable state and the two cannot disagree. Fixes from review of this commit: - mountedRef was set false on unmount and never re-armed, so under StrictMode the hook was permanently "unmounted" in dev and recording silently never started. - onStart guarded only on `phase`, which does not change until getUserMedia resolves; a second click during the permission prompt orphaned the first MediaStream, leaving the mic indicator lit. - Week start is now per locale. "Next Thursday" is week-relative, and hardcoding Saturday put an en/nl clinician's deadline a week out. - A missing `which` on a weekday intent is read as "this" rather than failing — a bare weekday carries no qualifier, and rejecting it discarded a real deadline. - durationMs is client-reported and so is a claim, not enforcement; the cap is now also checked against the vendor's own usage.seconds. - Blob type falls back to the recorder's actual mimeType before webm, so old Safari's mp4/aac clips are not mislabelled. Two review findings were rejected as incorrect, both re-verified against live sources: google/gemini-3.7-flash does exist on OpenRouter (1M context, $0.375/$1.875 per M), and base64 JSON input_audio is the documented primary path for /audio/transcriptions, with multipart as the OpenAI-compatible alternative. The spec's stale "unverified" note is corrected, and the provider now has unit tests covering the request shape, usage parsing, and that a vendor error body never reaches the thrown message. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 19:47:10 +03:30
// durationMs is client-reported and therefore not enforcement. usage.seconds is the
// vendor's own measurement of the audio it decoded, so a client under-reporting length
// to slip past the cap is caught here — after the ASR spend, but before the extraction
// call, and visibly in telemetry.
if (asrSeconds != null) {
this.assertWithinCap(asrSeconds * 1000);
}
feat(backend): voice extraction endpoint POST /voice/extract behind JwtAuthGuard + ClinicOrgGuard, plus GET /voice/availability so the frontend can decide whether to render the microphone — it cannot learn that from NEXT_PUBLIC_*, which are baked in at build time. Audio is held in memory for the request only: never written to disk, never a Prisma row. The transcript goes back to the client and is not persisted. What is logged is structured and patient-free — clip length, which fields resolved, unresolved count, vendor cost, outcome — with log lines as the interim sink until this repo has metrics infrastructure. On extraction failure the transcript still travels back in the error details, so the words the clinician already paid for can be salvaged into a note. v1 ships ungated beyond a configured locale profile; the Plan.features design is deferred, not dropped. From review of this commit, four of which were load-bearing: - Express's 100 kb default body limit rejected any recording past ~20 seconds, making the endpoint unusable at its own 2-minute cap. Body parsers are now registered explicitly with a 10 MB limit scoped to the voice route only. Verified empirically: 600 KB reaches /api/voice/extract, while /api/auth/login still 413s. - ThrottlerGuard keys on req.ip, so behind nginx the whole deployment would share one bucket and an abuser rotating IPs would bypass it. VoiceThrottlerGuard keys on the user id instead — with no plan gate, this is the only control on metered vendor spend. - ThrottlerException had no 429 fallback and surfaced as INTERNAL_ERROR; the guard now throws VOICE_RATE_LIMITED directly. - durationMs was optional, so omitting it bypassed VOICE_MAX_RECORDING_MS entirely. It is required. - VOICE_UNSUPPORTED_FORMAT was dead code — the DTO's @IsIn already rejects unknown containers — so it is gone rather than left unreachable. ThrottlerModule is deliberately not bound as a global APP_GUARD: a global ThrottlerGuard rate-limits every route against every named throttler, which would have capped the whole API at the voice limit. All seven remaining VOICE_* codes have errors.* keys in en, fa and nl. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 18:27:14 +03:30
if (!transcript.trim()) {
throw new AppException(
ErrorCode.VOICE_NOTHING_RECOGNIZED,
HttpStatus.UNPROCESSABLE_ENTITY,
);
}
// Stage 2 — structure it. On failure the transcript still goes back to the client so
// the words the clinician already paid for are not lost (transcript salvage).
const catalog = await this.buildCatalog(organizationId, catalogLocale);
let resolved: ResolvedExtraction;
let llmCost: number | null = null;
try {
const result = await extraction.extract(
transcript,
catalog,
catalogLocale,
signal,
);
llmCost = result.costUsd;
resolved = resolveVoiceIntent(result.intent, {
todayIso,
feat(frontend): split Add detail into a segmented control with voice The microphone becomes the second segment of the Add detail button, built like the detail chip's trash affordance in the same file — an overflow-hidden rounded wrapper holding two raw <button>s divided by border-s — rather than two shared Buttons, which each hardcode their own rounding and would fight a segmented control. border-s puts the mic at the logical end: visually right in en/nl, visually left in fa, on the same side as the chip's trash in both directions. The two halves share a wrapper and nothing else. Add keeps its exact behaviour. The control never changes size while recording; the timer and level meter live in a bar between the header row and the chip strip, because the header is sm:justify-between and growing the button would shove the row on every start and stop. The meter exists to prove the microphone is actually hearing something — silence and a dead mic look identical otherwise. Voice reaches the editor as one optional `voice` prop, so its absence *is* the unavailable state and the two cannot disagree. Fixes from review of this commit: - mountedRef was set false on unmount and never re-armed, so under StrictMode the hook was permanently "unmounted" in dev and recording silently never started. - onStart guarded only on `phase`, which does not change until getUserMedia resolves; a second click during the permission prompt orphaned the first MediaStream, leaving the mic indicator lit. - Week start is now per locale. "Next Thursday" is week-relative, and hardcoding Saturday put an en/nl clinician's deadline a week out. - A missing `which` on a weekday intent is read as "this" rather than failing — a bare weekday carries no qualifier, and rejecting it discarded a real deadline. - durationMs is client-reported and so is a claim, not enforcement; the cap is now also checked against the vendor's own usage.seconds. - Blob type falls back to the recorder's actual mimeType before webm, so old Safari's mp4/aac clips are not mislabelled. Two review findings were rejected as incorrect, both re-verified against live sources: google/gemini-3.7-flash does exist on OpenRouter (1M context, $0.375/$1.875 per M), and base64 JSON input_audio is the documented primary path for /audio/transcriptions, with multipart as the OpenAI-compatible alternative. The spec's stale "unverified" note is corrected, and the provider now has unit tests covering the request shape, usage parsing, and that a vendor error body never reaches the thrown message. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 19:47:10 +03:30
weekStartJs: weekStartForLocale(catalogLocale),
feat(backend): voice extraction endpoint POST /voice/extract behind JwtAuthGuard + ClinicOrgGuard, plus GET /voice/availability so the frontend can decide whether to render the microphone — it cannot learn that from NEXT_PUBLIC_*, which are baked in at build time. Audio is held in memory for the request only: never written to disk, never a Prisma row. The transcript goes back to the client and is not persisted. What is logged is structured and patient-free — clip length, which fields resolved, unresolved count, vendor cost, outcome — with log lines as the interim sink until this repo has metrics infrastructure. On extraction failure the transcript still travels back in the error details, so the words the clinician already paid for can be salvaged into a note. v1 ships ungated beyond a configured locale profile; the Plan.features design is deferred, not dropped. From review of this commit, four of which were load-bearing: - Express's 100 kb default body limit rejected any recording past ~20 seconds, making the endpoint unusable at its own 2-minute cap. Body parsers are now registered explicitly with a 10 MB limit scoped to the voice route only. Verified empirically: 600 KB reaches /api/voice/extract, while /api/auth/login still 413s. - ThrottlerGuard keys on req.ip, so behind nginx the whole deployment would share one bucket and an abuser rotating IPs would bypass it. VoiceThrottlerGuard keys on the user id instead — with no plan gate, this is the only control on metered vendor spend. - ThrottlerException had no 429 fallback and surfaced as INTERNAL_ERROR; the guard now throws VOICE_RATE_LIMITED directly. - durationMs was optional, so omitting it bypassed VOICE_MAX_RECORDING_MS entirely. It is required. - VOICE_UNSUPPORTED_FORMAT was dead code — the DTO's @IsIn already rejects unknown containers — so it is gone rather than left unreachable. ThrottlerModule is deliberately not bound as a global APP_GUARD: a global ThrottlerGuard rate-limits every route against every named throttler, which would have capped the whole API at the voice limit. All seven remaining VOICE_* codes have errors.* keys in en, fa and nl. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 18:27:14 +03:30
treatmentTypeCodes: new Set(catalog.treatmentTypes.map((t) => t.code)),
prosthesisTypeCodes: new Set(
catalog.prosthesisTypes.map((t) => t.code),
),
linkedLabIds: new Set(catalog.labs.map((l) => l.id)),
});
} catch (error) {
throw this.toAppException(error, 'extraction', transcript);
}
this.logTelemetry({
locale: catalogLocale,
feat(frontend): voice capture hook, API client and types MediaRecorder handling and the API call live in lib/, not in ui/, so TreatmentDetailsEditor can stay presentational and take only a `voice` prop. Container choice is made at record time and needs no transcode: Chrome and Android give webm/opus, Safari and iPad give mp4/aac, and the transcription endpoint accepts both. Safari's `audio/mp4` is sent as `m4a`, the name the vendor's container list actually uses, so iPad recordings do not fail while Chrome works. Older Safari shipped MediaRecorder without isTypeSupported, so that path lets the browser choose rather than refusing outright. From review of this commit: - The auto-stop at maxMs guaranteed a 413. The client measures the final length after the recorder has stopped, so a recording that runs to the cap always reports slightly over it, and the server rejected exactly the recording the auto-stop existed to save. The server now allows a documented 2s tolerance and the client keeps reporting the true length, so telemetry stays honest. - getUserMedia is async, so a permission granted after unmount installed a live stream the cleanup effect had already run past — leaving the browser's recording indicator lit with nothing listening. Guarded with a mounted ref. - Client-side failures are now ApiError-shaped ({code, statusCode}) rather than bare Errors, because getUserFacingError only resolves that shape; without it errors.VOICE_MIC_DENIED was dead in all three locales. Cancelling aborts the request, which closes the connection and aborts the metered vendor call server-side rather than letting it settle unseen. The level meter is best-effort: a blocked AudioContext costs the meter, not the recording. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 18:43:04 +03:30
durationMs: dto.durationMs,
feat(backend): voice extraction endpoint POST /voice/extract behind JwtAuthGuard + ClinicOrgGuard, plus GET /voice/availability so the frontend can decide whether to render the microphone — it cannot learn that from NEXT_PUBLIC_*, which are baked in at build time. Audio is held in memory for the request only: never written to disk, never a Prisma row. The transcript goes back to the client and is not persisted. What is logged is structured and patient-free — clip length, which fields resolved, unresolved count, vendor cost, outcome — with log lines as the interim sink until this repo has metrics infrastructure. On extraction failure the transcript still travels back in the error details, so the words the clinician already paid for can be salvaged into a note. v1 ships ungated beyond a configured locale profile; the Plan.features design is deferred, not dropped. From review of this commit, four of which were load-bearing: - Express's 100 kb default body limit rejected any recording past ~20 seconds, making the endpoint unusable at its own 2-minute cap. Body parsers are now registered explicitly with a 10 MB limit scoped to the voice route only. Verified empirically: 600 KB reaches /api/voice/extract, while /api/auth/login still 413s. - ThrottlerGuard keys on req.ip, so behind nginx the whole deployment would share one bucket and an abuser rotating IPs would bypass it. VoiceThrottlerGuard keys on the user id instead — with no plan gate, this is the only control on metered vendor spend. - ThrottlerException had no 429 fallback and surfaced as INTERNAL_ERROR; the guard now throws VOICE_RATE_LIMITED directly. - durationMs was optional, so omitting it bypassed VOICE_MAX_RECORDING_MS entirely. It is required. - VOICE_UNSUPPORTED_FORMAT was dead code — the DTO's @IsIn already rejects unknown containers — so it is gone rather than left unreachable. ThrottlerModule is deliberately not bound as a global APP_GUARD: a global ThrottlerGuard rate-limits every route against every named throttler, which would have capped the whole API at the voice limit. All seven remaining VOICE_* codes have errors.* keys in en, fa and nl. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 18:27:14 +03:30
elapsedMs: Date.now() - startedAt,
asrCost,
llmCost,
resolved,
});
return { ...resolved, transcript };
}
private assertOrganization(user: { organizationId?: string }): string {
if (!user?.organizationId) {
throw new AppException(
ErrorCode.AUTH_ORG_NOT_SELECTED,
HttpStatus.BAD_REQUEST,
);
}
return user.organizationId;
}
private async assertCanEditTreatment(userId: string, organizationId: string) {
const membership = await this.prisma.membership.findFirst({
where: {
userId,
organizationId,
OR: [{ isOwner: true }, { isActive: true }],
},
include: {
permissions: { include: { permission: true } },
organization: { include: { type: true, plan: true } },
},
});
if (!membership) {
throw new AppException(
ErrorCode.PERMISSION_NOT_MEMBER,
HttpStatus.FORBIDDEN,
);
}
if (!hasEffectivePermission(membership, 'TAB_TREATMENT_EDIT')) {
throw new AppException(
ErrorCode.PERMISSION_EDIT_TREATMENTS,
HttpStatus.FORBIDDEN,
);
}
}
private resolveProfile(locale: string): VoiceProfile {
const voice = this.voiceConfig;
const profile = voice.profiles[locale];
if (!profile || !voice.openRouter.apiKey) {
throw new AppException(
ErrorCode.VOICE_NOT_AVAILABLE,
HttpStatus.FORBIDDEN,
);
}
return profile;
}
feat(frontend): voice capture hook, API client and types MediaRecorder handling and the API call live in lib/, not in ui/, so TreatmentDetailsEditor can stay presentational and take only a `voice` prop. Container choice is made at record time and needs no transcode: Chrome and Android give webm/opus, Safari and iPad give mp4/aac, and the transcription endpoint accepts both. Safari's `audio/mp4` is sent as `m4a`, the name the vendor's container list actually uses, so iPad recordings do not fail while Chrome works. Older Safari shipped MediaRecorder without isTypeSupported, so that path lets the browser choose rather than refusing outright. From review of this commit: - The auto-stop at maxMs guaranteed a 413. The client measures the final length after the recorder has stopped, so a recording that runs to the cap always reports slightly over it, and the server rejected exactly the recording the auto-stop existed to save. The server now allows a documented 2s tolerance and the client keeps reporting the true length, so telemetry stays honest. - getUserMedia is async, so a permission granted after unmount installed a live stream the cleanup effect had already run past — leaving the browser's recording indicator lit with nothing listening. Guarded with a mounted ref. - Client-side failures are now ApiError-shaped ({code, statusCode}) rather than bare Errors, because getUserFacingError only resolves that shape; without it errors.VOICE_MIC_DENIED was dead in all three locales. Cancelling aborts the request, which closes the connection and aborts the metered vendor call server-side rather than letting it settle unseen. The level meter is best-effort: a blocked AudioContext costs the meter, not the recording. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 18:43:04 +03:30
/**
* Grace above the configured cap.
*
* The client auto-stops when elapsed >= maxMs, then measures the final length after the
* recorder has actually stopped so a recording that runs to the cap always reports
* slightly over it. Without this tolerance the auto-stop would guarantee a rejection,
* discarding exactly the recording it was meant to save. The client still reports the
* true length, so telemetry stays honest.
*/
private static readonly CAP_TOLERANCE_MS = 2_000;
private assertWithinCap(durationMs: number) {
feat(backend): voice extraction endpoint POST /voice/extract behind JwtAuthGuard + ClinicOrgGuard, plus GET /voice/availability so the frontend can decide whether to render the microphone — it cannot learn that from NEXT_PUBLIC_*, which are baked in at build time. Audio is held in memory for the request only: never written to disk, never a Prisma row. The transcript goes back to the client and is not persisted. What is logged is structured and patient-free — clip length, which fields resolved, unresolved count, vendor cost, outcome — with log lines as the interim sink until this repo has metrics infrastructure. On extraction failure the transcript still travels back in the error details, so the words the clinician already paid for can be salvaged into a note. v1 ships ungated beyond a configured locale profile; the Plan.features design is deferred, not dropped. From review of this commit, four of which were load-bearing: - Express's 100 kb default body limit rejected any recording past ~20 seconds, making the endpoint unusable at its own 2-minute cap. Body parsers are now registered explicitly with a 10 MB limit scoped to the voice route only. Verified empirically: 600 KB reaches /api/voice/extract, while /api/auth/login still 413s. - ThrottlerGuard keys on req.ip, so behind nginx the whole deployment would share one bucket and an abuser rotating IPs would bypass it. VoiceThrottlerGuard keys on the user id instead — with no plan gate, this is the only control on metered vendor spend. - ThrottlerException had no 429 fallback and surfaced as INTERNAL_ERROR; the guard now throws VOICE_RATE_LIMITED directly. - durationMs was optional, so omitting it bypassed VOICE_MAX_RECORDING_MS entirely. It is required. - VOICE_UNSUPPORTED_FORMAT was dead code — the DTO's @IsIn already rejects unknown containers — so it is gone rather than left unreachable. ThrottlerModule is deliberately not bound as a global APP_GUARD: a global ThrottlerGuard rate-limits every route against every named throttler, which would have capped the whole API at the voice limit. All seven remaining VOICE_* codes have errors.* keys in en, fa and nl. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 18:27:14 +03:30
const max = this.voiceConfig.maxRecordingMs;
feat(frontend): voice capture hook, API client and types MediaRecorder handling and the API call live in lib/, not in ui/, so TreatmentDetailsEditor can stay presentational and take only a `voice` prop. Container choice is made at record time and needs no transcode: Chrome and Android give webm/opus, Safari and iPad give mp4/aac, and the transcription endpoint accepts both. Safari's `audio/mp4` is sent as `m4a`, the name the vendor's container list actually uses, so iPad recordings do not fail while Chrome works. Older Safari shipped MediaRecorder without isTypeSupported, so that path lets the browser choose rather than refusing outright. From review of this commit: - The auto-stop at maxMs guaranteed a 413. The client measures the final length after the recorder has stopped, so a recording that runs to the cap always reports slightly over it, and the server rejected exactly the recording the auto-stop existed to save. The server now allows a documented 2s tolerance and the client keeps reporting the true length, so telemetry stays honest. - getUserMedia is async, so a permission granted after unmount installed a live stream the cleanup effect had already run past — leaving the browser's recording indicator lit with nothing listening. Guarded with a mounted ref. - Client-side failures are now ApiError-shaped ({code, statusCode}) rather than bare Errors, because getUserFacingError only resolves that shape; without it errors.VOICE_MIC_DENIED was dead in all three locales. Cancelling aborts the request, which closes the connection and aborts the metered vendor call server-side rather than letting it settle unseen. The level meter is best-effort: a blocked AudioContext costs the meter, not the recording. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 18:43:04 +03:30
if (max != null && durationMs > max + VoiceService.CAP_TOLERANCE_MS) {
feat(backend): voice extraction endpoint POST /voice/extract behind JwtAuthGuard + ClinicOrgGuard, plus GET /voice/availability so the frontend can decide whether to render the microphone — it cannot learn that from NEXT_PUBLIC_*, which are baked in at build time. Audio is held in memory for the request only: never written to disk, never a Prisma row. The transcript goes back to the client and is not persisted. What is logged is structured and patient-free — clip length, which fields resolved, unresolved count, vendor cost, outcome — with log lines as the interim sink until this repo has metrics infrastructure. On extraction failure the transcript still travels back in the error details, so the words the clinician already paid for can be salvaged into a note. v1 ships ungated beyond a configured locale profile; the Plan.features design is deferred, not dropped. From review of this commit, four of which were load-bearing: - Express's 100 kb default body limit rejected any recording past ~20 seconds, making the endpoint unusable at its own 2-minute cap. Body parsers are now registered explicitly with a 10 MB limit scoped to the voice route only. Verified empirically: 600 KB reaches /api/voice/extract, while /api/auth/login still 413s. - ThrottlerGuard keys on req.ip, so behind nginx the whole deployment would share one bucket and an abuser rotating IPs would bypass it. VoiceThrottlerGuard keys on the user id instead — with no plan gate, this is the only control on metered vendor spend. - ThrottlerException had no 429 fallback and surfaced as INTERNAL_ERROR; the guard now throws VOICE_RATE_LIMITED directly. - durationMs was optional, so omitting it bypassed VOICE_MAX_RECORDING_MS entirely. It is required. - VOICE_UNSUPPORTED_FORMAT was dead code — the DTO's @IsIn already rejects unknown containers — so it is gone rather than left unreachable. ThrottlerModule is deliberately not bound as a global APP_GUARD: a global ThrottlerGuard rate-limits every route against every named throttler, which would have capped the whole API at the voice limit. All seven remaining VOICE_* codes have errors.* keys in en, fa and nl. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 18:27:14 +03:30
throw new AppException(
ErrorCode.VOICE_CLIP_TOO_LONG,
HttpStatus.PAYLOAD_TOO_LARGE,
);
}
}
private buildProviders(profile: VoiceProfile): {
asr: AsrProvider;
extraction: ExtractionProvider;
} {
const { apiKey, baseUrl } = this.voiceConfig.openRouter;
const base = { apiKey: apiKey!, baseUrl };
return {
asr: new OpenRouterAsrProvider({ ...base, model: profile.asr.model }),
extraction: new OpenRouterExtractionProvider({
...base,
model: profile.llm.model,
}),
};
}
/** Codes with labels in the actor's locale, plus the clinic's linked labs. */
private async buildCatalog(
organizationId: string,
locale: string,
): Promise<ExtractionCatalog> {
const [treatmentTypes, prosthesisTypes, labs] = await Promise.all([
this.treatmentCatalog.list(locale, null),
this.prosthesisCatalog.list(locale),
this.listLinkedLabs(organizationId),
]);
return {
treatmentTypes: treatmentTypes
.filter((entry) => entry.availableInTreatment)
.map((entry) => ({ code: entry.code, label: entry.label })),
prosthesisTypes: prosthesisTypes.map((entry) => ({
code: entry.code,
label: entry.label,
})),
labs,
};
}
private async listLinkedLabs(
organizationId: string,
): Promise<{ id: string; name: string }[]> {
const [linksA, linksB] = await Promise.all([
this.prisma.organizationLink.findMany({
where: { organizationAId: organizationId, status: LinkStatus.ACTIVE },
include: { organizationB: { select: { id: true, name: true } } },
}),
this.prisma.organizationLink.findMany({
where: { organizationBId: organizationId, status: LinkStatus.ACTIVE },
include: { organizationA: { select: { id: true, name: true } } },
}),
]);
return [
...linksA.map((l) => ({
id: l.organizationB.id,
name: l.organizationB.name,
})),
...linksB.map((l) => ({
id: l.organizationA.id,
name: l.organizationA.name,
})),
];
}
private toAppException(
error: unknown,
stage: 'asr' | 'extraction',
transcript?: string,
): AppException {
if (error instanceof Error && error.name === 'AbortError') {
// The clinician cancelled; not a failure worth a translated message.
return new AppException(ErrorCode.BAD_REQUEST, HttpStatus.BAD_REQUEST);
}
if (error instanceof VoiceProviderError) {
this.logger.warn(`voice ${stage} failed: ${error.message}`);
} else {
this.logger.error(`voice ${stage} failed unexpectedly`, error as Error);
}
const code =
stage === 'asr'
? ErrorCode.VOICE_ASR_FAILED
: ErrorCode.VOICE_EXTRACT_FAILED;
return new AppException(
code,
HttpStatus.BAD_GATEWAY,
transcript ? { transcript } : undefined,
);
}
/**
* Structured, patient-free. Never the transcript, never audio, never a patient id.
* Log lines are the interim sink until this repo has metrics infrastructure.
*/
private logTelemetry(input: {
locale: string;
feat(frontend): voice capture hook, API client and types MediaRecorder handling and the API call live in lib/, not in ui/, so TreatmentDetailsEditor can stay presentational and take only a `voice` prop. Container choice is made at record time and needs no transcode: Chrome and Android give webm/opus, Safari and iPad give mp4/aac, and the transcription endpoint accepts both. Safari's `audio/mp4` is sent as `m4a`, the name the vendor's container list actually uses, so iPad recordings do not fail while Chrome works. Older Safari shipped MediaRecorder without isTypeSupported, so that path lets the browser choose rather than refusing outright. From review of this commit: - The auto-stop at maxMs guaranteed a 413. The client measures the final length after the recorder has stopped, so a recording that runs to the cap always reports slightly over it, and the server rejected exactly the recording the auto-stop existed to save. The server now allows a documented 2s tolerance and the client keeps reporting the true length, so telemetry stays honest. - getUserMedia is async, so a permission granted after unmount installed a live stream the cleanup effect had already run past — leaving the browser's recording indicator lit with nothing listening. Guarded with a mounted ref. - Client-side failures are now ApiError-shaped ({code, statusCode}) rather than bare Errors, because getUserFacingError only resolves that shape; without it errors.VOICE_MIC_DENIED was dead in all three locales. Cancelling aborts the request, which closes the connection and aborts the metered vendor call server-side rather than letting it settle unseen. The level meter is best-effort: a blocked AudioContext costs the meter, not the recording. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 18:43:04 +03:30
durationMs: number;
feat(backend): voice extraction endpoint POST /voice/extract behind JwtAuthGuard + ClinicOrgGuard, plus GET /voice/availability so the frontend can decide whether to render the microphone — it cannot learn that from NEXT_PUBLIC_*, which are baked in at build time. Audio is held in memory for the request only: never written to disk, never a Prisma row. The transcript goes back to the client and is not persisted. What is logged is structured and patient-free — clip length, which fields resolved, unresolved count, vendor cost, outcome — with log lines as the interim sink until this repo has metrics infrastructure. On extraction failure the transcript still travels back in the error details, so the words the clinician already paid for can be salvaged into a note. v1 ships ungated beyond a configured locale profile; the Plan.features design is deferred, not dropped. From review of this commit, four of which were load-bearing: - Express's 100 kb default body limit rejected any recording past ~20 seconds, making the endpoint unusable at its own 2-minute cap. Body parsers are now registered explicitly with a 10 MB limit scoped to the voice route only. Verified empirically: 600 KB reaches /api/voice/extract, while /api/auth/login still 413s. - ThrottlerGuard keys on req.ip, so behind nginx the whole deployment would share one bucket and an abuser rotating IPs would bypass it. VoiceThrottlerGuard keys on the user id instead — with no plan gate, this is the only control on metered vendor spend. - ThrottlerException had no 429 fallback and surfaced as INTERNAL_ERROR; the guard now throws VOICE_RATE_LIMITED directly. - durationMs was optional, so omitting it bypassed VOICE_MAX_RECORDING_MS entirely. It is required. - VOICE_UNSUPPORTED_FORMAT was dead code — the DTO's @IsIn already rejects unknown containers — so it is gone rather than left unreachable. ThrottlerModule is deliberately not bound as a global APP_GUARD: a global ThrottlerGuard rate-limits every route against every named throttler, which would have capped the whole API at the voice limit. All seven remaining VOICE_* codes have errors.* keys in en, fa and nl. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 18:27:14 +03:30
elapsedMs: number;
asrCost: number | null;
llmCost: number | null;
resolved: ResolvedExtraction;
}) {
const { resolved } = input;
this.logger.log(
JSON.stringify({
event: 'voice.extract',
locale: input.locale,
clipMs: input.durationMs,
elapsedMs: input.elapsedMs,
costUsd: (input.asrCost ?? 0) + (input.llmCost ?? 0),
resolvedFields: {
treatmentType: resolved.treatmentType != null,
teeth: resolved.teeth.length,
comment: resolved.comment != null,
prosthesisComplete: resolved.prosthesis?.complete ?? null,
lab: resolved.labId != null,
dueDate: resolved.dueDate != null,
},
unresolvedCount: resolved.unresolved.length,
}),
);
}
}