diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..898a35c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,108 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Read first + +Project conventions already live in **`AGENTS.md`** (project map + per-feature quick-reference), **`.cursor/rules/*.mdc`** (short always-on / file-scoped rules), and **`.cursor/skills/*/SKILL.md`** (multi-step workflow playbooks). They are plain markdown — read the ones covering the area you touch **before** editing. This file covers only what those do not: commands and cross-cutting architecture. + +Per `.cursor/rules/maintain-agent-docs.mdc`: when the user establishes a durable convention, update the matching `.mdc` rule or `SKILL.md` — not this file. + +## Commands + +There is **no root `package.json`**. Every npm command runs inside `backend/` or `frontend/`. + +### Backend (`cd backend`) + +| Command | Purpose | +|---|---| +| `npm run start:dev` | API on `http://localhost:3000/api`; Swagger `/api/docs`; AdminJS `/admin` | +| `npm run build` | **Verification gate for cross-cutting backend changes** | +| `npm test` | Jest (`src/**/*.spec.ts`) | +| `npm test -- lab-case-task.generator` | Single suite by path fragment | +| `npm test -- -t "merges teeth"` | Single test by name | +| `npm run test:e2e` | Jest with `test/jest-e2e.json` | +| `npm run lint` | ESLint with `--fix` | +| `docker compose -f docker-compose.postgres.yml up -d` | Dev Postgres (host port from `POSTGRES_PORT` in `.env`) | +| `npm run prisma:generate` / `prisma:migrate` / `prisma:seed` | Client, dev migration, reference-data upsert (seed never wipes) | +| `npx prisma migrate reset` | Dev clean slate — drop, re-migrate, re-seed. Never against staging/prod | +| `npm run prisma:wipe-app-data` / `prisma:reset-treatment` / `prisma:regenerate-tasks` | Targeted dev data scripts | + +`DATABASE_URL` must use `localhost` when Nest runs on the host and Postgres in Docker. + +### Frontend (`cd frontend`) + +| Command | Purpose | +|---|---| +| `npm run dev` | Dev server on **3001** (3000 is the API) | +| `npx tsc --noEmit` | **Verification gate for any type or cross-cutting frontend change** | +| `npm run build` | Production build (`output: 'standalone'`) | +| `npm run lint` | ESLint via Next | + +`NEXT_PUBLIC_*` values are baked in at build time — restart `npm run dev` after changing `.env.local`. + +### Git + +Do not commit, push, amend, force-push, or skip hooks unless the user explicitly asks. + +## Architecture + +Dental **clinic ↔ lab** platform. Every user acts inside one `Organization` whose `type` is `CLINIC` (patients, appointments, treatment) or `LAB` (cases, tasks). Most features exist only for one side. + +### Request identity: cookie JWT carrying the selected org + +There is no `Authorization` header. `JwtStrategy` reads the httpOnly **`accessToken` cookie**, and the JWT payload carries `organizationId` — the org the user currently acts as. `POST /auth/select-organization` re-issues the token with a different org, so **switching orgs means a new token**, and every service scopes queries by `req.user.organizationId`. + +On 401 the axios interceptor (`frontend/src/lib/api/client.ts`) refreshes, **re-selects** the org from `localStorage.currentOrganizationId`, then retries the original request — skipping that dance for auth endpoints and public invitation routes. `frontend/src/proxy.ts` (the Next middleware, exported as `proxy`) is a separate, cookie-only route gate that redirects unauthenticated users to `/{locale}/login?from=…`. + +### Permissions + +`TAB_*_READ` / `TAB_*_EDIT` codes in `backend/src/common/permissions.ts`; **EDIT implies READ**. Owners get org-type defaults merged with stored grants — always resolve via `hasEffectivePermission` / `getEffectivePermissionNames` in `common/membership-permissions.ts`, never by reading `membership.permissions` directly. Controllers stack `JwtAuthGuard` + `ClinicOrgGuard`/`LabOrgGuard`; feature-specific checks belong in the **service**. + +### Error contract (spans 3 layers — change all of them) + +`AppException(ErrorCode.X)` → `HttpExceptionFilter` → `{ success: false, error: { code } }` → axios normalizes to `ApiError` → `getUserFacingError(err, tErrors, fallback)` resolves `errors.X` from the message files. Adding a user-facing failure means: a code in `common/errors/error-codes.ts`, the throw site, and an `errors.X` key in **all three** of `frontend/messages/{en,fa,nl}.json`. Never throw raw English Nest exceptions for user-facing failures. + +### The core domain pipeline + +``` +Appointment ─┐ + ├→ Treatment (patient + day) → TreatmentDetail (treatment type + selected teeth) +Walk-in ─────┘ │ + │ "send to lab" (clinic side) + ▼ + LabCase + LabCaseToothProsthesis (per tooth, grouped by sourceKey) + │ generateLabCaseTasks() + ▼ + ProsthesisType → ProsthesisTypeStep → LabWorkflowStep ⇒ LabCaseTask rows + │ + ▼ + LAB org: Cases tab + Tasks tab +``` + +`backend/src/modules/cases/lab-case-task.generator.ts` is the expansion point: it is **idempotent** (returns early if tasks exist) and drives the entire lab-side task list from catalog data. Teeth carry `selectionGroupId` so bridges/connected units survive into task grouping. A `LabCase` can also be lab-origin (`LabCaseOrigin`), created without any clinic treatment. + +Clinics may only dispatch to labs they are linked to: `OrganizationLink` (A↔B, `LinkStatus`), plus `OrganizationInvitation` for counterparts not yet on the platform — the invite flow writes both rows in one transaction and stores only the token hash. + +### Catalog is code-based and DB-translated + +`TreatmentType`, `ProsthesisType`, and `LabWorkflowStep` store a stable `code` and **no label**. Labels come from `CatalogTranslation(entityKind, entityCode, locale)` resolved by `CatalogLabelService` (falls back locale → `en` → humanized code). So: never hardcode a catalog label in backend code, and pass the actor's locale into anything that materializes labels (task generation does). Frontend colors/labels for these codes live in `components/shared/treatmentTypeDisplay.ts` and `components/treatment/prosthesisTypeDisplay.ts`. + +### Realtime and unread state + +`modules/notifications/user-notification.service.ts` writes `UserNotification` rows and pushes them through the Socket.IO transport in `backend/src/realtime/` (`emitToUserOrg` → `notification.created`). On the frontend a single `notification.created` event drives three things: the header bell inbox, sidebar **tab badges**, and a *soft* refresh of whatever list is currently open — soft meaning it must not remount components or clear an in-progress treatment draft. Unread is per-user cursor state (`LabCaseUserReadState`, `LabCaseUserTabReadState`) plus the `LabCaseActivity` log — badges clear on opening a case, not on visiting a tab. + +### Layout conventions worth knowing before you create a file + +- **Prisma lives outside `src/`**: `backend/prisma/` holds `schema.prisma`, migrations, seeds *and* `prisma.module.ts` / `prisma.service.ts` — hence imports like `../../../prisma/prisma.service`. Register new Nest modules in `app.module.ts`. +- **Frontend layering** (`.cursor/rules/frontend-components.mdc`): `app/**/page.tsx` is a thin wrapper only → route logic in `components/ui/{feature}/{Feature}Page.tsx` → JSX in `components/ui/**` → pure helpers in `components/{feature}/` or `components/shared/`. No JSX outside `ui/`, no pure helpers inside it. +- **i18n is mandatory, not a follow-up**: every user-visible string goes into `en.json`, `fa.json`, **and** `nl.json`. `fa` is RTL, so use logical `text-start`/`text-end`, never `text-left`/`text-right`. Dates/times/numbers go through `lib/i18n/format.ts`; form dates use `AppDateInput`, never a native date input. +- Treatment attachments are written to disk at `backend/uploads/treatments` relative to `process.cwd()`. + +### Tests + +Jest covers pure logic only — permission normalization, phone/timezone helpers, task generation, lab-send validation (7 suites in `backend/src/**`). There are no frontend tests; `npx tsc --noEmit` is the frontend gate. + +## Deployment + +Images are built on a dev machine and pulled by the server; Compose files and scripts are in `infrastructure/` (`docker-compose.{prod,staging,registry}.yml`). Full guide: `infrastructure/DEPLOY.md`. Root `README.md` covers the Docker Hub + Let's Encrypt path and the Gitea registry path. Frontend `NEXT_PUBLIC_*` are **build args** — changing the public domain requires rebuilding the frontend image. diff --git a/backend/.env.example b/backend/.env.example index 3d2145e..d387d78 100644 --- a/backend/.env.example +++ b/backend/.env.example @@ -47,3 +47,28 @@ SMTP_PASSWORD=your_app_password # SMS_IR_API_KEY=4QKMiSU4Kh7tWPLCdRMV0QpDh8WgF33YkWRS18BcG3vf4QHi SMS_IR_API_KEY=lwbK7hxmjimNjFS4g5DWahh75EKCgJUfcUIinUQzfQXwXkSp SMS_IR_TEMPLATE_ID=123456 + +# ── Voice treatment entry ────────────────────────────────────────────────────── +# Without OPENROUTER_API_KEY the microphone button does not render at all. +OPENROUTER_API_KEY= +# OPENROUTER_BASE_URL=https://openrouter.ai/api/v1 + +# Locales the microphone is offered in. An unknown locale here fails at boot. +# VOICE_ENABLED_LOCALES=fa,en,nl + +# Models, overridable per locale (VOICE_ASR_MODEL_FA, VOICE_LLM_MODEL_NL, ...). +# All locales share these today; the per-locale override exists so Persian can be +# repointed at a specialist ASR vendor without a code change. +# VOICE_ASR_MODEL=openai/whisper-1 +# VOICE_LLM_MODEL=google/gemini-3.7-flash +# VOICE_ASR_PROVIDER_FA=openrouter +# VOICE_LLM_PROVIDER_FA=openrouter + +# Recording cap in ms (0 = uncapped). 2 minutes bounds worst-case vendor spend at +# about 1.3 cents per recording. +# VOICE_MAX_RECORDING_MS=120000 + +# Per-user rate limit on the extract endpoint. Unreachable by a human — a recording +# plus processing takes ten seconds at minimum — so it is purely an abuse guard. +# VOICE_THROTTLE_TTL=60 +# VOICE_THROTTLE_LIMIT=6 diff --git a/backend/src/app.module.ts b/backend/src/app.module.ts index 2dfad1d..66e6583 100644 --- a/backend/src/app.module.ts +++ b/backend/src/app.module.ts @@ -1,5 +1,6 @@ import { Module } from '@nestjs/common'; -import { ConfigModule } from '@nestjs/config'; +import { ConfigModule, ConfigService } from '@nestjs/config'; +import { ThrottlerModule } from '@nestjs/throttler'; import configurations from './configs/configurations'; import { AuthModule } from './modules/auth/auth.module'; import { AppController } from './app.controller'; @@ -20,6 +21,7 @@ import { LabCaseCommentsModule } from './modules/lab-case-comments/lab-case-comm import { TodayModule } from './modules/today/today.module'; import { NotificationsModule } from './modules/notifications/notifications.module'; import { RealtimeModule } from './realtime/realtime.module'; +import { VoiceModule } from './modules/voice/voice.module'; @Module({ imports: [ @@ -27,6 +29,22 @@ import { RealtimeModule } from './realtime/realtime.module'; isGlobal: true, load: [configurations], }), + // First use of @nestjs/throttler in this app. Deliberately NOT bound as a global + // APP_GUARD: a globally-bound ThrottlerGuard rate-limits every route against every + // named throttler, which would cap the whole API at the voice limit. ThrottlerGuard + // is applied to the one expensive route instead, so nothing else changes behaviour. + ThrottlerModule.forRootAsync({ + inject: [ConfigService], + useFactory: (config: ConfigService) => ({ + throttlers: [ + { + name: 'voice', + ttl: (config.get('voice.throttle.ttl') ?? 60) * 1000, + limit: config.get('voice.throttle.limit') ?? 6, + }, + ], + }), + }), PrismaModule, // ✅ ADD THIS CatalogModule, TreatmentCatalogModule, @@ -43,9 +61,10 @@ import { RealtimeModule } from './realtime/realtime.module'; TodayModule, NotificationsModule, RealtimeModule, + VoiceModule, AdminModule.forRoot(), ], controllers: [AppController], - providers: [AppService], + providers: [AppService], }) -export class AppModule {} \ No newline at end of file +export class AppModule {} diff --git a/backend/src/common/body-parsers.spec.ts b/backend/src/common/body-parsers.spec.ts new file mode 100644 index 0000000..e54019b --- /dev/null +++ b/backend/src/common/body-parsers.spec.ts @@ -0,0 +1,104 @@ +import express, { + type NextFunction, + type Request, + type Response, +} from 'express'; +import request from 'supertest'; +import { createJsonBodyParser, VOICE_EXTRACT_PATH } from './body-parsers'; + +/** + * Guards a bug that made the voice endpoint completely unusable while surfacing as a + * generic 500: the large-body limit stopped applying, so every real recording — anything + * past roughly 20 seconds of audio — was rejected by Express's 100 kb default. + */ + +type ProbeBody = { keys?: number; type?: string }; + +function buildApp(): express.Express { + const app = express(); + app.use(createJsonBodyParser()); + app.post('*splat', (req: Request, res: Response) => { + res.json({ keys: Object.keys((req.body ?? {}) as object).length }); + }); + // Surface body-parser's own error instead of Express's HTML default page. + app.use( + ( + err: { status?: number; type?: string }, + _req: Request, + res: Response, + next: NextFunction, + ) => { + if (res.headersSent) { + next(err); + return; + } + res.status(err.status ?? 500).json({ type: err.type }); + }, + ); + return app; +} + +const bodyOfKb = (kb: number) => ({ audio: 'A'.repeat(kb * 1024) }); + +describe('createJsonBodyParser', () => { + it('accepts a body far past the default limit on the voice route', async () => { + const res = await request(buildApp()) + .post(VOICE_EXTRACT_PATH) + .send(bodyOfKb(300)); + expect(res.status).toBe(200); + expect((res.body as ProbeBody).keys).toBe(1); + }); + + it('accepts a realistic worst-case recording', async () => { + // Two minutes of opus is well under 1 MB, but wav is far larger; 4 MB must pass. + const res = await request(buildApp()) + .post(VOICE_EXTRACT_PATH) + .send(bodyOfKb(4096)); + expect(res.status).toBe(200); + }); + + it('keeps the default limit on every other route', async () => { + // The larger limit must not leak app-wide as a side effect. + const res = await request(buildApp()) + .post('/api/auth/login') + .send(bodyOfKb(300)); + expect(res.status).toBe(413); + expect((res.body as ProbeBody).type).toBe('entity.too.large'); + }); + + it('still parses ordinary bodies on ordinary routes', async () => { + const res = await request(buildApp()) + .post('/api/auth/login') + .send({ email: 'a@b.c' }); + expect(res.status).toBe(200); + expect((res.body as ProbeBody).keys).toBe(1); + }); + + it('widens the limit for the spellings Express itself accepts', async () => { + // Express routes case-insensitively and ignores a trailing slash by default, so these + // all reach the voice controller. Any of them taking the 100 kb parser would 413 a + // real recording and read as a broken microphone. + for (const path of [ + '/api/voice/extract/', + '/API/Voice/Extract', + '/api/Voice/extract/', + ]) { + const res = await request(buildApp()).post(path).send(bodyOfKb(300)); + expect(res.status).toBe(200); + } + }); + + it('does not widen the limit for a path that merely looks similar', async () => { + for (const path of [ + '/api/voice/extract/extra', + '/api/voice', + '/voice/extract', + // Express ignores one trailing slash, not two — this one never routes, so it must + // not get the large parser either. + '/api/voice/extract//', + ]) { + const res = await request(buildApp()).post(path).send(bodyOfKb(300)); + expect(res.status).toBe(413); + } + }); +}); diff --git a/backend/src/common/body-parsers.ts b/backend/src/common/body-parsers.ts new file mode 100644 index 0000000..554c75e --- /dev/null +++ b/backend/src/common/body-parsers.ts @@ -0,0 +1,43 @@ +import { + json, + type NextFunction, + type Request, + type RequestHandler, + type Response, +} from 'express'; + +/** The one route that accepts a large body, and how large. */ +export const VOICE_EXTRACT_PATH = '/api/voice/extract'; +export const VOICE_BODY_LIMIT = '10mb'; + +/** + * JSON body parsing for the whole app. + * + * Voice recordings are base64 JSON and pass Express's 100 kb default at roughly 20 seconds + * of audio, so that one route needs a larger limit while every other endpoint keeps the + * default — a large body should not become acceptable everywhere. + * + * Deliberately one middleware that *chooses* a parser, not a path-mounted parser stacked in + * front of a default one: that arrangement depended on Express's mount-path stripping and on + * body-parser skipping an already-parsed request, and silently stopped applying whenever the + * middleware order shifted. One explicit branch has no such coupling. + */ +/** + * Express routes case-insensitively and ignores exactly one trailing slash, so + * `/API/Voice/Extract/` reaches the same controller and must get the same limit — otherwise + * it 413s every real recording, which reads as a broken microphone rather than a route. + * Two slashes never route, so they must not buy a 10 MB buffer either. + */ +function isVoiceExtractPath(path: string): boolean { + return path.toLowerCase().replace(/\/$/, '') === VOICE_EXTRACT_PATH; +} + +export function createJsonBodyParser(): RequestHandler { + const voiceParser = json({ limit: VOICE_BODY_LIMIT }); + const defaultParser = json(); + + return (req: Request, res: Response, next: NextFunction) => + isVoiceExtractPath(req.path) + ? voiceParser(req, res, next) + : defaultParser(req, res, next); +} diff --git a/backend/src/common/digits.spec.ts b/backend/src/common/digits.spec.ts new file mode 100644 index 0000000..01a834a --- /dev/null +++ b/backend/src/common/digits.spec.ts @@ -0,0 +1,31 @@ +import { toLatinDigits } from './digits'; + +/** + * Exercised by the voice pipeline on two untrusted inputs: spoken dates, and the tooth + * code the extraction model echoes back — a Persian-digit "۲۶" that fails to normalise + * costs the clinician a tooth, silently. + */ +describe('toLatinDigits', () => { + it('normalises Persian digits and leaves everything else alone', () => { + expect( + toLatinDigits('\u06F1\u06F4\u06F0\u06F4/\u06F0\u06F7/\u06F2\u06F5'), + ).toBe('1404/07/25'); + expect(toLatinDigits('1404/07/25')).toBe('1404/07/25'); + expect(toLatinDigits('\u062F\u0646\u062F\u0627\u0646 \u06F1\u06F4')).toBe( + '\u062F\u0646\u062F\u0627\u0646 14', + ); + }); + + it('also normalises the Arabic-Indic block, which ASR output can carry', () => { + // U+0660..U+0669, distinct code points from the Persian U+06F0..U+06F9 block. + expect( + toLatinDigits('\u0661\u0664\u0660\u0664/\u0660\u0667/\u0662\u0665'), + ).toBe('1404/07/25'); + }); + + it('normalises a transcript that mixes both blocks with ASCII', () => { + expect(toLatinDigits('\u06F1\u06F4 and \u0661\u0665 and 16')).toBe( + '14 and 15 and 16', + ); + }); +}); diff --git a/backend/src/common/digits.ts b/backend/src/common/digits.ts new file mode 100644 index 0000000..d78d538 --- /dev/null +++ b/backend/src/common/digits.ts @@ -0,0 +1,16 @@ +const PERSIAN_ZERO = 0x06f0; +const ARABIC_INDIC_ZERO = 0x0660; + +/** + * Normalise Persian and Arabic-Indic digits to ASCII. Non-digits pass through. + * + * Both blocks, not just the Persian one the frontend handles: ASR output can carry either, + * sometimes mixed with ASCII in a single transcript. + */ +export function toLatinDigits(value: string): string { + return value.replace(/[۰-۹٠-٩]/g, (ch) => { + const code = ch.charCodeAt(0); + const base = code >= PERSIAN_ZERO ? PERSIAN_ZERO : ARABIC_INDIC_ZERO; + return String(code - base); + }); +} diff --git a/backend/src/common/errors/error-codes.ts b/backend/src/common/errors/error-codes.ts index 5c17305..79dfe99 100644 --- a/backend/src/common/errors/error-codes.ts +++ b/backend/src/common/errors/error-codes.ts @@ -29,8 +29,10 @@ export const ErrorCode = { PERMISSION_LAB_ONLY: 'PERMISSION_LAB_ONLY', PERMISSION_NOT_MEMBER: 'PERMISSION_NOT_MEMBER', PERMISSION_OWNER_ONLY: 'PERMISSION_OWNER_ONLY', - PERMISSION_PARTICIPATION_SUBSCRIPTION: 'PERMISSION_PARTICIPATION_SUBSCRIPTION', - PERMISSION_ENABLE_PARTICIPATION_FIRST: 'PERMISSION_ENABLE_PARTICIPATION_FIRST', + PERMISSION_PARTICIPATION_SUBSCRIPTION: + 'PERMISSION_PARTICIPATION_SUBSCRIPTION', + PERMISSION_ENABLE_PARTICIPATION_FIRST: + 'PERMISSION_ENABLE_PARTICIPATION_FIRST', PERMISSION_CLINIC_WORKING_HOURS: 'PERMISSION_CLINIC_WORKING_HOURS', PERMISSION_ACCESS_APPOINTMENTS: 'PERMISSION_ACCESS_APPOINTMENTS', PERMISSION_EDIT_APPOINTMENTS: 'PERMISSION_EDIT_APPOINTMENTS', @@ -51,7 +53,8 @@ export const ErrorCode = { VALIDATION_PASSWORD_REQUIRED: 'VALIDATION_PASSWORD_REQUIRED', VALIDATION_MOBILE_INVALID: 'VALIDATION_MOBILE_INVALID', VALIDATION_NAME_TOO_SHORT: 'VALIDATION_NAME_TOO_SHORT', - VALIDATION_ORGANIZATION_NAME_REQUIRED: 'VALIDATION_ORGANIZATION_NAME_REQUIRED', + VALIDATION_ORGANIZATION_NAME_REQUIRED: + 'VALIDATION_ORGANIZATION_NAME_REQUIRED', VALIDATION_ORGANIZATION_TYPE_INVALID: 'VALIDATION_ORGANIZATION_TYPE_INVALID', VALIDATION_TOKEN_REQUIRED: 'VALIDATION_TOKEN_REQUIRED', VALIDATION_FIELD_REQUIRED: 'VALIDATION_FIELD_REQUIRED', @@ -69,8 +72,10 @@ export const ErrorCode = { APPOINTMENT_INVALID_DATE: 'APPOINTMENT_INVALID_DATE', APPOINTMENT_PROVIDER_NOT_MEMBER: 'APPOINTMENT_PROVIDER_NOT_MEMBER', APPOINTMENT_PROVIDER_INACTIVE: 'APPOINTMENT_PROVIDER_INACTIVE', - APPOINTMENT_PROVIDER_NO_TREATMENT_EDIT: 'APPOINTMENT_PROVIDER_NO_TREATMENT_EDIT', - APPOINTMENT_PROVIDER_NO_WORKING_HOURS: 'APPOINTMENT_PROVIDER_NO_WORKING_HOURS', + APPOINTMENT_PROVIDER_NO_TREATMENT_EDIT: + 'APPOINTMENT_PROVIDER_NO_TREATMENT_EDIT', + APPOINTMENT_PROVIDER_NO_WORKING_HOURS: + 'APPOINTMENT_PROVIDER_NO_WORKING_HOURS', APPOINTMENT_PROVIDER_NOT_WORKING_DAY: 'APPOINTMENT_PROVIDER_NOT_WORKING_DAY', APPOINTMENT_OUTSIDE_WORKING_HOURS: 'APPOINTMENT_OUTSIDE_WORKING_HOURS', APPOINTMENT_NOT_FOUND: 'APPOINTMENT_NOT_FOUND', @@ -82,7 +87,8 @@ export const ErrorCode = { WORKING_HOURS_INVALID: 'WORKING_HOURS_INVALID', WORKING_HOURS_OWNER_NOT_ALLOWED: 'WORKING_HOURS_OWNER_NOT_ALLOWED', - WORKING_HOURS_CONFLICTS_WITH_APPOINTMENTS: 'WORKING_HOURS_CONFLICTS_WITH_APPOINTMENTS', + WORKING_HOURS_CONFLICTS_WITH_APPOINTMENTS: + 'WORKING_HOURS_CONFLICTS_WITH_APPOINTMENTS', STAFF_UNKNOWN_PERMISSIONS: 'STAFF_UNKNOWN_PERMISSIONS', STAFF_NO_SUBSCRIPTION: 'STAFF_NO_SUBSCRIPTION', @@ -147,7 +153,8 @@ export const ErrorCode = { TREATMENT_ATTACHMENT_NOT_FOUND: 'TREATMENT_ATTACHMENT_NOT_FOUND', TREATMENT_FILE_UNAVAILABLE: 'TREATMENT_FILE_UNAVAILABLE', TREATMENT_CASE_INVALID_ATTACHMENTS: 'TREATMENT_CASE_INVALID_ATTACHMENTS', - TREATMENT_TOOTH_PROSTHESIS_INCOMPLETE: 'TREATMENT_TOOTH_PROSTHESIS_INCOMPLETE', + TREATMENT_TOOTH_PROSTHESIS_INCOMPLETE: + 'TREATMENT_TOOTH_PROSTHESIS_INCOMPLETE', TREATMENT_DETAIL_SENT: 'TREATMENT_DETAIL_SENT', TREATMENT_NOT_FOUND: 'TREATMENT_NOT_FOUND', TREATMENT_PATIENT_OR_WALK_IN: 'TREATMENT_PATIENT_OR_WALK_IN', @@ -177,6 +184,15 @@ export const ErrorCode = { TODAY_INVALID_RANGE: 'TODAY_INVALID_RANGE', TODAY_INVALID_RANGE_ORDER: 'TODAY_INVALID_RANGE_ORDER', + // Voice treatment entry + VOICE_NOT_AVAILABLE: 'VOICE_NOT_AVAILABLE', + VOICE_CLIP_TOO_LONG: 'VOICE_CLIP_TOO_LONG', + VOICE_UNSUPPORTED_FORMAT: 'VOICE_UNSUPPORTED_FORMAT', + VOICE_ASR_FAILED: 'VOICE_ASR_FAILED', + VOICE_EXTRACT_FAILED: 'VOICE_EXTRACT_FAILED', + VOICE_NOTHING_RECOGNIZED: 'VOICE_NOTHING_RECOGNIZED', + VOICE_RATE_LIMITED: 'VOICE_RATE_LIMITED', + // Generic HTTP NOT_FOUND: 'NOT_FOUND', CONFLICT: 'CONFLICT', diff --git a/backend/src/common/fdi.spec.ts b/backend/src/common/fdi.spec.ts new file mode 100644 index 0000000..3910963 --- /dev/null +++ b/backend/src/common/fdi.spec.ts @@ -0,0 +1,144 @@ +import { + areArchNeighbors, + sortInArchOrder, + FDI_TOOTH_IDS, + isFdiTooth, + sameArch, + teethBetweenInclusive, + toFdi, +} from './fdi'; + +describe('FDI geometry', () => { + describe('toFdi — the patient-right convention', () => { + // A mirrored quadrant produces a *valid* code for the wrong tooth, so no schema + // check can catch it. These four cases are the guard. + it('maps upper + patient right to quadrant 1', () => { + expect(toFdi('upper', 'patient_right', 6)).toBe('16'); + expect(toFdi('upper', 'patient_right', 1)).toBe('11'); + }); + + it('maps upper + patient left to quadrant 2', () => { + expect(toFdi('upper', 'patient_left', 6)).toBe('26'); + expect(toFdi('upper', 'patient_left', 8)).toBe('28'); + }); + + it('maps lower + patient left to quadrant 3', () => { + expect(toFdi('lower', 'patient_left', 6)).toBe('36'); + }); + + it('maps lower + patient right to quadrant 4', () => { + expect(toFdi('lower', 'patient_right', 6)).toBe('46'); + expect(toFdi('lower', 'patient_right', 8)).toBe('48'); + }); + + it('never clamps an out-of-range position', () => { + expect(toFdi('upper', 'patient_right', 9)).toBeNull(); + expect(toFdi('upper', 'patient_right', 0)).toBeNull(); + expect(toFdi('upper', 'patient_right', -1)).toBeNull(); + expect(toFdi('upper', 'patient_right', 1.5)).toBeNull(); + expect(toFdi('upper', 'patient_right', Number.NaN)).toBeNull(); + }); + }); + + describe('isFdiTooth', () => { + it('accepts all 32 permanent teeth', () => { + expect(FDI_TOOTH_IDS.size).toBe(32); + for (const tooth of FDI_TOOTH_IDS) expect(isFdiTooth(tooth)).toBe(true); + }); + + it('rejects deciduous teeth — the chart has no primary dentition', () => { + for (const tooth of ['51', '55', '61', '71', '85']) { + expect(isFdiTooth(tooth)).toBe(false); + } + }); + + it('rejects garbage', () => { + for (const value of [ + '', + '1', + '19', + '10', + '29', + '99', + 14, + null, + undefined, + {}, + ]) { + expect(isFdiTooth(value)).toBe(false); + } + }); + }); + + describe('adjacency', () => { + it('treats neighbours within a quadrant as adjacent', () => { + expect(areArchNeighbors('14', '15')).toBe(true); + expect(areArchNeighbors('15', '14')).toBe(true); + }); + + it('treats the midline pairs as adjacent', () => { + expect(areArchNeighbors('11', '21')).toBe(true); + expect(areArchNeighbors('41', '31')).toBe(true); + }); + + it('rejects non-neighbours and cross-arch pairs', () => { + expect(areArchNeighbors('14', '16')).toBe(false); + expect(areArchNeighbors('18', '28')).toBe(false); + expect(areArchNeighbors('14', '44')).toBe(false); + expect(areArchNeighbors('14', '14')).toBe(false); + }); + }); + + describe('sameArch', () => { + it('groups by arch, not by quadrant', () => { + expect(sameArch('18', '28')).toBe(true); + expect(sameArch('48', '38')).toBe(true); + expect(sameArch('18', '48')).toBe(false); + expect(sameArch('14', '99')).toBe(false); + }); + }); + + describe('teethBetweenInclusive', () => { + it('returns the span in arch order regardless of argument order', () => { + expect(teethBetweenInclusive('14', '16')).toEqual(['16', '15', '14']); + expect(teethBetweenInclusive('16', '14')).toEqual(['16', '15', '14']); + }); + + it('spans the midline', () => { + expect(teethBetweenInclusive('12', '22')).toEqual([ + '12', + '11', + '21', + '22', + ]); + }); + + it('returns a single tooth for identical endpoints', () => { + expect(teethBetweenInclusive('14', '14')).toEqual(['14']); + }); + + it('returns null across arches or for unknown teeth', () => { + expect(teethBetweenInclusive('14', '44')).toBeNull(); + expect(teethBetweenInclusive('14', '99')).toBeNull(); + expect(teethBetweenInclusive('99', '14')).toBeNull(); + }); + }); + describe('sortInArchOrder', () => { + it('sorts along the arch rather than lexically', () => { + expect(sortInArchOrder(['14', '16', '15'])).toEqual(['16', '15', '14']); + }); + + it('places 11 beside 21 across the midline', () => { + expect(sortInArchOrder(['21', '11', '12'])).toEqual(['12', '11', '21']); + }); + + it('is a no-op for an empty or single-tooth list', () => { + expect(sortInArchOrder([])).toEqual([]); + expect(sortInArchOrder(['14'])).toEqual(['14']); + }); + + it('does not drop teeth it cannot place', () => { + expect(sortInArchOrder(['99', '14']).sort()).toEqual(['14', '99']); + }); + }); +}); diff --git a/backend/src/common/fdi.ts b/backend/src/common/fdi.ts new file mode 100644 index 0000000..ae37fbe --- /dev/null +++ b/backend/src/common/fdi.ts @@ -0,0 +1,145 @@ +/** + * FDI tooth geometry — permanent dentition only. + * + * Mirrors `frontend/src/components/treatment/fdiToothMeta.ts` and the adjacency rules in + * `toothSelectionGroups.ts`. Adjacency is defined by position in the arch order, so the + * midline pairs (11–21, 41–31) are neighbours, exactly as the chart treats them. + */ + +import { toLatinDigits } from './digits'; + +export type Arch = 'upper' | 'lower'; + +/** Which side of the *patient*, not of the screen. Quadrant 1 is the patient's upper right. */ +export type PatientSide = 'patient_right' | 'patient_left'; + +/** + * Upper arch in chart order: patient's RIGHT (18) → midline → patient's LEFT (28) — the drawn + * layout, which mirrors the patient's own sides. Never read a position off this array by + * index; use `toFdi()`, which owns the side convention. + */ +export const FDI_UPPER_ARCH_ORDER = [ + '18', + '17', + '16', + '15', + '14', + '13', + '12', + '11', + '21', + '22', + '23', + '24', + '25', + '26', + '27', + '28', +] as const; + +/** Lower arch, same chart ordering: patient's RIGHT (48) → midline → patient's LEFT (38). */ +export const FDI_LOWER_ARCH_ORDER = [ + '48', + '47', + '46', + '45', + '44', + '43', + '42', + '41', + '31', + '32', + '33', + '34', + '35', + '36', + '37', + '38', +] as const; + +export const FDI_TOOTH_IDS: ReadonlySet = new Set([ + ...FDI_UPPER_ARCH_ORDER, + ...FDI_LOWER_ARCH_ORDER, +]); + +export function isFdiTooth(value: unknown): value is string { + return typeof value === 'string' && FDI_TOOTH_IDS.has(value); +} + +/** + * Clean up a tooth code the model echoed back. It is reading Persian speech, so it can hand + * back "۲۶" or "2 6" from digit-by-digit dictation; neither matches literally, and the + * near-miss does not fail loudly — the tooth just turns into "not understood". + */ +export function normalizeFdiCode(value: unknown): string { + if (typeof value !== 'string') return ''; + return toLatinDigits(value).replace(/\s+/g, ''); +} + +function archOrder(tooth: string): readonly string[] | null { + if ((FDI_UPPER_ARCH_ORDER as readonly string[]).includes(tooth)) + return FDI_UPPER_ARCH_ORDER; + if ((FDI_LOWER_ARCH_ORDER as readonly string[]).includes(tooth)) + return FDI_LOWER_ARCH_ORDER; + return null; +} + +export function sameArch(a: string, b: string): boolean { + const archA = archOrder(a); + const archB = archOrder(b); + return Boolean(archA && archB && archA === archB); +} + +export function areArchNeighbors(a: string, b: string): boolean { + const arch = archOrder(a); + if (!arch || !sameArch(a, b)) return false; + return Math.abs(arch.indexOf(a) - arch.indexOf(b)) === 1; +} + +/** Inclusive span between two teeth of the same arch, in arch order. Null if not comparable. */ +export function teethBetweenInclusive(a: string, b: string): string[] | null { + const arch = archOrder(a); + if (!arch || !sameArch(a, b)) return null; + const i = arch.indexOf(a); + const j = arch.indexOf(b); + if (i < 0 || j < 0) return null; + const [from, to] = i <= j ? [i, j] : [j, i]; + return [...arch.slice(from, to + 1)]; +} + +/** + * Arch + patient side + position (1 = central incisor … 8 = third molar) → FDI code. + * + * The single place the patient-right convention lives. Getting it backwards mirrors every + * quadrant into a valid-looking code for the wrong tooth, which no schema check can catch. + */ +export function toFdi( + arch: Arch, + side: PatientSide, + position: number, +): string | null { + if (!Number.isInteger(position) || position < 1 || position > 8) return null; + let quadrant: number; + if (arch === 'upper') { + quadrant = side === 'patient_right' ? 1 : 2; + } else { + quadrant = side === 'patient_left' ? 3 : 4; + } + const code = `${quadrant}${position}`; + return FDI_TOOTH_IDS.has(code) ? code : null; +} + +/** + * Along the arch, not lexically — a bridge reads 16-15-14, and 11 sits beside 21 across the + * midline. Teeth from another arch sort to the end, stably. + */ +export function sortInArchOrder(teeth: readonly string[]): string[] { + if (teeth.length === 0) return []; + const arch = teeth.map((t) => archOrder(t)).find((a) => a !== null) ?? null; + if (!arch) return [...teeth]; + const indexOf = (tooth: string) => { + const i = arch.indexOf(tooth); + return i === -1 ? Number.MAX_SAFE_INTEGER : i; + }; + return [...teeth].sort((a, b) => indexOf(a) - indexOf(b)); +} diff --git a/backend/src/common/jalali.spec.ts b/backend/src/common/jalali.spec.ts new file mode 100644 index 0000000..df1bb3c --- /dev/null +++ b/backend/src/common/jalali.spec.ts @@ -0,0 +1,89 @@ +import { + gregorianToJalali, + isJalaliLeapYear, + isValidJalaliDate, + jalaliDaysInMonth, + jalaliToGregorian, + jalaliToIsoDate, +} from './jalali'; + +describe('jalali calendar', () => { + // Anchors verified against frontend/src/lib/i18n/persianCalendar.ts, the source this + // was ported from, over every day between 1900 and 2100. + it('converts known Jalali dates to ISO', () => { + expect(jalaliToIsoDate(1404, 1, 1)).toBe('2025-03-21'); // Nowruz 1404 + expect(jalaliToIsoDate(1404, 7, 25)).toBe('2025-10-17'); + expect(jalaliToIsoDate(1404, 6, 31)).toBe('2025-09-22'); // 31-day first half + expect(jalaliToIsoDate(1405, 1, 1)).toBe('2026-03-21'); + }); + + it('round-trips Gregorian → Jalali → Gregorian', () => { + for (const [gy, gm, gd] of [ + [2025, 3, 21], + [2025, 10, 17], + [2026, 1, 1], + [2024, 2, 29], // Gregorian leap day + [1999, 12, 31], + ] as const) { + const [jy, jm, jd] = gregorianToJalali(gy, gm, gd); + expect(jalaliToGregorian(jy, jm, jd)).toEqual([gy, gm, gd]); + } + }); + + describe('leap years', () => { + it('gives Esfand 30 days in a leap year and 29 otherwise', () => { + expect(isJalaliLeapYear(1403)).toBe(true); + expect(jalaliDaysInMonth(1403, 12)).toBe(30); + expect(jalaliToIsoDate(1403, 12, 30)).toBe('2025-03-20'); + + expect(isJalaliLeapYear(1404)).toBe(false); + expect(jalaliDaysInMonth(1404, 12)).toBe(29); + expect(jalaliToIsoDate(1404, 12, 29)).toBe('2026-03-20'); + }); + + it('rejects Esfand 30 in a non-leap year', () => { + expect(isValidJalaliDate(1404, 12, 30)).toBe(false); + expect(jalaliToIsoDate(1404, 12, 30)).toBeNull(); + }); + }); + + describe('month lengths', () => { + it('is 31 days for months 1-6 and 30 for 7-11', () => { + for (let m = 1; m <= 6; m += 1) + expect(jalaliDaysInMonth(1404, m)).toBe(31); + for (let m = 7; m <= 11; m += 1) + expect(jalaliDaysInMonth(1404, m)).toBe(30); + }); + }); + + describe('invalid input degrades to null rather than throwing', () => { + // The resolver feeds this model-supplied values, which may be nonsense. + it.each([ + ['month 13', 1404, 13, 1], + ['month 0', 1404, 0, 1], + ['day 0', 1404, 1, 0], + ['day 32 in a 31-day month', 1404, 1, 32], + ['day 31 in a 30-day month', 1404, 7, 31], + ['year beyond the conversion table', 9999, 1, 1], + ['non-integer day', 1404, 1, 1.5], + ])('returns null for %s', (_label, jy, jm, jd) => { + expect(jalaliToIsoDate(jy, jm, jd)).toBeNull(); + }); + }); + + describe('no export throws on an unsupported year', () => { + // The whole module is reachable from model-supplied values, so it degrades instead of + // raising — jalaliToIsoDate's guard is not the only entry point. + it.each([9999, -9999, 3178, Number.NaN, 1.5])('year %s', (jy) => { + expect(() => isJalaliLeapYear(jy)).not.toThrow(); + expect(() => jalaliDaysInMonth(jy, 1)).not.toThrow(); + expect(isJalaliLeapYear(jy)).toBe(false); + expect(jalaliDaysInMonth(jy, 1)).toBe(0); + }); + + it('returns 0 days for an impossible month', () => { + expect(jalaliDaysInMonth(1404, 0)).toBe(0); + expect(jalaliDaysInMonth(1404, 13)).toBe(0); + }); + }); +}); diff --git a/backend/src/common/jalali.ts b/backend/src/common/jalali.ts new file mode 100644 index 0000000..b061e65 --- /dev/null +++ b/backend/src/common/jalali.ts @@ -0,0 +1,179 @@ +/** + * Jalali (Persian) calendar arithmetic, ported from + * `frontend/src/lib/i18n/persianCalendar.ts` (itself jalaali-js, MIT). The backend needs it + * because voice resolves spoken Jalali dates server-side, where the resolvers are tested. + * Keep the two copies in step; the underlying calendar does not change. + */ + +const BREAKS = [ + -61, 9, 38, 199, 426, 686, 756, 818, 1111, 1181, 1210, 1635, 2060, 2097, 2192, + 2262, 2324, 2394, 2456, 3178, +]; + +/** Inclusive lower / exclusive upper Jalali year bounds the conversion table covers. */ +export const MIN_JALALI_YEAR = BREAKS[0]; +export const MAX_JALALI_YEAR = BREAKS[BREAKS.length - 1]; + +function div(a: number, b: number): number { + return Math.trunc(a / b); +} + +function mod(a: number, b: number): number { + return a - Math.trunc(a / b) * b; +} + +function g2d(gy: number, gm: number, gd: number): number { + let d = + div((gy + div(gm - 8, 6) + 100100) * 1461, 4) + + div(153 * mod(gm + 9, 12) + 2, 5) + + gd - + 34840408; + d = d - div(div(gy + 100100 + div(gm - 8, 6), 100) * 3, 4) + 752; + return d; +} + +function d2g(jdn: number): { gy: number; gm: number; gd: number } { + let j = 4 * jdn + 139361631; + j = j + div(div(4 * jdn + 183187720, 146097) * 3, 4) * 4 - 3908; + const i = div(mod(j, 1461), 4) * 5 + 308; + const gd = div(mod(i, 153), 5) + 1; + const gm = mod(div(i, 153), 12) + 1; + const gy = div(j, 1461) - 100100 + div(8 - gm, 6); + return { gy, gm, gd }; +} + +function jalCal( + jy: number, + withoutLeap: boolean, +): { leap?: number; gy: number; march: number } { + const bl = BREAKS.length; + const gy = jy + 621; + let leapJ = -14; + let jp = BREAKS[0]; + let jump = 0; + let leap = 0; + let n = 0; + + if (jy < jp || jy >= BREAKS[bl - 1]) { + throw new Error(`Invalid Jalaali year ${jy}`); + } + + for (let i = 1; i < bl; i += 1) { + const jm = BREAKS[i]; + jump = jm - jp; + if (jy < jm) break; + leapJ = leapJ + div(jump, 33) * 8 + div(mod(jump, 33), 4); + jp = jm; + } + n = jy - jp; + leapJ = leapJ + div(n, 33) * 8 + div(mod(n, 33) + 3, 4); + if (mod(jump, 33) === 4 && jump - n === 4) leapJ += 1; + + const leapG = div(gy, 4) - div((div(gy, 100) + 1) * 3, 4) - 150; + const march = 20 + leapJ - leapG; + + if (withoutLeap) return { gy, march }; + + if (jump - n < 6) n = n - jump + div(jump + 4, 33) * 33; + leap = mod(mod(n + 1, 33) - 1, 4); + if (leap === -1) leap = 4; + return { leap, gy, march }; +} + +function j2d(jy: number, jm: number, jd: number): number { + const r = jalCal(jy, true); + return g2d(r.gy, 3, r.march) + (jm - 1) * 31 - div(jm, 7) * (jm - 7) + jd - 1; +} + +function d2j(jdn: number): { jy: number; jm: number; jd: number } { + const { gy } = d2g(jdn); + let jy = gy - 621; + const r = jalCal(jy, false); + const jdn1f = g2d(gy, 3, r.march); + let k = jdn - jdn1f; + let jm: number; + let jd: number; + + if (k >= 0) { + if (k <= 185) { + jm = 1 + div(k, 31); + jd = mod(k, 31) + 1; + return { jy, jm, jd }; + } + k -= 186; + } else { + jy -= 1; + k += 179; + if (r.leap === 1) k += 1; + } + jm = 7 + div(k, 30); + jd = mod(k, 30) + 1; + return { jy, jm, jd }; +} + +export function gregorianToJalali( + gy: number, + gm: number, + gd: number, +): [number, number, number] { + const { jy, jm, jd } = d2j(g2d(gy, gm, gd)); + return [jy, jm, jd]; +} + +export function jalaliToGregorian( + jy: number, + jm: number, + jd: number, +): [number, number, number] { + const { gy, gm, gd } = d2g(j2d(jy, jm, jd)); + return [gy, gm, gd]; +} + +/** True when the year is inside the conversion table's supported range. */ +export function isSupportedJalaliYear(jy: number): boolean { + return Number.isInteger(jy) && jy >= MIN_JALALI_YEAR && jy < MAX_JALALI_YEAR; +} + +/** False for unsupported years rather than throwing — see the module contract. */ +export function isJalaliLeapYear(jy: number): boolean { + if (!isSupportedJalaliYear(jy)) return false; + const r = jalCal(jy, false); + return r.leap === 0; +} + +/** + * Days in a Jalali month, or 0 when the year or month is not real. Zero rather than a throw: + * every export here is reachable from model-supplied values, so the module degrades. + */ +export function jalaliDaysInMonth(jy: number, jm: number): number { + if (!isSupportedJalaliYear(jy)) return 0; + if (!Number.isInteger(jm) || jm < 1 || jm > 12) return 0; + if (jm <= 6) return 31; + if (jm <= 11) return 30; + return isJalaliLeapYear(jy) ? 30 : 29; +} + +/** True when the triple is a real Jalali date inside the supported year range. */ +export function isValidJalaliDate(jy: number, jm: number, jd: number): boolean { + if (!Number.isInteger(jy) || !Number.isInteger(jm) || !Number.isInteger(jd)) { + return false; + } + if (jy < MIN_JALALI_YEAR || jy >= MAX_JALALI_YEAR) return false; + if (jm < 1 || jm > 12) return false; + if (jd < 1) return false; + return jd <= jalaliDaysInMonth(jy, jm); +} + +/** + * Jalali triple → `YYYY-MM-DD`, or null when the date is not real. Null rather than a throw, + * for the same reason: callers resolve model-supplied values, which may be nonsense. + */ +export function jalaliToIsoDate( + jy: number, + jm: number, + jd: number, +): string | null { + if (!isValidJalaliDate(jy, jm, jd)) return null; + const [gy, gm, gd] = jalaliToGregorian(jy, jm, jd); + return `${String(gy).padStart(4, '0')}-${String(gm).padStart(2, '0')}-${String(gd).padStart(2, '0')}`; +} diff --git a/backend/src/common/zoned-civil-time.spec.ts b/backend/src/common/zoned-civil-time.spec.ts index 99279ab..af0863f 100644 --- a/backend/src/common/zoned-civil-time.spec.ts +++ b/backend/src/common/zoned-civil-time.spec.ts @@ -1,5 +1,9 @@ import { appointmentWithinWorkingHours } from './working-hours'; -import { isValidIanaTimeZone, zonedWeekdayAndMinutes } from './zoned-civil-time'; +import { + civilDateInZone, + isValidIanaTimeZone, + zonedWeekdayAndMinutes, +} from './zoned-civil-time'; describe('zoned civil time', () => { it('accepts IANA zones and rejects garbage', () => { @@ -20,10 +24,48 @@ describe('zoned civil time', () => { const start = new Date('2026-08-20T06:15:00.000Z'); const end = new Date('2026-08-20T06:45:00.000Z'); expect( - appointmentWithinWorkingHours(start, end, [{ startMinute: 8 * 60, endMinute: 17 * 60 }], 'Asia/Tehran'), + appointmentWithinWorkingHours( + start, + end, + [{ startMinute: 8 * 60, endMinute: 17 * 60 }], + 'Asia/Tehran', + ), ).toBe(true); expect( - appointmentWithinWorkingHours(start, end, [{ startMinute: 8 * 60, endMinute: 17 * 60 }], 'UTC'), + appointmentWithinWorkingHours( + start, + end, + [{ startMinute: 8 * 60, endMinute: 17 * 60 }], + 'UTC', + ), ).toBe(false); }); + describe('civilDateInZone', () => { + it('gives the local civil date, which can differ from the UTC date', () => { + // 21:30 UTC is already the next day in Tehran (+03:30). + const instant = new Date('2025-10-11T21:30:00.000Z'); + expect(civilDateInZone(instant, 'UTC')).toBe('2025-10-11'); + expect(civilDateInZone(instant, 'Asia/Tehran')).toBe('2025-10-12'); + }); + + it('gives the previous day for zones behind UTC just after midnight', () => { + const instant = new Date('2025-10-11T02:00:00.000Z'); + expect(civilDateInZone(instant, 'America/New_York')).toBe('2025-10-10'); + expect(civilDateInZone(instant, 'Europe/Amsterdam')).toBe('2025-10-11'); + }); + + it('falls back to UTC on an invalid zone rather than throwing', () => { + // Intl raises RangeError on an unknown zone and this takes a client-supplied string. + const instant = new Date('2025-10-11T21:30:00.000Z'); + expect(() => civilDateInZone(instant, 'Not/AZone')).not.toThrow(); + expect(civilDateInZone(instant, 'Not/AZone')).toBe('2025-10-11'); + expect(civilDateInZone(instant, '')).toBe('2025-10-11'); + }); + + it('zero-pads single-digit months and days', () => { + expect(civilDateInZone(new Date('2025-01-05T12:00:00.000Z'), 'UTC')).toBe( + '2025-01-05', + ); + }); + }); }); diff --git a/backend/src/common/zoned-civil-time.ts b/backend/src/common/zoned-civil-time.ts index 88679ef..e61f161 100644 --- a/backend/src/common/zoned-civil-time.ts +++ b/backend/src/common/zoned-civil-time.ts @@ -53,3 +53,24 @@ export function civilDateJsWeekday(isoDate: string): number { const utcNoon = new Date(Date.UTC(y, m - 1, d, 12, 0, 0, 0)); return utcNoon.getUTCDay(); } + +/** + * Today's civil date (`YYYY-MM-DD`) in an IANA zone, so the server derives "today" from a + * client-supplied *zone* rather than trusting a client-supplied date. + */ +export function civilDateInZone(date: Date, timeZone: string): string { + // Intl throws RangeError on an unknown zone and this takes a client-supplied string; + // callers validate first, this is the backstop. + const zone = isValidIanaTimeZone(timeZone) ? timeZone : 'UTC'; + const parts = new Intl.DateTimeFormat('en-CA', { + timeZone: zone, + year: 'numeric', + month: '2-digit', + day: '2-digit', + }).formatToParts(date); + + const year = parts.find((p) => p.type === 'year')?.value ?? '1970'; + const month = parts.find((p) => p.type === 'month')?.value ?? '01'; + const day = parts.find((p) => p.type === 'day')?.value ?? '01'; + return `${year}-${month}-${day}`; +} diff --git a/backend/src/configs/configurations.ts b/backend/src/configs/configurations.ts index 473fa83..cc4c5eb 100644 --- a/backend/src/configs/configurations.ts +++ b/backend/src/configs/configurations.ts @@ -1,13 +1,14 @@ // backend/src/config/configuration.ts /** Matches values accepted by jsonwebtoken `expiresIn` (via ms), e.g. 7d, 15m, or plain seconds. */ -const JWT_TIMESPAN_PATTERN = - /^\d+(\.\d+)?(ms|s|m|h|d|w|y)?$/i; +const JWT_TIMESPAN_PATTERN = /^\d+(\.\d+)?(ms|s|m|h|d|w|y)?$/i; function assertJwtSecret(value: string, envKey: string): string { const trimmed = value.trim(); if (!trimmed) { - throw new Error(`❌ Environment variable ${envKey} is required but not set`); + throw new Error( + `❌ Environment variable ${envKey} is required but not set`, + ); } if (trimmed.length < 16) { throw new Error(`❌ ${envKey} must be at least 16 characters`); @@ -21,7 +22,9 @@ function assertJwtSecret(value: string, envKey: string): string { function assertJwtTimespan(value: string, envKey: string): string { const trimmed = value.trim(); if (!trimmed) { - throw new Error(`❌ Environment variable ${envKey} is required but not set`); + throw new Error( + `❌ Environment variable ${envKey} is required but not set`, + ); } if (!/^\d+$/.test(trimmed) && !JWT_TIMESPAN_PATTERN.test(trimmed)) { throw new Error( @@ -31,7 +34,10 @@ function assertJwtTimespan(value: string, envKey: string): string { return trimmed; } -function parseEnvBoolean(value: string | undefined, defaultValue: boolean): boolean { +function parseEnvBoolean( + value: string | undefined, + defaultValue: boolean, +): boolean { if (value === undefined || value.trim() === '') { return defaultValue; } @@ -67,8 +73,37 @@ export interface Config { apiKey: string | null; templateId: number; }; + voice: VoiceConfig; } +/** Only OpenRouter today. The indirection exists so a locale can diverge without code. */ +export type VoiceProviderId = 'openrouter'; + +export type VoiceProfile = { + asr: { provider: VoiceProviderId; model: string }; + llm: { provider: VoiceProviderId; model: string }; +}; + +export interface VoiceConfig { + openRouter: { + apiKey: string | null; + baseUrl: string; + }; + /** + * Locale → provider profile. A locale absent here has no microphone button at all — + * clean absence rather than a dead control. + */ + profiles: Record; + /** Recording cap in ms; null means uncapped. */ + maxRecordingMs: number | null; + throttle: { ttl: number; limit: number }; +} + +export const VOICE_LOCALES = ['fa', 'en', 'nl'] as const; + +const DEFAULT_ASR_MODEL = 'openai/whisper-1'; +const DEFAULT_LLM_MODEL = 'google/gemini-3.7-flash'; + export default (): Config => { // Helper function to get required env var with type safety const getEnvVar = (key: string): string => { @@ -129,5 +164,95 @@ export default (): Config => { apiKey: process.env.SMS_IR_API_KEY?.trim() || null, templateId: getEnvVarAsNumber('SMS_IR_TEMPLATE_ID', 123456), }, + voice: buildVoiceConfig(getEnvVarWithDefault, getEnvVarAsNumber), }; -}; \ No newline at end of file +}; + +const VOICE_PROVIDER_IDS: readonly VoiceProviderId[] = ['openrouter']; + +/** Unknown provider ids fail at boot; silently coercing a typo would ship the wrong vendor. */ +function parseProviderId( + value: string | undefined, + envKey: string, +): VoiceProviderId { + const trimmed = value?.trim(); + if (!trimmed) return 'openrouter'; + if ((VOICE_PROVIDER_IDS as readonly string[]).includes(trimmed)) { + return trimmed as VoiceProviderId; + } + throw new Error( + `❌ ${envKey}="${trimmed}" is not a known voice provider (${VOICE_PROVIDER_IDS.join(', ')})`, + ); +} + +/** + * Every enabled locale gets its own ASR and LLM provider+model, each independently + * overridable. They all point at the same OpenRouter models today; the per-locale + * indirection stays so a locale can diverge by configuration rather than by code. + */ +function buildVoiceConfig( + getEnvVarWithDefault: (key: string, defaultValue: string) => string, + getEnvVarAsNumber: (key: string, defaultValue: number) => number, +): VoiceConfig { + const enabled = getEnvVarWithDefault( + 'VOICE_ENABLED_LOCALES', + VOICE_LOCALES.join(','), + ) + .split(',') + .map((locale) => locale.trim().toLowerCase()) + .filter(Boolean); + for (const locale of enabled) { + // Fail loudly, like parseProviderId. Silently filtering a typo would disable the + // microphone everywhere with nothing to explain why. + if (!(VOICE_LOCALES as readonly string[]).includes(locale)) { + throw new Error( + `❌ VOICE_ENABLED_LOCALES contains unknown locale "${locale}" (known: ${VOICE_LOCALES.join(', ')})`, + ); + } + } + + const profiles: Record = {}; + for (const locale of enabled) { + const suffix = locale.toUpperCase(); + profiles[locale] = { + asr: { + provider: parseProviderId( + process.env[`VOICE_ASR_PROVIDER_${suffix}`], + `VOICE_ASR_PROVIDER_${suffix}`, + ), + model: getEnvVarWithDefault( + `VOICE_ASR_MODEL_${suffix}`, + getEnvVarWithDefault('VOICE_ASR_MODEL', DEFAULT_ASR_MODEL), + ), + }, + llm: { + provider: parseProviderId( + process.env[`VOICE_LLM_PROVIDER_${suffix}`], + `VOICE_LLM_PROVIDER_${suffix}`, + ), + model: getEnvVarWithDefault( + `VOICE_LLM_MODEL_${suffix}`, + getEnvVarWithDefault('VOICE_LLM_MODEL', DEFAULT_LLM_MODEL), + ), + }, + }; + } + + const maxRecordingMs = getEnvVarAsNumber('VOICE_MAX_RECORDING_MS', 120_000); + + return { + openRouter: { + apiKey: process.env.OPENROUTER_API_KEY?.trim() || null, + baseUrl: getEnvVarWithDefault( + 'OPENROUTER_BASE_URL', + 'https://openrouter.ai/api/v1', + ), + }, + profiles, + maxRecordingMs: maxRecordingMs > 0 ? maxRecordingMs : null, + throttle: { + ttl: getEnvVarAsNumber('VOICE_THROTTLE_TTL', 60), + limit: getEnvVarAsNumber('VOICE_THROTTLE_LIMIT', 6), + }, + }; +} diff --git a/backend/src/main.ts b/backend/src/main.ts index 4af42c2..b2ec46b 100644 --- a/backend/src/main.ts +++ b/backend/src/main.ts @@ -1,6 +1,8 @@ // backend/src/main.ts import { NestFactory } from '@nestjs/core'; +import { urlencoded } from 'express'; import { AppModule } from './app.module'; +import { createJsonBodyParser } from './common/body-parsers'; import { ValidationPipe } from '@nestjs/common'; import cookieParser from 'cookie-parser'; // 👈 Change this line! import { DocumentBuilder, SwaggerModule } from '@nestjs/swagger'; @@ -23,7 +25,13 @@ console.log = (...args) => { }; async function bootstrap() { - const app = await NestFactory.create(AppModule); + // bodyParser is disabled so the JSON parsers can be registered in an explicit order below; + // Nest's built-in one would otherwise reject a voice recording at 100 kb. + const app = await NestFactory.create(AppModule, { bodyParser: false }); + + // Voice needs a larger JSON limit than everything else; see body-parsers.ts. + app.use(createJsonBodyParser()); + app.use(urlencoded({ extended: true })); app.useGlobalFilters(new HttpExceptionFilter()); diff --git a/backend/src/modules/treatments/treatment.utils.ts b/backend/src/modules/treatments/treatment.utils.ts index 9a0e4b3..e22942e 100644 --- a/backend/src/modules/treatments/treatment.utils.ts +++ b/backend/src/modules/treatments/treatment.utils.ts @@ -1,9 +1,4 @@ -const FDI_TOOTH_IDS = new Set([ - '11', '12', '13', '14', '15', '16', '17', '18', - '21', '22', '23', '24', '25', '26', '27', '28', - '31', '32', '33', '34', '35', '36', '37', '38', - '41', '42', '43', '44', '45', '46', '47', '48', -]); +import { FDI_TOOTH_IDS } from '../../common/fdi'; export function normalizeTeeth(teeth: unknown): string[] { if (!Array.isArray(teeth)) { @@ -35,7 +30,10 @@ export function normalizeToothSelectionGroups( for (const row of value) { if (!row || typeof row !== 'object') continue; const rec = row as Record; - const groupId = typeof rec.groupId === 'string' && rec.groupId.trim() ? rec.groupId.trim() : ''; + const groupId = + typeof rec.groupId === 'string' && rec.groupId.trim() + ? rec.groupId.trim() + : ''; if (!groupId) continue; const kind = rec.kind === 'connected' ? 'connected' : 'single'; const teeth = normalizeTeeth(rec.teeth); @@ -64,7 +62,8 @@ export function generateTreatmentTitle( } const parts = cases.map((c) => { - const label = c.treatmentType.charAt(0).toUpperCase() + c.treatmentType.slice(1); + const label = + c.treatmentType.charAt(0).toUpperCase() + c.treatmentType.slice(1); if (c.teeth.length > 0) { return `${label} ${c.teeth.join(', ')}`; } diff --git a/backend/src/modules/voice/dto/voice.dto.ts b/backend/src/modules/voice/dto/voice.dto.ts new file mode 100644 index 0000000..7a8492b --- /dev/null +++ b/backend/src/modules/voice/dto/voice.dto.ts @@ -0,0 +1,58 @@ +import { + IsBase64, + IsIn, + IsInt, + IsString, + MaxLength, + Min, +} from 'class-validator'; +import { ErrorCode } from '../../../common/errors/error-codes'; + +/** Containers OpenRouter's transcription endpoint accepts, and MediaRecorder can produce. */ +export const VOICE_AUDIO_FORMATS = [ + 'webm', + 'mp4', + 'm4a', + 'aac', + 'ogg', + 'wav', + 'mp3', + 'flac', +] as const; + +export type VoiceAudioFormat = (typeof VOICE_AUDIO_FORMATS)[number]; + +/** Locales the app ships; a profile still has to be configured for one to be usable. */ +export const VOICE_LOCALES = ['en', 'fa', 'nl'] as const; + +export class ExtractVoiceDto { + /** Base64 audio, no data: prefix. Well above a 2-minute opus clip (~400 KB). */ + @IsString() + @IsBase64() + // Both name their own code: the shared map sends `maxLength` to + // VALIDATION_FIELD_REQUIRED and `isIn` to VALIDATION_LANGUAGE_INVALID, neither of which + // is true here. + @MaxLength(8_000_000, { message: ErrorCode.VOICE_CLIP_TOO_LONG }) + audio: string; + + @IsIn(VOICE_AUDIO_FORMATS, { message: ErrorCode.VOICE_UNSUPPORTED_FORMAT }) + format: VoiceAudioFormat; + + /** The clinician's IANA zone; "today" is derived from it, never sent by the client. */ + @IsString() + @MaxLength(64) + timeZone: string; + + /** Required, not optional — omitting it would bypass VOICE_MAX_RECORDING_MS entirely. */ + @IsInt() + @Min(0) + durationMs: number; + + /** + * The locale the UI offered the microphone in. Sent explicitly because `user.language` can + * diverge from the URL locale, and a mismatch transcribes Persian with an English hint and + * anchors "next Thursday" to the wrong week start. + */ + @IsIn(VOICE_LOCALES) + locale: string; +} diff --git a/backend/src/modules/voice/due-date.resolver.spec.ts b/backend/src/modules/voice/due-date.resolver.spec.ts new file mode 100644 index 0000000..74b7167 --- /dev/null +++ b/backend/src/modules/voice/due-date.resolver.spec.ts @@ -0,0 +1,385 @@ +import { resolveDueDate, weekStartForLocale } from './due-date.resolver'; +import type { DueIntent } from './voice.types'; + +// 2025-10-11 is a Saturday — the first day of the Iranian week. +const FA_WEEK = 6; // Saturday +const EU_WEEK = 1; // Monday +const SATURDAY = '2025-10-11'; +const THURSDAY = '2025-10-16'; + +describe('resolveDueDate', () => { + describe('weekday intents', () => { + it('resolves "this " to the coming occurrence', () => { + const result = resolveDueDate( + { kind: 'weekday', weekday: 'thursday', which: 'this' }, + SATURDAY, + FA_WEEK, + ); + expect(result.dueDate).toBe(THURSDAY); // Sat -> Thu is 5 days + }); + + it('resolves "next " to that weekday in the following week', () => { + const result = resolveDueDate( + { kind: 'weekday', weekday: 'thursday', which: 'next' }, + SATURDAY, + FA_WEEK, + ); + expect(result.dueDate).toBe('2025-10-23'); + }); + + it('anchors "next" to the week, not to "this" plus seven', () => { + // Said on Thursday 2025-10-16, the Iranian week runs Sat 10-18 .. Fri 10-24, so its + // Thursday is 10-23. Adding a week to "this Thursday" (already 10-23) would + // overshoot to 10-30 — a lab case a week late. + expect( + resolveDueDate( + { kind: 'weekday', weekday: 'thursday', which: 'next' }, + THURSDAY, + FA_WEEK, + ).dueDate, + ).toBe('2025-10-23'); + }); + + it('lets "this" and "next" coincide when they name the same day', () => { + // On a Thursday, "the coming Saturday" and "Saturday next week" are both 10-18. + for (const which of ['this', 'next'] as const) { + expect( + resolveDueDate( + { kind: 'weekday', weekday: 'saturday', which }, + THURSDAY, + FA_WEEK, + ).dueDate, + ).toBe('2025-10-18'); + } + }); + + it('reads "by Thursday" said on a Thursday as the next one, not today', () => { + // A deadline of today is almost never what was meant. + expect( + resolveDueDate( + { kind: 'weekday', weekday: 'thursday', which: 'this' }, + THURSDAY, + FA_WEEK, + ).dueDate, + ).toBe('2025-10-23'); + }); + + it('never resolves a weekday into the past', () => { + for (const which of ['this', 'next'] as const) { + const result = resolveDueDate( + { kind: 'weekday', weekday: 'sunday', which }, + THURSDAY, + FA_WEEK, + ); + expect(result.dueDate).not.toBeNull(); + expect(result.dueDate! > THURSDAY).toBe(true); + } + }); + + it('anchors "next" to a Monday week for en and nl', () => { + // The same sentence means a different day depending on where the week starts. + // Said on Saturday 10-11: the Monday-start week is 10-13..10-19, Thursday = 10-16. + // The Saturday-start week is 10-18..10-24, Thursday = 10-23. + const intent = { + kind: 'weekday', + weekday: 'thursday', + which: 'next', + } as const; + expect(resolveDueDate(intent, SATURDAY, EU_WEEK).dueDate).toBe( + '2025-10-16', + ); + expect(resolveDueDate(intent, SATURDAY, FA_WEEK).dueDate).toBe( + '2025-10-23', + ); + }); + + it('maps each locale to its week start', () => { + expect(weekStartForLocale('fa')).toBe(FA_WEEK); + expect(weekStartForLocale('en')).toBe(EU_WEEK); + expect(weekStartForLocale('nl')).toBe(EU_WEEK); + expect(weekStartForLocale('unknown')).toBe(EU_WEEK); + }); + + it('treats a missing qualifier as "this" rather than failing', () => { + // A bare weekday carries no qualifier; failing would discard a real spoken deadline. + expect( + resolveDueDate( + { + kind: 'weekday', + weekday: 'thursday', + which: null, + } as unknown as DueIntent, + SATURDAY, + FA_WEEK, + ).dueDate, + ).toBe(THURSDAY); + }); + + it('rejects an unknown weekday', () => { + expect( + resolveDueDate( + { + kind: 'weekday', + weekday: 'caturday', + which: 'this', + } as unknown as DueIntent, + SATURDAY, + FA_WEEK, + ).dueDate, + ).toBeNull(); + }); + }); + + describe('offset intents', () => { + it('adds days, weeks and months', () => { + expect( + resolveDueDate( + { kind: 'offset', unit: 'day', amount: 1 }, + SATURDAY, + FA_WEEK, + ).dueDate, + ).toBe('2025-10-12'); + expect( + resolveDueDate( + { kind: 'offset', unit: 'week', amount: 1 }, + SATURDAY, + FA_WEEK, + ).dueDate, + ).toBe('2025-10-18'); + expect( + resolveDueDate( + { kind: 'offset', unit: 'month', amount: 1 }, + SATURDAY, + FA_WEEK, + ).dueDate, + ).toBe('2025-11-11'); + }); + + it('clamps to the end of a shorter month', () => { + expect( + resolveDueDate( + { kind: 'offset', unit: 'month', amount: 1 }, + '2025-01-31', + ).dueDate, + ).toBe('2025-02-28'); + expect( + resolveDueDate( + { kind: 'offset', unit: 'month', amount: 1 }, + '2024-01-31', + ).dueDate, + ).toBe('2024-02-29'); + }); + + it('crosses a year boundary', () => { + expect( + resolveDueDate( + { kind: 'offset', unit: 'month', amount: 3 }, + '2025-11-15', + ).dueDate, + ).toBe('2026-02-15'); + expect( + resolveDueDate( + { kind: 'offset', unit: 'day', amount: 30 }, + '2025-12-20', + ).dueDate, + ).toBe('2026-01-19'); + }); + + it('rejects negative, fractional and absurd amounts', () => { + for (const amount of [-1, 1.5, 10_000, Number.NaN]) { + expect( + resolveDueDate( + { kind: 'offset', unit: 'day', amount }, + SATURDAY, + FA_WEEK, + ).dueDate, + ).toBeNull(); + } + }); + }); + + describe('jalali intents', () => { + it('converts by arithmetic, not inference', () => { + expect( + resolveDueDate( + { kind: 'jalali', jy: 1404, jm: 7, jd: 25 }, + SATURDAY, + FA_WEEK, + ).dueDate, + ).toBe('2025-10-17'); + }); + + it('handles the leap-year Esfand 30', () => { + expect( + resolveDueDate( + { kind: 'jalali', jy: 1403, jm: 12, jd: 30 }, + '2025-03-01', + ).dueDate, + ).toBe('2025-03-20'); + }); + + it('rejects Esfand 30 in a non-leap year', () => { + const result = resolveDueDate( + { kind: 'jalali', jy: 1404, jm: 12, jd: 30 }, + SATURDAY, + ); + expect(result.dueDate).toBeNull(); + expect(result.unresolved?.reason).toBe('invalid_date'); + }); + }); + + describe('gregorian intents', () => { + it('accepts a real date and rejects an impossible one', () => { + expect( + resolveDueDate( + { kind: 'gregorian', y: 2025, m: 10, d: 17 }, + SATURDAY, + FA_WEEK, + ).dueDate, + ).toBe('2025-10-17'); + expect( + resolveDueDate( + { kind: 'gregorian', y: 2025, m: 2, d: 30 }, + SATURDAY, + FA_WEEK, + ).dueDate, + ).toBeNull(); + expect( + resolveDueDate( + { kind: 'gregorian', y: 2025, m: 13, d: 1 }, + SATURDAY, + FA_WEEK, + ).dueDate, + ).toBeNull(); + }); + }); + + describe('guard rails', () => { + it('treats a past date as unresolved', () => { + const result = resolveDueDate( + { kind: 'gregorian', y: 2020, m: 1, d: 1 }, + SATURDAY, + ); + expect(result.dueDate).toBeNull(); + expect(result.unresolved?.reason).toBe('invalid_date'); + }); + + it('treats a date decades away as unresolved', () => { + expect( + resolveDueDate( + { kind: 'gregorian', y: 2099, m: 1, d: 1 }, + SATURDAY, + FA_WEEK, + ).dueDate, + ).toBeNull(); + }); + + it('accepts today itself via a zero-day offset', () => { + expect( + resolveDueDate( + { kind: 'offset', unit: 'day', amount: 0 }, + SATURDAY, + FA_WEEK, + ).dueDate, + ).toBe(SATURDAY); + }); + + it('reports no due date at all when the model said nothing, without flagging it', () => { + expect(resolveDueDate(null, SATURDAY)).toEqual({ + dueDate: null, + unresolved: null, + }); + expect(resolveDueDate(undefined, SATURDAY)).toEqual({ + dueDate: null, + unresolved: null, + }); + }); + + it('treats an object with no kind as no deadline, not a lost one', () => { + // A blank "heard but lost" row in front of a clinician who never mentioned a + // deadline is worse than saying nothing. + expect(resolveDueDate({} as never, SATURDAY)).toEqual({ + dueDate: null, + unresolved: null, + }); + }); + + it('flags an unrecognised kind and names it', () => { + const result = resolveDueDate({ kind: 'lunar_month' } as never, SATURDAY); + expect(result.dueDate).toBeNull(); + expect(result.unresolved).toEqual({ + spoken: 'lunar_month', + reason: 'invalid_date', + }); + }); + + it('flags a non-object deadline instead of silently dropping it', () => { + // A bare string is a deadline we failed to parse, not an absent one — the clinician + // must see that something was heard and lost. + for (const bad of ['next thursday', 42, true]) { + const result = resolveDueDate(bad as never, SATURDAY); + expect(result.dueDate).toBeNull(); + expect(result.unresolved?.reason).toBe('invalid_date'); + } + expect( + resolveDueDate('next thursday' as never, SATURDAY).unresolved?.spoken, + ).toBe('next thursday'); + }); + + it('degrades rather than throwing on a malformed today or intent', () => { + expect( + resolveDueDate({ kind: 'offset', unit: 'day', amount: 1 }, 'not-a-date') + .dueDate, + ).toBeNull(); + expect( + resolveDueDate({ kind: 'nope' } as unknown as DueIntent, SATURDAY) + .dueDate, + ).toBeNull(); + }); + + it('echoes what was heard so the review sheet can show it', () => { + const result = resolveDueDate( + { kind: 'jalali', jy: 1404, jm: 12, jd: 30 }, + SATURDAY, + ); + expect(result.unresolved?.spoken).toBe('1404/12/30'); + }); + }); +}); + +describe('what an unresolvable deadline quotes back', () => { + // The wire shape allows nulls in every field and toVoiceIntent casts rather than + // checks, so these reach the resolver intact. The sheet renders `spoken` verbatim. + it('never puts "null" or "NaN" in front of the clinician', () => { + const bad = [ + { kind: 'weekday', weekday: null, which: null }, + { kind: 'offset', unit: null, amount: null }, + { kind: 'offset', unit: 'day', amount: Number.NaN }, + { kind: 'jalali', jy: null, jm: 7, jd: 25 }, + { kind: 'gregorian', y: 2026, m: null, d: null }, + ]; + for (const intent of bad) { + const result = resolveDueDate( + intent as unknown as DueIntent, + SATURDAY, + FA_WEEK, + ); + expect(result.dueDate).toBeNull(); + expect(result.unresolved?.spoken ?? '').not.toMatch(/null|NaN/); + } + }); + + it('still quotes a deadline it did understand the words of', () => { + const result = resolveDueDate( + { + kind: 'weekday', + weekday: 'thursday', + which: null, + } as unknown as DueIntent, + SATURDAY, + FA_WEEK, + ); + // A weekday with no "this/next" resolves, so nothing is quoted back at all. + expect(result.dueDate).not.toBeNull(); + }); +}); diff --git a/backend/src/modules/voice/due-date.resolver.ts b/backend/src/modules/voice/due-date.resolver.ts new file mode 100644 index 0000000..d65d2bf --- /dev/null +++ b/backend/src/modules/voice/due-date.resolver.ts @@ -0,0 +1,234 @@ +import { jalaliToIsoDate } from '../../common/jalali'; +import { civilDateJsWeekday } from '../../common/zoned-civil-time'; +import type { DueIntent, UnresolvedItem, Weekday } from './voice.types'; + +/** + * Spoken deadline → ISO civil date. + * + * The model never does calendar arithmetic; it says what it heard and this decides what + * that means. Jalali conversion in particular is arithmetic here, not inference — an LLM + * asked to turn "۲۵ مهر" into ISO answers confidently and is often wrong, and + * `@IsDateString()` would accept the wrong answer. + * + * Works entirely in civil dates. The caller derives `todayIso` from the actor's IANA zone + * (see `civilDateInZone`) rather than passing a zone in here, so nothing in this file has + * to reason about instants. + */ + +const WEEKDAY_TO_JS: Record = { + saturday: 6, + sunday: 0, + monday: 1, + tuesday: 2, + wednesday: 3, + thursday: 4, + friday: 5, +}; + +const MAX_DAYS_AHEAD = 365 * 5; + +export type DueResolution = { + dueDate: string | null; + unresolved: UnresolvedItem | null; +}; + +const ISO_DATE = /^\d{4}-\d{2}-\d{2}$/; + +function isRealCivilDate(iso: string): boolean { + if (!ISO_DATE.test(iso)) return false; + const [y, m, d] = iso.split('-').map(Number); + if (m < 1 || m > 12 || d < 1 || d > 31) return false; + const probe = new Date(Date.UTC(y, m - 1, d)); + return ( + probe.getUTCFullYear() === y && + probe.getUTCMonth() === m - 1 && + probe.getUTCDate() === d + ); +} + +function toIso(utcMs: number): string { + return new Date(utcMs).toISOString().slice(0, 10); +} + +function utcMsOf(iso: string): number { + const [y, m, d] = iso.split('-').map(Number); + return Date.UTC(y, m - 1, d); +} + +function addDays(iso: string, days: number): string { + return toIso(utcMsOf(iso) + days * 86_400_000); +} + +/** Calendar-month addition with end-of-month clamping (31 Jan + 1 month = 28/29 Feb). */ +function addMonths(iso: string, months: number): string { + const [y, m, d] = iso.split('-').map(Number); + const targetMonthIndex = m - 1 + months; + const targetYear = y + Math.floor(targetMonthIndex / 12); + const targetMonth = ((targetMonthIndex % 12) + 12) % 12; + const lastDay = new Date( + Date.UTC(targetYear, targetMonth + 1, 0), + ).getUTCDate(); + return toIso(Date.UTC(targetYear, targetMonth, Math.min(d, lastDay))); +} + +function unresolved(spoken: string): DueResolution { + return { dueDate: null, unresolved: { spoken, reason: 'invalid_date' } }; +} + +/** + * What to quote back when a deadline could not be resolved. Every field is nullable on the + * wire and `toVoiceIntent` casts rather than checks, and the sheet renders this verbatim — + * so a half-classified deadline must fall back to '', not to `"null null"`. + */ +function describe(intent: DueIntent): string { + const usable = (value: unknown): value is number => + typeof value === 'number' && Number.isFinite(value); + + switch (intent?.kind) { + case 'weekday': + // `which` is legitimately null (it means "this"), the weekday is not. + return [intent.which, intent.weekday] + .filter((part) => typeof part === 'string') + .join(' '); + case 'offset': + return usable(intent.amount) + ? `+${intent.amount} ${intent.unit ?? ''}`.trim() + : ''; + case 'jalali': + return [intent.jy, intent.jm, intent.jd].every(usable) + ? `${intent.jy}/${intent.jm}/${intent.jd}` + : ''; + case 'gregorian': + return [intent.y, intent.m, intent.d].every(usable) + ? `${intent.y}-${intent.m}-${intent.d}` + : ''; + default: { + // An unrecognised `kind`, already established as a string — echo what was heard. + const kind = (intent as { kind?: unknown })?.kind; + return typeof kind === 'string' ? kind : ''; + } + } +} + +/** + * "Next Thursday" is week-relative, so this changes the answer: the Iranian week starts + * Saturday, the Dutch and English week Monday. Hardcoding either puts the other locale's + * deadline a week out. + */ +const WEEK_START_BY_LOCALE: Record = { + fa: WEEKDAY_TO_JS.saturday, + en: WEEKDAY_TO_JS.monday, + nl: WEEKDAY_TO_JS.monday, +}; + +const DEFAULT_WEEK_START = WEEKDAY_TO_JS.monday; + +export function weekStartForLocale(locale: string): number { + return WEEK_START_BY_LOCALE[locale] ?? DEFAULT_WEEK_START; +} + +function startOfWeek(iso: string, weekStartJs: number): string { + const back = (civilDateJsWeekday(iso) - weekStartJs + 7) % 7; + return addDays(iso, -back); +} + +/** + * `'this'` is occurrence-anchored: the soonest occurrence strictly after today, so it can + * never resolve into the past. + * + * `'next'` is *week*-anchored, not "this plus seven" — adding a week to `'this'` overshoots + * by seven days whenever `'this'` has already rolled into next week. The two legitimately + * coincide: said on a Thursday, "the coming Saturday" and "Saturday next week" are one day. + */ +function resolveWeekday( + intent: Extract, + todayIso: string, + weekStartJs: number, +) { + const targetJs = WEEKDAY_TO_JS[intent.weekday]; + if (targetJs === undefined) return null; + + // A bare weekday ("پنجشنبه") carries no qualifier, and the model may leave `which` + // null. Treat that as 'this' rather than failing an utterance that named a real day. + const which = intent.which === 'next' ? 'next' : 'this'; + + if (which === 'this') { + const todayJs = civilDateJsWeekday(todayIso); + let delta = (targetJs - todayJs + 7) % 7; + if (delta === 0) delta = 7; + return addDays(todayIso, delta); + } + + if (which === 'next') { + const offsetInWeek = (targetJs - weekStartJs + 7) % 7; + return addDays(startOfWeek(todayIso, weekStartJs), 7 + offsetInWeek); + } + + return null; +} + +function resolveOffset( + intent: Extract, + todayIso: string, +) { + const { amount, unit } = intent; + if (!Number.isInteger(amount) || amount < 0) return null; + if (unit === 'day') + return amount <= MAX_DAYS_AHEAD ? addDays(todayIso, amount) : null; + if (unit === 'week') + return amount <= 260 ? addDays(todayIso, amount * 7) : null; + if (unit === 'month') + return amount <= 60 ? addMonths(todayIso, amount) : null; + return null; +} + +export function resolveDueDate( + intent: DueIntent | null | undefined, + todayIso: string, + weekStartJs: number = DEFAULT_WEEK_START, +): DueResolution { + // Absent is not an error — most utterances carry no deadline. + if (intent === null || intent === undefined) { + return { dueDate: null, unresolved: null }; + } + if (typeof intent !== 'object') { + return unresolved(String(intent).slice(0, 120)); + } + // No `kind` at all says nothing about a deadline, so it is not "heard but lost". An + // *unrecognised* kind did try to say something, and is flagged below. + if (typeof (intent as { kind?: unknown }).kind !== 'string') { + return { dueDate: null, unresolved: null }; + } + if (!isRealCivilDate(todayIso)) { + return unresolved(describe(intent)); + } + + let resolved: string | null = null; + switch (intent.kind) { + case 'weekday': + resolved = resolveWeekday(intent, todayIso, weekStartJs); + break; + case 'offset': + resolved = resolveOffset(intent, todayIso); + break; + case 'jalali': + resolved = jalaliToIsoDate(intent.jy, intent.jm, intent.jd); + break; + case 'gregorian': { + const candidate = `${String(intent.y).padStart(4, '0')}-${String(intent.m).padStart(2, '0')}-${String(intent.d).padStart(2, '0')}`; + resolved = isRealCivilDate(candidate) ? candidate : null; + break; + } + default: + resolved = null; + } + + if (!resolved) return unresolved(describe(intent)); + + // A date the model invented can land anywhere; past or decades away is not a deadline. + const daysAhead = (utcMsOf(resolved) - utcMsOf(todayIso)) / 86_400_000; + if (daysAhead < 0 || daysAhead > MAX_DAYS_AHEAD) + return unresolved(describe(intent)); + + return { dueDate: resolved, unresolved: null }; +} diff --git a/backend/src/modules/voice/extraction.prompt.ts b/backend/src/modules/voice/extraction.prompt.ts new file mode 100644 index 0000000..0df6c22 --- /dev/null +++ b/backend/src/modules/voice/extraction.prompt.ts @@ -0,0 +1,93 @@ +import type { ExtractionCatalog } from './voice.providers'; + +/** Locale-specific guidance. Only the tooth vocabulary and numbering habits differ. */ +const LOCALE_NOTES: Record = { + fa: [ + 'The clinician is speaking Persian. A tooth number can be said as a whole number', + '("بیست و شش" = 26), digit by digit ("دو شش" = 26), or with a lead-in', + '("دندون شماره ۲۶"). Digits may arrive in Persian or Latin script — either way, copy', + 'the number into "fdi" as two Latin digits. The descriptive form is quadrant-relative:', + '"شش بالا راست" = upper right six -> arch "upper", side "patient_right", position 6.', + ].join(' '), + nl: [ + 'The clinician is speaking Dutch, where FDI is standard. "zesentwintig" and "26" are', + 'tooth 26. The descriptive form is "rechtsboven zes" = upper right six.', + ].join(' '), + en: [ + 'The clinician is speaking English and uses FDI. "twenty-six", "two six" and "26" are', + 'all tooth 26. The descriptive form is "upper right six".', + ].join(' '), +}; + +function codeList(entries: { code: string; label: string }[]): string { + if (entries.length === 0) return '(none available)'; + return entries.map((e) => `- ${e.code} = ${e.label}`).join('\n'); +} + +export function buildExtractionPrompt( + transcript: string, + catalog: ExtractionCatalog, + localeHint: string, +) { + const localeNote = LOCALE_NOTES[localeHint] ?? LOCALE_NOTES.en; + + const system = [ + 'You extract structured dental treatment data from a transcript of a clinician speaking.', + 'You are a parser, not an assistant: report only what was said.', + '', + 'HARD RULES', + '1. Never invent a code. treatmentType, prosthesisDefaultType and prosthesisOverrides[].type', + ' must be codes from the lists below. labId must be an id from the lab list. If what you', + ' heard is not in a list, use null.', + '2. "side" is always the PATIENT\'s side. The patient\'s upper right is quadrant 1. Never', + " flip to the viewer's point of view.", + '3. Never do calendar arithmetic. Report the deadline as it was said, using due.kind.', + ' If no deadline was mentioned, use due.kind = "none".', + '4. Copy the exact spoken words for each tooth into "spoken", so the clinician can see', + ' what was heard.', + '5. If you are unsure about a value, use null. A missing field is recoverable; a wrong', + ' one is not.', + '', + 'TOOTH NUMBERS', + 'A number the clinician says for a tooth IS that tooth\'s FDI code. Put it in "fdi" as', + 'two digits. FDI is built from the two digits:', + " first digit = quadrant, from the PATIENT's own point of view —", + ' 1 upper right, 2 upper left, 3 lower left, 4 lower right.', + ' (5-8 are those same four quadrants in primary/deciduous teeth.)', + ' second digit = position from the midline — 1 central incisor ... 8 third molar.', + 'So 26 is the upper left first molar, and 47 is the lower right second molar.', + '', + '- Use "arch" + "side" + "position" only when the tooth is DESCRIBED rather than', + ' numbered ("upper right six" -> arch "upper", side "patient_right", position 6).', + '- A single digit is a position, never an FDI code. If a single digit is said with no', + ' quadrant words at all, set "position" and leave "arch" and "side" null. Never pick a', + ' quadrant that was not said.', + '- If a number is given AND the quadrant is spelled out as well, still use "fdi".', + '- Not every number is a tooth. Dates, counts and quantities ("two teeth", "the 26th")', + ' are not teeth, and must never appear in the teeth list.', + '', + localeNote, + '', + 'TREATMENT TYPE CODES', + codeList(catalog.treatmentTypes), + '', + 'PROSTHESIS TYPE CODES', + codeList(catalog.prosthesisTypes), + '', + 'LABS THIS CLINIC CAN SEND TO', + catalog.labs.length > 0 + ? catalog.labs.map((l) => `- ${l.id} = ${l.name}`).join('\n') + : '(none linked — labId must be null)', + '', + 'OTHER FIELDS', + '- connectedSpans: only for bridges or splinted units. Endpoints inclusive.', + '- comment: clinical notes, in the language spoken. Omit the parts already captured as', + ' treatment type, teeth or deadline.', + '- labMatchExact: true only when the spoken name matched a lab name exactly.', + ].join('\n'); + + return [ + { role: 'system' as const, content: system }, + { role: 'user' as const, content: transcript }, + ]; +} diff --git a/backend/src/modules/voice/extraction.resolver.spec.ts b/backend/src/modules/voice/extraction.resolver.spec.ts new file mode 100644 index 0000000..b86f577 --- /dev/null +++ b/backend/src/modules/voice/extraction.resolver.spec.ts @@ -0,0 +1,356 @@ +import { + resolveConnectedSpans, + resolveProsthesis, + resolveVoiceIntent, + type ResolveContext, +} from './extraction.resolver'; +import type { ToothIntent, VoiceIntent } from './voice.types'; + +const tooth = (fdi: string, spoken = fdi): ToothIntent => ({ + kind: 'explicit', + fdi, + spoken, +}); + +const CTX: ResolveContext = { + todayIso: '2025-10-11', + weekStartJs: 6, // Saturday — the fa week + + treatmentTypeCodes: new Set(['restoration', 'prosthesis', 'extraction']), + prosthesisTypeCodes: new Set(['monolithic_zirconia', 'pfm_crown']), + linkedLabIds: new Set(['lab-sina', 'lab-mehr']), +}; + +describe('resolveConnectedSpans', () => { + it('selects the teeth between the endpoints, which were never named', () => { + // "a bridge from 14 to 16" must select 15 too. + const result = resolveConnectedSpans( + [{ from: tooth('14'), to: tooth('16') }], + [], + ); + expect(result.teeth).toEqual(['14', '15', '16']); + expect(result.groups).toEqual([ + { groupId: 'voice-c1', kind: 'connected', teeth: ['16', '15', '14'] }, + ]); + }); + + it('orders group teeth along the arch, not lexically', () => { + const result = resolveConnectedSpans( + [{ from: tooth('16'), to: tooth('14') }], + [], + ); + expect(result.groups[0].teeth).toEqual(['16', '15', '14']); + }); + + it('spans the midline', () => { + const result = resolveConnectedSpans( + [{ from: tooth('12'), to: tooth('22') }], + [], + ); + expect(result.groups[0].teeth).toEqual(['12', '11', '21', '22']); + }); + + it('merges overlapping spans into one bridge', () => { + const result = resolveConnectedSpans( + [ + { from: tooth('14'), to: tooth('16') }, + { from: tooth('15'), to: tooth('17') }, + ], + [], + ); + const connected = result.groups.filter((g) => g.kind === 'connected'); + expect(connected).toHaveLength(1); + expect(connected[0].teeth).toEqual(['17', '16', '15', '14']); + }); + + it('gives loose teeth their own single groups', () => { + const result = resolveConnectedSpans( + [{ from: tooth('14'), to: tooth('15') }], + ['26'], + ); + expect(result.groups).toEqual([ + { groupId: 'voice-c1', kind: 'connected', teeth: ['15', '14'] }, + { groupId: 'voice-s-26', kind: 'single', teeth: ['26'] }, + ]); + }); + + it('never produces a one-tooth connected group', () => { + const result = resolveConnectedSpans( + [{ from: tooth('14'), to: tooth('14') }], + [], + ); + expect(result.groups).toEqual([ + { groupId: 'voice-s-14', kind: 'single', teeth: ['14'] }, + ]); + expect(result.unresolved).toEqual([]); + }); + + it('reports a cross-arch span rather than guessing', () => { + const result = resolveConnectedSpans( + [{ from: tooth('14', 'چهارده'), to: tooth('44', 'چهل و چهار') }], + [], + ); + expect(result.groups).toEqual([]); + expect(result.unresolved).toEqual([ + { spoken: 'چهارده → چهل و چهار', reason: 'span_not_same_arch' }, + ]); + }); + + it('reports a span with an unresolvable endpoint', () => { + const result = resolveConnectedSpans( + [{ from: tooth('14'), to: tooth('99') }], + [], + ); + expect(result.unresolved[0].reason).toBe('malformed'); + }); + + it('survives a non-array', () => { + expect(resolveConnectedSpans(undefined as never, ['14']).teeth).toEqual([ + '14', + ]); + }); +}); + +describe('resolveProsthesis', () => { + const allowed = CTX.prosthesisTypeCodes; + + it('expands the default across every tooth', () => { + const result = resolveProsthesis( + { defaultType: 'monolithic_zirconia', overrides: [] }, + ['14', '15'], + allowed, + ); + expect(result.prosthesis?.byTooth).toEqual({ + '14': 'monolithic_zirconia', + '15': 'monolithic_zirconia', + }); + expect(result.prosthesis?.complete).toBe(true); + }); + + it('applies per-tooth overrides on top of the default', () => { + const result = resolveProsthesis( + { + defaultType: 'monolithic_zirconia', + overrides: [{ tooth: tooth('26'), type: 'pfm_crown' }], + }, + ['14', '26'], + allowed, + ); + expect(result.prosthesis?.byTooth).toEqual({ + '14': 'monolithic_zirconia', + '26': 'pfm_crown', + }); + expect(result.prosthesis?.complete).toBe(true); + }); + + it('marks the map incomplete when a tooth ends up untyped', () => { + // Unshippable: assertCompleteToothProsthesisMap would reject this at dispatch. + const result = resolveProsthesis( + { + defaultType: null, + overrides: [{ tooth: tooth('14'), type: 'pfm_crown' }], + }, + ['14', '15'], + allowed, + ); + expect(result.prosthesis?.complete).toBe(false); + expect(result.prosthesis?.missingTeeth).toEqual(['15']); + }); + + it('rejects a catalog code the clinic does not have', () => { + const result = resolveProsthesis( + { defaultType: 'gold_foil', overrides: [] }, + ['14'], + allowed, + ); + // Nothing usable was said, so there is no prosthesis to show — not an empty one. + expect(result.prosthesis).toBeNull(); + expect(result.unresolved).toEqual([ + { spoken: 'gold_foil', reason: 'unknown_catalog_code' }, + ]); + }); + + it('ignores an override for a tooth that is not selected', () => { + const result = resolveProsthesis( + { + defaultType: 'monolithic_zirconia', + overrides: [{ tooth: tooth('37', 'سی و هفت'), type: 'pfm_crown' }], + }, + ['14'], + allowed, + ); + expect(result.prosthesis?.byTooth).toEqual({ '14': 'monolithic_zirconia' }); + expect(result.unresolved).toEqual([ + { spoken: 'سی و هفت', reason: 'tooth_not_selected' }, + ]); + }); + + it('reports no prosthesis at all when the object carries nothing usable', () => { + // An empty-but-present map would paint a plain restoration with a fabricated + // "incomplete, cannot ship" warning. + for (const empty of [{ defaultType: null, overrides: [] }, {} as never]) { + expect( + resolveProsthesis(empty, ['14', '15'], allowed).prosthesis, + ).toBeNull(); + } + }); + + it('reports no prosthesis when there are no teeth to type', () => { + const result = resolveProsthesis( + { defaultType: 'monolithic_zirconia', overrides: [] }, + [], + allowed, + ); + expect(result.prosthesis).toBeNull(); + }); + + it('distinguishes a tooth it could not understand from one that is not selected', () => { + // Different corrective actions: add the tooth, versus repeat yourself. + const result = resolveProsthesis( + { + defaultType: 'monolithic_zirconia', + overrides: [ + { + tooth: { kind: 'explicit', fdi: '99', spoken: 'نود و نه' }, + type: 'pfm_crown', + }, + ], + }, + ['14'], + allowed, + ); + expect(result.unresolved).toEqual([ + { spoken: 'نود و نه', reason: 'malformed' }, + ]); + }); + + it('returns null when no prosthesis was spoken', () => { + expect(resolveProsthesis(null, ['14'], allowed).prosthesis).toBeNull(); + }); +}); + +describe('resolveVoiceIntent', () => { + const base: VoiceIntent = { + treatmentType: 'restoration', + teeth: [tooth('14'), tooth('15')], + connectedSpans: [], + comment: ' حساسیت به سرما ', + prosthesis: null, + labId: null, + labMatchExact: false, + due: null, + }; + + it('composes a plain restoration', () => { + const result = resolveVoiceIntent(base, CTX); + expect(result.treatmentType).toBe('restoration'); + expect(result.teeth).toEqual(['14', '15']); + expect(result.comment).toBe('حساسیت به سرما'); + expect(result.prosthesis).toBeNull(); + expect(result.unresolved).toEqual([]); + }); + + it('rejects a treatment type outside the catalog', () => { + const result = resolveVoiceIntent( + { ...base, treatmentType: 'teeth_whitening' }, + CTX, + ); + expect(result.treatmentType).toBeNull(); + expect(result.unresolved).toContainEqual({ + spoken: 'teeth_whitening', + reason: 'unknown_catalog_code', + }); + }); + + it('drops a lab id the clinic is not linked to', () => { + // Shipping to a lab the clinic never named is worse than shipping nowhere. + const result = resolveVoiceIntent( + { ...base, labId: 'lab-elsewhere', labMatchExact: true }, + CTX, + ); + expect(result.labId).toBeNull(); + expect(result.labMatchExact).toBe(false); + }); + + it('keeps a linked lab and its exactness flag', () => { + const result = resolveVoiceIntent( + { ...base, labId: 'lab-sina', labMatchExact: true }, + CTX, + ); + expect(result.labId).toBe('lab-sina'); + expect(result.labMatchExact).toBe(true); + }); + + it('reports a hallucinated lab rather than dropping it silently', () => { + // A near-miss lab id must not look identical to "no lab was spoken". + const result = resolveVoiceIntent({ ...base, labId: 'lab-elsewhere' }, CTX); + // The id is not what the clinician said — quoting it back shows them a raw UUID. + expect(result.unresolved).toContainEqual({ + spoken: '', + reason: 'unknown_catalog_code', + }); + }); + + it('never reports an inexact match as exact when the lab was dropped', () => { + const result = resolveVoiceIntent( + { ...base, labId: null, labMatchExact: true }, + CTX, + ); + expect(result.labMatchExact).toBe(false); + }); + + it('applies prosthesis over the span-expanded tooth set', () => { + const result = resolveVoiceIntent( + { + ...base, + treatmentType: 'prosthesis', + teeth: [tooth('14')], + connectedSpans: [{ from: tooth('14'), to: tooth('16') }], + prosthesis: { defaultType: 'monolithic_zirconia', overrides: [] }, + }, + CTX, + ); + // 15 was never spoken but is part of the bridge, so it must carry a type too. + expect(result.teeth).toEqual(['14', '15', '16']); + expect(result.prosthesis?.complete).toBe(true); + expect(Object.keys(result.prosthesis!.byTooth).sort()).toEqual([ + '14', + '15', + '16', + ]); + }); + + it('resolves a due date through the same context', () => { + const result = resolveVoiceIntent( + { ...base, due: { kind: 'weekday', weekday: 'thursday', which: 'this' } }, + CTX, + ); + expect(result.dueDate).toBe('2025-10-16'); + }); + + it('collects unresolved items from every stage', () => { + const result = resolveVoiceIntent( + { + ...base, + treatmentType: 'nope', + teeth: [tooth('51', 'شیری')], + connectedSpans: [{ from: tooth('14'), to: tooth('44') }], + due: { kind: 'jalali', jy: 1404, jm: 12, jd: 30 }, + }, + CTX, + ); + const reasons = result.unresolved.map((u) => u.reason).sort(); + expect(reasons).toEqual([ + 'invalid_date', + 'not_permanent_tooth', + 'span_not_same_arch', + 'unknown_catalog_code', + ]); + }); + + it('treats an empty comment as absent', () => { + expect( + resolveVoiceIntent({ ...base, comment: ' ' }, CTX).comment, + ).toBeNull(); + }); +}); diff --git a/backend/src/modules/voice/extraction.resolver.ts b/backend/src/modules/voice/extraction.resolver.ts new file mode 100644 index 0000000..e7f6714 --- /dev/null +++ b/backend/src/modules/voice/extraction.resolver.ts @@ -0,0 +1,300 @@ +import { + sameArch, + sortInArchOrder, + teethBetweenInclusive, +} from '../../common/fdi'; +import { resolveDueDate } from './due-date.resolver'; +import { + resolveToothIntent, + resolveToothIntents, +} from './tooth-intent.resolver'; +import type { + ConnectedSpanIntent, + ProsthesisIntent, + ToothIntent, + UnresolvedItem, + VoiceIntent, +} from './voice.types'; + +export type ResolvedToothGroup = { + groupId: string; + kind: 'connected' | 'single'; + teeth: string[]; +}; + +export type ResolvedProsthesis = { + /** FDI code → prosthesis type code. */ + byTooth: Record; + /** + * True when every selected tooth carries a code. A prosthesis detail cannot be shipped + * otherwise (`assertCompleteToothProsthesisMap`), so the review sheet surfaces the gap + * here rather than letting it fail at dispatch. + */ + complete: boolean; + missingTeeth: string[]; +}; + +export type ResolvedExtraction = { + treatmentType: string | null; + teeth: string[]; + toothSelectionGroups: ResolvedToothGroup[]; + comment: string | null; + prosthesis: ResolvedProsthesis | null; + labId: string | null; + labMatchExact: boolean; + dueDate: string | null; + unresolved: UnresolvedItem[]; +}; + +export type ResolveContext = { + todayIso: string; + /** JS weekday index the clinician's week starts on — see weekStartForLocale. */ + weekStartJs: number; + treatmentTypeCodes: ReadonlySet; + prosthesisTypeCodes: ReadonlySet; + linkedLabIds: ReadonlySet; +}; + +function spokenOf(intent: ToothIntent): string { + const spoken = (intent as { spoken?: unknown })?.spoken; + return typeof spoken === 'string' && spoken.trim() ? spoken.trim() : ''; +} + +/** A code the model returned is only usable if it exists in the catalog we supplied it. */ +function resolveCatalogCode( + value: unknown, + allowed: ReadonlySet, +): string | null { + if (typeof value !== 'string') return null; + const trimmed = value.trim(); + return trimmed && allowed.has(trimmed) ? trimmed : null; +} + +/** Merge any span sets that share a tooth, so overlapping bridges become one group. */ +function mergeOverlapping(sets: string[][]): string[][] { + const merged: string[][] = []; + for (const candidate of sets) { + let current = [...candidate]; + let index = 0; + while (index < merged.length) { + if (merged[index].some((tooth) => current.includes(tooth))) { + current = [...new Set([...merged[index], ...current])]; + merged.splice(index, 1); + index = 0; + continue; + } + index += 1; + } + merged.push(current); + } + return merged; +} + +/** + * Span teeth join the selection: "a bridge from 14 to 16" selects 15 though it was never + * named. A cross-arch span is reported rather than guessed at, and a span collapsing to one + * tooth degrades to a single — there is no one-tooth bridge. + */ +export function resolveConnectedSpans( + spans: readonly ConnectedSpanIntent[], + selectedTeeth: readonly string[], +): { + groups: ResolvedToothGroup[]; + teeth: string[]; + unresolved: UnresolvedItem[]; +} { + const unresolved: UnresolvedItem[] = []; + const connectedSets: string[][] = []; + // A span that collapses to one tooth still selected that tooth — it must not vanish. + const loneSpanTeeth: string[] = []; + const list: readonly ConnectedSpanIntent[] = Array.isArray(spans) + ? (spans as readonly ConnectedSpanIntent[]) + : []; + + for (const span of list) { + const from = resolveToothIntent(span?.from); + const to = resolveToothIntent(span?.to); + const spoken = [spokenOf(span?.from), spokenOf(span?.to)] + .filter(Boolean) + .join(' → '); + + if (!from || !to) { + unresolved.push({ spoken, reason: 'malformed' }); + continue; + } + if (!sameArch(from, to)) { + unresolved.push({ spoken, reason: 'span_not_same_arch' }); + continue; + } + const between = teethBetweenInclusive(from, to); + if (!between || between.length === 0) { + unresolved.push({ spoken, reason: 'malformed' }); + continue; + } + if (between.length === 1) { + loneSpanTeeth.push(between[0]); + continue; // degrades to a single, below + } + connectedSets.push(between); + } + + const groups: ResolvedToothGroup[] = []; + const claimed = new Set(); + mergeOverlapping(connectedSets).forEach((set, i) => { + const teeth = sortInArchOrder(set); + teeth.forEach((tooth) => claimed.add(tooth)); + groups.push({ groupId: `voice-c${i + 1}`, kind: 'connected', teeth }); + }); + + const spanTeeth = [...groups.flatMap((g) => g.teeth), ...loneSpanTeeth]; + const singles = [...new Set([...selectedTeeth, ...spanTeeth])] + .filter((tooth) => !claimed.has(tooth)) + .sort(); + for (const tooth of singles) { + groups.push({ + groupId: `voice-s-${tooth}`, + kind: 'single', + teeth: [tooth], + }); + } + + return { + groups, + teeth: [...new Set([...selectedTeeth, ...spanTeeth])].sort(), + unresolved, + }; +} + +/** + * A default across the selection, then per-tooth overrides — "همه زیرکونیا، ۲۶ پی‌اف‌ام" is + * how clinicians actually speak. + */ +export function resolveProsthesis( + intent: ProsthesisIntent | null | undefined, + teeth: readonly string[], + allowed: ReadonlySet, +): { prosthesis: ResolvedProsthesis | null; unresolved: UnresolvedItem[] } { + if (!intent || typeof intent !== 'object') + return { prosthesis: null, unresolved: [] }; + + const unresolved: UnresolvedItem[] = []; + const defaultType = resolveCatalogCode(intent.defaultType, allowed); + if (intent.defaultType != null && !defaultType) { + unresolved.push({ + spoken: String(intent.defaultType), + reason: 'unknown_catalog_code', + }); + } + + const byTooth: Record = {}; + const selected = new Set(teeth); + if (defaultType) { + for (const tooth of teeth) byTooth[tooth] = defaultType; + } + + const overrides: ProsthesisIntent['overrides'] = Array.isArray( + intent.overrides, + ) + ? intent.overrides + : []; + for (const override of overrides) { + const tooth = resolveToothIntent(override?.tooth); + const type = resolveCatalogCode(override?.type, allowed); + const spoken = spokenOf(override?.tooth) || String(override?.type ?? ''); + if (!tooth) { + unresolved.push({ spoken, reason: 'malformed' }); + continue; + } + // A tooth we understood perfectly well but which is not part of this detail. Saying + // so is actionable ("add tooth 37, or drop it"); calling it malformed is not. + if (!selected.has(tooth)) { + unresolved.push({ spoken, reason: 'tooth_not_selected' }); + continue; + } + if (!type) { + unresolved.push({ spoken, reason: 'unknown_catalog_code' }); + continue; + } + byTooth[tooth] = type; + } + + // Nothing usable was said about prosthesis. Returning an empty-but-present map would + // paint a plain restoration with a fabricated "incomplete, cannot ship" warning. + if (Object.keys(byTooth).length === 0) { + return { prosthesis: null, unresolved }; + } + + const missingTeeth = teeth.filter((tooth) => !byTooth[tooth]); + return { + prosthesis: { + byTooth, + complete: missingTeeth.length === 0, + missingTeeth, + }, + unresolved, + }; +} + +/** Compose every resolver into the payload the review sheet renders. */ +export function resolveVoiceIntent( + intent: VoiceIntent, + ctx: ResolveContext, +): ResolvedExtraction { + const unresolved: UnresolvedItem[] = []; + + const toothResult = resolveToothIntents(intent?.teeth ?? []); + unresolved.push(...toothResult.unresolved); + + const spanResult = resolveConnectedSpans( + intent?.connectedSpans ?? [], + toothResult.teeth, + ); + unresolved.push(...spanResult.unresolved); + + const treatmentType = resolveCatalogCode( + intent?.treatmentType, + ctx.treatmentTypeCodes, + ); + if (intent?.treatmentType != null && !treatmentType) { + unresolved.push({ + spoken: String(intent.treatmentType), + reason: 'unknown_catalog_code', + }); + } + + const prosthesisResult = resolveProsthesis( + intent?.prosthesis, + spanResult.teeth, + ctx.prosthesisTypeCodes, + ); + unresolved.push(...prosthesisResult.unresolved); + + const due = resolveDueDate(intent?.due, ctx.todayIso, ctx.weekStartJs); + if (due.unresolved) unresolved.push(due.unresolved); + + const comment = + typeof intent?.comment === 'string' && intent.comment.trim() + ? intent.comment.trim() + : null; + + // An invented lab id would ship a case to a lab the clinic never named. A rejected one is + // reported, so it cannot look identical to "no lab was spoken". + const labId = resolveCatalogCode(intent?.labId, ctx.linkedLabIds); + if (intent?.labId != null && !labId) { + // `spoken` is what the clinician said — quoting an invented id back would put a raw + // UUID in front of the user. + unresolved.push({ spoken: '', reason: 'unknown_catalog_code' }); + } + + return { + treatmentType, + teeth: spanResult.teeth, + toothSelectionGroups: spanResult.groups, + comment, + prosthesis: prosthesisResult.prosthesis, + labId, + labMatchExact: labId ? intent?.labMatchExact === true : false, + dueDate: due.dueDate, + unresolved, + }; +} diff --git a/backend/src/modules/voice/extraction.wire.spec.ts b/backend/src/modules/voice/extraction.wire.spec.ts new file mode 100644 index 0000000..f40bcc7 --- /dev/null +++ b/backend/src/modules/voice/extraction.wire.spec.ts @@ -0,0 +1,236 @@ +import { + toVoiceIntent, + VOICE_INTENT_JSON_SCHEMA, + type WireVoiceIntent, +} from './extraction.wire'; + +const emptyDue = { + kind: 'none' as const, + weekday: null, + which: null, + unit: null, + amount: null, + jy: null, + jm: null, + jd: null, + y: null, + m: null, + d: null, +}; + +const wire = (overrides: Partial = {}): WireVoiceIntent => ({ + treatmentType: null, + teeth: [], + connectedSpans: [], + comment: null, + prosthesisDefaultType: null, + prosthesisOverrides: [], + labId: null, + labMatchExact: false, + due: emptyDue, + ...overrides, +}); + +const positionalTooth = { + spoken: 'شش بالا راست', + fdi: null, + arch: 'upper' as const, + side: 'patient_right' as const, + position: 6, +}; + +describe('VOICE_INTENT_JSON_SCHEMA', () => { + it('is strict — every property required, no extras', () => { + expect(VOICE_INTENT_JSON_SCHEMA.additionalProperties).toBe(false); + expect([...VOICE_INTENT_JSON_SCHEMA.required].sort()).toEqual( + Object.keys(VOICE_INTENT_JSON_SCHEMA.properties).sort(), + ); + }); + + it('requires every field of the due object, since strict mode allows no optionals', () => { + const due = VOICE_INTENT_JSON_SCHEMA.properties.due; + expect([...due.required].sort()).toEqual( + Object.keys(due.properties).sort(), + ); + }); +}); + +describe('toVoiceIntent', () => { + it('takes the explicit branch for a code the model wrote in Persian digits', () => { + // The model is reading Persian text back, so "۲۶" and a digit-by-digit "2 6" both + // reach us. Matching only ASCII drops the tooth into the positional branch with no + // quadrant, where it is reported as unresolved — the clinician loses a tooth and is + // told the words were the problem. + for (const raw of ['\u06F2\u06F6', '2 6', ' 26 ', '\u0662\u0666']) { + const [tooth] = toVoiceIntent( + wire({ + teeth: [ + { + spoken: '\u0628\u06CC\u0633\u062A \u0648 \u0634\u0634', + fdi: raw, + arch: null, + side: null, + position: null, + }, + ], + }), + ).teeth; + expect(tooth).toEqual({ + kind: 'explicit', + fdi: '26', + spoken: '\u0628\u06CC\u0633\u062A \u0648 \u0634\u0634', + }); + } + }); + + it('narrows a positional tooth', () => { + const result = toVoiceIntent(wire({ teeth: [positionalTooth] })); + expect(result.teeth[0]).toEqual({ + kind: 'positional', + arch: 'upper', + side: 'patient_right', + position: 6, + spoken: 'شش بالا راست', + }); + }); + + it('narrows an explicit FDI tooth, which wins over positional fields', () => { + const result = toVoiceIntent( + wire({ teeth: [{ ...positionalTooth, fdi: '14', spoken: 'یک چهار' }] }), + ); + expect(result.teeth[0]).toEqual({ + kind: 'explicit', + fdi: '14', + spoken: 'یک چهار', + }); + }); + + it('maps due.kind "none" to no deadline', () => { + expect(toVoiceIntent(wire()).due).toBeNull(); + }); + + it('narrows each due kind', () => { + expect( + toVoiceIntent( + wire({ + due: { + ...emptyDue, + kind: 'weekday', + weekday: 'thursday', + which: 'next', + }, + }), + ).due, + ).toEqual({ kind: 'weekday', weekday: 'thursday', which: 'next' }); + + expect( + toVoiceIntent( + wire({ due: { ...emptyDue, kind: 'offset', unit: 'week', amount: 2 } }), + ).due, + ).toEqual({ kind: 'offset', unit: 'week', amount: 2 }); + + expect( + toVoiceIntent( + wire({ due: { ...emptyDue, kind: 'jalali', jy: 1404, jm: 7, jd: 25 } }), + ).due, + ).toEqual({ kind: 'jalali', jy: 1404, jm: 7, jd: 25 }); + + expect( + toVoiceIntent( + wire({ + due: { ...emptyDue, kind: 'gregorian', y: 2025, m: 10, d: 17 }, + }), + ).due, + ).toEqual({ kind: 'gregorian', y: 2025, m: 10, d: 17 }); + }); + + it('ignores a non-FDI-shaped fdi and keeps the positional fields', () => { + // A model emitting fdi:"6" alongside correct arch/side/position must still yield 16, + // not lose the tooth to the explicit branch. + const result = toVoiceIntent( + wire({ teeth: [{ ...positionalTooth, fdi: '6' }] }), + ); + expect(result.teeth[0]).toEqual({ + kind: 'positional', + arch: 'upper', + side: 'patient_right', + position: 6, + spoken: 'شش بالا راست', + }); + }); + + it('rejects impossible FDI shapes from the explicit branch', () => { + for (const fdi of ['99', '0', '140', '9', 'ab']) { + expect( + toVoiceIntent(wire({ teeth: [{ ...positionalTooth, fdi }] })).teeth[0] + .kind, + ).toBe('positional'); + } + }); + + it('passes an unrecognised due kind through so it can be flagged', () => { + // Collapsing it to null would make a misunderstood deadline indistinguishable from + // no deadline at all, and the resolver's flagging path unreachable. + const result = toVoiceIntent( + wire({ due: { ...emptyDue, kind: 'lunar_month' as never } }), + ); + expect(result.due).toEqual({ kind: 'lunar_month' }); + }); + + it('reports no prosthesis when neither a default nor an override was given', () => { + expect(toVoiceIntent(wire()).prosthesis).toBeNull(); + }); + + it('builds a prosthesis intent from a default alone', () => { + const result = toVoiceIntent(wire({ prosthesisDefaultType: 'pfm_crown' })); + expect(result.prosthesis).toEqual({ + defaultType: 'pfm_crown', + overrides: [], + }); + }); + + it('builds a prosthesis intent from overrides alone', () => { + const result = toVoiceIntent( + wire({ + prosthesisOverrides: [{ tooth: positionalTooth, type: 'pfm_crown' }], + }), + ); + expect(result.prosthesis?.defaultType).toBeNull(); + expect(result.prosthesis?.overrides).toHaveLength(1); + }); + + it('narrows connected spans', () => { + const result = toVoiceIntent( + wire({ + connectedSpans: [ + { + from: { ...positionalTooth, fdi: '14' }, + to: { ...positionalTooth, fdi: '16' }, + }, + ], + }), + ); + expect(result.connectedSpans[0].from).toEqual({ + kind: 'explicit', + fdi: '14', + spoken: 'شش بالا راست', + }); + }); + + it('is total — a missing or malformed payload yields a resolvable shape, not a throw', () => { + // Whatever survives here is reported as unresolved downstream rather than crashing. + for (const bad of [undefined, null, {}, { teeth: 'nope', due: 5 }]) { + expect(() => toVoiceIntent(bad as never)).not.toThrow(); + const result = toVoiceIntent(bad as never); + expect(result.teeth).toEqual([]); + expect(result.due).toBeNull(); + expect(result.labMatchExact).toBe(false); + } + }); + + it('coerces a non-boolean labMatchExact to false', () => { + expect( + toVoiceIntent(wire({ labMatchExact: 'yes' as never })).labMatchExact, + ).toBe(false); + }); +}); diff --git a/backend/src/modules/voice/extraction.wire.ts b/backend/src/modules/voice/extraction.wire.ts new file mode 100644 index 0000000..fecf194 --- /dev/null +++ b/backend/src/modules/voice/extraction.wire.ts @@ -0,0 +1,273 @@ +import { normalizeFdiCode } from '../../common/fdi'; +import type { + ConnectedSpanIntent, + DueIntent, + ProsthesisIntent, + ToothIntent, + VoiceIntent, + Weekday, +} from './voice.types'; +import { WEEKDAYS } from './voice.types'; + +/** + * Deliberately flat: strict `json_schema` mode has poor support for discriminated unions, + * so every variant field is present and nullable. `toVoiceIntent` narrows it into the + * internal union and is total — anything it cannot classify becomes a shape the resolvers + * report as unresolved rather than something that throws here. + */ + +export type WireToothIntent = { + spoken: string; + /** The two-digit FDI code the clinician spoke; null when the tooth was described. */ + fdi: string | null; + arch: 'upper' | 'lower' | null; + side: 'patient_right' | 'patient_left' | null; + position: number | null; +}; + +export type WireDue = { + kind: 'weekday' | 'offset' | 'jalali' | 'gregorian' | 'none'; + weekday: Weekday | null; + which: 'this' | 'next' | null; + unit: 'day' | 'week' | 'month' | null; + amount: number | null; + jy: number | null; + jm: number | null; + jd: number | null; + y: number | null; + m: number | null; + d: number | null; +}; + +export type WireVoiceIntent = { + treatmentType: string | null; + teeth: WireToothIntent[]; + connectedSpans: { from: WireToothIntent; to: WireToothIntent }[]; + comment: string | null; + prosthesisDefaultType: string | null; + prosthesisOverrides: { tooth: WireToothIntent; type: string }[]; + labId: string | null; + labMatchExact: boolean; + due: WireDue; +}; + +const TOOTH_SCHEMA = { + type: 'object', + additionalProperties: false, + required: ['spoken', 'fdi', 'arch', 'side', 'position'], + properties: { + spoken: { + type: 'string', + description: 'The exact transcript words for this tooth.', + }, + fdi: { + type: ['string', 'null'], + description: + 'The two-digit FDI code the clinician said for this tooth, e.g. "26". Null only ' + + 'when the tooth was described in words instead of numbered.', + }, + arch: { type: ['string', 'null'], enum: ['upper', 'lower', null] }, + side: { + type: ['string', 'null'], + enum: ['patient_right', 'patient_left', null], + description: "The PATIENT's side, never the viewer's.", + }, + position: { + type: ['integer', 'null'], + description: + 'Position from the midline: 1 = central incisor … 8 = third molar. Never an FDI code.', + }, + }, +} as const; + +export const VOICE_INTENT_JSON_SCHEMA = { + type: 'object', + additionalProperties: false, + required: [ + 'treatmentType', + 'teeth', + 'connectedSpans', + 'comment', + 'prosthesisDefaultType', + 'prosthesisOverrides', + 'labId', + 'labMatchExact', + 'due', + ], + properties: { + treatmentType: { + type: ['string', 'null'], + description: 'A treatment type CODE from the supplied list, or null.', + }, + teeth: { type: 'array', items: TOOTH_SCHEMA }, + connectedSpans: { + type: 'array', + description: 'Bridges / splinted units. Endpoints inclusive.', + items: { + type: 'object', + additionalProperties: false, + required: ['from', 'to'], + properties: { from: TOOTH_SCHEMA, to: TOOTH_SCHEMA }, + }, + }, + comment: { + type: ['string', 'null'], + description: 'Clinical notes, in the spoken language.', + }, + prosthesisDefaultType: { + type: ['string', 'null'], + description: + 'A prosthesis type CODE applied to every tooth unless overridden.', + }, + prosthesisOverrides: { + type: 'array', + items: { + type: 'object', + additionalProperties: false, + required: ['tooth', 'type'], + properties: { tooth: TOOTH_SCHEMA, type: { type: 'string' } }, + }, + }, + labId: { + type: ['string', 'null'], + description: 'An id from the supplied lab list. Never invent one.', + }, + labMatchExact: { + type: 'boolean', + description: 'True only when the spoken name matched a lab name exactly.', + }, + due: { + type: 'object', + additionalProperties: false, + required: [ + 'kind', + 'weekday', + 'which', + 'unit', + 'amount', + 'jy', + 'jm', + 'jd', + 'y', + 'm', + 'd', + ], + properties: { + kind: { + type: 'string', + enum: ['weekday', 'offset', 'jalali', 'gregorian', 'none'], + }, + weekday: { type: ['string', 'null'], enum: [...WEEKDAYS, null] }, + which: { type: ['string', 'null'], enum: ['this', 'next', null] }, + unit: { + type: ['string', 'null'], + enum: ['day', 'week', 'month', null], + }, + amount: { type: ['integer', 'null'] }, + jy: { type: ['integer', 'null'] }, + jm: { type: ['integer', 'null'] }, + jd: { type: ['integer', 'null'] }, + y: { type: ['integer', 'null'] }, + m: { type: ['integer', 'null'] }, + d: { type: ['integer', 'null'] }, + }, + }, + }, +} as const; + +/** Two digits, quadrant 1-8, position 1-8 — the only thing that can be an FDI code. */ +const FDI_SHAPE = /^[1-8][1-8]$/; + +function toToothIntent(wire: WireToothIntent | undefined | null): ToothIntent { + const spoken = typeof wire?.spoken === 'string' ? wire.spoken : ''; + // "۲۶" and "2 6" are FDI codes that do not match literally; unnormalised they fall + // through to the positional branch with no quadrant and read as unresolved. + const fdi = normalizeFdiCode(wire?.fdi); + // Only take the explicit branch for something actually FDI-shaped. A model that emits + // fdi:"6" alongside correct arch/side/position would otherwise lose the tooth entirely. + if (FDI_SHAPE.test(fdi)) { + return { kind: 'explicit', fdi, spoken }; + } + return { + kind: 'positional', + arch: wire?.arch as 'upper' | 'lower', + side: wire?.side as 'patient_right' | 'patient_left', + position: typeof wire?.position === 'number' ? wire.position : Number.NaN, + spoken, + }; +} + +function toDueIntent(wire: WireDue | undefined | null): DueIntent | null { + switch (wire?.kind) { + case 'weekday': + return { + kind: 'weekday', + weekday: wire.weekday as Weekday, + which: wire.which as 'this', + }; + case 'offset': + return { + kind: 'offset', + unit: wire.unit as 'day', + amount: typeof wire.amount === 'number' ? wire.amount : Number.NaN, + }; + case 'jalali': + return { + kind: 'jalali', + jy: wire.jy as number, + jm: wire.jm as number, + jd: wire.jd as number, + }; + case 'gregorian': + return { + kind: 'gregorian', + y: wire.y as number, + m: wire.m as number, + d: wire.d as number, + }; + case 'none': + case undefined: + return null; + default: + // An unrecognised kind means a deadline WAS spoken and we failed to classify it. + // Passing it through lets the resolver flag it; collapsing it to null would make a + // misunderstood deadline indistinguishable from no deadline at all. + return { kind: wire?.kind } as unknown as DueIntent; + } +} + +export function toVoiceIntent(wire: WireVoiceIntent): VoiceIntent { + const teeth = Array.isArray(wire?.teeth) ? wire.teeth : []; + const spans = Array.isArray(wire?.connectedSpans) ? wire.connectedSpans : []; + const overrides = Array.isArray(wire?.prosthesisOverrides) + ? wire.prosthesisOverrides + : []; + + const connectedSpans: ConnectedSpanIntent[] = spans.map((span) => ({ + from: toToothIntent(span?.from), + to: toToothIntent(span?.to), + })); + + const hasProsthesis = + wire?.prosthesisDefaultType != null || overrides.length > 0; + const prosthesis: ProsthesisIntent | null = hasProsthesis + ? { + defaultType: wire?.prosthesisDefaultType ?? null, + overrides: overrides.map((o) => ({ + tooth: toToothIntent(o?.tooth), + type: o?.type, + })), + } + : null; + + return { + treatmentType: wire?.treatmentType ?? null, + teeth: teeth.map(toToothIntent), + connectedSpans, + comment: wire?.comment ?? null, + prosthesis, + labId: wire?.labId ?? null, + labMatchExact: wire?.labMatchExact === true, + due: toDueIntent(wire?.due), + }; +} diff --git a/backend/src/modules/voice/openrouter.provider.spec.ts b/backend/src/modules/voice/openrouter.provider.spec.ts new file mode 100644 index 0000000..a18f0e6 --- /dev/null +++ b/backend/src/modules/voice/openrouter.provider.spec.ts @@ -0,0 +1,197 @@ +import { + OpenRouterAsrProvider, + OpenRouterExtractionProvider, +} from './openrouter.provider'; +import { VoiceProviderError } from './voice.providers'; + +const CONFIG = { + apiKey: 'test-key', + baseUrl: 'https://openrouter.test/api/v1', + model: 'm', +}; + +const CATALOG = { + treatmentTypes: [{ code: 'prosthesis', label: 'پروتز' }], + prosthesisTypes: [{ code: 'pfm_crown', label: 'روکش پی‌اف‌ام' }], + labs: [{ id: 'lab-1', name: 'لابراتوار سینا' }], +}; + +type ChatBody = { + temperature: number; + provider: { require_parameters: boolean }; + response_format: { type: string; json_schema: { strict: boolean } }; + messages: { role: string; content: string }[]; +}; + +function parseBody(spy: jest.Mock): ChatBody { + const init = (spy.mock.calls[0] as [string, RequestInit])[1]; + return JSON.parse(init.body as string) as ChatBody; +} + +function mockFetch(response: { + ok: boolean; + status?: number; + body?: unknown; + text?: string; +}) { + const spy = jest.fn().mockResolvedValue({ + ok: response.ok, + status: response.status ?? (response.ok ? 200 : 500), + json: () => Promise.resolve(response.body), + text: () => Promise.resolve(response.text ?? ''), + }); + global.fetch = spy; + return spy; +} + +afterEach(() => { + jest.restoreAllMocks(); +}); + +describe('OpenRouterAsrProvider', () => { + it('posts base64 JSON with the language hint, per the documented STT contract', async () => { + const spy = mockFetch({ + ok: true, + body: { text: ' سلام ', usage: { seconds: 12, cost: 0.002 } }, + }); + + const result = await new OpenRouterAsrProvider(CONFIG).transcribe( + { data: 'BASE64', format: 'webm' }, + 'fa', + ); + + const [url, init] = spy.mock.calls[0] as [string, RequestInit]; + expect(url).toBe('https://openrouter.test/api/v1/audio/transcriptions'); + expect(init.headers).toMatchObject({ Authorization: 'Bearer test-key' }); + expect(JSON.parse(init.body as string)).toEqual({ + model: 'm', + input_audio: { data: 'BASE64', format: 'webm' }, + language: 'fa', + }); + expect(result.text).toBe('سلام'); + expect(result.usage).toEqual({ seconds: 12, costUsd: 0.002 }); + }); + + it('reports missing usage as null rather than zero', async () => { + mockFetch({ ok: true, body: { text: 'x' } }); + const result = await new OpenRouterAsrProvider(CONFIG).transcribe( + { data: 'B', format: 'webm' }, + 'en', + ); + expect(result.usage).toEqual({ seconds: null, costUsd: null }); + }); + + it('raises a staged error without leaking the vendor body', async () => { + // A 4xx can echo the request back, transcript included. + mockFetch({ + ok: false, + status: 400, + text: 'transcript: patient name here', + }); + const provider = new OpenRouterAsrProvider(CONFIG); + + await expect( + provider.transcribe({ data: 'B', format: 'webm' }, 'fa'), + ).rejects.toMatchObject({ + name: 'VoiceProviderError', + stage: 'asr', + status: 400, + }); + await expect( + provider.transcribe({ data: 'B', format: 'webm' }, 'fa'), + ).rejects.not.toThrow(/patient name/); + }); +}); + +describe('OpenRouterExtractionProvider', () => { + const wireContent = JSON.stringify({ + treatmentType: 'prosthesis', + teeth: [ + { spoken: 'یک چهار', fdi: '14', arch: null, side: null, position: null }, + ], + connectedSpans: [], + comment: null, + prosthesisDefaultType: 'pfm_crown', + prosthesisOverrides: [], + labId: 'lab-1', + labMatchExact: true, + due: { + kind: 'none', + weekday: null, + which: null, + unit: null, + amount: null, + jy: null, + jm: null, + jd: null, + y: null, + m: null, + d: null, + }, + }); + + it('constrains output with a strict JSON schema and a schema-honouring provider', async () => { + const spy = mockFetch({ + ok: true, + body: { + choices: [{ message: { content: wireContent } }], + usage: { cost: 0.0008 }, + }, + }); + + const result = await new OpenRouterExtractionProvider(CONFIG).extract( + 'روی دندان ۱۴ روکش', + CATALOG, + 'fa', + ); + + const body = parseBody(spy); + expect(body.response_format.type).toBe('json_schema'); + expect(body.response_format.json_schema.strict).toBe(true); + // Without require_parameters OpenRouter may route to a provider that treats the + // schema as a hint and returns prose, failing parsing intermittently. + expect(body.provider).toEqual({ require_parameters: true }); + expect(body.temperature).toBe(0); + + expect(result.intent.treatmentType).toBe('prosthesis'); + expect(result.intent.teeth[0]).toEqual({ + kind: 'explicit', + fdi: '14', + spoken: 'یک چهار', + }); + expect(result.intent.due).toBeNull(); + expect(result.costUsd).toBe(0.0008); + }); + + it('sends the catalog codes and lab ids the model is allowed to choose from', async () => { + const spy = mockFetch({ + ok: true, + body: { choices: [{ message: { content: wireContent } }] }, + }); + await new OpenRouterExtractionProvider(CONFIG).extract('x', CATALOG, 'fa'); + + const body = parseBody(spy); + const system = body.messages[0].content; + expect(system).toContain('prosthesis'); + expect(system).toContain('pfm_crown'); + expect(system).toContain('lab-1'); + expect(body.messages[1]).toEqual({ role: 'user', content: 'x' }); + }); + + it('fails loudly on unparseable content rather than passing rubbish downstream', async () => { + mockFetch({ + ok: true, + body: { choices: [{ message: { content: 'I think tooth 14?' } }] }, + }); + await expect( + new OpenRouterExtractionProvider(CONFIG).extract('x', CATALOG, 'fa'), + ).rejects.toBeInstanceOf(VoiceProviderError); + }); + + it('fails when the model returns no content at all', async () => { + mockFetch({ ok: true, body: { choices: [] } }); + await expect( + new OpenRouterExtractionProvider(CONFIG).extract('x', CATALOG, 'fa'), + ).rejects.toMatchObject({ stage: 'extraction' }); + }); +}); diff --git a/backend/src/modules/voice/openrouter.provider.ts b/backend/src/modules/voice/openrouter.provider.ts new file mode 100644 index 0000000..7753fa2 --- /dev/null +++ b/backend/src/modules/voice/openrouter.provider.ts @@ -0,0 +1,170 @@ +import { Logger } from '@nestjs/common'; +import { + type AsrProvider, + type AsrResult, + type AudioInput, + type ExtractionCatalog, + type ExtractionProvider, + type ExtractionResult, + VoiceProviderError, +} from './voice.providers'; +import { buildExtractionPrompt } from './extraction.prompt'; +import { + VOICE_INTENT_JSON_SCHEMA, + toVoiceIntent, + type WireVoiceIntent, +} from './extraction.wire'; + +type OpenRouterConfig = { + apiKey: string; + baseUrl: string; + model: string; +}; + +type TranscriptionResponse = { + text?: unknown; + usage?: { seconds?: unknown; cost?: unknown }; +}; + +type ChatResponse = { + choices?: { message?: { content?: unknown } }[]; + usage?: { cost?: unknown }; +}; + +function numberOrNull(value: unknown): number | null { + return typeof value === 'number' && Number.isFinite(value) ? value : null; +} + +const errorLogger = new Logger('OpenRouterVoice'); + +async function readError( + response: Response, + stage: 'asr' | 'extraction', +): Promise { + // A vendor 4xx can echo the request back, transcript included. Keep the body out of the + // thrown message — which callers log and could forward — and out of the default log + // level; it stays available at debug when someone is actively diagnosing. + try { + errorLogger.debug( + `${stage} ${response.status} body: ${(await response.text()).slice(0, 500)}`, + ); + } catch { + errorLogger.debug(`${stage} ${response.status} body unreadable`); + } + throw new VoiceProviderError( + stage, + `OpenRouter ${stage} failed with status ${response.status}`, + response.status, + ); +} + +/** Speech → text via OpenRouter's transcription endpoint (whisper-1 and friends). */ +export class OpenRouterAsrProvider implements AsrProvider { + private readonly logger = new Logger(OpenRouterAsrProvider.name); + + constructor(private readonly config: OpenRouterConfig) {} + + async transcribe( + audio: AudioInput, + localeHint: string, + signal?: AbortSignal, + ): Promise { + const response = await fetch( + `${this.config.baseUrl}/audio/transcriptions`, + { + method: 'POST', + signal, + headers: { + Authorization: `Bearer ${this.config.apiKey}`, + 'Content-Type': 'application/json', + }, + body: JSON.stringify({ + model: this.config.model, + input_audio: { data: audio.data, format: audio.format }, + language: localeHint, + }), + }, + ); + + if (!response.ok) await readError(response, 'asr'); + + const body = (await response.json()) as TranscriptionResponse; + const text = typeof body.text === 'string' ? body.text.trim() : ''; + this.logger.debug( + `transcribed ${numberOrNull(body.usage?.seconds) ?? '?'}s`, + ); + + return { + text, + usage: { + seconds: numberOrNull(body.usage?.seconds), + costUsd: numberOrNull(body.usage?.cost), + }, + }; + } +} + +/** Transcript → VoiceIntent via OpenRouter chat completions with a JSON schema. */ +export class OpenRouterExtractionProvider implements ExtractionProvider { + constructor(private readonly config: OpenRouterConfig) {} + + async extract( + transcript: string, + catalog: ExtractionCatalog, + localeHint: string, + signal?: AbortSignal, + ): Promise { + const response = await fetch(`${this.config.baseUrl}/chat/completions`, { + method: 'POST', + signal, + headers: { + Authorization: `Bearer ${this.config.apiKey}`, + 'Content-Type': 'application/json', + }, + body: JSON.stringify({ + model: this.config.model, + temperature: 0, + // Only route to endpoints that actually honour the JSON schema — otherwise OpenRouter may + // pick a provider that treats it as a hint and returns prose, failing intermittently. + provider: { require_parameters: true }, + messages: buildExtractionPrompt(transcript, catalog, localeHint), + response_format: { + type: 'json_schema', + json_schema: { + name: 'voice_intent', + strict: true, + schema: VOICE_INTENT_JSON_SCHEMA, + }, + }, + }), + }); + + if (!response.ok) await readError(response, 'extraction'); + + const body = (await response.json()) as ChatResponse; + const content = body.choices?.[0]?.message?.content; + if (typeof content !== 'string' || !content.trim()) { + throw new VoiceProviderError( + 'extraction', + 'OpenRouter returned no content', + ); + } + + let wire: WireVoiceIntent; + try { + wire = JSON.parse(content) as WireVoiceIntent; + } catch { + // Schema-constrained output should be valid JSON; if it is not, the resolvers can do + // nothing with it, so fail here rather than pass rubbish downstream. + throw new VoiceProviderError( + 'extraction', + 'OpenRouter returned unparseable JSON', + ); + } + + return { + intent: toVoiceIntent(wire), + costUsd: numberOrNull(body.usage?.cost), + }; + } +} diff --git a/backend/src/modules/voice/tooth-intent.resolver.spec.ts b/backend/src/modules/voice/tooth-intent.resolver.spec.ts new file mode 100644 index 0000000..f145309 --- /dev/null +++ b/backend/src/modules/voice/tooth-intent.resolver.spec.ts @@ -0,0 +1,265 @@ +import { + resolveToothIntent, + resolveToothIntents, +} from './tooth-intent.resolver'; +import type { ToothIntent } from './voice.types'; + +const positional = ( + arch: 'upper' | 'lower', + side: 'patient_right' | 'patient_left', + position: number, + spoken = 'x', +): ToothIntent => ({ kind: 'positional', arch, side, position, spoken }); + +const explicit = (fdi: string, spoken = 'x'): ToothIntent => ({ + kind: 'explicit', + fdi, + spoken, +}); + +describe('resolveToothIntent', () => { + describe('positional intents', () => { + // "شش بالا راست" — upper right six — must be 16, not 26. A mirrored quadrant is a + // valid code for the wrong tooth and reaches the lab unnoticed. + it('resolves each quadrant from the patient perspective', () => { + expect(resolveToothIntent(positional('upper', 'patient_right', 6))).toBe( + '16', + ); + expect(resolveToothIntent(positional('upper', 'patient_left', 6))).toBe( + '26', + ); + expect(resolveToothIntent(positional('lower', 'patient_left', 6))).toBe( + '36', + ); + expect(resolveToothIntent(positional('lower', 'patient_right', 6))).toBe( + '46', + ); + }); + + it('returns null for an out-of-range position instead of clamping', () => { + expect( + resolveToothIntent(positional('upper', 'patient_right', 9)), + ).toBeNull(); + expect( + resolveToothIntent(positional('upper', 'patient_right', 0)), + ).toBeNull(); + }); + + it('returns null for a malformed arch or side', () => { + expect( + resolveToothIntent( + positional('sideways' as 'upper', 'patient_right', 6), + ), + ).toBeNull(); + expect( + resolveToothIntent( + positional('upper', 'viewer_right' as 'patient_right', 6), + ), + ).toBeNull(); + }); + }); + + describe('explicit intents', () => { + it('accepts a permanent FDI code', () => { + expect(resolveToothIntent(explicit('14'))).toBe('14'); + expect(resolveToothIntent(explicit('48'))).toBe('48'); + }); + + it('rejects deciduous codes rather than snapping to a permanent tooth', () => { + expect(resolveToothIntent(explicit('51'))).toBeNull(); + expect(resolveToothIntent(explicit('85'))).toBeNull(); + }); + + it('rejects nonsense', () => { + for (const value of ['', '1', '99', '140']) { + expect(resolveToothIntent(explicit(value))).toBeNull(); + } + }); + }); + + it('returns null for a missing or unknown intent shape', () => { + expect(resolveToothIntent(undefined as unknown as ToothIntent)).toBeNull(); + expect( + resolveToothIntent({ kind: 'guess' } as unknown as ToothIntent), + ).toBeNull(); + }); +}); + +describe('resolveToothIntents', () => { + it('resolves a mixed list and sorts the result', () => { + const result = resolveToothIntents([ + positional('upper', 'patient_right', 5, 'پنج بالا راست'), + explicit('14', 'یک چهار'), + ]); + expect(result.teeth).toEqual(['14', '15']); + expect(result.unresolved).toEqual([]); + }); + + it('collapses a tooth named twice', () => { + const result = resolveToothIntents([ + explicit('14', 'چهارده'), + positional('upper', 'patient_right', 4, 'چهار بالا راست'), + ]); + expect(result.teeth).toEqual(['14']); + }); + + it('reports what it could not understand instead of dropping it', () => { + const result = resolveToothIntents([ + explicit('14', 'یک چهار'), + explicit('51', 'دندان شیری'), + positional('upper', 'patient_right', 9, 'نه بالا راست'), + ]); + expect(result.teeth).toEqual(['14']); + expect(result.unresolved).toEqual([ + { spoken: 'دندان شیری', reason: 'not_permanent_tooth' }, + { spoken: 'نه بالا راست', reason: 'position_out_of_range' }, + ]); + }); + + it('does not repeat an identical unresolved item', () => { + const result = resolveToothIntents([ + explicit('51', 'شیری'), + explicit('51', 'شیری'), + ]); + expect(result.unresolved).toHaveLength(1); + }); + + it('survives a non-array where the model should have sent a list', () => { + // The model can return an object or a number here; that must degrade, not 500. + for (const bad of [undefined, null, 5, 'teeth', { fdi: '14' }]) { + expect(resolveToothIntents(bad as unknown as ToothIntent[])).toEqual({ + teeth: [], + unresolved: [], + }); + } + }); + + it('reads a spoken number as its FDI code, digits in any script', () => { + // The product rule: the number the clinician says IS the tooth. 26 = quadrant 2 + // (patient's upper left) + position 6 = first molar. + for (const raw of ['26', ' 26 ', '2 6', '\u06F2\u06F6', '\u0662\u0666']) { + expect(resolveToothIntents([explicit(raw, 'x')]).teeth).toEqual(['26']); + } + }); + + it('trims an explicit code, matching normalizeTeeth', () => { + expect(resolveToothIntents([explicit(' 14 ', 'x')]).teeth).toEqual(['14']); + }); + + it('distinguishes a deciduous tooth from nonsense in the reason it reports', () => { + // '51' really is a (primary) tooth the chart cannot show; '99' is not a tooth at all. + expect( + resolveToothIntents([explicit('51', 'shiri')]).unresolved[0].reason, + ).toBe('not_permanent_tooth'); + for (const junk of ['99', '19', '140', '', 'ab']) { + expect( + resolveToothIntents([explicit(junk, `j-${junk}`)]).unresolved[0].reason, + ).toBe('malformed'); + } + }); + + it('says the quadrant is missing rather than blaming the words', () => { + // Regression: "ترمیم برای دندون دو" reported "could not be read", sending the + // clinician to look for a transcription fault. Position 2 was understood fine — + // what is missing is the quadrant, and four teeth carry position 2. + const bare = { + kind: 'positional', + arch: null, + side: null, + position: 2, + spoken: 'دندون دو', + } as unknown as ToothIntent; + + const result = resolveToothIntents([bare]); + expect(result.teeth).toEqual([]); + expect(result.unresolved).toEqual([ + { + spoken: 'دندون دو', + reason: 'tooth_missing_quadrant', + // Every reading of "position 2", for the clinician to pick from. + candidates: ['12', '22', '32', '42'], + }, + ]); + }); + + it('narrows the candidates by whatever the clinician did say', () => { + const half = (arch: string | null, side: string | null) => + resolveToothIntents([ + { + kind: 'positional', + arch, + side, + position: 2, + spoken: 'دو', + } as unknown as ToothIntent, + ]).unresolved[0].candidates; + + expect(half('upper', null)).toEqual(['12', '22']); + expect(half('lower', null)).toEqual(['32', '42']); + // Quadrant 1 is the patient's upper right, 4 the lower right. + expect(half(null, 'patient_right')).toEqual(['12', '42']); + expect(half(null, 'patient_left')).toEqual(['22', '32']); + }); + + it('offers no candidates for a reason a choice cannot settle', () => { + // Nothing to choose between when the position itself was wrong, or the tooth is + // deciduous — offering chips there would invent options. + for (const intent of [ + positional('upper', 'patient_right', 9, 'نه'), + explicit('51', 'شیری'), + ]) { + expect( + resolveToothIntents([intent]).unresolved[0].candidates, + ).toBeUndefined(); + } + }); + + it('reports a missing quadrant for a half-specified tooth too', () => { + // "دو بالا" narrows it to 12 or 22 — still not one tooth, and still not our guess. + for (const half of [ + { arch: 'upper', side: null }, + { arch: null, side: 'patient_right' }, + ]) { + const result = resolveToothIntents([ + { + kind: 'positional', + ...half, + position: 2, + spoken: 'دو', + } as unknown as ToothIntent, + ]); + expect(result.unresolved[0].reason).toBe('tooth_missing_quadrant'); + } + }); + + it('still calls an out-of-range position out of range when the quadrant is missing', () => { + // Position wins: "nine" is wrong however completely it was said. + const result = resolveToothIntents([ + { + kind: 'positional', + arch: null, + side: null, + position: 9, + spoken: 'نه', + } as unknown as ToothIntent, + ]); + expect(result.unresolved[0].reason).toBe('position_out_of_range'); + }); + + it('keeps unresolved items separate when the model omits the spoken span', () => { + // Without `spoken` these are indistinguishable; collapsing them would hide a lost tooth. + const result = resolveToothIntents([ + { kind: 'explicit', fdi: '51' } as ToothIntent, + { kind: 'explicit', fdi: '52' } as ToothIntent, + ]); + expect(result.unresolved).toHaveLength(2); + }); + + it('handles an empty or missing list', () => { + expect(resolveToothIntents([])).toEqual({ teeth: [], unresolved: [] }); + expect(resolveToothIntents(undefined as unknown as ToothIntent[])).toEqual({ + teeth: [], + unresolved: [], + }); + }); +}); diff --git a/backend/src/modules/voice/tooth-intent.resolver.ts b/backend/src/modules/voice/tooth-intent.resolver.ts new file mode 100644 index 0000000..fd1e719 --- /dev/null +++ b/backend/src/modules/voice/tooth-intent.resolver.ts @@ -0,0 +1,145 @@ +import { + isFdiTooth, + normalizeFdiCode, + toFdi, + type Arch, + type PatientSide, +} from '../../common/fdi'; +import type { ToothIntent, UnresolvedItem } from './voice.types'; + +const ARCHES: readonly Arch[] = ['upper', 'lower']; +const SIDES: readonly PatientSide[] = ['patient_right', 'patient_left']; + +export type ToothResolution = { + /** Unique FDI codes, sorted (matching normalizeTeeth's ordering). */ + teeth: string[]; + unresolved: UnresolvedItem[]; +}; + +/** Everything here parses untrusted model output, so nothing may throw. */ +function normalizedFdi(intent: ToothIntent): string { + // The same normalisation the wire layer used to pick this branch, so the two agree. + return normalizeFdiCode((intent as { fdi?: unknown }).fdi); +} + +/** + * Never guesses and never clamps: position 9, a deciduous tooth or a malformed intent all + * resolve to null, so the caller surfaces "not understood" rather than silently selecting a + * neighbouring tooth. + */ +export function resolveToothIntent(intent: ToothIntent): string | null { + if (!intent || typeof intent !== 'object') return null; + + if (intent.kind === 'explicit') { + const fdi = normalizedFdi(intent); + return isFdiTooth(fdi) ? fdi : null; + } + + if (intent.kind === 'positional') { + if (intent.arch !== 'upper' && intent.arch !== 'lower') return null; + if (intent.side !== 'patient_right' && intent.side !== 'patient_left') + return null; + return toFdi(intent.arch, intent.side, intent.position); + } + + return null; +} + +function unresolvedReason(intent: ToothIntent): UnresolvedItem['reason'] { + if (!intent || typeof intent !== 'object') return 'malformed'; + + if (intent.kind === 'explicit') { + const fdi = normalizedFdi(intent); + // Quadrants 1-4 are permanent and would already have resolved, so a well-formed + // quadrant+position reaching here is quadrant 5-8: deciduous. Anything else is noise. + return /^[1-8][1-8]$/.test(fdi) ? 'not_permanent_tooth' : 'malformed'; + } + + if (intent.kind === 'positional') { + const positionBad = + !Number.isInteger(intent.position) || + intent.position < 1 || + intent.position > 8; + if (positionBad) return 'position_out_of_range'; + // The position was understood; the quadrant was never said. "دندون دو" names four + // teeth, so "could not be read" would send the clinician after the wrong fault. + const archMissing = intent.arch !== 'upper' && intent.arch !== 'lower'; + const sideMissing = + intent.side !== 'patient_right' && intent.side !== 'patient_left'; + return archMissing || sideMissing ? 'tooth_missing_quadrant' : 'malformed'; + } + + return 'malformed'; +} + +/** + * The teeth still consistent with what *was* heard — "دو" leaves four, "دو بالا" two. Not + * a guess: the full set of readings, for the clinician to choose from. + */ +function quadrantCandidates(intent: ToothIntent): string[] { + if (intent.kind !== 'positional') return []; + const arches = + intent.arch === 'upper' || intent.arch === 'lower' ? [intent.arch] : ARCHES; + const sides = + intent.side === 'patient_right' || intent.side === 'patient_left' + ? [intent.side] + : SIDES; + + const codes: string[] = []; + for (const arch of arches) { + for (const side of sides) { + const fdi = toFdi(arch, side, intent.position); + if (fdi) codes.push(fdi); + } + } + return codes.sort(); +} + +function spokenOf(intent: ToothIntent): string { + const spoken = (intent as { spoken?: unknown })?.spoken; + return typeof spoken === 'string' && spoken.trim() ? spoken.trim() : ''; +} + +/** + * Duplicates collapse; anything unresolvable is reported rather than dropped, so the sheet + * can show which words were not understood. + */ +export function resolveToothIntents( + intents: readonly ToothIntent[], +): ToothResolution { + const teeth = new Set(); + const unresolved: UnresolvedItem[] = []; + const seenUnresolved = new Set(); + + // Not `intents ?? []`: a model may return an object or a number here, and a + // non-iterable must degrade like any other malformed shape rather than throw. + const list: readonly ToothIntent[] = Array.isArray(intents) + ? (intents as readonly ToothIntent[]) + : []; + + for (const intent of list) { + const fdi = resolveToothIntent(intent); + if (fdi) { + teeth.add(fdi); + continue; + } + const reason = unresolvedReason(intent); + const spoken = spokenOf(intent); + const candidates = + reason === 'tooth_missing_quadrant' ? quadrantCandidates(intent) : []; + // Only dedupe what we can tell apart: without `spoken`, two lost references collapse + // into one blank row and a tooth vanishes. Candidates are part of the identity. + if (spoken) { + const key = `${spoken}::${reason}::${candidates.join(',')}`; + if (seenUnresolved.has(key)) continue; + seenUnresolved.add(key); + } + unresolved.push( + candidates.length > 0 + ? { spoken, reason, candidates } + : { spoken, reason }, + ); + } + + return { teeth: [...teeth].sort(), unresolved }; +} diff --git a/backend/src/modules/voice/voice-throttler.guard.ts b/backend/src/modules/voice/voice-throttler.guard.ts new file mode 100644 index 0000000..f28bdad --- /dev/null +++ b/backend/src/modules/voice/voice-throttler.guard.ts @@ -0,0 +1,29 @@ +import { HttpStatus, Injectable } from '@nestjs/common'; +import { ThrottlerGuard } from '@nestjs/throttler'; +import { AppException, ErrorCode } from '../../common/errors'; + +/** + * Rate limits voice extraction per user, not per IP: the default tracker keys on `req.ip`, + * which behind nginx means the whole deployment shares one bucket unless `trust proxy` is + * set, and one clinic could then lock out every other. + */ +@Injectable() +export class VoiceThrottlerGuard extends ThrottlerGuard { + protected getTracker(req: Record): Promise { + const user = req?.user as { id?: unknown } | undefined; + if (typeof user?.id === 'string' && user.id) { + return Promise.resolve(`voice:user:${user.id}`); + } + // Unauthenticated requests never reach here, but fall back rather than share a bucket. + const ip = typeof req?.ip === 'string' ? req.ip : 'unknown'; + return Promise.resolve(`voice:ip:${ip}`); + } + + /** Without this, ThrottlerException surfaces as INTERNAL_ERROR — there is no 429 fallback. */ + protected throwThrottlingException(): Promise { + throw new AppException( + ErrorCode.VOICE_RATE_LIMITED, + HttpStatus.TOO_MANY_REQUESTS, + ); + } +} diff --git a/backend/src/modules/voice/voice.controller.ts b/backend/src/modules/voice/voice.controller.ts new file mode 100644 index 0000000..0b93657 --- /dev/null +++ b/backend/src/modules/voice/voice.controller.ts @@ -0,0 +1,68 @@ +import { + Body, + Controller, + Get, + Post, + Req, + Res, + UseGuards, +} from '@nestjs/common'; +import { ApiBearerAuth, ApiOperation, ApiTags } from '@nestjs/swagger'; +import type { Request, Response } from 'express'; +import { ClinicOrgGuard } from '../../common/guards/clinic-org.guard'; +import { JwtAuthGuard } from '../auth/guards/jwt-auth.guard'; +import { ExtractVoiceDto } from './dto/voice.dto'; +import { VoiceThrottlerGuard } from './voice-throttler.guard'; +import { VoiceService } from './voice.service'; + +type VoiceRequestUser = { id: string; organizationId?: string }; + +@ApiTags('voice') +@ApiBearerAuth('JWT-auth') +@UseGuards(JwtAuthGuard, ClinicOrgGuard) +@Controller('voice') +export class VoiceController { + constructor(private readonly voiceService: VoiceService) {} + + @Get('availability') + @ApiOperation({ + summary: 'Whether voice entry is configured, and for which locales', + }) + getAvailability() { + // The frontend cannot learn this from NEXT_PUBLIC_* — those are baked in at build + // time, so enabling a locale would otherwise require rebuilding the image. + return { success: true, data: this.voiceService.getAvailability() }; + } + + @Post('extract') + // Guarded here rather than globally, and configured by VOICE_THROTTLE_* rather than + // hardcoded. Availability is deliberately left unthrottled — it is cheap and the + // frontend calls it on load. + @UseGuards(VoiceThrottlerGuard) + @ApiOperation({ + summary: + 'Transcribe a recording and extract treatment detail intents (TAB_TREATMENT_EDIT)', + }) + async extract( + @Req() req: Request & { user: VoiceRequestUser }, + @Res({ passthrough: true }) res: Response, + @Body() dto: ExtractVoiceDto, + ) { + // Cancelling in the browser closes the connection; propagate it as an abort so the vendor + // call stops rather than settling unseen. It is metered per minute. + const aborter = new AbortController(); + res.on('close', () => { + if (!res.writableFinished) aborter.abort(); + }); + + // dto.locale, not req.user.language: the client sends the locale the microphone was + // actually offered in, so the ASR hint, catalog labels and week start all match it. + const data = await this.voiceService.extract( + req.user, + dto, + dto.locale, + aborter.signal, + ); + return { success: true, data }; + } +} diff --git a/backend/src/modules/voice/voice.module.ts b/backend/src/modules/voice/voice.module.ts new file mode 100644 index 0000000..7c5c366 --- /dev/null +++ b/backend/src/modules/voice/voice.module.ts @@ -0,0 +1,12 @@ +import { Module } from '@nestjs/common'; +import { PrismaService } from '../../../prisma/prisma.service'; +import { ProsthesisCatalogModule } from '../prosthesis-catalog/prosthesis-catalog.module'; +import { VoiceController } from './voice.controller'; +import { VoiceService } from './voice.service'; + +@Module({ + imports: [ProsthesisCatalogModule], + controllers: [VoiceController], + providers: [VoiceService, PrismaService], +}) +export class VoiceModule {} diff --git a/backend/src/modules/voice/voice.providers.ts b/backend/src/modules/voice/voice.providers.ts new file mode 100644 index 0000000..7511963 --- /dev/null +++ b/backend/src/modules/voice/voice.providers.ts @@ -0,0 +1,66 @@ +import type { VoiceIntent } from './voice.types'; + +/** + * ASR and extraction are separate, independently swappable roles — they will not come from + * the same vendor for every locale. Both resolve per locale from `config.voice.profiles`. + */ + +export type AudioInput = { + /** Raw base64, no data: prefix. */ + data: string; + /** Container as the recorder produced it: webm, mp4, m4a, wav, … */ + format: string; +}; + +export type AsrUsage = { + seconds: number | null; + costUsd: number | null; +}; + +export type AsrResult = { + text: string; + usage: AsrUsage; +}; + +export interface AsrProvider { + /** `localeHint` is ISO-639-1 and materially improves accuracy — always pass it. */ + transcribe( + audio: AudioInput, + localeHint: string, + signal?: AbortSignal, + ): Promise; +} + +export type ExtractionCatalog = { + /** Catalog codes with their labels in the actor's locale, so the model matches spoken words. */ + treatmentTypes: { code: string; label: string }[]; + prosthesisTypes: { code: string; label: string }[]; + /** The clinic's linked labs — a closed choice list. */ + labs: { id: string; name: string }[]; +}; + +export type ExtractionResult = { + intent: VoiceIntent; + costUsd: number | null; +}; + +export interface ExtractionProvider { + extract( + transcript: string, + catalog: ExtractionCatalog, + localeHint: string, + signal?: AbortSignal, + ): Promise; +} + +/** Raised when a vendor call fails; the service maps this onto the staged error codes. */ +export class VoiceProviderError extends Error { + constructor( + readonly stage: 'asr' | 'extraction', + message: string, + readonly status?: number, + ) { + super(message); + this.name = 'VoiceProviderError'; + } +} diff --git a/backend/src/modules/voice/voice.service.ts b/backend/src/modules/voice/voice.service.ts new file mode 100644 index 0000000..f36666d --- /dev/null +++ b/backend/src/modules/voice/voice.service.ts @@ -0,0 +1,344 @@ +import { HttpStatus, Injectable, Logger } from '@nestjs/common'; +import { ConfigService } from '@nestjs/config'; +import { LinkStatus } from '@prisma/client'; +import { PrismaService } from '../../../prisma/prisma.service'; +import { AppException, ErrorCode } from '../../common/errors'; +import { + civilDateInZone, + isValidIanaTimeZone, +} from '../../common/zoned-civil-time'; +import type { VoiceConfig, VoiceProfile } from '../../configs/configurations'; +import { hasEffectivePermission } from '../../common/membership-permissions'; +import { normalizeCatalogLocale } from '../catalog/catalog-label.service'; +import { ProsthesisCatalogService } from '../prosthesis-catalog/prosthesis-catalog.service'; +import { TreatmentCatalogService } from '../treatment-catalog/treatment-catalog.service'; +import { weekStartForLocale } from './due-date.resolver'; +import { + resolveVoiceIntent, + type ResolvedExtraction, +} from './extraction.resolver'; +import { + OpenRouterAsrProvider, + OpenRouterExtractionProvider, +} from './openrouter.provider'; +import { + VoiceProviderError, + type AsrProvider, + type ExtractionCatalog, + type ExtractionProvider, +} from './voice.providers'; +import type { ExtractVoiceDto } from './dto/voice.dto'; + +export type VoiceAvailability = { + enabled: boolean; + locales: string[]; + maxRecordingMs: number | null; +}; + +export type VoiceExtractionResponse = ResolvedExtraction & { + transcript: string; +}; + +@Injectable() +export class VoiceService { + private readonly logger = new Logger(VoiceService.name); + + constructor( + private readonly prisma: PrismaService, + private readonly config: ConfigService, + private readonly treatmentCatalog: TreatmentCatalogService, + private readonly prosthesisCatalog: ProsthesisCatalogService, + ) {} + + private get voiceConfig(): VoiceConfig { + return this.config.get('voice')!; + } + + /** + * What the frontend needs to decide whether to render the microphone. v1 is ungated beyond + * a configured locale profile; the Plan.features design is deferred, not dropped. + */ + getAvailability(): VoiceAvailability { + const voice = this.voiceConfig; + const hasKey = Boolean(voice.openRouter.apiKey); + const locales = hasKey ? Object.keys(voice.profiles) : []; + return { + enabled: locales.length > 0, + locales, + maxRecordingMs: voice.maxRecordingMs, + }; + } + + async extract( + user: { id: string; organizationId?: string }, + dto: ExtractVoiceDto, + locale: string, + signal?: AbortSignal, + ): Promise { + const startedAt = Date.now(); + const organizationId = this.assertOrganization(user); + await this.assertCanEditTreatment(user.id, organizationId); + + const catalogLocale = normalizeCatalogLocale(locale); + const profile = this.resolveProfile(catalogLocale); + this.assertWithinCap(dto.durationMs); + + const timeZone = isValidIanaTimeZone(dto.timeZone) ? dto.timeZone : 'UTC'; + const todayIso = civilDateInZone(new Date(), timeZone); + + const { asr, extraction } = this.buildProviders(profile); + + // Stage 1 — audio never touches disk and is not retained beyond this call. + let transcript: string; + let asrCost: number | null = null; + let asrSeconds: number | null = null; + try { + const result = await asr.transcribe( + { data: dto.audio, format: dto.format }, + catalogLocale, + signal, + ); + transcript = result.text; + asrCost = result.usage.costUsd; + asrSeconds = result.usage.seconds; + } catch (error) { + throw this.toAppException(error, 'asr'); + } + + // durationMs is client-reported, so not enforcement. usage.seconds is the vendor's own + // measurement — a client under-reporting to slip past the cap is caught here, after the + // ASR spend but before the more expensive extraction call. + if (asrSeconds != null) { + this.assertWithinCap(asrSeconds * 1000); + } + + if (!transcript.trim()) { + throw new AppException( + ErrorCode.VOICE_NOTHING_RECOGNIZED, + HttpStatus.UNPROCESSABLE_ENTITY, + ); + } + + // Stage 2 — structure it. On failure the transcript still goes back to the client so + // the words the clinician already paid for are not lost (transcript salvage). + let resolved: ResolvedExtraction; + let llmCost: number | null = null; + try { + // Inside the try: the transcript is already paid for, so a catalog/DB failure here + // must still salvage it rather than becoming a generic 500 that throws it away. + const catalog = await this.buildCatalog(organizationId, catalogLocale); + const result = await extraction.extract( + transcript, + catalog, + catalogLocale, + signal, + ); + llmCost = result.costUsd; + resolved = resolveVoiceIntent(result.intent, { + todayIso, + weekStartJs: weekStartForLocale(catalogLocale), + treatmentTypeCodes: new Set(catalog.treatmentTypes.map((t) => t.code)), + prosthesisTypeCodes: new Set( + catalog.prosthesisTypes.map((t) => t.code), + ), + linkedLabIds: new Set(catalog.labs.map((l) => l.id)), + }); + } catch (error) { + throw this.toAppException(error, 'extraction', transcript); + } + + this.logTelemetry({ + locale: catalogLocale, + durationMs: dto.durationMs, + elapsedMs: Date.now() - startedAt, + asrCost, + llmCost, + resolved, + }); + + return { ...resolved, transcript }; + } + + private assertOrganization(user: { organizationId?: string }): string { + if (!user?.organizationId) { + throw new AppException( + ErrorCode.AUTH_ORG_NOT_SELECTED, + HttpStatus.BAD_REQUEST, + ); + } + return user.organizationId; + } + + private async assertCanEditTreatment(userId: string, organizationId: string) { + const membership = await this.prisma.membership.findFirst({ + where: { + userId, + organizationId, + OR: [{ isOwner: true }, { isActive: true }], + }, + include: { + permissions: { include: { permission: true } }, + organization: { include: { type: true, plan: true } }, + }, + }); + if (!membership) { + throw new AppException( + ErrorCode.PERMISSION_NOT_MEMBER, + HttpStatus.FORBIDDEN, + ); + } + if (!hasEffectivePermission(membership, 'TAB_TREATMENT_EDIT')) { + throw new AppException( + ErrorCode.PERMISSION_EDIT_TREATMENTS, + HttpStatus.FORBIDDEN, + ); + } + } + + private resolveProfile(locale: string): VoiceProfile { + const voice = this.voiceConfig; + const profile = voice.profiles[locale]; + if (!profile || !voice.openRouter.apiKey) { + throw new AppException( + ErrorCode.VOICE_NOT_AVAILABLE, + HttpStatus.FORBIDDEN, + ); + } + return profile; + } + + /** + * The client auto-stops at maxMs and only then measures, so a capped recording always + * reports slightly over. Without this tolerance every auto-stopped recording — the exact + * case the cap exists for — would be rejected as too long. + */ + private static readonly CAP_TOLERANCE_MS = 2_000; + + private assertWithinCap(durationMs: number) { + const max = this.voiceConfig.maxRecordingMs; + if (max != null && durationMs > max + VoiceService.CAP_TOLERANCE_MS) { + throw new AppException( + ErrorCode.VOICE_CLIP_TOO_LONG, + HttpStatus.PAYLOAD_TOO_LARGE, + ); + } + } + + private buildProviders(profile: VoiceProfile): { + asr: AsrProvider; + extraction: ExtractionProvider; + } { + const { apiKey, baseUrl } = this.voiceConfig.openRouter; + const base = { apiKey: apiKey!, baseUrl }; + return { + asr: new OpenRouterAsrProvider({ ...base, model: profile.asr.model }), + extraction: new OpenRouterExtractionProvider({ + ...base, + model: profile.llm.model, + }), + }; + } + + /** Codes with labels in the actor's locale, plus the clinic's linked labs. */ + private async buildCatalog( + organizationId: string, + locale: string, + ): Promise { + const [treatmentTypes, prosthesisTypes, labs] = await Promise.all([ + this.treatmentCatalog.list(locale, null), + this.prosthesisCatalog.list(locale), + this.listLinkedLabs(organizationId), + ]); + + return { + treatmentTypes: treatmentTypes + .filter((entry) => entry.availableInTreatment) + .map((entry) => ({ code: entry.code, label: entry.label })), + prosthesisTypes: prosthesisTypes.map((entry) => ({ + code: entry.code, + label: entry.label, + })), + labs, + }; + } + + private async listLinkedLabs( + organizationId: string, + ): Promise<{ id: string; name: string }[]> { + const [linksA, linksB] = await Promise.all([ + this.prisma.organizationLink.findMany({ + where: { organizationAId: organizationId, status: LinkStatus.ACTIVE }, + include: { organizationB: { select: { id: true, name: true } } }, + }), + this.prisma.organizationLink.findMany({ + where: { organizationBId: organizationId, status: LinkStatus.ACTIVE }, + include: { organizationA: { select: { id: true, name: true } } }, + }), + ]); + return [ + ...linksA.map((l) => ({ + id: l.organizationB.id, + name: l.organizationB.name, + })), + ...linksB.map((l) => ({ + id: l.organizationA.id, + name: l.organizationA.name, + })), + ]; + } + + private toAppException( + error: unknown, + stage: 'asr' | 'extraction', + transcript?: string, + ): AppException { + if (error instanceof Error && error.name === 'AbortError') { + // The clinician cancelled; not a failure worth a translated message. + return new AppException(ErrorCode.BAD_REQUEST, HttpStatus.BAD_REQUEST); + } + if (error instanceof VoiceProviderError) { + this.logger.warn(`voice ${stage} failed: ${error.message}`); + } else { + this.logger.error(`voice ${stage} failed unexpectedly`, error as Error); + } + const code = + stage === 'asr' + ? ErrorCode.VOICE_ASR_FAILED + : ErrorCode.VOICE_EXTRACT_FAILED; + return new AppException( + code, + HttpStatus.BAD_GATEWAY, + transcript ? { transcript } : undefined, + ); + } + + /** Structured and patient-free: never the transcript, never audio, never a patient id. */ + private logTelemetry(input: { + locale: string; + durationMs: number; + elapsedMs: number; + asrCost: number | null; + llmCost: number | null; + resolved: ResolvedExtraction; + }) { + const { resolved } = input; + this.logger.log( + JSON.stringify({ + event: 'voice.extract', + locale: input.locale, + clipMs: input.durationMs, + elapsedMs: input.elapsedMs, + costUsd: (input.asrCost ?? 0) + (input.llmCost ?? 0), + resolvedFields: { + treatmentType: resolved.treatmentType != null, + teeth: resolved.teeth.length, + comment: resolved.comment != null, + prosthesisComplete: resolved.prosthesis?.complete ?? null, + lab: resolved.labId != null, + dueDate: resolved.dueDate != null, + }, + unresolvedCount: resolved.unresolved.length, + }), + ); + } +} diff --git a/backend/src/modules/voice/voice.types.ts b/backend/src/modules/voice/voice.types.ts new file mode 100644 index 0000000..f557110 --- /dev/null +++ b/backend/src/modules/voice/voice.types.ts @@ -0,0 +1,85 @@ +import type { Arch, PatientSide } from '../../common/fdi'; + +/** + * What the extraction model is allowed to return. + * + * The model emits *intents*, never resolved values: no FDI codes, no ISO dates. Pure, + * unit-tested resolvers turn intents into domain values, so the two highest-consequence + * mappings — quadrant mirroring and Jalali conversion — are testable rather than hopeful. + */ + +/** A single spoken tooth reference. `spoken` is the transcript span, echoed back to the user. */ +export type ToothIntent = + | { kind: 'explicit'; fdi: string; spoken: string } + | { + kind: 'positional'; + arch: Arch; + side: PatientSide; + /** 1 = central incisor … 8 = third molar. */ + position: number; + spoken: string; + }; + +/** A spoken deadline. The model never does calendar arithmetic. */ +export type DueIntent = + | { kind: 'weekday'; weekday: Weekday; which: 'this' | 'next' } + | { kind: 'offset'; unit: 'day' | 'week' | 'month'; amount: number } + | { kind: 'jalali'; jy: number; jm: number; jd: number } + | { kind: 'gregorian'; y: number; m: number; d: number }; + +export const WEEKDAYS = [ + 'saturday', + 'sunday', + 'monday', + 'tuesday', + 'wednesday', + 'thursday', + 'friday', +] as const; + +export type Weekday = (typeof WEEKDAYS)[number]; + +/** Two teeth defining an inclusive connected (bridge) span. */ +export type ConnectedSpanIntent = { from: ToothIntent; to: ToothIntent }; + +export type ProsthesisIntent = { + /** Catalog code applied to every tooth unless overridden. */ + defaultType: string | null; + overrides: { tooth: ToothIntent; type: string }[]; +}; + +export type VoiceIntent = { + treatmentType: string | null; + teeth: ToothIntent[]; + connectedSpans: ConnectedSpanIntent[]; + comment: string | null; + prosthesis: ProsthesisIntent | null; + /** Must be one of the linked-lab ids supplied in the prompt, or null. */ + labId: string | null; + /** False when the spoken name only approximately matched — the UI then requires an explicit tick. */ + labMatchExact: boolean; + due: DueIntent | null; +}; + +/** Why a spoken item could not be turned into a domain value. Shown to the user. */ +export type UnresolvedReason = + | 'not_permanent_tooth' + | 'position_out_of_range' + /** A position was understood but no quadrant was spoken — four teeth match. */ + | 'tooth_missing_quadrant' + | 'malformed' + | 'span_not_same_arch' + | 'unknown_catalog_code' + | 'tooth_not_selected' + | 'invalid_date'; + +export type UnresolvedItem = { + /** The transcript span that could not be resolved, so the user can see what was heard. */ + spoken: string; + reason: UnresolvedReason; + /** + * FDI codes still consistent with what was heard — "دو" leaves four, "دو بالا" two. Only + * `tooth_missing_quadrant` carries them; the sheet offers them as chips. + */ + candidates?: string[]; +}; diff --git a/docs/specs/voice-treatment-entry/spec.md b/docs/specs/voice-treatment-entry/spec.md new file mode 100644 index 0000000..3bb3f3c --- /dev/null +++ b/docs/specs/voice-treatment-entry/spec.md @@ -0,0 +1,802 @@ +# Voice treatment entry + +**Status:** Implemented on `feat/voice-treatment-entry`, with one specified piece missing — +the transcript-salvage dialog (§9). First live test on 2026-08-21 sent the tooth path back +for revision — a spoken number is now read as its FDI code (§6). +Still blocked on the ASR spike (§11 item 1) before it is trustworthy in front of patients +**Area:** Treatment workspace (CLINIC orgs) +**Created:** 2026-08-20 + +Fill a `TreatmentDetail` — including its lab dispatch — by speaking, instead of by +tapping through the type dropdown, the FDI chart, the prosthesis wizard and the lab +picker. + +--- + +## 1. Goal + +A clinician on the Treatment tab taps a microphone, describes the treatment for the +already-selected patient in one utterance, and is shown a **review sheet** of what was +understood. Fields they tick are applied to the open detail chip. Nothing is written to +the form without confirmation. + +### In scope + +One recording produces **exactly one** `TreatmentDetail`, and may fill every field of it: + +| Field | Source | +|---|---| +| `treatmentType` | catalog code, matched against locale labels | +| `teeth` | FDI codes, via tooth-intent resolver | +| `toothSelectionGroups` | connected (bridge) / single spans | +| `comment` | cleaned dictated notes | +| lab: `prosthesisTypeCode` per tooth | default type + per-tooth overrides | +| lab: `destinationOrganizationId` | matched against the clinic's linked labs | +| lab: `dueDate` | via due-date intent resolver | + +### Out of scope (v1) + +- Multiple detail chips from one recording. +- `attachmentIds` — files cannot be dictated. +- Editing an existing detail by voice ("no, make that 15"). Confirming a recording always + creates a **new** detail (see §2). +- Creating the treatment or selecting the patient by voice. A patient is already + selected; voice only fills the form. +- Lab-side (`LAB` org) usage. Clinic only. + +--- + +## 2. User flow and UI integration + +### The control: Add detail, split + +The `Add detail` button gains a second segment holding the mic. The halves read as +siblings — both end in a new detail — but they are **independent actions**: + +- **Add half — unchanged.** Same `onAddDetail`, same seeding, same `setEntryStep`. It + gains a neighbour and nothing else. Its logic is not modified, wrapped or made + conditional. +- **Mic half** — starts a recording. Nothing is created until confirm (below). + +The `Add detail` ` + {voice ? ( + + ) : ( + + )} + {voice ? : null} +
{details.map((d, idx) => { const detailLocked = isDetailLocked(d); @@ -312,3 +334,77 @@ function NotesField({ ); } + +/** + * "Add detail", split into two segments with the microphone at the logical end. + * + * Built like the detail chip's trash affordance in this same file — a wrapper holding two + * raw ` + +
+ ); +} diff --git a/frontend/src/components/ui/treatment/TreatmentWorkspace.tsx b/frontend/src/components/ui/treatment/TreatmentWorkspace.tsx index 0d22bba..d7b39c2 100644 --- a/frontend/src/components/ui/treatment/TreatmentWorkspace.tsx +++ b/frontend/src/components/ui/treatment/TreatmentWorkspace.tsx @@ -29,6 +29,14 @@ import { import { appointmentsApi } from '@/lib/api/appointments'; import { treatmentCatalogApi } from '@/lib/api/treatment-catalog'; import { prosthesisCatalogApi } from '@/lib/api/prosthesis-catalog'; +import { voiceApi } from '@/lib/api/voice'; +import { useVoiceCapture } from '@/lib/voice/useVoiceCapture'; +import { VoiceReviewSheet } from '@/components/ui/treatment/VoiceReviewSheet'; +import type { + VoiceApplySelection, + VoiceAvailability, + VoiceExtractionResult, +} from '@/types/voice'; import { treatmentsApi } from '@/lib/api/treatments'; import { notificationsApi } from '@/lib/api/notifications'; import { pickAutoAppointment } from '@/components/shared/treatmentSelection'; @@ -464,6 +472,11 @@ export function TreatmentWorkspace({ const [showWholeTreatmentPlan, setShowWholeTreatmentPlan] = useState(false); const [entryStep, setEntryStep] = useState('treatment'); + + const [voiceAvailability, setVoiceAvailability] = useState(null); + const [voiceResult, setVoiceResult] = useState(null); + + const isDetailLocked = useCallback( (detail: TreatmentDetailDraft) => labCaseDrafts.some((lc) => lc.sentAt && lc.detailClientId === detail.clientId), @@ -489,6 +502,21 @@ export function TreatmentWorkspace({ [appointments, selectedAppointmentId], ); + + const voice = useVoiceCapture({ + // The locale the clinician is actually reading and speaking in. Sent explicitly so + // the server's ASR hint, catalog labels and week start match what the microphone was + // offered for — req.user.language can drift from the URL locale. + locale, + maxMs: voiceAvailability?.maxRecordingMs ?? null, + onExtracted: setVoiceResult, + onError: (error) => showError(getUserFacingError(error, tErrors, t('voiceFailed'))), + }); + + /** Absence is the unavailable state — the Add button then renders unsplit. */ + const voiceForEditor = + voiceAvailability?.enabled && voiceAvailability.locales.includes(locale) ? voice : undefined; + const selectedStandalone = useMemo( () => standaloneTreatments.find((t) => t.id === selectedStandaloneId) ?? null, [standaloneTreatments, selectedStandaloneId], @@ -927,12 +955,18 @@ export function TreatmentWorkspace({ let cancelled = false; void (async () => { try { - const [orgsResponse, catalogResponse, prosthesisResponse] = await Promise.all([ - treatmentsApi.listLinkedOrganizations(), - treatmentCatalogApi.list(), - prosthesisCatalogApi.list(), - ]); + const [orgsResponse, catalogResponse, prosthesisResponse, voiceResponse] = + await Promise.all([ + treatmentsApi.listLinkedOrganizations(), + treatmentCatalogApi.list(), + prosthesisCatalogApi.list(), + // Voice availability comes from the API, not a NEXT_PUBLIC_* var: those are + // baked in at build time, so enabling a locale would need a frontend rebuild. + // A failure here must not take the whole treatment tab down with it. + voiceApi.availability().catch(() => null), + ]); if (cancelled) return; + setVoiceAvailability(voiceResponse?.data ?? null); setOrgs(orgsResponse.data); setTreatmentCatalog(catalogResponse.data); setProsthesisCatalog(prosthesisResponse.data); @@ -1935,6 +1969,114 @@ export function TreatmentWorkspace({ ], ); + /** + * Voice entry. + * + * Confirm always appends a NEW detail — it never edits an existing one, and never + * touches onAddDetail. Nothing is created until this runs, so cancelling or a failed + * recording leaves the chip strip untouched. + */ + const applyVoiceResult = useCallback( + (result: VoiceExtractionResult, selection: VoiceApplySelection) => { + const detail = newDetail( + defaultTreatmentTypeForAppointment(selectedAppointment?.purpose, treatmentCatalog), + ); + + // Ticked rows land on top of the seeded defaults, so unticking the type row leaves + // the appointment-purpose default rather than a blank. + if (selection.treatmentType && result.treatmentType) { + detail.treatmentType = result.treatmentType; + } + if (selection.teeth) { + detail.teeth = [...result.teeth]; + detail.toothSelectionGroups = result.toothSelectionGroups.map((group) => ({ + ...group, + teeth: [...group.teeth], + })); + } + if (selection.comment && result.comment) { + detail.comment = result.comment; + } + + const nextDetails = [...detailsRef.current, detail]; + setDetails(nextDetails); + // persistDraft reads detailsRef, and setDetails has not rendered yet. + detailsRef.current = nextDetails; + setActiveDetailId(detail.clientId); + setEntryStep('treatment'); + + // Lab-side rows ride on a lab case draft keyed by the detail's *client* id, so a + // brand-new unsaved detail can still carry one; it is persisted after the detail is. + const wantsLabDraft = + (selection.prosthesis && result.prosthesis) || + (selection.lab && result.labId) || + (selection.dueDate && result.dueDate); + + if (wantsLabDraft) { + const draft = newLabCaseDraft(); + draft.detailClientId = detail.clientId; + if (selection.lab && result.labId) { + draft.destinationOrganizationId = result.labId; + } + if (selection.dueDate && result.dueDate) { + draft.dueDate = result.dueDate; + } + if (selection.prosthesis && result.prosthesis) { + // byTooth keys are plain strings; the group's teeth are FdiToothId. + const groupOf = (tooth: string) => + result.toothSelectionGroups.find((group) => + (group.teeth as readonly string[]).includes(tooth), + )?.groupId ?? ''; + // Only teeth that actually landed on the detail. Unticking "teeth" while + // leaving "prosthesis" ticked would otherwise attach prosthesis rows for teeth + // the treatment does not contain — nothing downstream filters them, and they + // would reach task generation as work for teeth nobody is treating. + const detailTeeth = new Set(detail.teeth); + draft.toothProsthesis = Object.entries(result.prosthesis.byTooth) + .filter(([tooth]) => detailTeeth.has(tooth)) + .map(([tooth, prosthesisTypeCode]) => ({ + detailClientId: detail.clientId, + tooth, + prosthesisTypeCode, + selectionGroupId: groupOf(tooth), + })); + } + const updatedLabCases = [...labCaseDrafts, draft]; + setLabCaseDrafts(updatedLabCases); + + // Autosave only watches `details`, so a lab draft left in state alone loses the + // lab, the due date and the prosthesis map on reload — silently, because the + // detail itself survives. + void (async () => { + try { + const saved = await persistDraft({ force: true }); + // persistDraft returns a *preview* when the details are not persistable — one + // blank detail is enough — and a preview's detail id falls back to the client + // id. Check what came back, not the precondition, so this holds for every early + // return persistDraft has. + const savedDetail = saved.details.find((d) => d.clientId === detail.clientId); + if (!savedDetail?.id || savedDetail.id === detail.clientId) return; + await persistLabCases(saved, updatedLabCases); + } catch (error: unknown) { + showError(getUserFacingError(error, tErrors, t('errorSaveLabShipments'))); + } + })(); + } + + setVoiceResult(null); + }, + [ + labCaseDrafts, + persistDraft, + persistLabCases, + selectedAppointment?.purpose, + showError, + t, + tErrors, + treatmentCatalog, + ], + ); + const handleRemoveDetail = useCallback( (detailClientId: string) => { if (!canEditTreatmentForDay) return; @@ -2478,6 +2620,7 @@ export function TreatmentWorkspace({ }} showChrome showFields={entryStep === 'treatment'} + voice={voiceForEditor} chartLocked={ entryStep === 'treatment' && !activeTypeSelected && !showWholeTreatmentPlan } @@ -2725,6 +2868,16 @@ export function TreatmentWorkspace({ )} + {voiceResult ? ( + applyVoiceResult(applied, selection)} + onDiscard={() => setVoiceResult(null)} + /> + ) : null} ); } diff --git a/frontend/src/components/ui/treatment/VoiceRecordingBar.tsx b/frontend/src/components/ui/treatment/VoiceRecordingBar.tsx new file mode 100644 index 0000000..a1b65a1 --- /dev/null +++ b/frontend/src/components/ui/treatment/VoiceRecordingBar.tsx @@ -0,0 +1,90 @@ +'use client'; + +import { useTranslations } from 'next-intl'; +import { Loader2, X } from 'lucide-react'; +import type { VoiceCaptureState } from '@/lib/voice/useVoiceCapture'; + +const METER_BARS = 9; + +function formatElapsed(ms: number): string { + const totalSeconds = Math.floor(Math.max(0, ms) / 1000); + const minutes = Math.floor(totalSeconds / 60); + const seconds = totalSeconds % 60; + return `${minutes}:${String(seconds).padStart(2, '0')}`; +} + +/** + * Live recording / processing strip. + * + * Sits between the header row and the chip strip rather than inside the segmented control: + * the header is `sm:justify-between`, so growing the button mid-recording would shift the + * whole row. + */ +export function VoiceRecordingBar({ voice }: { voice: VoiceCaptureState }) { + const t = useTranslations('treatment'); + + if (voice.phase === 'idle') return null; + + const isRecording = voice.phase === 'recording'; + + return ( +
+ {isRecording ? ( + <> + + + {formatElapsed(voice.elapsedMs)} + {voice.maxMs != null ? ( + / {formatElapsed(voice.maxMs)} + ) : null} + + + + ) : ( + <> + + {t('voiceProcessing')} + + )} + + +
+ ); +} + +/** Proves the microphone is actually hearing something — silence looks identical otherwise. */ +function LevelMeter({ level }: { level: number }) { + return ( + + {Array.from({ length: METER_BARS }, (_, index) => { + // Bars light up left to right as the level rises, with a floor so the meter never + // looks dead while a quiet voice is still being captured. + const threshold = (index + 1) / METER_BARS; + const active = level >= threshold * 0.9; + const height = active ? 30 + threshold * 70 : 20; + return ( + + ); + })} + + ); +} diff --git a/frontend/src/components/ui/treatment/VoiceReviewSheet.tsx b/frontend/src/components/ui/treatment/VoiceReviewSheet.tsx new file mode 100644 index 0000000..ef5da7a --- /dev/null +++ b/frontend/src/components/ui/treatment/VoiceReviewSheet.tsx @@ -0,0 +1,306 @@ +'use client'; + +import { useMemo, useState } from 'react'; +import { useTranslations } from 'next-intl'; +import { AlertTriangle } from 'lucide-react'; +import { Button } from '@/components/ui/shared/Button'; +import { Checkbox } from '@/components/ui/shared/Checkbox'; +import { + ResponsiveDialogOverlay, + ResponsiveDialogPanel, +} from '@/components/ui/shared/ResponsiveDialog'; +import { FdiToothChart } from '@/components/ui/treatment/FdiToothChart'; +import { + connectedTeethFromResult, + countSelected, + hasAnythingToApply, + initialVoiceSelection, + voiceRowAvailability, + withChosenTeeth, +} from '@/components/treatment/voiceReviewRows'; +import { useLocale } from 'next-intl'; +import { useAppFormatters } from '@/lib/hooks/useAppFormatters'; +import type { TreatmentCatalogEntry } from '@/types/treatment-catalog'; +import type { ProsthesisCatalogEntry } from '@/types/treatment-catalog'; +import type { FdiToothId, LinkedOrganizationOption } from '@/types/treatment'; +import type { VoiceApplySelection, VoiceExtractionResult } from '@/types/voice'; + +interface VoiceReviewSheetProps { + result: VoiceExtractionResult; + treatmentCatalog: TreatmentCatalogEntry[]; + prosthesisCatalog: ProsthesisCatalogEntry[]; + labs: LinkedOrganizationOption[]; + /** The result is handed back because the sheet may have added teeth the model missed. */ + onApply: (selection: VoiceApplySelection, result: VoiceExtractionResult) => void; + onDiscard: () => void; +} + +/** + * Confirmation step between the model's output and the form. + * + * Modal on desktop, bottom sheet on mobile — an overlay and not a route, because navigating + * would unmount TreatmentWorkspace and destroy the in-progress draft. + */ +export function VoiceReviewSheet({ + result, + treatmentCatalog, + prosthesisCatalog, + labs, + onApply, + onDiscard, +}: VoiceReviewSheetProps) { + const t = useTranslations('treatment'); + const locale = useLocale(); + const { formatDate } = useAppFormatters(); + const [selection, setSelection] = useState(() => + initialVoiceSelection(result), + ); + const [chosen, setChosen] = useState([]); + + // Everything below renders from `effective`, never from `result` — a tooth picked from + // the candidate chips has to reach the rows, the chart and the apply count alike. + const effective = useMemo(() => withChosenTeeth(result, chosen), [result, chosen]); + + const available = useMemo(() => voiceRowAvailability(effective), [effective]); + const connectedTeeth = useMemo(() => connectedTeethFromResult(effective), [effective]); + const selectedTeeth = useMemo(() => new Set(effective.teeth), [effective.teeth]); + const nothingToApply = !hasAnythingToApply(effective); + const selectedCount = countSelected(selection, available); + + const pickCandidate = (tooth: FdiToothId) => { + const nextChosen = chosen.includes(tooth) + ? chosen.filter((t) => t !== tooth) + : [...chosen, tooth]; + setChosen(nextChosen); + setSelection((prev) => ({ + ...prev, + // The teeth row starts unticked whenever the recording produced no teeth of its own, + // and a picked tooth that is not ticked applies nothing. + teeth: true, + // A picked tooth has no prosthesis type, so the map is no longer shippable — leaving the + // row ticked would apply a map dispatch rejects. Only ever unticks; re-ticking is the + // clinician's call. + prosthesis: + prev.prosthesis && + withChosenTeeth(result, nextChosen).prosthesis?.complete !== false, + })); + }; + + const labelFor = (code: string | null, catalog: { code: string; label: string }[]) => + catalog.find((entry) => entry.code === code)?.label ?? code ?? ''; + + const toggle = (key: keyof VoiceApplySelection) => (checked: boolean) => + setSelection((prev) => ({ ...prev, [key]: checked })); + + return ( + + +

+ {t('voiceReviewTitle')} +

+ +

+ {effective.transcript} +

+ + {nothingToApply ? ( +

{t('voiceNothingExtracted')}

+ ) : ( +
+ {available.treatmentType ? ( + + + {labelFor(effective.treatmentType, treatmentCatalog)} + + + ) : null} + + {available.teeth ? ( + +
+ +
+ + ) : null} + + {available.comment ? ( + + + {effective.comment} + + + ) : null} + + {available.prosthesis && effective.prosthesis ? ( + + + {Object.entries(effective.prosthesis.byTooth) + .map( + ([tooth, code]) => `${tooth}: ${labelFor(code, prosthesisCatalog)}`, + ) + .join(' · ')} + + + ) : null} + + {available.lab ? ( + + + {labs.find((lab) => lab.id === effective.labId)?.name ?? effective.labId} + + + ) : null} + + {available.dueDate && effective.dueDate ? ( + + + {formatDate(civilDateToLocalDate(effective.dueDate))} + + + ) : null} +
+ )} + + {effective.unresolved.length > 0 ? ( +
+

+ {t('voiceNotUnderstood')} +

+
    + {effective.unresolved.map((item, index) => ( +
  • + {item.spoken ? `“${item.spoken}” — ` : ''} + {t(`voiceUnresolved.${item.reason}`)} + {item.candidates && item.candidates.length > 0 ? ( + + {t('voicePickTooth')} + {item.candidates.map((tooth) => { + const picked = chosen.includes(tooth as FdiToothId); + return ( + + ); + })} + + ) : null} +
    • + ))} +
+
+ ) : null} + +
+ + +
+ + + ); +} + +function Row({ + label, + checked, + onChange, + warning, + children, +}: { + label: string; + checked: boolean; + onChange: (checked: boolean) => void; + warning?: string; + children: React.ReactNode; +}) { + return ( +
+ +
{children}
+ {warning ? ( +

+ + {warning} +

+ ) : null} +
+ ); +} + +/** + * `new Date('2025-10-17')` parses a civil date as UTC midnight, which renders as the 16th + * west of Greenwich. Build it from its parts so it means the same day everywhere. + */ +function civilDateToLocalDate(iso: string): Date { + const [year, month, day] = iso.split('-').map(Number); + return new Date(year, (month ?? 1) - 1, day ?? 1); +} + +/** Locale-aware list separator — the Arabic comma is not correct in en or nl. */ +function formatToothList(teeth: readonly string[], locale: string): string { + try { + return new Intl.ListFormat(locale, { style: 'short', type: 'unit' }).format([...teeth]); + } catch { + return teeth.join(', '); + } +} diff --git a/frontend/src/lib/api/voice.ts b/frontend/src/lib/api/voice.ts new file mode 100644 index 0000000..f0e1eaf --- /dev/null +++ b/frontend/src/lib/api/voice.ts @@ -0,0 +1,29 @@ +import { apiClient } from './client'; +import type { VoiceAvailability, VoiceExtractionResult } from '@/types/voice'; + +export interface ExtractVoicePayload { + /** Base64 audio, no data: prefix. */ + audio: string; + format: string; + /** IANA zone — the server derives "today" from it for relative deadlines. */ + timeZone: string; + durationMs: number; + /** Locale the clinician is speaking; the server uses it rather than the stored one. */ + locale: string; +} + +export const voiceApi = { + availability: async (): Promise<{ success: boolean; data: VoiceAvailability }> => { + const response = await apiClient.get('/voice/availability'); + return response.data; + }, + + extract: async ( + payload: ExtractVoicePayload, + signal?: AbortSignal, + ): Promise<{ success: boolean; data: VoiceExtractionResult }> => { + // Forwarded so cancelling closes the connection; the controller turns that into an abort. + const response = await apiClient.post('/voice/extract', payload, { signal }); + return response.data; + }, +}; diff --git a/frontend/src/lib/voice/audioFormat.ts b/frontend/src/lib/voice/audioFormat.ts new file mode 100644 index 0000000..4b94945 --- /dev/null +++ b/frontend/src/lib/voice/audioFormat.ts @@ -0,0 +1,60 @@ +/** Containers the backend accepts, in the order we prefer to record them. */ +const PREFERRED_MIME_TYPES = [ + 'audio/webm;codecs=opus', + 'audio/webm', + 'audio/mp4', + 'audio/aac', + 'audio/ogg;codecs=opus', + 'audio/ogg', +] as const; + +/** + * Pick a container this browser can record AND the backend accepts. Chrome and Android give + * webm/opus, Safari and iPad mp4/aac; both go to the vendor unmodified, so there is no + * transcode step and the list is an intersection, not a preference. + */ +export function pickRecordingMimeType(): string | null { + if (typeof MediaRecorder === 'undefined') return null; + if (typeof MediaRecorder.isTypeSupported !== 'function') { + // Safari <14.1 shipped MediaRecorder without the feature check; let it choose. + return ''; + } + for (const type of PREFERRED_MIME_TYPES) { + if (MediaRecorder.isTypeSupported(type)) return type; + } + return null; +} + +/** `audio/webm;codecs=opus` → `webm`, which is what the API's `format` field wants. */ +export function mimeTypeToFormat(mimeType: string): string { + const base = mimeType.split(';')[0]?.trim().toLowerCase() ?? ''; + const subtype = base.startsWith('audio/') ? base.slice('audio/'.length) : base; + // Safari/iOS records `audio/mp4`, but the transcription endpoint's documented container + // list names m4a, not mp4. Same container; send the name the vendor documents, so iPad + // recordings do not fail while Chrome's webm works. + if (subtype === 'x-m4a' || subtype === 'm4a' || subtype === 'mp4') return 'm4a'; + if (subtype === 'mpeg') return 'mp3'; + return subtype || 'webm'; +} + +/** Blob → base64 without the `data:` prefix, which the API does not want. */ +export async function blobToBase64(blob: Blob): Promise { + const buffer = await blob.arrayBuffer(); + let binary = ''; + const bytes = new Uint8Array(buffer); + // Chunked to avoid blowing the argument limit on a two-minute recording. + const chunkSize = 0x8000; + for (let i = 0; i < bytes.length; i += chunkSize) { + binary += String.fromCharCode(...bytes.subarray(i, i + chunkSize)); + } + return btoa(binary); +} + +export function isMediaRecorderSupported(): boolean { + return ( + typeof window !== 'undefined' && + typeof MediaRecorder !== 'undefined' && + typeof navigator !== 'undefined' && + Boolean(navigator.mediaDevices?.getUserMedia) + ); +} diff --git a/frontend/src/lib/voice/useVoiceCapture.ts b/frontend/src/lib/voice/useVoiceCapture.ts new file mode 100644 index 0000000..5c3161a --- /dev/null +++ b/frontend/src/lib/voice/useVoiceCapture.ts @@ -0,0 +1,300 @@ +'use client'; + +import { useCallback, useEffect, useRef, useState } from 'react'; +import { voiceApi } from '@/lib/api/voice'; +import type { ApiError } from '@/types/api'; +import type { VoiceExtractionResult, VoicePhase } from '@/types/voice'; +import { + blobToBase64, + isMediaRecorderSupported, + mimeTypeToFormat, + pickRecordingMimeType, +} from './audioFormat'; + +export interface UseVoiceCaptureOptions { + /** The locale the clinician is speaking, sent so the server does not have to guess. */ + locale: string; + /** null means uncapped; otherwise the recorder auto-stops here. */ + maxMs: number | null; + onExtracted: (result: VoiceExtractionResult) => void; + onError: (error: unknown) => void; +} + +export interface VoiceCaptureState { + phase: VoicePhase; + elapsedMs: number; + /** 0..1, for the level meter — proves the microphone is actually hearing something. */ + level: number; + maxMs: number | null; + onStart: () => void; + onStop: () => void; + onCancel: () => void; +} + +const LEVEL_POLL_MS = 100; + +/** + * Client-side failures must be ApiError-shaped or getUserFacingError cannot resolve them + * and every one renders the generic fallback, leaving errors.VOICE_MIC_DENIED dead. + */ +function clientError(code: string): ApiError { + return { statusCode: 0, code }; +} + +/** + * Microphone capture for treatment voice entry. + * + * Lives in lib/ rather than in the editor: TreatmentDetailsEditor stays presentational + * and receives only a `voice` prop, so MediaRecorder and the API call never enter ui/. + */ +export function useVoiceCapture({ + locale, + maxMs, + onExtracted, + onError, +}: UseVoiceCaptureOptions): VoiceCaptureState { + const [phase, setPhase] = useState('idle'); + const [elapsedMs, setElapsedMs] = useState(0); + const [level, setLevel] = useState(0); + + const recorderRef = useRef(null); + const streamRef = useRef(null); + const chunksRef = useRef([]); + const startedAtRef = useRef(0); + const timerRef = useRef | null>(null); + const audioContextRef = useRef(null); + const abortRef = useRef(null); + /** Set when the user cancels, so the recorder's stop handler discards instead of sending. */ + const cancelledRef = useRef(false); + /** getUserMedia is async; without this a permission granted after unmount leaks the mic. */ + const mountedRef = useRef(true); + /** + * Set synchronously on click: `phase` only becomes 'recording' once getUserMedia resolves, + * so a second click during the permission prompt would orphan the first stream. + */ + const startingRef = useRef(false); + + const teardown = useCallback(() => { + if (timerRef.current) { + clearInterval(timerRef.current); + timerRef.current = null; + } + streamRef.current?.getTracks().forEach((track) => track.stop()); + streamRef.current = null; + void audioContextRef.current?.close().catch(() => undefined); + audioContextRef.current = null; + recorderRef.current = null; + setLevel(0); + }, []); + + // Releasing the microphone on unmount matters: the browser shows a recording indicator + // for as long as the track is live, and an orphaned one looks like the app is listening. + useEffect(() => { + // Re-armed on every mount: React StrictMode runs mount → unmount → mount in dev, and + // a ref that is only ever set false would leave the hook permanently "unmounted". + mountedRef.current = true; + return () => { + mountedRef.current = false; + cancelledRef.current = true; + abortRef.current?.abort(); + try { + recorderRef.current?.stop(); + } catch { + // already stopped + } + teardown(); + }; + }, [teardown]); + + const send = useCallback( + async (blob: Blob, mimeType: string, durationMs: number) => { + setPhase('processing'); + const controller = new AbortController(); + abortRef.current = controller; + try { + const audio = await blobToBase64(blob); + const response = await voiceApi.extract( + { + audio, + format: mimeTypeToFormat(mimeType), + timeZone: Intl.DateTimeFormat().resolvedOptions().timeZone, + durationMs, + locale, + }, + controller.signal, + ); + if (cancelledRef.current) return; + onExtracted(response.data); + } catch (error) { + if (cancelledRef.current || controller.signal.aborted) return; + onError(error); + } finally { + abortRef.current = null; + setPhase('idle'); + setElapsedMs(0); + } + }, + [locale, onExtracted, onError], + ); + + const stop = useCallback(() => { + // No recorder means nothing will fire `onstop`, so nothing else moves the phase. + if (!recorderRef.current) { + teardown(); + setPhase('idle'); + return; + } + try { + recorderRef.current.stop(); + } catch { + teardown(); + setPhase('idle'); + } + }, [teardown]); + + const onStart = useCallback(() => { + if (phase !== 'idle' || startingRef.current) return; + if (!isMediaRecorderSupported()) { + // VOICE_UNSUPPORTED_FORMAT, not MIC_DENIED: nothing asked for a permission yet, and + // blaming the microphone sends the clinician into site settings for no reason. Same + // for the two paths below. + onError(clientError('VOICE_UNSUPPORTED_FORMAT')); + return; + } + + cancelledRef.current = false; + startingRef.current = true; + + void (async () => { + try { + let stream: MediaStream; + try { + stream = await navigator.mediaDevices.getUserMedia({ audio: true }); + } catch { + // Permission refused, or no input device. Never a server round-trip. + onError(clientError('VOICE_MIC_DENIED')); + return; + } + + if (!mountedRef.current) { + // Permission resolved after the component went away — release it immediately + // rather than leaving the browser's recording indicator lit. + stream.getTracks().forEach((track) => track.stop()); + return; + } + + const mimeType = pickRecordingMimeType(); + if (mimeType === null) { + stream.getTracks().forEach((track) => track.stop()); + onError(clientError('VOICE_UNSUPPORTED_FORMAT')); + return; + } + + streamRef.current = stream; + chunksRef.current = []; + const recorder = new MediaRecorder(stream, mimeType ? { mimeType } : undefined); + recorderRef.current = recorder; + + recorder.ondataavailable = (event) => { + if (event.data.size > 0) chunksRef.current.push(event.data); + }; + recorder.onstop = () => { + const durationMs = Date.now() - startedAtRef.current; + const blob = new Blob(chunksRef.current, { type: recorder.mimeType || mimeType }); + teardown(); + if (cancelledRef.current || blob.size === 0) { + setPhase('idle'); + setElapsedMs(0); + return; + } + // Prefer what the recorder actually produced, then the blob's own type. Old + // Safari accepts no mimeType hint, and defaulting to webm would mislabel its + // mp4/aac clips as something they are not. + void send(blob, recorder.mimeType || blob.type || mimeType || 'audio/webm', durationMs); + }; + + attachLevelMeter(stream, audioContextRef, setLevel); + + startedAtRef.current = Date.now(); + recorder.start(); + setPhase('recording'); + setElapsedMs(0); + + timerRef.current = setInterval(() => { + const elapsed = Date.now() - startedAtRef.current; + setElapsedMs(elapsed); + // Auto-stop proceeds to processing with what was captured; discarding two + // minutes of dictation because a timer expired would be the worst failure. + if (maxMs != null && elapsed >= maxMs) stop(); + }, LEVEL_POLL_MS); + } catch { + // `new MediaRecorder()` and `recorder.start()` both throw on some browsers, and by + // then the stream is live — without this the mic indicator stays lit until unmount. + teardown(); + setPhase('idle'); + onError(clientError('VOICE_UNSUPPORTED_FORMAT')); + } finally { + startingRef.current = false; + } + })(); + }, [maxMs, onError, phase, send, stop, teardown]); + + const onCancel = useCallback(() => { + cancelledRef.current = true; + // Aborting closes the connection, which aborts the vendor call server-side. + abortRef.current?.abort(); + try { + recorderRef.current?.stop(); + } catch { + // already stopped + } + teardown(); + setPhase('idle'); + setElapsedMs(0); + }, [teardown]); + + return { phase, elapsedMs, level, maxMs, onStart, onStop: stop, onCancel }; +} + +/** Drives the level meter from the live stream; failure here must not stop recording. */ +function attachLevelMeter( + stream: MediaStream, + contextRef: React.MutableRefObject, + setLevel: (value: number) => void, +) { + try { + const AudioContextCtor = + window.AudioContext ?? (window as { webkitAudioContext?: typeof AudioContext }).webkitAudioContext; + if (!AudioContextCtor) return; + + const context = new AudioContextCtor(); + contextRef.current = context; + const source = context.createMediaStreamSource(stream); + const analyser = context.createAnalyser(); + analyser.fftSize = 512; + source.connect(analyser); + + const data = new Uint8Array(analyser.frequencyBinCount); + // Sample every frame so a transient is not missed, publish at LEVEL_POLL_MS. The hook + // lives in TreatmentWorkspace, so an unthrottled setLevel is ~7,200 whole-tree renders + // across a two-minute recording. + let peakSinceEmit = 0; + let lastEmit = 0; + const tick = (now: number) => { + if (contextRef.current !== context || context.state === 'closed') return; + analyser.getByteTimeDomainData(data); + let peak = 0; + for (const sample of data) peak = Math.max(peak, Math.abs(sample - 128)); + peakSinceEmit = Math.max(peakSinceEmit, peak); + if (now - lastEmit >= LEVEL_POLL_MS) { + lastEmit = now; + setLevel(Math.min(1, peakSinceEmit / 128)); + peakSinceEmit = 0; + } + requestAnimationFrame(tick); + }; + requestAnimationFrame(tick); + } catch { + // A missing or blocked AudioContext costs the meter, not the recording. + } +} diff --git a/frontend/src/types/voice.ts b/frontend/src/types/voice.ts new file mode 100644 index 0000000..b77665d --- /dev/null +++ b/frontend/src/types/voice.ts @@ -0,0 +1,64 @@ +import type { FdiToothId, ToothSelectionGroup } from '@/types/treatment'; + +/** Mirrors the backend's ResolvedExtraction — values already resolved, plus what was not. */ + +export type VoiceUnresolvedReason = + | 'not_permanent_tooth' + | 'position_out_of_range' + | 'tooth_missing_quadrant' + | 'malformed' + | 'span_not_same_arch' + | 'unknown_catalog_code' + | 'tooth_not_selected' + | 'invalid_date'; + +export interface VoiceUnresolvedItem { + /** The transcript span that could not be resolved, so the clinician sees what was heard. */ + spoken: string; + reason: VoiceUnresolvedReason; + /** + * FDI codes still consistent with what was heard, when a choice would settle it — the + * review sheet offers them as chips. Only `tooth_missing_quadrant` carries these. + */ + candidates?: string[]; +} + +export interface VoiceProsthesisResult { + byTooth: Record; + /** False means the case cannot ship — every tooth needs a prosthesis type. */ + complete: boolean; + missingTeeth: FdiToothId[]; +} + +export interface VoiceExtractionResult { + transcript: string; + treatmentType: string | null; + teeth: FdiToothId[]; + toothSelectionGroups: ToothSelectionGroup[]; + comment: string | null; + prosthesis: VoiceProsthesisResult | null; + labId: string | null; + /** When false, the lab row must not tick itself — the name only approximately matched. */ + labMatchExact: boolean; + dueDate: string | null; + unresolved: VoiceUnresolvedItem[]; +} + +export interface VoiceAvailability { + enabled: boolean; + locales: string[]; + /** null means uncapped. */ + maxRecordingMs: number | null; +} + +/** Which review rows the clinician ticked. */ +export interface VoiceApplySelection { + treatmentType: boolean; + teeth: boolean; + comment: boolean; + prosthesis: boolean; + lab: boolean; + dueDate: boolean; +} + +export type VoicePhase = 'idle' | 'recording' | 'processing';