admin/dyolink
0
0
Fork 0
Code Issues 24 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

Merge pull request 'feat(treatment): add voice detail entry' (#65) from feat/voice-treatment-entry into master
Some checks failed
Registry — build, push, deploy / build-and-push (push) Successful in 7m32s
Details
Registry — build, push, deploy / deploy (push) Failing after 45s
Details

Browse Source 
Reviewed-on: http://host.docker.internal:3000/admin/dyolink/pulls/65
Reviewed-by: admin <admin@localhost>
This commit was merged in pull request #65.
This commit is contained in:
admin
2026-08-24 11:13:45 +03:30
parent dc9401e13f 8c16a46740
commit 2a0f9dbfee
48 changed files with 6570 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files

108
CLAUDE.md Normal file
View File

@@ -0,0 +1,108 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Read first
Project conventions already live in **`AGENTS.md`** (project map + per-feature quick-reference), **`.cursor/rules/*.mdc`** (short always-on / file-scoped rules), and **`.cursor/skills/*/SKILL.md`** (multi-step workflow playbooks). They are plain markdown — read the ones covering the area you touch **before** editing. This file covers only what those do not: commands and cross-cutting architecture.
Per `.cursor/rules/maintain-agent-docs.mdc`: when the user establishes a durable convention, update the matching `.mdc` rule or `SKILL.md` — not this file.
## Commands
There is **no root `package.json`**. Every npm command runs inside `backend/` or `frontend/`.
### Backend (`cd backend`)
| Command | Purpose |
|---|---|
| `npm run start:dev` | API on `http://localhost:3000/api`; Swagger `/api/docs`; AdminJS `/admin` |
| `npm run build` | **Verification gate for cross-cutting backend changes** |
| `npm test` | Jest (`src/**/*.spec.ts`) |
| `npm test -- lab-case-task.generator` | Single suite by path fragment |
| `npm test -- -t "merges teeth"` | Single test by name |
| `npm run test:e2e` | Jest with `test/jest-e2e.json` |
| `npm run lint` | ESLint with `--fix` |
| `docker compose -f docker-compose.postgres.yml up -d` | Dev Postgres (host port from `POSTGRES_PORT` in `.env`) |
| `npm run prisma:generate` / `prisma:migrate` / `prisma:seed` | Client, dev migration, reference-data upsert (seed never wipes) |
| `npx prisma migrate reset` | Dev clean slate — drop, re-migrate, re-seed. Never against staging/prod |
| `npm run prisma:wipe-app-data` / `prisma:reset-treatment` / `prisma:regenerate-tasks` | Targeted dev data scripts |
`DATABASE_URL` must use `localhost` when Nest runs on the host and Postgres in Docker.
### Frontend (`cd frontend`)
| Command | Purpose |
|---|---|
| `npm run dev` | Dev server on **3001** (3000 is the API) |
| `npx tsc --noEmit` | **Verification gate for any type or cross-cutting frontend change** |
| `npm run build` | Production build (`output: 'standalone'`) |
| `npm run lint` | ESLint via Next |
`NEXT_PUBLIC_*` values are baked in at build time — restart `npm run dev` after changing `.env.local`.
### Git
Do not commit, push, amend, force-push, or skip hooks unless the user explicitly asks.
## Architecture
Dental **clinic ↔ lab** platform. Every user acts inside one `Organization` whose `type` is `CLINIC` (patients, appointments, treatment) or `LAB` (cases, tasks). Most features exist only for one side.
### Request identity: cookie JWT carrying the selected org
There is no `Authorization` header. `JwtStrategy` reads the httpOnly **`accessToken` cookie**, and the JWT payload carries `organizationId` — the org the user currently acts as. `POST /auth/select-organization` re-issues the token with a different org, so **switching orgs means a new token**, and every service scopes queries by `req.user.organizationId`.
On 401 the axios interceptor (`frontend/src/lib/api/client.ts`) refreshes, **re-selects** the org from `localStorage.currentOrganizationId`, then retries the original request — skipping that dance for auth endpoints and public invitation routes. `frontend/src/proxy.ts` (the Next middleware, exported as `proxy`) is a separate, cookie-only route gate that redirects unauthenticated users to `/{locale}/login?from=…`.
### Permissions
`TAB_*_READ` / `TAB_*_EDIT` codes in `backend/src/common/permissions.ts`; **EDIT implies READ**. Owners get org-type defaults merged with stored grants — always resolve via `hasEffectivePermission` / `getEffectivePermissionNames` in `common/membership-permissions.ts`, never by reading `membership.permissions` directly. Controllers stack `JwtAuthGuard` + `ClinicOrgGuard`/`LabOrgGuard`; feature-specific checks belong in the **service**.
### Error contract (spans 3 layers — change all of them)
`AppException(ErrorCode.X)` → `HttpExceptionFilter` → `{ success: false, error: { code } }` → axios normalizes to `ApiError` → `getUserFacingError(err, tErrors, fallback)` resolves `errors.X` from the message files. Adding a user-facing failure means: a code in `common/errors/error-codes.ts`, the throw site, and an `errors.X` key in **all three** of `frontend/messages/{en,fa,nl}.json`. Never throw raw English Nest exceptions for user-facing failures.
### The core domain pipeline
```
Appointment ─┐
├→ Treatment (patient + day) → TreatmentDetail (treatment type + selected teeth)
Walk-in ─────┘ │
│ "send to lab" (clinic side)
▼
LabCase + LabCaseToothProsthesis (per tooth, grouped by sourceKey)
│ generateLabCaseTasks()
▼
ProsthesisType → ProsthesisTypeStep → LabWorkflowStep ⇒ LabCaseTask rows
│
▼
LAB org: Cases tab + Tasks tab
```
`backend/src/modules/cases/lab-case-task.generator.ts` is the expansion point: it is **idempotent** (returns early if tasks exist) and drives the entire lab-side task list from catalog data. Teeth carry `selectionGroupId` so bridges/connected units survive into task grouping. A `LabCase` can also be lab-origin (`LabCaseOrigin`), created without any clinic treatment.
Clinics may only dispatch to labs they are linked to: `OrganizationLink` (A↔B, `LinkStatus`), plus `OrganizationInvitation` for counterparts not yet on the platform — the invite flow writes both rows in one transaction and stores only the token hash.
### Catalog is code-based and DB-translated
`TreatmentType`, `ProsthesisType`, and `LabWorkflowStep` store a stable `code` and **no label**. Labels come from `CatalogTranslation(entityKind, entityCode, locale)` resolved by `CatalogLabelService` (falls back locale → `en` → humanized code). So: never hardcode a catalog label in backend code, and pass the actor's locale into anything that materializes labels (task generation does). Frontend colors/labels for these codes live in `components/shared/treatmentTypeDisplay.ts` and `components/treatment/prosthesisTypeDisplay.ts`.
### Realtime and unread state
`modules/notifications/user-notification.service.ts` writes `UserNotification` rows and pushes them through the Socket.IO transport in `backend/src/realtime/` (`emitToUserOrg` → `notification.created`). On the frontend a single `notification.created` event drives three things: the header bell inbox, sidebar **tab badges**, and a *soft* refresh of whatever list is currently open — soft meaning it must not remount components or clear an in-progress treatment draft. Unread is per-user cursor state (`LabCaseUserReadState`, `LabCaseUserTabReadState`) plus the `LabCaseActivity` log — badges clear on opening a case, not on visiting a tab.
### Layout conventions worth knowing before you create a file
- **Prisma lives outside `src/`**: `backend/prisma/` holds `schema.prisma`, migrations, seeds *and* `prisma.module.ts` / `prisma.service.ts` — hence imports like `../../../prisma/prisma.service`. Register new Nest modules in `app.module.ts`.
- **Frontend layering** (`.cursor/rules/frontend-components.mdc`): `app/**/page.tsx` is a thin wrapper only → route logic in `components/ui/{feature}/{Feature}Page.tsx` → JSX in `components/ui/**` → pure helpers in `components/{feature}/` or `components/shared/`. No JSX outside `ui/`, no pure helpers inside it.
- **i18n is mandatory, not a follow-up**: every user-visible string goes into `en.json`, `fa.json`, **and** `nl.json`. `fa` is RTL, so use logical `text-start`/`text-end`, never `text-left`/`text-right`. Dates/times/numbers go through `lib/i18n/format.ts`; form dates use `AppDateInput`, never a native date input.
- Treatment attachments are written to disk at `backend/uploads/treatments` relative to `process.cwd()`.
### Tests
Jest covers pure logic only — permission normalization, phone/timezone helpers, task generation, lab-send validation (7 suites in `backend/src/**`). There are no frontend tests; `npx tsc --noEmit` is the frontend gate.
## Deployment
Images are built on a dev machine and pulled by the server; Compose files and scripts are in `infrastructure/` (`docker-compose.{prod,staging,registry}.yml`). Full guide: `infrastructure/DEPLOY.md`. Root `README.md` covers the Docker Hub + Let's Encrypt path and the Gitea registry path. Frontend `NEXT_PUBLIC_*` are **build args** — changing the public domain requires rebuilding the frontend image.

25
backend/.env.example
View File

@@ -47,3 +47,28 @@ SMTP_PASSWORD=your_app_password
# SMS_IR_API_KEY=4QKMiSU4Kh7tWPLCdRMV0QpDh8WgF33YkWRS18BcG3vf4QHi
SMS_IR_API_KEY=lwbK7hxmjimNjFS4g5DWahh75EKCgJUfcUIinUQzfQXwXkSp
SMS_IR_TEMPLATE_ID=123456
# ── Voice treatment entry ──────────────────────────────────────────────────────
# Without OPENROUTER_API_KEY the microphone button does not render at all.
OPENROUTER_API_KEY=
# OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
# Locales the microphone is offered in. An unknown locale here fails at boot.
# VOICE_ENABLED_LOCALES=fa,en,nl
# Models, overridable per locale (VOICE_ASR_MODEL_FA, VOICE_LLM_MODEL_NL, ...).
# All locales share these today; the per-locale override exists so Persian can be
# repointed at a specialist ASR vendor without a code change.
# VOICE_ASR_MODEL=openai/whisper-1
# VOICE_LLM_MODEL=google/gemini-3.7-flash
# VOICE_ASR_PROVIDER_FA=openrouter
# VOICE_LLM_PROVIDER_FA=openrouter
# Recording cap in ms (0 = uncapped). 2 minutes bounds worst-case vendor spend at
# about 1.3 cents per recording.
# VOICE_MAX_RECORDING_MS=120000
# Per-user rate limit on the extract endpoint. Unreachable by a human — a recording
# plus processing takes ten seconds at minimum — so it is purely an abuse guard.
# VOICE_THROTTLE_TTL=60
# VOICE_THROTTLE_LIMIT=6

25
backend/src/app.module.ts
View File

@@ -1,5 +1,6 @@
import { Module } from '@nestjs/common';
import { ConfigModule } from '@nestjs/config';
import { ConfigModule, ConfigService } from '@nestjs/config';
import { ThrottlerModule } from '@nestjs/throttler';
import configurations from './configs/configurations';
import { AuthModule } from './modules/auth/auth.module';
import { AppController } from './app.controller';
@@ -20,6 +21,7 @@ import { LabCaseCommentsModule } from './modules/lab-case-comments/lab-case-comm
import { TodayModule } from './modules/today/today.module';
import { NotificationsModule } from './modules/notifications/notifications.module';
import { RealtimeModule } from './realtime/realtime.module';
import { VoiceModule } from './modules/voice/voice.module';
@Module({
imports: [
@@ -27,6 +29,22 @@ import { RealtimeModule } from './realtime/realtime.module';
isGlobal: true,
load: [configurations],
}),
// First use of @nestjs/throttler in this app. Deliberately NOT bound as a global
// APP_GUARD: a globally-bound ThrottlerGuard rate-limits every route against every
// named throttler, which would cap the whole API at the voice limit. ThrottlerGuard
// is applied to the one expensive route instead, so nothing else changes behaviour.
ThrottlerModule.forRootAsync({
inject: [ConfigService],
useFactory: (config: ConfigService) => ({
throttlers: [
{
name: 'voice',
ttl: (config.get<number>('voice.throttle.ttl') ?? 60) * 1000,
limit: config.get<number>('voice.throttle.limit') ?? 6,
},
],
}),
}),
PrismaModule, // ✅ ADD THIS
CatalogModule,
TreatmentCatalogModule,
@@ -43,9 +61,10 @@ import { RealtimeModule } from './realtime/realtime.module';
TodayModule,
NotificationsModule,
RealtimeModule,
VoiceModule,
AdminModule.forRoot(),
],
controllers: [AppController],
providers: [AppService],
providers: [AppService],
})
export class AppModule {}
export class AppModule {}

104
backend/src/common/body-parsers.spec.ts Normal file
View File

@@ -0,0 +1,104 @@
import express, {
type NextFunction,
type Request,
type Response,
} from 'express';
import request from 'supertest';
import { createJsonBodyParser, VOICE_EXTRACT_PATH } from './body-parsers';
/**
* Guards a bug that made the voice endpoint completely unusable while surfacing as a
* generic 500: the large-body limit stopped applying, so every real recording — anything
* past roughly 20 seconds of audio — was rejected by Express's 100 kb default.
*/
type ProbeBody = { keys?: number; type?: string };
function buildApp(): express.Express {
const app = express();
app.use(createJsonBodyParser());
app.post('*splat', (req: Request, res: Response) => {
res.json({ keys: Object.keys((req.body ?? {}) as object).length });
});
// Surface body-parser's own error instead of Express's HTML default page.
app.use(
(
err: { status?: number; type?: string },
_req: Request,
res: Response,
next: NextFunction,
) => {
if (res.headersSent) {
next(err);
return;
}
res.status(err.status ?? 500).json({ type: err.type });
},
);
return app;
}
const bodyOfKb = (kb: number) => ({ audio: 'A'.repeat(kb * 1024) });
describe('createJsonBodyParser', () => {
it('accepts a body far past the default limit on the voice route', async () => {
const res = await request(buildApp())
.post(VOICE_EXTRACT_PATH)
.send(bodyOfKb(300));
expect(res.status).toBe(200);
expect((res.body as ProbeBody).keys).toBe(1);
});
it('accepts a realistic worst-case recording', async () => {
// Two minutes of opus is well under 1 MB, but wav is far larger; 4 MB must pass.
const res = await request(buildApp())
.post(VOICE_EXTRACT_PATH)
.send(bodyOfKb(4096));
expect(res.status).toBe(200);
});
it('keeps the default limit on every other route', async () => {
// The larger limit must not leak app-wide as a side effect.
const res = await request(buildApp())
.post('/api/auth/login')
.send(bodyOfKb(300));
expect(res.status).toBe(413);
expect((res.body as ProbeBody).type).toBe('entity.too.large');
});
it('still parses ordinary bodies on ordinary routes', async () => {
const res = await request(buildApp())
.post('/api/auth/login')
.send({ email: 'a@b.c' });
expect(res.status).toBe(200);
expect((res.body as ProbeBody).keys).toBe(1);
});
it('widens the limit for the spellings Express itself accepts', async () => {
// Express routes case-insensitively and ignores a trailing slash by default, so these
// all reach the voice controller. Any of them taking the 100 kb parser would 413 a
// real recording and read as a broken microphone.
for (const path of [
'/api/voice/extract/',
'/API/Voice/Extract',
'/api/Voice/extract/',
]) {
const res = await request(buildApp()).post(path).send(bodyOfKb(300));
expect(res.status).toBe(200);
}
});
it('does not widen the limit for a path that merely looks similar', async () => {
for (const path of [
'/api/voice/extract/extra',
'/api/voice',
'/voice/extract',
// Express ignores one trailing slash, not two — this one never routes, so it must
// not get the large parser either.
'/api/voice/extract//',
]) {
const res = await request(buildApp()).post(path).send(bodyOfKb(300));
expect(res.status).toBe(413);
}
});
});

43
backend/src/common/body-parsers.ts Normal file
View File

@@ -0,0 +1,43 @@
import {
json,
type NextFunction,
type Request,
type RequestHandler,
type Response,
} from 'express';
/** The one route that accepts a large body, and how large. */
export const VOICE_EXTRACT_PATH = '/api/voice/extract';
export const VOICE_BODY_LIMIT = '10mb';
/**
* JSON body parsing for the whole app.
*
* Voice recordings are base64 JSON and pass Express's 100 kb default at roughly 20 seconds
* of audio, so that one route needs a larger limit while every other endpoint keeps the
* default — a large body should not become acceptable everywhere.
*
* Deliberately one middleware that *chooses* a parser, not a path-mounted parser stacked in
* front of a default one: that arrangement depended on Express's mount-path stripping and on
* body-parser skipping an already-parsed request, and silently stopped applying whenever the
* middleware order shifted. One explicit branch has no such coupling.
*/
/**
* Express routes case-insensitively and ignores exactly one trailing slash, so
* `/API/Voice/Extract/` reaches the same controller and must get the same limit — otherwise
* it 413s every real recording, which reads as a broken microphone rather than a route.
* Two slashes never route, so they must not buy a 10 MB buffer either.
*/
function isVoiceExtractPath(path: string): boolean {
return path.toLowerCase().replace(/\/$/, '') === VOICE_EXTRACT_PATH;
}
export function createJsonBodyParser(): RequestHandler {
const voiceParser = json({ limit: VOICE_BODY_LIMIT });
const defaultParser = json();
return (req: Request, res: Response, next: NextFunction) =>
isVoiceExtractPath(req.path)
? voiceParser(req, res, next)
: defaultParser(req, res, next);
}

31
backend/src/common/digits.spec.ts Normal file
View File

@@ -0,0 +1,31 @@
import { toLatinDigits } from './digits';
/**
* Exercised by the voice pipeline on two untrusted inputs: spoken dates, and the tooth
* code the extraction model echoes back — a Persian-digit "۲۶" that fails to normalise
* costs the clinician a tooth, silently.
*/
describe('toLatinDigits', () => {
it('normalises Persian digits and leaves everything else alone', () => {
expect(
toLatinDigits('\u06F1\u06F4\u06F0\u06F4/\u06F0\u06F7/\u06F2\u06F5'),
).toBe('1404/07/25');
expect(toLatinDigits('1404/07/25')).toBe('1404/07/25');
expect(toLatinDigits('\u062F\u0646\u062F\u0627\u0646 \u06F1\u06F4')).toBe(
'\u062F\u0646\u062F\u0627\u0646 14',
);
});
it('also normalises the Arabic-Indic block, which ASR output can carry', () => {
// U+0660..U+0669, distinct code points from the Persian U+06F0..U+06F9 block.
expect(
toLatinDigits('\u0661\u0664\u0660\u0664/\u0660\u0667/\u0662\u0665'),
).toBe('1404/07/25');
});
it('normalises a transcript that mixes both blocks with ASCII', () => {
expect(toLatinDigits('\u06F1\u06F4 and \u0661\u0665 and 16')).toBe(
'14 and 15 and 16',
);
});
});

16
backend/src/common/digits.ts Normal file
View File

@@ -0,0 +1,16 @@
const PERSIAN_ZERO = 0x06f0;
const ARABIC_INDIC_ZERO = 0x0660;
/**
* Normalise Persian and Arabic-Indic digits to ASCII. Non-digits pass through.
*
* Both blocks, not just the Persian one the frontend handles: ASR output can carry either,
* sometimes mixed with ASCII in a single transcript.
*/
export function toLatinDigits(value: string): string {
return value.replace(/[۰-۹٠-٩]/g, (ch) => {
const code = ch.charCodeAt(0);
const base = code >= PERSIAN_ZERO ? PERSIAN_ZERO : ARABIC_INDIC_ZERO;
return String(code - base);
});
}

30
backend/src/common/errors/error-codes.ts
View File

@@ -29,8 +29,10 @@ export const ErrorCode = {
PERMISSION_LAB_ONLY: 'PERMISSION_LAB_ONLY',
PERMISSION_NOT_MEMBER: 'PERMISSION_NOT_MEMBER',
PERMISSION_OWNER_ONLY: 'PERMISSION_OWNER_ONLY',
PERMISSION_PARTICIPATION_SUBSCRIPTION: 'PERMISSION_PARTICIPATION_SUBSCRIPTION',
PERMISSION_ENABLE_PARTICIPATION_FIRST: 'PERMISSION_ENABLE_PARTICIPATION_FIRST',
PERMISSION_PARTICIPATION_SUBSCRIPTION:
'PERMISSION_PARTICIPATION_SUBSCRIPTION',
PERMISSION_ENABLE_PARTICIPATION_FIRST:
'PERMISSION_ENABLE_PARTICIPATION_FIRST',
PERMISSION_CLINIC_WORKING_HOURS: 'PERMISSION_CLINIC_WORKING_HOURS',
PERMISSION_ACCESS_APPOINTMENTS: 'PERMISSION_ACCESS_APPOINTMENTS',
PERMISSION_EDIT_APPOINTMENTS: 'PERMISSION_EDIT_APPOINTMENTS',
@@ -51,7 +53,8 @@ export const ErrorCode = {
VALIDATION_PASSWORD_REQUIRED: 'VALIDATION_PASSWORD_REQUIRED',
VALIDATION_MOBILE_INVALID: 'VALIDATION_MOBILE_INVALID',
VALIDATION_NAME_TOO_SHORT: 'VALIDATION_NAME_TOO_SHORT',
VALIDATION_ORGANIZATION_NAME_REQUIRED: 'VALIDATION_ORGANIZATION_NAME_REQUIRED',
VALIDATION_ORGANIZATION_NAME_REQUIRED:
'VALIDATION_ORGANIZATION_NAME_REQUIRED',
VALIDATION_ORGANIZATION_TYPE_INVALID: 'VALIDATION_ORGANIZATION_TYPE_INVALID',
VALIDATION_TOKEN_REQUIRED: 'VALIDATION_TOKEN_REQUIRED',
VALIDATION_FIELD_REQUIRED: 'VALIDATION_FIELD_REQUIRED',
@@ -69,8 +72,10 @@ export const ErrorCode = {
APPOINTMENT_INVALID_DATE: 'APPOINTMENT_INVALID_DATE',
APPOINTMENT_PROVIDER_NOT_MEMBER: 'APPOINTMENT_PROVIDER_NOT_MEMBER',
APPOINTMENT_PROVIDER_INACTIVE: 'APPOINTMENT_PROVIDER_INACTIVE',
APPOINTMENT_PROVIDER_NO_TREATMENT_EDIT: 'APPOINTMENT_PROVIDER_NO_TREATMENT_EDIT',
APPOINTMENT_PROVIDER_NO_WORKING_HOURS: 'APPOINTMENT_PROVIDER_NO_WORKING_HOURS',
APPOINTMENT_PROVIDER_NO_TREATMENT_EDIT:
'APPOINTMENT_PROVIDER_NO_TREATMENT_EDIT',
APPOINTMENT_PROVIDER_NO_WORKING_HOURS:
'APPOINTMENT_PROVIDER_NO_WORKING_HOURS',
APPOINTMENT_PROVIDER_NOT_WORKING_DAY: 'APPOINTMENT_PROVIDER_NOT_WORKING_DAY',
APPOINTMENT_OUTSIDE_WORKING_HOURS: 'APPOINTMENT_OUTSIDE_WORKING_HOURS',
APPOINTMENT_NOT_FOUND: 'APPOINTMENT_NOT_FOUND',
@@ -82,7 +87,8 @@ export const ErrorCode = {
WORKING_HOURS_INVALID: 'WORKING_HOURS_INVALID',
WORKING_HOURS_OWNER_NOT_ALLOWED: 'WORKING_HOURS_OWNER_NOT_ALLOWED',
WORKING_HOURS_CONFLICTS_WITH_APPOINTMENTS: 'WORKING_HOURS_CONFLICTS_WITH_APPOINTMENTS',
WORKING_HOURS_CONFLICTS_WITH_APPOINTMENTS:
'WORKING_HOURS_CONFLICTS_WITH_APPOINTMENTS',
STAFF_UNKNOWN_PERMISSIONS: 'STAFF_UNKNOWN_PERMISSIONS',
STAFF_NO_SUBSCRIPTION: 'STAFF_NO_SUBSCRIPTION',
@@ -147,7 +153,8 @@ export const ErrorCode = {
TREATMENT_ATTACHMENT_NOT_FOUND: 'TREATMENT_ATTACHMENT_NOT_FOUND',
TREATMENT_FILE_UNAVAILABLE: 'TREATMENT_FILE_UNAVAILABLE',
TREATMENT_CASE_INVALID_ATTACHMENTS: 'TREATMENT_CASE_INVALID_ATTACHMENTS',
TREATMENT_TOOTH_PROSTHESIS_INCOMPLETE: 'TREATMENT_TOOTH_PROSTHESIS_INCOMPLETE',
TREATMENT_TOOTH_PROSTHESIS_INCOMPLETE:
'TREATMENT_TOOTH_PROSTHESIS_INCOMPLETE',
TREATMENT_DETAIL_SENT: 'TREATMENT_DETAIL_SENT',
TREATMENT_NOT_FOUND: 'TREATMENT_NOT_FOUND',
TREATMENT_PATIENT_OR_WALK_IN: 'TREATMENT_PATIENT_OR_WALK_IN',
@@ -177,6 +184,15 @@ export const ErrorCode = {
TODAY_INVALID_RANGE: 'TODAY_INVALID_RANGE',
TODAY_INVALID_RANGE_ORDER: 'TODAY_INVALID_RANGE_ORDER',
// Voice treatment entry
VOICE_NOT_AVAILABLE: 'VOICE_NOT_AVAILABLE',
VOICE_CLIP_TOO_LONG: 'VOICE_CLIP_TOO_LONG',
VOICE_UNSUPPORTED_FORMAT: 'VOICE_UNSUPPORTED_FORMAT',
VOICE_ASR_FAILED: 'VOICE_ASR_FAILED',
VOICE_EXTRACT_FAILED: 'VOICE_EXTRACT_FAILED',
VOICE_NOTHING_RECOGNIZED: 'VOICE_NOTHING_RECOGNIZED',
VOICE_RATE_LIMITED: 'VOICE_RATE_LIMITED',
// Generic HTTP
NOT_FOUND: 'NOT_FOUND',
CONFLICT: 'CONFLICT',

144
backend/src/common/fdi.spec.ts Normal file
View File

@@ -0,0 +1,144 @@
import {
areArchNeighbors,
sortInArchOrder,
FDI_TOOTH_IDS,
isFdiTooth,
sameArch,
teethBetweenInclusive,
toFdi,
} from './fdi';
describe('FDI geometry', () => {
describe('toFdi — the patient-right convention', () => {
// A mirrored quadrant produces a *valid* code for the wrong tooth, so no schema
// check can catch it. These four cases are the guard.
it('maps upper + patient right to quadrant 1', () => {
expect(toFdi('upper', 'patient_right', 6)).toBe('16');
expect(toFdi('upper', 'patient_right', 1)).toBe('11');
});
it('maps upper + patient left to quadrant 2', () => {
expect(toFdi('upper', 'patient_left', 6)).toBe('26');
expect(toFdi('upper', 'patient_left', 8)).toBe('28');
});
it('maps lower + patient left to quadrant 3', () => {
expect(toFdi('lower', 'patient_left', 6)).toBe('36');
});
it('maps lower + patient right to quadrant 4', () => {
expect(toFdi('lower', 'patient_right', 6)).toBe('46');
expect(toFdi('lower', 'patient_right', 8)).toBe('48');
});
it('never clamps an out-of-range position', () => {
expect(toFdi('upper', 'patient_right', 9)).toBeNull();
expect(toFdi('upper', 'patient_right', 0)).toBeNull();
expect(toFdi('upper', 'patient_right', -1)).toBeNull();
expect(toFdi('upper', 'patient_right', 1.5)).toBeNull();
expect(toFdi('upper', 'patient_right', Number.NaN)).toBeNull();
});
});
describe('isFdiTooth', () => {
it('accepts all 32 permanent teeth', () => {
expect(FDI_TOOTH_IDS.size).toBe(32);
for (const tooth of FDI_TOOTH_IDS) expect(isFdiTooth(tooth)).toBe(true);
});
it('rejects deciduous teeth — the chart has no primary dentition', () => {
for (const tooth of ['51', '55', '61', '71', '85']) {
expect(isFdiTooth(tooth)).toBe(false);
}
});
it('rejects garbage', () => {
for (const value of [
'',
'1',
'19',
'10',
'29',
'99',
14,
null,
undefined,
{},
]) {
expect(isFdiTooth(value)).toBe(false);
}
});
});
describe('adjacency', () => {
it('treats neighbours within a quadrant as adjacent', () => {
expect(areArchNeighbors('14', '15')).toBe(true);
expect(areArchNeighbors('15', '14')).toBe(true);
});
it('treats the midline pairs as adjacent', () => {
expect(areArchNeighbors('11', '21')).toBe(true);
expect(areArchNeighbors('41', '31')).toBe(true);
});
it('rejects non-neighbours and cross-arch pairs', () => {
expect(areArchNeighbors('14', '16')).toBe(false);
expect(areArchNeighbors('18', '28')).toBe(false);
expect(areArchNeighbors('14', '44')).toBe(false);
expect(areArchNeighbors('14', '14')).toBe(false);
});
});
describe('sameArch', () => {
it('groups by arch, not by quadrant', () => {
expect(sameArch('18', '28')).toBe(true);
expect(sameArch('48', '38')).toBe(true);
expect(sameArch('18', '48')).toBe(false);
expect(sameArch('14', '99')).toBe(false);
});
});
describe('teethBetweenInclusive', () => {
it('returns the span in arch order regardless of argument order', () => {
expect(teethBetweenInclusive('14', '16')).toEqual(['16', '15', '14']);
expect(teethBetweenInclusive('16', '14')).toEqual(['16', '15', '14']);
});
it('spans the midline', () => {
expect(teethBetweenInclusive('12', '22')).toEqual([
'12',
'11',
'21',
'22',
]);
});
it('returns a single tooth for identical endpoints', () => {
expect(teethBetweenInclusive('14', '14')).toEqual(['14']);
});
it('returns null across arches or for unknown teeth', () => {
expect(teethBetweenInclusive('14', '44')).toBeNull();
expect(teethBetweenInclusive('14', '99')).toBeNull();
expect(teethBetweenInclusive('99', '14')).toBeNull();
});
});
describe('sortInArchOrder', () => {
it('sorts along the arch rather than lexically', () => {
expect(sortInArchOrder(['14', '16', '15'])).toEqual(['16', '15', '14']);
});
it('places 11 beside 21 across the midline', () => {
expect(sortInArchOrder(['21', '11', '12'])).toEqual(['12', '11', '21']);
});
it('is a no-op for an empty or single-tooth list', () => {
expect(sortInArchOrder([])).toEqual([]);
expect(sortInArchOrder(['14'])).toEqual(['14']);
});
it('does not drop teeth it cannot place', () => {
expect(sortInArchOrder(['99', '14']).sort()).toEqual(['14', '99']);
});
});
});

145
backend/src/common/fdi.ts Normal file
View File

@@ -0,0 +1,145 @@
/**
* FDI tooth geometry — permanent dentition only.
*
* Mirrors `frontend/src/components/treatment/fdiToothMeta.ts` and the adjacency rules in
* `toothSelectionGroups.ts`. Adjacency is defined by position in the arch order, so the
* midline pairs (11–21, 41–31) are neighbours, exactly as the chart treats them.
*/
import { toLatinDigits } from './digits';
export type Arch = 'upper' | 'lower';
/** Which side of the *patient*, not of the screen. Quadrant 1 is the patient's upper right. */
export type PatientSide = 'patient_right' | 'patient_left';
/**
* Upper arch in chart order: patient's RIGHT (18) → midline → patient's LEFT (28) — the drawn
* layout, which mirrors the patient's own sides. Never read a position off this array by
* index; use `toFdi()`, which owns the side convention.
*/
export const FDI_UPPER_ARCH_ORDER = [
'18',
'17',
'16',
'15',
'14',
'13',
'12',
'11',
'21',
'22',
'23',
'24',
'25',
'26',
'27',
'28',
] as const;
/** Lower arch, same chart ordering: patient's RIGHT (48) → midline → patient's LEFT (38). */
export const FDI_LOWER_ARCH_ORDER = [
'48',
'47',
'46',
'45',
'44',
'43',
'42',
'41',
'31',
'32',
'33',
'34',
'35',
'36',
'37',
'38',
] as const;
export const FDI_TOOTH_IDS: ReadonlySet<string> = new Set<string>([
...FDI_UPPER_ARCH_ORDER,
...FDI_LOWER_ARCH_ORDER,
]);
export function isFdiTooth(value: unknown): value is string {
return typeof value === 'string' && FDI_TOOTH_IDS.has(value);
}
/**
* Clean up a tooth code the model echoed back. It is reading Persian speech, so it can hand
* back "۲۶" or "2 6" from digit-by-digit dictation; neither matches literally, and the
* near-miss does not fail loudly — the tooth just turns into "not understood".
*/
export function normalizeFdiCode(value: unknown): string {
if (typeof value !== 'string') return '';
return toLatinDigits(value).replace(/\s+/g, '');
}
function archOrder(tooth: string): readonly string[] | null {
if ((FDI_UPPER_ARCH_ORDER as readonly string[]).includes(tooth))
return FDI_UPPER_ARCH_ORDER;
if ((FDI_LOWER_ARCH_ORDER as readonly string[]).includes(tooth))
return FDI_LOWER_ARCH_ORDER;
return null;
}
export function sameArch(a: string, b: string): boolean {
const archA = archOrder(a);
const archB = archOrder(b);
return Boolean(archA && archB && archA === archB);
}
export function areArchNeighbors(a: string, b: string): boolean {
const arch = archOrder(a);
if (!arch || !sameArch(a, b)) return false;
return Math.abs(arch.indexOf(a) - arch.indexOf(b)) === 1;
}
/** Inclusive span between two teeth of the same arch, in arch order. Null if not comparable. */
export function teethBetweenInclusive(a: string, b: string): string[] | null {
const arch = archOrder(a);
if (!arch || !sameArch(a, b)) return null;
const i = arch.indexOf(a);
const j = arch.indexOf(b);
if (i < 0 || j < 0) return null;
const [from, to] = i <= j ? [i, j] : [j, i];
return [...arch.slice(from, to + 1)];
}
/**
* Arch + patient side + position (1 = central incisor … 8 = third molar) → FDI code.
*
* The single place the patient-right convention lives. Getting it backwards mirrors every
* quadrant into a valid-looking code for the wrong tooth, which no schema check can catch.
*/
export function toFdi(
arch: Arch,
side: PatientSide,
position: number,
): string | null {
if (!Number.isInteger(position) || position < 1 || position > 8) return null;
let quadrant: number;
if (arch === 'upper') {
quadrant = side === 'patient_right' ? 1 : 2;
} else {
quadrant = side === 'patient_left' ? 3 : 4;
}
const code = `${quadrant}${position}`;
return FDI_TOOTH_IDS.has(code) ? code : null;
}
/**
* Along the arch, not lexically — a bridge reads 16-15-14, and 11 sits beside 21 across the
* midline. Teeth from another arch sort to the end, stably.
*/
export function sortInArchOrder(teeth: readonly string[]): string[] {
if (teeth.length === 0) return [];
const arch = teeth.map((t) => archOrder(t)).find((a) => a !== null) ?? null;
if (!arch) return [...teeth];
const indexOf = (tooth: string) => {
const i = arch.indexOf(tooth);
return i === -1 ? Number.MAX_SAFE_INTEGER : i;
};
return [...teeth].sort((a, b) => indexOf(a) - indexOf(b));
}

89
backend/src/common/jalali.spec.ts Normal file
View File

@@ -0,0 +1,89 @@
import {
gregorianToJalali,
isJalaliLeapYear,
isValidJalaliDate,
jalaliDaysInMonth,
jalaliToGregorian,
jalaliToIsoDate,
} from './jalali';
describe('jalali calendar', () => {
// Anchors verified against frontend/src/lib/i18n/persianCalendar.ts, the source this
// was ported from, over every day between 1900 and 2100.
it('converts known Jalali dates to ISO', () => {
expect(jalaliToIsoDate(1404, 1, 1)).toBe('2025-03-21'); // Nowruz 1404
expect(jalaliToIsoDate(1404, 7, 25)).toBe('2025-10-17');
expect(jalaliToIsoDate(1404, 6, 31)).toBe('2025-09-22'); // 31-day first half
expect(jalaliToIsoDate(1405, 1, 1)).toBe('2026-03-21');
});
it('round-trips Gregorian → Jalali → Gregorian', () => {
for (const [gy, gm, gd] of [
[2025, 3, 21],
[2025, 10, 17],
[2026, 1, 1],
[2024, 2, 29], // Gregorian leap day
[1999, 12, 31],
] as const) {
const [jy, jm, jd] = gregorianToJalali(gy, gm, gd);
expect(jalaliToGregorian(jy, jm, jd)).toEqual([gy, gm, gd]);
}
});
describe('leap years', () => {
it('gives Esfand 30 days in a leap year and 29 otherwise', () => {
expect(isJalaliLeapYear(1403)).toBe(true);
expect(jalaliDaysInMonth(1403, 12)).toBe(30);
expect(jalaliToIsoDate(1403, 12, 30)).toBe('2025-03-20');
expect(isJalaliLeapYear(1404)).toBe(false);
expect(jalaliDaysInMonth(1404, 12)).toBe(29);
expect(jalaliToIsoDate(1404, 12, 29)).toBe('2026-03-20');
});
it('rejects Esfand 30 in a non-leap year', () => {
expect(isValidJalaliDate(1404, 12, 30)).toBe(false);
expect(jalaliToIsoDate(1404, 12, 30)).toBeNull();
});
});
describe('month lengths', () => {
it('is 31 days for months 1-6 and 30 for 7-11', () => {
for (let m = 1; m <= 6; m += 1)
expect(jalaliDaysInMonth(1404, m)).toBe(31);
for (let m = 7; m <= 11; m += 1)
expect(jalaliDaysInMonth(1404, m)).toBe(30);
});
});
describe('invalid input degrades to null rather than throwing', () => {
// The resolver feeds this model-supplied values, which may be nonsense.
it.each([
['month 13', 1404, 13, 1],
['month 0', 1404, 0, 1],
['day 0', 1404, 1, 0],
['day 32 in a 31-day month', 1404, 1, 32],
['day 31 in a 30-day month', 1404, 7, 31],
['year beyond the conversion table', 9999, 1, 1],
['non-integer day', 1404, 1, 1.5],
])('returns null for %s', (_label, jy, jm, jd) => {
expect(jalaliToIsoDate(jy, jm, jd)).toBeNull();
});
});
describe('no export throws on an unsupported year', () => {
// The whole module is reachable from model-supplied values, so it degrades instead of
// raising — jalaliToIsoDate's guard is not the only entry point.
it.each([9999, -9999, 3178, Number.NaN, 1.5])('year %s', (jy) => {
expect(() => isJalaliLeapYear(jy)).not.toThrow();
expect(() => jalaliDaysInMonth(jy, 1)).not.toThrow();
expect(isJalaliLeapYear(jy)).toBe(false);
expect(jalaliDaysInMonth(jy, 1)).toBe(0);
});
it('returns 0 days for an impossible month', () => {
expect(jalaliDaysInMonth(1404, 0)).toBe(0);
expect(jalaliDaysInMonth(1404, 13)).toBe(0);
});
});
});

179
backend/src/common/jalali.ts Normal file
View File

@@ -0,0 +1,179 @@
/**
* Jalali (Persian) calendar arithmetic, ported from
* `frontend/src/lib/i18n/persianCalendar.ts` (itself jalaali-js, MIT). The backend needs it
* because voice resolves spoken Jalali dates server-side, where the resolvers are tested.
* Keep the two copies in step; the underlying calendar does not change.
*/
const BREAKS = [
-61, 9, 38, 199, 426, 686, 756, 818, 1111, 1181, 1210, 1635, 2060, 2097, 2192,
2262, 2324, 2394, 2456, 3178,
];
/** Inclusive lower / exclusive upper Jalali year bounds the conversion table covers. */
export const MIN_JALALI_YEAR = BREAKS[0];
export const MAX_JALALI_YEAR = BREAKS[BREAKS.length - 1];
function div(a: number, b: number): number {
return Math.trunc(a / b);
}
function mod(a: number, b: number): number {
return a - Math.trunc(a / b) * b;
}
function g2d(gy: number, gm: number, gd: number): number {
let d =
div((gy + div(gm - 8, 6) + 100100) * 1461, 4) +
div(153 * mod(gm + 9, 12) + 2, 5) +
gd -
34840408;
d = d - div(div(gy + 100100 + div(gm - 8, 6), 100) * 3, 4) + 752;
return d;
}
function d2g(jdn: number): { gy: number; gm: number; gd: number } {
let j = 4 * jdn + 139361631;
j = j + div(div(4 * jdn + 183187720, 146097) * 3, 4) * 4 - 3908;
const i = div(mod(j, 1461), 4) * 5 + 308;
const gd = div(mod(i, 153), 5) + 1;
const gm = mod(div(i, 153), 12) + 1;
const gy = div(j, 1461) - 100100 + div(8 - gm, 6);
return { gy, gm, gd };
}
function jalCal(
jy: number,
withoutLeap: boolean,
): { leap?: number; gy: number; march: number } {
const bl = BREAKS.length;
const gy = jy + 621;
let leapJ = -14;
let jp = BREAKS[0];
let jump = 0;
let leap = 0;
let n = 0;
if (jy < jp || jy >= BREAKS[bl - 1]) {
throw new Error(`Invalid Jalaali year ${jy}`);
}
for (let i = 1; i < bl; i += 1) {
const jm = BREAKS[i];
jump = jm - jp;
if (jy < jm) break;
leapJ = leapJ + div(jump, 33) * 8 + div(mod(jump, 33), 4);
jp = jm;
}
n = jy - jp;
leapJ = leapJ + div(n, 33) * 8 + div(mod(n, 33) + 3, 4);
if (mod(jump, 33) === 4 && jump - n === 4) leapJ += 1;
const leapG = div(gy, 4) - div((div(gy, 100) + 1) * 3, 4) - 150;
const march = 20 + leapJ - leapG;
if (withoutLeap) return { gy, march };
if (jump - n < 6) n = n - jump + div(jump + 4, 33) * 33;
leap = mod(mod(n + 1, 33) - 1, 4);
if (leap === -1) leap = 4;
return { leap, gy, march };
}
function j2d(jy: number, jm: number, jd: number): number {
const r = jalCal(jy, true);
return g2d(r.gy, 3, r.march) + (jm - 1) * 31 - div(jm, 7) * (jm - 7) + jd - 1;
}
function d2j(jdn: number): { jy: number; jm: number; jd: number } {
const { gy } = d2g(jdn);
let jy = gy - 621;
const r = jalCal(jy, false);
const jdn1f = g2d(gy, 3, r.march);
let k = jdn - jdn1f;
let jm: number;
let jd: number;
if (k >= 0) {
if (k <= 185) {
jm = 1 + div(k, 31);
jd = mod(k, 31) + 1;
return { jy, jm, jd };
}
k -= 186;
} else {
jy -= 1;
k += 179;
if (r.leap === 1) k += 1;
}
jm = 7 + div(k, 30);
jd = mod(k, 30) + 1;
return { jy, jm, jd };
}
export function gregorianToJalali(
gy: number,
gm: number,
gd: number,
): [number, number, number] {
const { jy, jm, jd } = d2j(g2d(gy, gm, gd));
return [jy, jm, jd];
}
export function jalaliToGregorian(
jy: number,
jm: number,
jd: number,
): [number, number, number] {
const { gy, gm, gd } = d2g(j2d(jy, jm, jd));
return [gy, gm, gd];
}
/** True when the year is inside the conversion table's supported range. */
export function isSupportedJalaliYear(jy: number): boolean {
return Number.isInteger(jy) && jy >= MIN_JALALI_YEAR && jy < MAX_JALALI_YEAR;
}
/** False for unsupported years rather than throwing — see the module contract. */
export function isJalaliLeapYear(jy: number): boolean {
if (!isSupportedJalaliYear(jy)) return false;
const r = jalCal(jy, false);
return r.leap === 0;
}
/**
* Days in a Jalali month, or 0 when the year or month is not real. Zero rather than a throw:
* every export here is reachable from model-supplied values, so the module degrades.
*/
export function jalaliDaysInMonth(jy: number, jm: number): number {
if (!isSupportedJalaliYear(jy)) return 0;
if (!Number.isInteger(jm) || jm < 1 || jm > 12) return 0;
if (jm <= 6) return 31;
if (jm <= 11) return 30;
return isJalaliLeapYear(jy) ? 30 : 29;
}
/** True when the triple is a real Jalali date inside the supported year range. */
export function isValidJalaliDate(jy: number, jm: number, jd: number): boolean {
if (!Number.isInteger(jy) || !Number.isInteger(jm) || !Number.isInteger(jd)) {
return false;
}
if (jy < MIN_JALALI_YEAR || jy >= MAX_JALALI_YEAR) return false;
if (jm < 1 || jm > 12) return false;
if (jd < 1) return false;
return jd <= jalaliDaysInMonth(jy, jm);
}
/**
* Jalali triple → `YYYY-MM-DD`, or null when the date is not real. Null rather than a throw,
* for the same reason: callers resolve model-supplied values, which may be nonsense.
*/
export function jalaliToIsoDate(
jy: number,
jm: number,
jd: number,
): string | null {
if (!isValidJalaliDate(jy, jm, jd)) return null;
const [gy, gm, gd] = jalaliToGregorian(jy, jm, jd);
return `${String(gy).padStart(4, '0')}-${String(gm).padStart(2, '0')}-${String(gd).padStart(2, '0')}`;
}

48
backend/src/common/zoned-civil-time.spec.ts
View File

@@ -1,5 +1,9 @@
import { appointmentWithinWorkingHours } from './working-hours';
import { isValidIanaTimeZone, zonedWeekdayAndMinutes } from './zoned-civil-time';
import {
civilDateInZone,
isValidIanaTimeZone,
zonedWeekdayAndMinutes,
} from './zoned-civil-time';
describe('zoned civil time', () => {
it('accepts IANA zones and rejects garbage', () => {
@@ -20,10 +24,48 @@ describe('zoned civil time', () => {
const start = new Date('2026-08-20T06:15:00.000Z');
const end = new Date('2026-08-20T06:45:00.000Z');
expect(
appointmentWithinWorkingHours(start, end, [{ startMinute: 8 * 60, endMinute: 17 * 60 }], 'Asia/Tehran'),
appointmentWithinWorkingHours(
start,
end,
[{ startMinute: 8 * 60, endMinute: 17 * 60 }],
'Asia/Tehran',
),
).toBe(true);
expect(
appointmentWithinWorkingHours(start, end, [{ startMinute: 8 * 60, endMinute: 17 * 60 }], 'UTC'),
appointmentWithinWorkingHours(
start,
end,
[{ startMinute: 8 * 60, endMinute: 17 * 60 }],
'UTC',
),
).toBe(false);
});
describe('civilDateInZone', () => {
it('gives the local civil date, which can differ from the UTC date', () => {
// 21:30 UTC is already the next day in Tehran (+03:30).
const instant = new Date('2025-10-11T21:30:00.000Z');
expect(civilDateInZone(instant, 'UTC')).toBe('2025-10-11');
expect(civilDateInZone(instant, 'Asia/Tehran')).toBe('2025-10-12');
});
it('gives the previous day for zones behind UTC just after midnight', () => {
const instant = new Date('2025-10-11T02:00:00.000Z');
expect(civilDateInZone(instant, 'America/New_York')).toBe('2025-10-10');
expect(civilDateInZone(instant, 'Europe/Amsterdam')).toBe('2025-10-11');
});
it('falls back to UTC on an invalid zone rather than throwing', () => {
// Intl raises RangeError on an unknown zone and this takes a client-supplied string.
const instant = new Date('2025-10-11T21:30:00.000Z');
expect(() => civilDateInZone(instant, 'Not/AZone')).not.toThrow();
expect(civilDateInZone(instant, 'Not/AZone')).toBe('2025-10-11');
expect(civilDateInZone(instant, '')).toBe('2025-10-11');
});
it('zero-pads single-digit months and days', () => {
expect(civilDateInZone(new Date('2025-01-05T12:00:00.000Z'), 'UTC')).toBe(
'2025-01-05',
);
});
});
});

21
backend/src/common/zoned-civil-time.ts
View File

@@ -53,3 +53,24 @@ export function civilDateJsWeekday(isoDate: string): number {
const utcNoon = new Date(Date.UTC(y, m - 1, d, 12, 0, 0, 0));
return utcNoon.getUTCDay();
}
/**
* Today's civil date (`YYYY-MM-DD`) in an IANA zone, so the server derives "today" from a
* client-supplied *zone* rather than trusting a client-supplied date.
*/
export function civilDateInZone(date: Date, timeZone: string): string {
// Intl throws RangeError on an unknown zone and this takes a client-supplied string;
// callers validate first, this is the backstop.
const zone = isValidIanaTimeZone(timeZone) ? timeZone : 'UTC';
const parts = new Intl.DateTimeFormat('en-CA', {
timeZone: zone,
year: 'numeric',
month: '2-digit',
day: '2-digit',
}).formatToParts(date);
const year = parts.find((p) => p.type === 'year')?.value ?? '1970';
const month = parts.find((p) => p.type === 'month')?.value ?? '01';
const day = parts.find((p) => p.type === 'day')?.value ?? '01';
return `${year}-${month}-${day}`;
}

137
backend/src/configs/configurations.ts
View File

@@ -1,13 +1,14 @@
// backend/src/config/configuration.ts
/** Matches values accepted by jsonwebtoken `expiresIn` (via ms), e.g. 7d, 15m, or plain seconds. */
const JWT_TIMESPAN_PATTERN =
/^\d+(\.\d+)?(ms|s|m|h|d|w|y)?$/i;
const JWT_TIMESPAN_PATTERN = /^\d+(\.\d+)?(ms|s|m|h|d|w|y)?$/i;
function assertJwtSecret(value: string, envKey: string): string {
const trimmed = value.trim();
if (!trimmed) {
throw new Error(`❌ Environment variable ${envKey} is required but not set`);
throw new Error(
`❌ Environment variable ${envKey} is required but not set`,
);
}
if (trimmed.length < 16) {
throw new Error(`❌ ${envKey} must be at least 16 characters`);
@@ -21,7 +22,9 @@ function assertJwtSecret(value: string, envKey: string): string {
function assertJwtTimespan(value: string, envKey: string): string {
const trimmed = value.trim();
if (!trimmed) {
throw new Error(`❌ Environment variable ${envKey} is required but not set`);
throw new Error(
`❌ Environment variable ${envKey} is required but not set`,
);
}
if (!/^\d+$/.test(trimmed) && !JWT_TIMESPAN_PATTERN.test(trimmed)) {
throw new Error(
@@ -31,7 +34,10 @@ function assertJwtTimespan(value: string, envKey: string): string {
return trimmed;
}
function parseEnvBoolean(value: string | undefined, defaultValue: boolean): boolean {
function parseEnvBoolean(
value: string | undefined,
defaultValue: boolean,
): boolean {
if (value === undefined || value.trim() === '') {
return defaultValue;
}
@@ -67,8 +73,37 @@ export interface Config {
apiKey: string | null;
templateId: number;
};
voice: VoiceConfig;
}
/** Only OpenRouter today. The indirection exists so a locale can diverge without code. */
export type VoiceProviderId = 'openrouter';
export type VoiceProfile = {
asr: { provider: VoiceProviderId; model: string };
llm: { provider: VoiceProviderId; model: string };
};
export interface VoiceConfig {
openRouter: {
apiKey: string | null;
baseUrl: string;
};
/**
* Locale → provider profile. A locale absent here has no microphone button at all —
* clean absence rather than a dead control.
*/
profiles: Record<string, VoiceProfile>;
/** Recording cap in ms; null means uncapped. */
maxRecordingMs: number | null;
throttle: { ttl: number; limit: number };
}
export const VOICE_LOCALES = ['fa', 'en', 'nl'] as const;
const DEFAULT_ASR_MODEL = 'openai/whisper-1';
const DEFAULT_LLM_MODEL = 'google/gemini-3.7-flash';
export default (): Config => {
// Helper function to get required env var with type safety
const getEnvVar = (key: string): string => {
@@ -129,5 +164,95 @@ export default (): Config => {
apiKey: process.env.SMS_IR_API_KEY?.trim() || null,
templateId: getEnvVarAsNumber('SMS_IR_TEMPLATE_ID', 123456),
},
voice: buildVoiceConfig(getEnvVarWithDefault, getEnvVarAsNumber),
};
};
};
const VOICE_PROVIDER_IDS: readonly VoiceProviderId[] = ['openrouter'];
/** Unknown provider ids fail at boot; silently coercing a typo would ship the wrong vendor. */
function parseProviderId(
value: string | undefined,
envKey: string,
): VoiceProviderId {
const trimmed = value?.trim();
if (!trimmed) return 'openrouter';
if ((VOICE_PROVIDER_IDS as readonly string[]).includes(trimmed)) {
return trimmed as VoiceProviderId;
}
throw new Error(
`❌ ${envKey}="${trimmed}" is not a known voice provider (${VOICE_PROVIDER_IDS.join(', ')})`,
);
}
/**
* Every enabled locale gets its own ASR and LLM provider+model, each independently
* overridable. They all point at the same OpenRouter models today; the per-locale
* indirection stays so a locale can diverge by configuration rather than by code.
*/
function buildVoiceConfig(
getEnvVarWithDefault: (key: string, defaultValue: string) => string,
getEnvVarAsNumber: (key: string, defaultValue: number) => number,
): VoiceConfig {
const enabled = getEnvVarWithDefault(
'VOICE_ENABLED_LOCALES',
VOICE_LOCALES.join(','),
)
.split(',')
.map((locale) => locale.trim().toLowerCase())
.filter(Boolean);
for (const locale of enabled) {
// Fail loudly, like parseProviderId. Silently filtering a typo would disable the
// microphone everywhere with nothing to explain why.
if (!(VOICE_LOCALES as readonly string[]).includes(locale)) {
throw new Error(
`❌ VOICE_ENABLED_LOCALES contains unknown locale "${locale}" (known: ${VOICE_LOCALES.join(', ')})`,
);
}
}
const profiles: Record<string, VoiceProfile> = {};
for (const locale of enabled) {
const suffix = locale.toUpperCase();
profiles[locale] = {
asr: {
provider: parseProviderId(
process.env[`VOICE_ASR_PROVIDER_${suffix}`],
`VOICE_ASR_PROVIDER_${suffix}`,
),
model: getEnvVarWithDefault(
`VOICE_ASR_MODEL_${suffix}`,
getEnvVarWithDefault('VOICE_ASR_MODEL', DEFAULT_ASR_MODEL),
),
},
llm: {
provider: parseProviderId(
process.env[`VOICE_LLM_PROVIDER_${suffix}`],
`VOICE_LLM_PROVIDER_${suffix}`,
),
model: getEnvVarWithDefault(
`VOICE_LLM_MODEL_${suffix}`,
getEnvVarWithDefault('VOICE_LLM_MODEL', DEFAULT_LLM_MODEL),
),
},
};
}
const maxRecordingMs = getEnvVarAsNumber('VOICE_MAX_RECORDING_MS', 120_000);
return {
openRouter: {
apiKey: process.env.OPENROUTER_API_KEY?.trim() || null,
baseUrl: getEnvVarWithDefault(
'OPENROUTER_BASE_URL',
'https://openrouter.ai/api/v1',
),
},
profiles,
maxRecordingMs: maxRecordingMs > 0 ? maxRecordingMs : null,
throttle: {
ttl: getEnvVarAsNumber('VOICE_THROTTLE_TTL', 60),
limit: getEnvVarAsNumber('VOICE_THROTTLE_LIMIT', 6),
},
};
}

10
backend/src/main.ts
View File

@@ -1,6 +1,8 @@
// backend/src/main.ts
import { NestFactory } from '@nestjs/core';
import { urlencoded } from 'express';
import { AppModule } from './app.module';
import { createJsonBodyParser } from './common/body-parsers';
import { ValidationPipe } from '@nestjs/common';
import cookieParser from 'cookie-parser'; // 👈 Change this line!
import { DocumentBuilder, SwaggerModule } from '@nestjs/swagger';
@@ -23,7 +25,13 @@ console.log = (...args) => {
};
async function bootstrap() {
const app = await NestFactory.create(AppModule);
// bodyParser is disabled so the JSON parsers can be registered in an explicit order below;
// Nest's built-in one would otherwise reject a voice recording at 100 kb.
const app = await NestFactory.create(AppModule, { bodyParser: false });
// Voice needs a larger JSON limit than everything else; see body-parsers.ts.
app.use(createJsonBodyParser());
app.use(urlencoded({ extended: true }));
app.useGlobalFilters(new HttpExceptionFilter());

15
backend/src/modules/treatments/treatment.utils.ts
View File

@@ -1,9 +1,4 @@
const FDI_TOOTH_IDS = new Set([
'11', '12', '13', '14', '15', '16', '17', '18',
'21', '22', '23', '24', '25', '26', '27', '28',
'31', '32', '33', '34', '35', '36', '37', '38',
'41', '42', '43', '44', '45', '46', '47', '48',
]);
import { FDI_TOOTH_IDS } from '../../common/fdi';
export function normalizeTeeth(teeth: unknown): string[] {
if (!Array.isArray(teeth)) {
@@ -35,7 +30,10 @@ export function normalizeToothSelectionGroups(
for (const row of value) {
if (!row || typeof row !== 'object') continue;
const rec = row as Record<string, unknown>;
const groupId = typeof rec.groupId === 'string' && rec.groupId.trim() ? rec.groupId.trim() : '';
const groupId =
typeof rec.groupId === 'string' && rec.groupId.trim()
? rec.groupId.trim()
: '';
if (!groupId) continue;
const kind = rec.kind === 'connected' ? 'connected' : 'single';
const teeth = normalizeTeeth(rec.teeth);
@@ -64,7 +62,8 @@ export function generateTreatmentTitle(
}
const parts = cases.map((c) => {
const label = c.treatmentType.charAt(0).toUpperCase() + c.treatmentType.slice(1);
const label =
c.treatmentType.charAt(0).toUpperCase() + c.treatmentType.slice(1);
if (c.teeth.length > 0) {
return `${label} ${c.teeth.join(', ')}`;
}

58
backend/src/modules/voice/dto/voice.dto.ts Normal file
View File

@@ -0,0 +1,58 @@
import {
IsBase64,
IsIn,
IsInt,
IsString,
MaxLength,
Min,
} from 'class-validator';
import { ErrorCode } from '../../../common/errors/error-codes';
/** Containers OpenRouter's transcription endpoint accepts, and MediaRecorder can produce. */
export const VOICE_AUDIO_FORMATS = [
'webm',
'mp4',
'm4a',
'aac',
'ogg',
'wav',
'mp3',
'flac',
] as const;
export type VoiceAudioFormat = (typeof VOICE_AUDIO_FORMATS)[number];
/** Locales the app ships; a profile still has to be configured for one to be usable. */
export const VOICE_LOCALES = ['en', 'fa', 'nl'] as const;
export class ExtractVoiceDto {
/** Base64 audio, no data: prefix. Well above a 2-minute opus clip (~400 KB). */
@IsString()
@IsBase64()
// Both name their own code: the shared map sends `maxLength` to
// VALIDATION_FIELD_REQUIRED and `isIn` to VALIDATION_LANGUAGE_INVALID, neither of which
// is true here.
@MaxLength(8_000_000, { message: ErrorCode.VOICE_CLIP_TOO_LONG })
audio: string;
@IsIn(VOICE_AUDIO_FORMATS, { message: ErrorCode.VOICE_UNSUPPORTED_FORMAT })
format: VoiceAudioFormat;
/** The clinician's IANA zone; "today" is derived from it, never sent by the client. */
@IsString()
@MaxLength(64)
timeZone: string;
/** Required, not optional — omitting it would bypass VOICE_MAX_RECORDING_MS entirely. */
@IsInt()
@Min(0)
durationMs: number;
/**
* The locale the UI offered the microphone in. Sent explicitly because `user.language` can
* diverge from the URL locale, and a mismatch transcribes Persian with an English hint and
* anchors "next Thursday" to the wrong week start.
*/
@IsIn(VOICE_LOCALES)
locale: string;
}

385
backend/src/modules/voice/due-date.resolver.spec.ts Normal file
View File

@@ -0,0 +1,385 @@
import { resolveDueDate, weekStartForLocale } from './due-date.resolver';
import type { DueIntent } from './voice.types';
// 2025-10-11 is a Saturday — the first day of the Iranian week.
const FA_WEEK = 6; // Saturday
const EU_WEEK = 1; // Monday
const SATURDAY = '2025-10-11';
const THURSDAY = '2025-10-16';
describe('resolveDueDate', () => {
describe('weekday intents', () => {
it('resolves "this <weekday>" to the coming occurrence', () => {
const result = resolveDueDate(
{ kind: 'weekday', weekday: 'thursday', which: 'this' },
SATURDAY,
FA_WEEK,
);
expect(result.dueDate).toBe(THURSDAY); // Sat -> Thu is 5 days
});
it('resolves "next <weekday>" to that weekday in the following week', () => {
const result = resolveDueDate(
{ kind: 'weekday', weekday: 'thursday', which: 'next' },
SATURDAY,
FA_WEEK,
);
expect(result.dueDate).toBe('2025-10-23');
});
it('anchors "next" to the week, not to "this" plus seven', () => {
// Said on Thursday 2025-10-16, the Iranian week runs Sat 10-18 .. Fri 10-24, so its
// Thursday is 10-23. Adding a week to "this Thursday" (already 10-23) would
// overshoot to 10-30 — a lab case a week late.
expect(
resolveDueDate(
{ kind: 'weekday', weekday: 'thursday', which: 'next' },
THURSDAY,
FA_WEEK,
).dueDate,
).toBe('2025-10-23');
});
it('lets "this" and "next" coincide when they name the same day', () => {
// On a Thursday, "the coming Saturday" and "Saturday next week" are both 10-18.
for (const which of ['this', 'next'] as const) {
expect(
resolveDueDate(
{ kind: 'weekday', weekday: 'saturday', which },
THURSDAY,
FA_WEEK,
).dueDate,
).toBe('2025-10-18');
}
});
it('reads "by Thursday" said on a Thursday as the next one, not today', () => {
// A deadline of today is almost never what was meant.
expect(
resolveDueDate(
{ kind: 'weekday', weekday: 'thursday', which: 'this' },
THURSDAY,
FA_WEEK,
).dueDate,
).toBe('2025-10-23');
});
it('never resolves a weekday into the past', () => {
for (const which of ['this', 'next'] as const) {
const result = resolveDueDate(
{ kind: 'weekday', weekday: 'sunday', which },
THURSDAY,
FA_WEEK,
);
expect(result.dueDate).not.toBeNull();
expect(result.dueDate! > THURSDAY).toBe(true);
}
});
it('anchors "next" to a Monday week for en and nl', () => {
// The same sentence means a different day depending on where the week starts.
// Said on Saturday 10-11: the Monday-start week is 10-13..10-19, Thursday = 10-16.
// The Saturday-start week is 10-18..10-24, Thursday = 10-23.
const intent = {
kind: 'weekday',
weekday: 'thursday',
which: 'next',
} as const;
expect(resolveDueDate(intent, SATURDAY, EU_WEEK).dueDate).toBe(
'2025-10-16',
);
expect(resolveDueDate(intent, SATURDAY, FA_WEEK).dueDate).toBe(
'2025-10-23',
);
});
it('maps each locale to its week start', () => {
expect(weekStartForLocale('fa')).toBe(FA_WEEK);
expect(weekStartForLocale('en')).toBe(EU_WEEK);
expect(weekStartForLocale('nl')).toBe(EU_WEEK);
expect(weekStartForLocale('unknown')).toBe(EU_WEEK);
});
it('treats a missing qualifier as "this" rather than failing', () => {
// A bare weekday carries no qualifier; failing would discard a real spoken deadline.
expect(
resolveDueDate(
{
kind: 'weekday',
weekday: 'thursday',
which: null,
} as unknown as DueIntent,
SATURDAY,
FA_WEEK,
).dueDate,
).toBe(THURSDAY);
});
it('rejects an unknown weekday', () => {
expect(
resolveDueDate(
{
kind: 'weekday',
weekday: 'caturday',
which: 'this',
} as unknown as DueIntent,
SATURDAY,
FA_WEEK,
).dueDate,
).toBeNull();
});
});
describe('offset intents', () => {
it('adds days, weeks and months', () => {
expect(
resolveDueDate(
{ kind: 'offset', unit: 'day', amount: 1 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBe('2025-10-12');
expect(
resolveDueDate(
{ kind: 'offset', unit: 'week', amount: 1 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBe('2025-10-18');
expect(
resolveDueDate(
{ kind: 'offset', unit: 'month', amount: 1 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBe('2025-11-11');
});
it('clamps to the end of a shorter month', () => {
expect(
resolveDueDate(
{ kind: 'offset', unit: 'month', amount: 1 },
'2025-01-31',
).dueDate,
).toBe('2025-02-28');
expect(
resolveDueDate(
{ kind: 'offset', unit: 'month', amount: 1 },
'2024-01-31',
).dueDate,
).toBe('2024-02-29');
});
it('crosses a year boundary', () => {
expect(
resolveDueDate(
{ kind: 'offset', unit: 'month', amount: 3 },
'2025-11-15',
).dueDate,
).toBe('2026-02-15');
expect(
resolveDueDate(
{ kind: 'offset', unit: 'day', amount: 30 },
'2025-12-20',
).dueDate,
).toBe('2026-01-19');
});
it('rejects negative, fractional and absurd amounts', () => {
for (const amount of [-1, 1.5, 10_000, Number.NaN]) {
expect(
resolveDueDate(
{ kind: 'offset', unit: 'day', amount },
SATURDAY,
FA_WEEK,
).dueDate,
).toBeNull();
}
});
});
describe('jalali intents', () => {
it('converts by arithmetic, not inference', () => {
expect(
resolveDueDate(
{ kind: 'jalali', jy: 1404, jm: 7, jd: 25 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBe('2025-10-17');
});
it('handles the leap-year Esfand 30', () => {
expect(
resolveDueDate(
{ kind: 'jalali', jy: 1403, jm: 12, jd: 30 },
'2025-03-01',
).dueDate,
).toBe('2025-03-20');
});
it('rejects Esfand 30 in a non-leap year', () => {
const result = resolveDueDate(
{ kind: 'jalali', jy: 1404, jm: 12, jd: 30 },
SATURDAY,
);
expect(result.dueDate).toBeNull();
expect(result.unresolved?.reason).toBe('invalid_date');
});
});
describe('gregorian intents', () => {
it('accepts a real date and rejects an impossible one', () => {
expect(
resolveDueDate(
{ kind: 'gregorian', y: 2025, m: 10, d: 17 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBe('2025-10-17');
expect(
resolveDueDate(
{ kind: 'gregorian', y: 2025, m: 2, d: 30 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBeNull();
expect(
resolveDueDate(
{ kind: 'gregorian', y: 2025, m: 13, d: 1 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBeNull();
});
});
describe('guard rails', () => {
it('treats a past date as unresolved', () => {
const result = resolveDueDate(
{ kind: 'gregorian', y: 2020, m: 1, d: 1 },
SATURDAY,
);
expect(result.dueDate).toBeNull();
expect(result.unresolved?.reason).toBe('invalid_date');
});
it('treats a date decades away as unresolved', () => {
expect(
resolveDueDate(
{ kind: 'gregorian', y: 2099, m: 1, d: 1 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBeNull();
});
it('accepts today itself via a zero-day offset', () => {
expect(
resolveDueDate(
{ kind: 'offset', unit: 'day', amount: 0 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBe(SATURDAY);
});
it('reports no due date at all when the model said nothing, without flagging it', () => {
expect(resolveDueDate(null, SATURDAY)).toEqual({
dueDate: null,
unresolved: null,
});
expect(resolveDueDate(undefined, SATURDAY)).toEqual({
dueDate: null,
unresolved: null,
});
});
it('treats an object with no kind as no deadline, not a lost one', () => {
// A blank "heard but lost" row in front of a clinician who never mentioned a
// deadline is worse than saying nothing.
expect(resolveDueDate({} as never, SATURDAY)).toEqual({
dueDate: null,
unresolved: null,
});
});
it('flags an unrecognised kind and names it', () => {
const result = resolveDueDate({ kind: 'lunar_month' } as never, SATURDAY);
expect(result.dueDate).toBeNull();
expect(result.unresolved).toEqual({
spoken: 'lunar_month',
reason: 'invalid_date',
});
});
it('flags a non-object deadline instead of silently dropping it', () => {
// A bare string is a deadline we failed to parse, not an absent one — the clinician
// must see that something was heard and lost.
for (const bad of ['next thursday', 42, true]) {
const result = resolveDueDate(bad as never, SATURDAY);
expect(result.dueDate).toBeNull();
expect(result.unresolved?.reason).toBe('invalid_date');
}
expect(
resolveDueDate('next thursday' as never, SATURDAY).unresolved?.spoken,
).toBe('next thursday');
});
it('degrades rather than throwing on a malformed today or intent', () => {
expect(
resolveDueDate({ kind: 'offset', unit: 'day', amount: 1 }, 'not-a-date')
.dueDate,
).toBeNull();
expect(
resolveDueDate({ kind: 'nope' } as unknown as DueIntent, SATURDAY)
.dueDate,
).toBeNull();
});
it('echoes what was heard so the review sheet can show it', () => {
const result = resolveDueDate(
{ kind: 'jalali', jy: 1404, jm: 12, jd: 30 },
SATURDAY,
);
expect(result.unresolved?.spoken).toBe('1404/12/30');
});
});
});
describe('what an unresolvable deadline quotes back', () => {
// The wire shape allows nulls in every field and toVoiceIntent casts rather than
// checks, so these reach the resolver intact. The sheet renders `spoken` verbatim.
it('never puts "null" or "NaN" in front of the clinician', () => {
const bad = [
{ kind: 'weekday', weekday: null, which: null },
{ kind: 'offset', unit: null, amount: null },
{ kind: 'offset', unit: 'day', amount: Number.NaN },
{ kind: 'jalali', jy: null, jm: 7, jd: 25 },
{ kind: 'gregorian', y: 2026, m: null, d: null },
];
for (const intent of bad) {
const result = resolveDueDate(
intent as unknown as DueIntent,
SATURDAY,
FA_WEEK,
);
expect(result.dueDate).toBeNull();
expect(result.unresolved?.spoken ?? '').not.toMatch(/null|NaN/);
}
});
it('still quotes a deadline it did understand the words of', () => {
const result = resolveDueDate(
{
kind: 'weekday',
weekday: 'thursday',
which: null,
} as unknown as DueIntent,
SATURDAY,
FA_WEEK,
);
// A weekday with no "this/next" resolves, so nothing is quoted back at all.
expect(result.dueDate).not.toBeNull();
});
});

234
backend/src/modules/voice/due-date.resolver.ts Normal file
View File

@@ -0,0 +1,234 @@
import { jalaliToIsoDate } from '../../common/jalali';
import { civilDateJsWeekday } from '../../common/zoned-civil-time';
import type { DueIntent, UnresolvedItem, Weekday } from './voice.types';
/**
* Spoken deadline → ISO civil date.
*
* The model never does calendar arithmetic; it says what it heard and this decides what
* that means. Jalali conversion in particular is arithmetic here, not inference — an LLM
* asked to turn "۲۵ مهر" into ISO answers confidently and is often wrong, and
* `@IsDateString()` would accept the wrong answer.
*
* Works entirely in civil dates. The caller derives `todayIso` from the actor's IANA zone
* (see `civilDateInZone`) rather than passing a zone in here, so nothing in this file has
* to reason about instants.
*/
const WEEKDAY_TO_JS: Record<Weekday, number> = {
saturday: 6,
sunday: 0,
monday: 1,
tuesday: 2,
wednesday: 3,
thursday: 4,
friday: 5,
};
const MAX_DAYS_AHEAD = 365 * 5;
export type DueResolution = {
dueDate: string | null;
unresolved: UnresolvedItem | null;
};
const ISO_DATE = /^\d{4}-\d{2}-\d{2}$/;
function isRealCivilDate(iso: string): boolean {
if (!ISO_DATE.test(iso)) return false;
const [y, m, d] = iso.split('-').map(Number);
if (m < 1 || m > 12 || d < 1 || d > 31) return false;
const probe = new Date(Date.UTC(y, m - 1, d));
return (
probe.getUTCFullYear() === y &&
probe.getUTCMonth() === m - 1 &&
probe.getUTCDate() === d
);
}
function toIso(utcMs: number): string {
return new Date(utcMs).toISOString().slice(0, 10);
}
function utcMsOf(iso: string): number {
const [y, m, d] = iso.split('-').map(Number);
return Date.UTC(y, m - 1, d);
}
function addDays(iso: string, days: number): string {
return toIso(utcMsOf(iso) + days * 86_400_000);
}
/** Calendar-month addition with end-of-month clamping (31 Jan + 1 month = 28/29 Feb). */
function addMonths(iso: string, months: number): string {
const [y, m, d] = iso.split('-').map(Number);
const targetMonthIndex = m - 1 + months;
const targetYear = y + Math.floor(targetMonthIndex / 12);
const targetMonth = ((targetMonthIndex % 12) + 12) % 12;
const lastDay = new Date(
Date.UTC(targetYear, targetMonth + 1, 0),
).getUTCDate();
return toIso(Date.UTC(targetYear, targetMonth, Math.min(d, lastDay)));
}
function unresolved(spoken: string): DueResolution {
return { dueDate: null, unresolved: { spoken, reason: 'invalid_date' } };
}
/**
* What to quote back when a deadline could not be resolved. Every field is nullable on the
* wire and `toVoiceIntent` casts rather than checks, and the sheet renders this verbatim —
* so a half-classified deadline must fall back to '', not to `"null null"`.
*/
function describe(intent: DueIntent): string {
const usable = (value: unknown): value is number =>
typeof value === 'number' && Number.isFinite(value);
switch (intent?.kind) {
case 'weekday':
// `which` is legitimately null (it means "this"), the weekday is not.
return [intent.which, intent.weekday]
.filter((part) => typeof part === 'string')
.join(' ');
case 'offset':
return usable(intent.amount)
? `+${intent.amount} ${intent.unit ?? ''}`.trim()
: '';
case 'jalali':
return [intent.jy, intent.jm, intent.jd].every(usable)
? `${intent.jy}/${intent.jm}/${intent.jd}`
: '';
case 'gregorian':
return [intent.y, intent.m, intent.d].every(usable)
? `${intent.y}-${intent.m}-${intent.d}`
: '';
default: {
// An unrecognised `kind`, already established as a string — echo what was heard.
const kind = (intent as { kind?: unknown })?.kind;
return typeof kind === 'string' ? kind : '';
}
}
}
/**
* "Next Thursday" is week-relative, so this changes the answer: the Iranian week starts
* Saturday, the Dutch and English week Monday. Hardcoding either puts the other locale's
* deadline a week out.
*/
const WEEK_START_BY_LOCALE: Record<string, number> = {
fa: WEEKDAY_TO_JS.saturday,
en: WEEKDAY_TO_JS.monday,
nl: WEEKDAY_TO_JS.monday,
};
const DEFAULT_WEEK_START = WEEKDAY_TO_JS.monday;
export function weekStartForLocale(locale: string): number {
return WEEK_START_BY_LOCALE[locale] ?? DEFAULT_WEEK_START;
}
function startOfWeek(iso: string, weekStartJs: number): string {
const back = (civilDateJsWeekday(iso) - weekStartJs + 7) % 7;
return addDays(iso, -back);
}
/**
* `'this'` is occurrence-anchored: the soonest occurrence strictly after today, so it can
* never resolve into the past.
*
* `'next'` is *week*-anchored, not "this plus seven" — adding a week to `'this'` overshoots
* by seven days whenever `'this'` has already rolled into next week. The two legitimately
* coincide: said on a Thursday, "the coming Saturday" and "Saturday next week" are one day.
*/
function resolveWeekday(
intent: Extract<DueIntent, { kind: 'weekday' }>,
todayIso: string,
weekStartJs: number,
) {
const targetJs = WEEKDAY_TO_JS[intent.weekday];
if (targetJs === undefined) return null;
// A bare weekday ("پنجشنبه") carries no qualifier, and the model may leave `which`
// null. Treat that as 'this' rather than failing an utterance that named a real day.
const which = intent.which === 'next' ? 'next' : 'this';
if (which === 'this') {
const todayJs = civilDateJsWeekday(todayIso);
let delta = (targetJs - todayJs + 7) % 7;
if (delta === 0) delta = 7;
return addDays(todayIso, delta);
}
if (which === 'next') {
const offsetInWeek = (targetJs - weekStartJs + 7) % 7;
return addDays(startOfWeek(todayIso, weekStartJs), 7 + offsetInWeek);
}
return null;
}
function resolveOffset(
intent: Extract<DueIntent, { kind: 'offset' }>,
todayIso: string,
) {
const { amount, unit } = intent;
if (!Number.isInteger(amount) || amount < 0) return null;
if (unit === 'day')
return amount <= MAX_DAYS_AHEAD ? addDays(todayIso, amount) : null;
if (unit === 'week')
return amount <= 260 ? addDays(todayIso, amount * 7) : null;
if (unit === 'month')
return amount <= 60 ? addMonths(todayIso, amount) : null;
return null;
}
export function resolveDueDate(
intent: DueIntent | null | undefined,
todayIso: string,
weekStartJs: number = DEFAULT_WEEK_START,
): DueResolution {
// Absent is not an error — most utterances carry no deadline.
if (intent === null || intent === undefined) {
return { dueDate: null, unresolved: null };
}
if (typeof intent !== 'object') {
return unresolved(String(intent).slice(0, 120));
}
// No `kind` at all says nothing about a deadline, so it is not "heard but lost". An
// *unrecognised* kind did try to say something, and is flagged below.
if (typeof (intent as { kind?: unknown }).kind !== 'string') {
return { dueDate: null, unresolved: null };
}
if (!isRealCivilDate(todayIso)) {
return unresolved(describe(intent));
}
let resolved: string | null = null;
switch (intent.kind) {
case 'weekday':
resolved = resolveWeekday(intent, todayIso, weekStartJs);
break;
case 'offset':
resolved = resolveOffset(intent, todayIso);
break;
case 'jalali':
resolved = jalaliToIsoDate(intent.jy, intent.jm, intent.jd);
break;
case 'gregorian': {
const candidate = `${String(intent.y).padStart(4, '0')}-${String(intent.m).padStart(2, '0')}-${String(intent.d).padStart(2, '0')}`;
resolved = isRealCivilDate(candidate) ? candidate : null;
break;
}
default:
resolved = null;
}
if (!resolved) return unresolved(describe(intent));
// A date the model invented can land anywhere; past or decades away is not a deadline.
const daysAhead = (utcMsOf(resolved) - utcMsOf(todayIso)) / 86_400_000;
if (daysAhead < 0 || daysAhead > MAX_DAYS_AHEAD)
return unresolved(describe(intent));
return { dueDate: resolved, unresolved: null };
}

93
backend/src/modules/voice/extraction.prompt.ts Normal file
View File

@@ -0,0 +1,93 @@
import type { ExtractionCatalog } from './voice.providers';
/** Locale-specific guidance. Only the tooth vocabulary and numbering habits differ. */
const LOCALE_NOTES: Record<string, string> = {
fa: [
'The clinician is speaking Persian. A tooth number can be said as a whole number',
'("بیست و شش" = 26), digit by digit ("دو شش" = 26), or with a lead-in',
'("دندون شماره ۲۶"). Digits may arrive in Persian or Latin script — either way, copy',
'the number into "fdi" as two Latin digits. The descriptive form is quadrant-relative:',
'"شش بالا راست" = upper right six -> arch "upper", side "patient_right", position 6.',
].join(' '),
nl: [
'The clinician is speaking Dutch, where FDI is standard. "zesentwintig" and "26" are',
'tooth 26. The descriptive form is "rechtsboven zes" = upper right six.',
].join(' '),
en: [
'The clinician is speaking English and uses FDI. "twenty-six", "two six" and "26" are',
'all tooth 26. The descriptive form is "upper right six".',
].join(' '),
};
function codeList(entries: { code: string; label: string }[]): string {
if (entries.length === 0) return '(none available)';
return entries.map((e) => `- ${e.code} = ${e.label}`).join('\n');
}
export function buildExtractionPrompt(
transcript: string,
catalog: ExtractionCatalog,
localeHint: string,
) {
const localeNote = LOCALE_NOTES[localeHint] ?? LOCALE_NOTES.en;
const system = [
'You extract structured dental treatment data from a transcript of a clinician speaking.',
'You are a parser, not an assistant: report only what was said.',
'',
'HARD RULES',
'1. Never invent a code. treatmentType, prosthesisDefaultType and prosthesisOverrides[].type',
' must be codes from the lists below. labId must be an id from the lab list. If what you',
' heard is not in a list, use null.',
'2. "side" is always the PATIENT\'s side. The patient\'s upper right is quadrant 1. Never',
" flip to the viewer's point of view.",
'3. Never do calendar arithmetic. Report the deadline as it was said, using due.kind.',
' If no deadline was mentioned, use due.kind = "none".',
'4. Copy the exact spoken words for each tooth into "spoken", so the clinician can see',
' what was heard.',
'5. If you are unsure about a value, use null. A missing field is recoverable; a wrong',
' one is not.',
'',
'TOOTH NUMBERS',
'A number the clinician says for a tooth IS that tooth\'s FDI code. Put it in "fdi" as',
'two digits. FDI is built from the two digits:',
" first digit = quadrant, from the PATIENT's own point of view —",
' 1 upper right, 2 upper left, 3 lower left, 4 lower right.',
' (5-8 are those same four quadrants in primary/deciduous teeth.)',
' second digit = position from the midline — 1 central incisor ... 8 third molar.',
'So 26 is the upper left first molar, and 47 is the lower right second molar.',
'',
'- Use "arch" + "side" + "position" only when the tooth is DESCRIBED rather than',
' numbered ("upper right six" -> arch "upper", side "patient_right", position 6).',
'- A single digit is a position, never an FDI code. If a single digit is said with no',
' quadrant words at all, set "position" and leave "arch" and "side" null. Never pick a',
' quadrant that was not said.',
'- If a number is given AND the quadrant is spelled out as well, still use "fdi".',
'- Not every number is a tooth. Dates, counts and quantities ("two teeth", "the 26th")',
' are not teeth, and must never appear in the teeth list.',
'',
localeNote,
'',
'TREATMENT TYPE CODES',
codeList(catalog.treatmentTypes),
'',
'PROSTHESIS TYPE CODES',
codeList(catalog.prosthesisTypes),
'',
'LABS THIS CLINIC CAN SEND TO',
catalog.labs.length > 0
? catalog.labs.map((l) => `- ${l.id} = ${l.name}`).join('\n')
: '(none linked — labId must be null)',
'',
'OTHER FIELDS',
'- connectedSpans: only for bridges or splinted units. Endpoints inclusive.',
'- comment: clinical notes, in the language spoken. Omit the parts already captured as',
' treatment type, teeth or deadline.',
'- labMatchExact: true only when the spoken name matched a lab name exactly.',
].join('\n');
return [
{ role: 'system' as const, content: system },
{ role: 'user' as const, content: transcript },
];
}

356
backend/src/modules/voice/extraction.resolver.spec.ts Normal file
View File

@@ -0,0 +1,356 @@
import {
resolveConnectedSpans,
resolveProsthesis,
resolveVoiceIntent,
type ResolveContext,
} from './extraction.resolver';
import type { ToothIntent, VoiceIntent } from './voice.types';
const tooth = (fdi: string, spoken = fdi): ToothIntent => ({
kind: 'explicit',
fdi,
spoken,
});
const CTX: ResolveContext = {
todayIso: '2025-10-11',
weekStartJs: 6, // Saturday — the fa week
treatmentTypeCodes: new Set(['restoration', 'prosthesis', 'extraction']),
prosthesisTypeCodes: new Set(['monolithic_zirconia', 'pfm_crown']),
linkedLabIds: new Set(['lab-sina', 'lab-mehr']),
};
describe('resolveConnectedSpans', () => {
it('selects the teeth between the endpoints, which were never named', () => {
// "a bridge from 14 to 16" must select 15 too.
const result = resolveConnectedSpans(
[{ from: tooth('14'), to: tooth('16') }],
[],
);
expect(result.teeth).toEqual(['14', '15', '16']);
expect(result.groups).toEqual([
{ groupId: 'voice-c1', kind: 'connected', teeth: ['16', '15', '14'] },
]);
});
it('orders group teeth along the arch, not lexically', () => {
const result = resolveConnectedSpans(
[{ from: tooth('16'), to: tooth('14') }],
[],
);
expect(result.groups[0].teeth).toEqual(['16', '15', '14']);
});
it('spans the midline', () => {
const result = resolveConnectedSpans(
[{ from: tooth('12'), to: tooth('22') }],
[],
);
expect(result.groups[0].teeth).toEqual(['12', '11', '21', '22']);
});
it('merges overlapping spans into one bridge', () => {
const result = resolveConnectedSpans(
[
{ from: tooth('14'), to: tooth('16') },
{ from: tooth('15'), to: tooth('17') },
],
[],
);
const connected = result.groups.filter((g) => g.kind === 'connected');
expect(connected).toHaveLength(1);
expect(connected[0].teeth).toEqual(['17', '16', '15', '14']);
});
it('gives loose teeth their own single groups', () => {
const result = resolveConnectedSpans(
[{ from: tooth('14'), to: tooth('15') }],
['26'],
);
expect(result.groups).toEqual([
{ groupId: 'voice-c1', kind: 'connected', teeth: ['15', '14'] },
{ groupId: 'voice-s-26', kind: 'single', teeth: ['26'] },
]);
});
it('never produces a one-tooth connected group', () => {
const result = resolveConnectedSpans(
[{ from: tooth('14'), to: tooth('14') }],
[],
);
expect(result.groups).toEqual([
{ groupId: 'voice-s-14', kind: 'single', teeth: ['14'] },
]);
expect(result.unresolved).toEqual([]);
});
it('reports a cross-arch span rather than guessing', () => {
const result = resolveConnectedSpans(
[{ from: tooth('14', 'چهارده'), to: tooth('44', 'چهل و چهار') }],
[],
);
expect(result.groups).toEqual([]);
expect(result.unresolved).toEqual([
{ spoken: 'چهارده → چهل و چهار', reason: 'span_not_same_arch' },
]);
});
it('reports a span with an unresolvable endpoint', () => {
const result = resolveConnectedSpans(
[{ from: tooth('14'), to: tooth('99') }],
[],
);
expect(result.unresolved[0].reason).toBe('malformed');
});
it('survives a non-array', () => {
expect(resolveConnectedSpans(undefined as never, ['14']).teeth).toEqual([
'14',
]);
});
});
describe('resolveProsthesis', () => {
const allowed = CTX.prosthesisTypeCodes;
it('expands the default across every tooth', () => {
const result = resolveProsthesis(
{ defaultType: 'monolithic_zirconia', overrides: [] },
['14', '15'],
allowed,
);
expect(result.prosthesis?.byTooth).toEqual({
'14': 'monolithic_zirconia',
'15': 'monolithic_zirconia',
});
expect(result.prosthesis?.complete).toBe(true);
});
it('applies per-tooth overrides on top of the default', () => {
const result = resolveProsthesis(
{
defaultType: 'monolithic_zirconia',
overrides: [{ tooth: tooth('26'), type: 'pfm_crown' }],
},
['14', '26'],
allowed,
);
expect(result.prosthesis?.byTooth).toEqual({
'14': 'monolithic_zirconia',
'26': 'pfm_crown',
});
expect(result.prosthesis?.complete).toBe(true);
});
it('marks the map incomplete when a tooth ends up untyped', () => {
// Unshippable: assertCompleteToothProsthesisMap would reject this at dispatch.
const result = resolveProsthesis(
{
defaultType: null,
overrides: [{ tooth: tooth('14'), type: 'pfm_crown' }],
},
['14', '15'],
allowed,
);
expect(result.prosthesis?.complete).toBe(false);
expect(result.prosthesis?.missingTeeth).toEqual(['15']);
});
it('rejects a catalog code the clinic does not have', () => {
const result = resolveProsthesis(
{ defaultType: 'gold_foil', overrides: [] },
['14'],
allowed,
);
// Nothing usable was said, so there is no prosthesis to show — not an empty one.
expect(result.prosthesis).toBeNull();
expect(result.unresolved).toEqual([
{ spoken: 'gold_foil', reason: 'unknown_catalog_code' },
]);
});
it('ignores an override for a tooth that is not selected', () => {
const result = resolveProsthesis(
{
defaultType: 'monolithic_zirconia',
overrides: [{ tooth: tooth('37', 'سی و هفت'), type: 'pfm_crown' }],
},
['14'],
allowed,
);
expect(result.prosthesis?.byTooth).toEqual({ '14': 'monolithic_zirconia' });
expect(result.unresolved).toEqual([
{ spoken: 'سی و هفت', reason: 'tooth_not_selected' },
]);
});
it('reports no prosthesis at all when the object carries nothing usable', () => {
// An empty-but-present map would paint a plain restoration with a fabricated
// "incomplete, cannot ship" warning.
for (const empty of [{ defaultType: null, overrides: [] }, {} as never]) {
expect(
resolveProsthesis(empty, ['14', '15'], allowed).prosthesis,
).toBeNull();
}
});
it('reports no prosthesis when there are no teeth to type', () => {
const result = resolveProsthesis(
{ defaultType: 'monolithic_zirconia', overrides: [] },
[],
allowed,
);
expect(result.prosthesis).toBeNull();
});
it('distinguishes a tooth it could not understand from one that is not selected', () => {
// Different corrective actions: add the tooth, versus repeat yourself.
const result = resolveProsthesis(
{
defaultType: 'monolithic_zirconia',
overrides: [
{
tooth: { kind: 'explicit', fdi: '99', spoken: 'نود و نه' },
type: 'pfm_crown',
},
],
},
['14'],
allowed,
);
expect(result.unresolved).toEqual([
{ spoken: 'نود و نه', reason: 'malformed' },
]);
});
it('returns null when no prosthesis was spoken', () => {
expect(resolveProsthesis(null, ['14'], allowed).prosthesis).toBeNull();
});
});
describe('resolveVoiceIntent', () => {
const base: VoiceIntent = {
treatmentType: 'restoration',
teeth: [tooth('14'), tooth('15')],
connectedSpans: [],
comment: ' حساسیت به سرما ',
prosthesis: null,
labId: null,
labMatchExact: false,
due: null,
};
it('composes a plain restoration', () => {
const result = resolveVoiceIntent(base, CTX);
expect(result.treatmentType).toBe('restoration');
expect(result.teeth).toEqual(['14', '15']);
expect(result.comment).toBe('حساسیت به سرما');
expect(result.prosthesis).toBeNull();
expect(result.unresolved).toEqual([]);
});
it('rejects a treatment type outside the catalog', () => {
const result = resolveVoiceIntent(
{ ...base, treatmentType: 'teeth_whitening' },
CTX,
);
expect(result.treatmentType).toBeNull();
expect(result.unresolved).toContainEqual({
spoken: 'teeth_whitening',
reason: 'unknown_catalog_code',
});
});
it('drops a lab id the clinic is not linked to', () => {
// Shipping to a lab the clinic never named is worse than shipping nowhere.
const result = resolveVoiceIntent(
{ ...base, labId: 'lab-elsewhere', labMatchExact: true },
CTX,
);
expect(result.labId).toBeNull();
expect(result.labMatchExact).toBe(false);
});
it('keeps a linked lab and its exactness flag', () => {
const result = resolveVoiceIntent(
{ ...base, labId: 'lab-sina', labMatchExact: true },
CTX,
);
expect(result.labId).toBe('lab-sina');
expect(result.labMatchExact).toBe(true);
});
it('reports a hallucinated lab rather than dropping it silently', () => {
// A near-miss lab id must not look identical to "no lab was spoken".
const result = resolveVoiceIntent({ ...base, labId: 'lab-elsewhere' }, CTX);
// The id is not what the clinician said — quoting it back shows them a raw UUID.
expect(result.unresolved).toContainEqual({
spoken: '',
reason: 'unknown_catalog_code',
});
});
it('never reports an inexact match as exact when the lab was dropped', () => {
const result = resolveVoiceIntent(
{ ...base, labId: null, labMatchExact: true },
CTX,
);
expect(result.labMatchExact).toBe(false);
});
it('applies prosthesis over the span-expanded tooth set', () => {
const result = resolveVoiceIntent(
{
...base,
treatmentType: 'prosthesis',
teeth: [tooth('14')],
connectedSpans: [{ from: tooth('14'), to: tooth('16') }],
prosthesis: { defaultType: 'monolithic_zirconia', overrides: [] },
},
CTX,
);
// 15 was never spoken but is part of the bridge, so it must carry a type too.
expect(result.teeth).toEqual(['14', '15', '16']);
expect(result.prosthesis?.complete).toBe(true);
expect(Object.keys(result.prosthesis!.byTooth).sort()).toEqual([
'14',
'15',
'16',
]);
});
it('resolves a due date through the same context', () => {
const result = resolveVoiceIntent(
{ ...base, due: { kind: 'weekday', weekday: 'thursday', which: 'this' } },
CTX,
);
expect(result.dueDate).toBe('2025-10-16');
});
it('collects unresolved items from every stage', () => {
const result = resolveVoiceIntent(
{
...base,
treatmentType: 'nope',
teeth: [tooth('51', 'شیری')],
connectedSpans: [{ from: tooth('14'), to: tooth('44') }],
due: { kind: 'jalali', jy: 1404, jm: 12, jd: 30 },
},
CTX,
);
const reasons = result.unresolved.map((u) => u.reason).sort();
expect(reasons).toEqual([
'invalid_date',
'not_permanent_tooth',
'span_not_same_arch',
'unknown_catalog_code',
]);
});
it('treats an empty comment as absent', () => {
expect(
resolveVoiceIntent({ ...base, comment: ' ' }, CTX).comment,
).toBeNull();
});
});

300
backend/src/modules/voice/extraction.resolver.ts Normal file
View File

@@ -0,0 +1,300 @@
import {
sameArch,
sortInArchOrder,
teethBetweenInclusive,
} from '../../common/fdi';
import { resolveDueDate } from './due-date.resolver';
import {
resolveToothIntent,
resolveToothIntents,
} from './tooth-intent.resolver';
import type {
ConnectedSpanIntent,
ProsthesisIntent,
ToothIntent,
UnresolvedItem,
VoiceIntent,
} from './voice.types';
export type ResolvedToothGroup = {
groupId: string;
kind: 'connected' | 'single';
teeth: string[];
};
export type ResolvedProsthesis = {
/** FDI code → prosthesis type code. */
byTooth: Record<string, string>;
/**
* True when every selected tooth carries a code. A prosthesis detail cannot be shipped
* otherwise (`assertCompleteToothProsthesisMap`), so the review sheet surfaces the gap
* here rather than letting it fail at dispatch.
*/
complete: boolean;
missingTeeth: string[];
};
export type ResolvedExtraction = {
treatmentType: string | null;
teeth: string[];
toothSelectionGroups: ResolvedToothGroup[];
comment: string | null;
prosthesis: ResolvedProsthesis | null;
labId: string | null;
labMatchExact: boolean;
dueDate: string | null;
unresolved: UnresolvedItem[];
};
export type ResolveContext = {
todayIso: string;
/** JS weekday index the clinician's week starts on — see weekStartForLocale. */
weekStartJs: number;
treatmentTypeCodes: ReadonlySet<string>;
prosthesisTypeCodes: ReadonlySet<string>;
linkedLabIds: ReadonlySet<string>;
};
function spokenOf(intent: ToothIntent): string {
const spoken = (intent as { spoken?: unknown })?.spoken;
return typeof spoken === 'string' && spoken.trim() ? spoken.trim() : '';
}
/** A code the model returned is only usable if it exists in the catalog we supplied it. */
function resolveCatalogCode(
value: unknown,
allowed: ReadonlySet<string>,
): string | null {
if (typeof value !== 'string') return null;
const trimmed = value.trim();
return trimmed && allowed.has(trimmed) ? trimmed : null;
}
/** Merge any span sets that share a tooth, so overlapping bridges become one group. */
function mergeOverlapping(sets: string[][]): string[][] {
const merged: string[][] = [];
for (const candidate of sets) {
let current = [...candidate];
let index = 0;
while (index < merged.length) {
if (merged[index].some((tooth) => current.includes(tooth))) {
current = [...new Set([...merged[index], ...current])];
merged.splice(index, 1);
index = 0;
continue;
}
index += 1;
}
merged.push(current);
}
return merged;
}
/**
* Span teeth join the selection: "a bridge from 14 to 16" selects 15 though it was never
* named. A cross-arch span is reported rather than guessed at, and a span collapsing to one
* tooth degrades to a single — there is no one-tooth bridge.
*/
export function resolveConnectedSpans(
spans: readonly ConnectedSpanIntent[],
selectedTeeth: readonly string[],
): {
groups: ResolvedToothGroup[];
teeth: string[];
unresolved: UnresolvedItem[];
} {
const unresolved: UnresolvedItem[] = [];
const connectedSets: string[][] = [];
// A span that collapses to one tooth still selected that tooth — it must not vanish.
const loneSpanTeeth: string[] = [];
const list: readonly ConnectedSpanIntent[] = Array.isArray(spans)
? (spans as readonly ConnectedSpanIntent[])
: [];
for (const span of list) {
const from = resolveToothIntent(span?.from);
const to = resolveToothIntent(span?.to);
const spoken = [spokenOf(span?.from), spokenOf(span?.to)]
.filter(Boolean)
.join(' → ');
if (!from || !to) {
unresolved.push({ spoken, reason: 'malformed' });
continue;
}
if (!sameArch(from, to)) {
unresolved.push({ spoken, reason: 'span_not_same_arch' });
continue;
}
const between = teethBetweenInclusive(from, to);
if (!between || between.length === 0) {
unresolved.push({ spoken, reason: 'malformed' });
continue;
}
if (between.length === 1) {
loneSpanTeeth.push(between[0]);
continue; // degrades to a single, below
}
connectedSets.push(between);
}
const groups: ResolvedToothGroup[] = [];
const claimed = new Set<string>();
mergeOverlapping(connectedSets).forEach((set, i) => {
const teeth = sortInArchOrder(set);
teeth.forEach((tooth) => claimed.add(tooth));
groups.push({ groupId: `voice-c${i + 1}`, kind: 'connected', teeth });
});
const spanTeeth = [...groups.flatMap((g) => g.teeth), ...loneSpanTeeth];
const singles = [...new Set([...selectedTeeth, ...spanTeeth])]
.filter((tooth) => !claimed.has(tooth))
.sort();
for (const tooth of singles) {
groups.push({
groupId: `voice-s-${tooth}`,
kind: 'single',
teeth: [tooth],
});
}
return {
groups,
teeth: [...new Set([...selectedTeeth, ...spanTeeth])].sort(),
unresolved,
};
}
/**
* A default across the selection, then per-tooth overrides — "همه زیرکونیا، ۲۶ پی‌اف‌ام" is
* how clinicians actually speak.
*/
export function resolveProsthesis(
intent: ProsthesisIntent | null | undefined,
teeth: readonly string[],
allowed: ReadonlySet<string>,
): { prosthesis: ResolvedProsthesis | null; unresolved: UnresolvedItem[] } {
if (!intent || typeof intent !== 'object')
return { prosthesis: null, unresolved: [] };
const unresolved: UnresolvedItem[] = [];
const defaultType = resolveCatalogCode(intent.defaultType, allowed);
if (intent.defaultType != null && !defaultType) {
unresolved.push({
spoken: String(intent.defaultType),
reason: 'unknown_catalog_code',
});
}
const byTooth: Record<string, string> = {};
const selected = new Set(teeth);
if (defaultType) {
for (const tooth of teeth) byTooth[tooth] = defaultType;
}
const overrides: ProsthesisIntent['overrides'] = Array.isArray(
intent.overrides,
)
? intent.overrides
: [];
for (const override of overrides) {
const tooth = resolveToothIntent(override?.tooth);
const type = resolveCatalogCode(override?.type, allowed);
const spoken = spokenOf(override?.tooth) || String(override?.type ?? '');
if (!tooth) {
unresolved.push({ spoken, reason: 'malformed' });
continue;
}
// A tooth we understood perfectly well but which is not part of this detail. Saying
// so is actionable ("add tooth 37, or drop it"); calling it malformed is not.
if (!selected.has(tooth)) {
unresolved.push({ spoken, reason: 'tooth_not_selected' });
continue;
}
if (!type) {
unresolved.push({ spoken, reason: 'unknown_catalog_code' });
continue;
}
byTooth[tooth] = type;
}
// Nothing usable was said about prosthesis. Returning an empty-but-present map would
// paint a plain restoration with a fabricated "incomplete, cannot ship" warning.
if (Object.keys(byTooth).length === 0) {
return { prosthesis: null, unresolved };
}
const missingTeeth = teeth.filter((tooth) => !byTooth[tooth]);
return {
prosthesis: {
byTooth,
complete: missingTeeth.length === 0,
missingTeeth,
},
unresolved,
};
}
/** Compose every resolver into the payload the review sheet renders. */
export function resolveVoiceIntent(
intent: VoiceIntent,
ctx: ResolveContext,
): ResolvedExtraction {
const unresolved: UnresolvedItem[] = [];
const toothResult = resolveToothIntents(intent?.teeth ?? []);
unresolved.push(...toothResult.unresolved);
const spanResult = resolveConnectedSpans(
intent?.connectedSpans ?? [],
toothResult.teeth,
);
unresolved.push(...spanResult.unresolved);
const treatmentType = resolveCatalogCode(
intent?.treatmentType,
ctx.treatmentTypeCodes,
);
if (intent?.treatmentType != null && !treatmentType) {
unresolved.push({
spoken: String(intent.treatmentType),
reason: 'unknown_catalog_code',
});
}
const prosthesisResult = resolveProsthesis(
intent?.prosthesis,
spanResult.teeth,
ctx.prosthesisTypeCodes,
);
unresolved.push(...prosthesisResult.unresolved);
const due = resolveDueDate(intent?.due, ctx.todayIso, ctx.weekStartJs);
if (due.unresolved) unresolved.push(due.unresolved);
const comment =
typeof intent?.comment === 'string' && intent.comment.trim()
? intent.comment.trim()
: null;
// An invented lab id would ship a case to a lab the clinic never named. A rejected one is
// reported, so it cannot look identical to "no lab was spoken".
const labId = resolveCatalogCode(intent?.labId, ctx.linkedLabIds);
if (intent?.labId != null && !labId) {
// `spoken` is what the clinician said — quoting an invented id back would put a raw
// UUID in front of the user.
unresolved.push({ spoken: '', reason: 'unknown_catalog_code' });
}
return {
treatmentType,
teeth: spanResult.teeth,
toothSelectionGroups: spanResult.groups,
comment,
prosthesis: prosthesisResult.prosthesis,
labId,
labMatchExact: labId ? intent?.labMatchExact === true : false,
dueDate: due.dueDate,
unresolved,
};
}

236
backend/src/modules/voice/extraction.wire.spec.ts Normal file
View File

@@ -0,0 +1,236 @@
import {
toVoiceIntent,
VOICE_INTENT_JSON_SCHEMA,
type WireVoiceIntent,
} from './extraction.wire';
const emptyDue = {
kind: 'none' as const,
weekday: null,
which: null,
unit: null,
amount: null,
jy: null,
jm: null,
jd: null,
y: null,
m: null,
d: null,
};
const wire = (overrides: Partial<WireVoiceIntent> = {}): WireVoiceIntent => ({
treatmentType: null,
teeth: [],
connectedSpans: [],
comment: null,
prosthesisDefaultType: null,
prosthesisOverrides: [],
labId: null,
labMatchExact: false,
due: emptyDue,
...overrides,
});
const positionalTooth = {
spoken: 'شش بالا راست',
fdi: null,
arch: 'upper' as const,
side: 'patient_right' as const,
position: 6,
};
describe('VOICE_INTENT_JSON_SCHEMA', () => {
it('is strict — every property required, no extras', () => {
expect(VOICE_INTENT_JSON_SCHEMA.additionalProperties).toBe(false);
expect([...VOICE_INTENT_JSON_SCHEMA.required].sort()).toEqual(
Object.keys(VOICE_INTENT_JSON_SCHEMA.properties).sort(),
);
});
it('requires every field of the due object, since strict mode allows no optionals', () => {
const due = VOICE_INTENT_JSON_SCHEMA.properties.due;
expect([...due.required].sort()).toEqual(
Object.keys(due.properties).sort(),
);
});
});
describe('toVoiceIntent', () => {
it('takes the explicit branch for a code the model wrote in Persian digits', () => {
// The model is reading Persian text back, so "۲۶" and a digit-by-digit "2 6" both
// reach us. Matching only ASCII drops the tooth into the positional branch with no
// quadrant, where it is reported as unresolved — the clinician loses a tooth and is
// told the words were the problem.
for (const raw of ['\u06F2\u06F6', '2 6', ' 26 ', '\u0662\u0666']) {
const [tooth] = toVoiceIntent(
wire({
teeth: [
{
spoken: '\u0628\u06CC\u0633\u062A \u0648 \u0634\u0634',
fdi: raw,
arch: null,
side: null,
position: null,
},
],
}),
).teeth;
expect(tooth).toEqual({
kind: 'explicit',
fdi: '26',
spoken: '\u0628\u06CC\u0633\u062A \u0648 \u0634\u0634',
});
}
});
it('narrows a positional tooth', () => {
const result = toVoiceIntent(wire({ teeth: [positionalTooth] }));
expect(result.teeth[0]).toEqual({
kind: 'positional',
arch: 'upper',
side: 'patient_right',
position: 6,
spoken: 'شش بالا راست',
});
});
it('narrows an explicit FDI tooth, which wins over positional fields', () => {
const result = toVoiceIntent(
wire({ teeth: [{ ...positionalTooth, fdi: '14', spoken: 'یک چهار' }] }),
);
expect(result.teeth[0]).toEqual({
kind: 'explicit',
fdi: '14',
spoken: 'یک چهار',
});
});
it('maps due.kind "none" to no deadline', () => {
expect(toVoiceIntent(wire()).due).toBeNull();
});
it('narrows each due kind', () => {
expect(
toVoiceIntent(
wire({
due: {
...emptyDue,
kind: 'weekday',
weekday: 'thursday',
which: 'next',
},
}),
).due,
).toEqual({ kind: 'weekday', weekday: 'thursday', which: 'next' });
expect(
toVoiceIntent(
wire({ due: { ...emptyDue, kind: 'offset', unit: 'week', amount: 2 } }),
).due,
).toEqual({ kind: 'offset', unit: 'week', amount: 2 });
expect(
toVoiceIntent(
wire({ due: { ...emptyDue, kind: 'jalali', jy: 1404, jm: 7, jd: 25 } }),
).due,
).toEqual({ kind: 'jalali', jy: 1404, jm: 7, jd: 25 });
expect(
toVoiceIntent(
wire({
due: { ...emptyDue, kind: 'gregorian', y: 2025, m: 10, d: 17 },
}),
).due,
).toEqual({ kind: 'gregorian', y: 2025, m: 10, d: 17 });
});
it('ignores a non-FDI-shaped fdi and keeps the positional fields', () => {
// A model emitting fdi:"6" alongside correct arch/side/position must still yield 16,
// not lose the tooth to the explicit branch.
const result = toVoiceIntent(
wire({ teeth: [{ ...positionalTooth, fdi: '6' }] }),
);
expect(result.teeth[0]).toEqual({
kind: 'positional',
arch: 'upper',
side: 'patient_right',
position: 6,
spoken: 'شش بالا راست',
});
});
it('rejects impossible FDI shapes from the explicit branch', () => {
for (const fdi of ['99', '0', '140', '9', 'ab']) {
expect(
toVoiceIntent(wire({ teeth: [{ ...positionalTooth, fdi }] })).teeth[0]
.kind,
).toBe('positional');
}
});
it('passes an unrecognised due kind through so it can be flagged', () => {
// Collapsing it to null would make a misunderstood deadline indistinguishable from
// no deadline at all, and the resolver's flagging path unreachable.
const result = toVoiceIntent(
wire({ due: { ...emptyDue, kind: 'lunar_month' as never } }),
);
expect(result.due).toEqual({ kind: 'lunar_month' });
});
it('reports no prosthesis when neither a default nor an override was given', () => {
expect(toVoiceIntent(wire()).prosthesis).toBeNull();
});
it('builds a prosthesis intent from a default alone', () => {
const result = toVoiceIntent(wire({ prosthesisDefaultType: 'pfm_crown' }));
expect(result.prosthesis).toEqual({
defaultType: 'pfm_crown',
overrides: [],
});
});
it('builds a prosthesis intent from overrides alone', () => {
const result = toVoiceIntent(
wire({
prosthesisOverrides: [{ tooth: positionalTooth, type: 'pfm_crown' }],
}),
);
expect(result.prosthesis?.defaultType).toBeNull();
expect(result.prosthesis?.overrides).toHaveLength(1);
});
it('narrows connected spans', () => {
const result = toVoiceIntent(
wire({
connectedSpans: [
{
from: { ...positionalTooth, fdi: '14' },
to: { ...positionalTooth, fdi: '16' },
},
],
}),
);
expect(result.connectedSpans[0].from).toEqual({
kind: 'explicit',
fdi: '14',
spoken: 'شش بالا راست',
});
});
it('is total — a missing or malformed payload yields a resolvable shape, not a throw', () => {
// Whatever survives here is reported as unresolved downstream rather than crashing.
for (const bad of [undefined, null, {}, { teeth: 'nope', due: 5 }]) {
expect(() => toVoiceIntent(bad as never)).not.toThrow();
const result = toVoiceIntent(bad as never);
expect(result.teeth).toEqual([]);
expect(result.due).toBeNull();
expect(result.labMatchExact).toBe(false);
}
});
it('coerces a non-boolean labMatchExact to false', () => {
expect(
toVoiceIntent(wire({ labMatchExact: 'yes' as never })).labMatchExact,
).toBe(false);
});
});

273
backend/src/modules/voice/extraction.wire.ts Normal file
View File

@@ -0,0 +1,273 @@
import { normalizeFdiCode } from '../../common/fdi';
import type {
ConnectedSpanIntent,
DueIntent,
ProsthesisIntent,
ToothIntent,
VoiceIntent,
Weekday,
} from './voice.types';
import { WEEKDAYS } from './voice.types';
/**
* Deliberately flat: strict `json_schema` mode has poor support for discriminated unions,
* so every variant field is present and nullable. `toVoiceIntent` narrows it into the
* internal union and is total — anything it cannot classify becomes a shape the resolvers
* report as unresolved rather than something that throws here.
*/
export type WireToothIntent = {
spoken: string;
/** The two-digit FDI code the clinician spoke; null when the tooth was described. */
fdi: string | null;
arch: 'upper' | 'lower' | null;
side: 'patient_right' | 'patient_left' | null;
position: number | null;
};
export type WireDue = {
kind: 'weekday' | 'offset' | 'jalali' | 'gregorian' | 'none';
weekday: Weekday | null;
which: 'this' | 'next' | null;
unit: 'day' | 'week' | 'month' | null;
amount: number | null;
jy: number | null;
jm: number | null;
jd: number | null;
y: number | null;
m: number | null;
d: number | null;
};
export type WireVoiceIntent = {
treatmentType: string | null;
teeth: WireToothIntent[];
connectedSpans: { from: WireToothIntent; to: WireToothIntent }[];
comment: string | null;
prosthesisDefaultType: string | null;
prosthesisOverrides: { tooth: WireToothIntent; type: string }[];
labId: string | null;
labMatchExact: boolean;
due: WireDue;
};
const TOOTH_SCHEMA = {
type: 'object',
additionalProperties: false,
required: ['spoken', 'fdi', 'arch', 'side', 'position'],
properties: {
spoken: {
type: 'string',
description: 'The exact transcript words for this tooth.',
},
fdi: {
type: ['string', 'null'],
description:
'The two-digit FDI code the clinician said for this tooth, e.g. "26". Null only ' +
'when the tooth was described in words instead of numbered.',
},
arch: { type: ['string', 'null'], enum: ['upper', 'lower', null] },
side: {
type: ['string', 'null'],
enum: ['patient_right', 'patient_left', null],
description: "The PATIENT's side, never the viewer's.",
},
position: {
type: ['integer', 'null'],
description:
'Position from the midline: 1 = central incisor … 8 = third molar. Never an FDI code.',
},
},
} as const;
export const VOICE_INTENT_JSON_SCHEMA = {
type: 'object',
additionalProperties: false,
required: [
'treatmentType',
'teeth',
'connectedSpans',
'comment',
'prosthesisDefaultType',
'prosthesisOverrides',
'labId',
'labMatchExact',
'due',
],
properties: {
treatmentType: {
type: ['string', 'null'],
description: 'A treatment type CODE from the supplied list, or null.',
},
teeth: { type: 'array', items: TOOTH_SCHEMA },
connectedSpans: {
type: 'array',
description: 'Bridges / splinted units. Endpoints inclusive.',
items: {
type: 'object',
additionalProperties: false,
required: ['from', 'to'],
properties: { from: TOOTH_SCHEMA, to: TOOTH_SCHEMA },
},
},
comment: {
type: ['string', 'null'],
description: 'Clinical notes, in the spoken language.',
},
prosthesisDefaultType: {
type: ['string', 'null'],
description:
'A prosthesis type CODE applied to every tooth unless overridden.',
},
prosthesisOverrides: {
type: 'array',
items: {
type: 'object',
additionalProperties: false,
required: ['tooth', 'type'],
properties: { tooth: TOOTH_SCHEMA, type: { type: 'string' } },
},
},
labId: {
type: ['string', 'null'],
description: 'An id from the supplied lab list. Never invent one.',
},
labMatchExact: {
type: 'boolean',
description: 'True only when the spoken name matched a lab name exactly.',
},
due: {
type: 'object',
additionalProperties: false,
required: [
'kind',
'weekday',
'which',
'unit',
'amount',
'jy',
'jm',
'jd',
'y',
'm',
'd',
],
properties: {
kind: {
type: 'string',
enum: ['weekday', 'offset', 'jalali', 'gregorian', 'none'],
},
weekday: { type: ['string', 'null'], enum: [...WEEKDAYS, null] },
which: { type: ['string', 'null'], enum: ['this', 'next', null] },
unit: {
type: ['string', 'null'],
enum: ['day', 'week', 'month', null],
},
amount: { type: ['integer', 'null'] },
jy: { type: ['integer', 'null'] },
jm: { type: ['integer', 'null'] },
jd: { type: ['integer', 'null'] },
y: { type: ['integer', 'null'] },
m: { type: ['integer', 'null'] },
d: { type: ['integer', 'null'] },
},
},
},
} as const;
/** Two digits, quadrant 1-8, position 1-8 — the only thing that can be an FDI code. */
const FDI_SHAPE = /^[1-8][1-8]$/;
function toToothIntent(wire: WireToothIntent | undefined | null): ToothIntent {
const spoken = typeof wire?.spoken === 'string' ? wire.spoken : '';
// "۲۶" and "2 6" are FDI codes that do not match literally; unnormalised they fall
// through to the positional branch with no quadrant and read as unresolved.
const fdi = normalizeFdiCode(wire?.fdi);
// Only take the explicit branch for something actually FDI-shaped. A model that emits
// fdi:"6" alongside correct arch/side/position would otherwise lose the tooth entirely.
if (FDI_SHAPE.test(fdi)) {
return { kind: 'explicit', fdi, spoken };
}
return {
kind: 'positional',
arch: wire?.arch as 'upper' | 'lower',
side: wire?.side as 'patient_right' | 'patient_left',
position: typeof wire?.position === 'number' ? wire.position : Number.NaN,
spoken,
};
}
function toDueIntent(wire: WireDue | undefined | null): DueIntent | null {
switch (wire?.kind) {
case 'weekday':
return {
kind: 'weekday',
weekday: wire.weekday as Weekday,
which: wire.which as 'this',
};
case 'offset':
return {
kind: 'offset',
unit: wire.unit as 'day',
amount: typeof wire.amount === 'number' ? wire.amount : Number.NaN,
};
case 'jalali':
return {
kind: 'jalali',
jy: wire.jy as number,
jm: wire.jm as number,
jd: wire.jd as number,
};
case 'gregorian':
return {
kind: 'gregorian',
y: wire.y as number,
m: wire.m as number,
d: wire.d as number,
};
case 'none':
case undefined:
return null;
default:
// An unrecognised kind means a deadline WAS spoken and we failed to classify it.
// Passing it through lets the resolver flag it; collapsing it to null would make a
// misunderstood deadline indistinguishable from no deadline at all.
return { kind: wire?.kind } as unknown as DueIntent;
}
}
export function toVoiceIntent(wire: WireVoiceIntent): VoiceIntent {
const teeth = Array.isArray(wire?.teeth) ? wire.teeth : [];
const spans = Array.isArray(wire?.connectedSpans) ? wire.connectedSpans : [];
const overrides = Array.isArray(wire?.prosthesisOverrides)
? wire.prosthesisOverrides
: [];
const connectedSpans: ConnectedSpanIntent[] = spans.map((span) => ({
from: toToothIntent(span?.from),
to: toToothIntent(span?.to),
}));
const hasProsthesis =
wire?.prosthesisDefaultType != null || overrides.length > 0;
const prosthesis: ProsthesisIntent | null = hasProsthesis
? {
defaultType: wire?.prosthesisDefaultType ?? null,
overrides: overrides.map((o) => ({
tooth: toToothIntent(o?.tooth),
type: o?.type,
})),
}
: null;
return {
treatmentType: wire?.treatmentType ?? null,
teeth: teeth.map(toToothIntent),
connectedSpans,
comment: wire?.comment ?? null,
prosthesis,
labId: wire?.labId ?? null,
labMatchExact: wire?.labMatchExact === true,
due: toDueIntent(wire?.due),
};
}

197
backend/src/modules/voice/openrouter.provider.spec.ts Normal file
View File

@@ -0,0 +1,197 @@
import {
OpenRouterAsrProvider,
OpenRouterExtractionProvider,
} from './openrouter.provider';
import { VoiceProviderError } from './voice.providers';
const CONFIG = {
apiKey: 'test-key',
baseUrl: 'https://openrouter.test/api/v1',
model: 'm',
};
const CATALOG = {
treatmentTypes: [{ code: 'prosthesis', label: 'پروتز' }],
prosthesisTypes: [{ code: 'pfm_crown', label: 'روکش پی‌اف‌ام' }],
labs: [{ id: 'lab-1', name: 'لابراتوار سینا' }],
};
type ChatBody = {
temperature: number;
provider: { require_parameters: boolean };
response_format: { type: string; json_schema: { strict: boolean } };
messages: { role: string; content: string }[];
};
function parseBody(spy: jest.Mock): ChatBody {
const init = (spy.mock.calls[0] as [string, RequestInit])[1];
return JSON.parse(init.body as string) as ChatBody;
}
function mockFetch(response: {
ok: boolean;
status?: number;
body?: unknown;
text?: string;
}) {
const spy = jest.fn().mockResolvedValue({
ok: response.ok,
status: response.status ?? (response.ok ? 200 : 500),
json: () => Promise.resolve(response.body),
text: () => Promise.resolve(response.text ?? ''),
});
global.fetch = spy;
return spy;
}
afterEach(() => {
jest.restoreAllMocks();
});
describe('OpenRouterAsrProvider', () => {
it('posts base64 JSON with the language hint, per the documented STT contract', async () => {
const spy = mockFetch({
ok: true,
body: { text: ' سلام ', usage: { seconds: 12, cost: 0.002 } },
});
const result = await new OpenRouterAsrProvider(CONFIG).transcribe(
{ data: 'BASE64', format: 'webm' },
'fa',
);
const [url, init] = spy.mock.calls[0] as [string, RequestInit];
expect(url).toBe('https://openrouter.test/api/v1/audio/transcriptions');
expect(init.headers).toMatchObject({ Authorization: 'Bearer test-key' });
expect(JSON.parse(init.body as string)).toEqual({
model: 'm',
input_audio: { data: 'BASE64', format: 'webm' },
language: 'fa',
});
expect(result.text).toBe('سلام');
expect(result.usage).toEqual({ seconds: 12, costUsd: 0.002 });
});
it('reports missing usage as null rather than zero', async () => {
mockFetch({ ok: true, body: { text: 'x' } });
const result = await new OpenRouterAsrProvider(CONFIG).transcribe(
{ data: 'B', format: 'webm' },
'en',
);
expect(result.usage).toEqual({ seconds: null, costUsd: null });
});
it('raises a staged error without leaking the vendor body', async () => {
// A 4xx can echo the request back, transcript included.
mockFetch({
ok: false,
status: 400,
text: 'transcript: patient name here',
});
const provider = new OpenRouterAsrProvider(CONFIG);
await expect(
provider.transcribe({ data: 'B', format: 'webm' }, 'fa'),
).rejects.toMatchObject({
name: 'VoiceProviderError',
stage: 'asr',
status: 400,
});
await expect(
provider.transcribe({ data: 'B', format: 'webm' }, 'fa'),
).rejects.not.toThrow(/patient name/);
});
});
describe('OpenRouterExtractionProvider', () => {
const wireContent = JSON.stringify({
treatmentType: 'prosthesis',
teeth: [
{ spoken: 'یک چهار', fdi: '14', arch: null, side: null, position: null },
],
connectedSpans: [],
comment: null,
prosthesisDefaultType: 'pfm_crown',
prosthesisOverrides: [],
labId: 'lab-1',
labMatchExact: true,
due: {
kind: 'none',
weekday: null,
which: null,
unit: null,
amount: null,
jy: null,
jm: null,
jd: null,
y: null,
m: null,
d: null,
},
});
it('constrains output with a strict JSON schema and a schema-honouring provider', async () => {
const spy = mockFetch({
ok: true,
body: {
choices: [{ message: { content: wireContent } }],
usage: { cost: 0.0008 },
},
});
const result = await new OpenRouterExtractionProvider(CONFIG).extract(
'روی دندان ۱۴ روکش',
CATALOG,
'fa',
);
const body = parseBody(spy);
expect(body.response_format.type).toBe('json_schema');
expect(body.response_format.json_schema.strict).toBe(true);
// Without require_parameters OpenRouter may route to a provider that treats the
// schema as a hint and returns prose, failing parsing intermittently.
expect(body.provider).toEqual({ require_parameters: true });
expect(body.temperature).toBe(0);
expect(result.intent.treatmentType).toBe('prosthesis');
expect(result.intent.teeth[0]).toEqual({
kind: 'explicit',
fdi: '14',
spoken: 'یک چهار',
});
expect(result.intent.due).toBeNull();
expect(result.costUsd).toBe(0.0008);
});
it('sends the catalog codes and lab ids the model is allowed to choose from', async () => {
const spy = mockFetch({
ok: true,
body: { choices: [{ message: { content: wireContent } }] },
});
await new OpenRouterExtractionProvider(CONFIG).extract('x', CATALOG, 'fa');
const body = parseBody(spy);
const system = body.messages[0].content;
expect(system).toContain('prosthesis');
expect(system).toContain('pfm_crown');
expect(system).toContain('lab-1');
expect(body.messages[1]).toEqual({ role: 'user', content: 'x' });
});
it('fails loudly on unparseable content rather than passing rubbish downstream', async () => {
mockFetch({
ok: true,
body: { choices: [{ message: { content: 'I think tooth 14?' } }] },
});
await expect(
new OpenRouterExtractionProvider(CONFIG).extract('x', CATALOG, 'fa'),
).rejects.toBeInstanceOf(VoiceProviderError);
});
it('fails when the model returns no content at all', async () => {
mockFetch({ ok: true, body: { choices: [] } });
await expect(
new OpenRouterExtractionProvider(CONFIG).extract('x', CATALOG, 'fa'),
).rejects.toMatchObject({ stage: 'extraction' });
});
});

170
backend/src/modules/voice/openrouter.provider.ts Normal file
View File

@@ -0,0 +1,170 @@
import { Logger } from '@nestjs/common';
import {
type AsrProvider,
type AsrResult,
type AudioInput,
type ExtractionCatalog,
type ExtractionProvider,
type ExtractionResult,
VoiceProviderError,
} from './voice.providers';
import { buildExtractionPrompt } from './extraction.prompt';
import {
VOICE_INTENT_JSON_SCHEMA,
toVoiceIntent,
type WireVoiceIntent,
} from './extraction.wire';
type OpenRouterConfig = {
apiKey: string;
baseUrl: string;
model: string;
};
type TranscriptionResponse = {
text?: unknown;
usage?: { seconds?: unknown; cost?: unknown };
};
type ChatResponse = {
choices?: { message?: { content?: unknown } }[];
usage?: { cost?: unknown };
};
function numberOrNull(value: unknown): number | null {
return typeof value === 'number' && Number.isFinite(value) ? value : null;
}
const errorLogger = new Logger('OpenRouterVoice');
async function readError(
response: Response,
stage: 'asr' | 'extraction',
): Promise<never> {
// A vendor 4xx can echo the request back, transcript included. Keep the body out of the
// thrown message — which callers log and could forward — and out of the default log
// level; it stays available at debug when someone is actively diagnosing.
try {
errorLogger.debug(
`${stage} ${response.status} body: ${(await response.text()).slice(0, 500)}`,
);
} catch {
errorLogger.debug(`${stage} ${response.status} body unreadable`);
}
throw new VoiceProviderError(
stage,
`OpenRouter ${stage} failed with status ${response.status}`,
response.status,
);
}
/** Speech → text via OpenRouter's transcription endpoint (whisper-1 and friends). */
export class OpenRouterAsrProvider implements AsrProvider {
private readonly logger = new Logger(OpenRouterAsrProvider.name);
constructor(private readonly config: OpenRouterConfig) {}
async transcribe(
audio: AudioInput,
localeHint: string,
signal?: AbortSignal,
): Promise<AsrResult> {
const response = await fetch(
`${this.config.baseUrl}/audio/transcriptions`,
{
method: 'POST',
signal,
headers: {
Authorization: `Bearer ${this.config.apiKey}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: this.config.model,
input_audio: { data: audio.data, format: audio.format },
language: localeHint,
}),
},
);
if (!response.ok) await readError(response, 'asr');
const body = (await response.json()) as TranscriptionResponse;
const text = typeof body.text === 'string' ? body.text.trim() : '';
this.logger.debug(
`transcribed ${numberOrNull(body.usage?.seconds) ?? '?'}s`,
);
return {
text,
usage: {
seconds: numberOrNull(body.usage?.seconds),
costUsd: numberOrNull(body.usage?.cost),
},
};
}
}
/** Transcript → VoiceIntent via OpenRouter chat completions with a JSON schema. */
export class OpenRouterExtractionProvider implements ExtractionProvider {
constructor(private readonly config: OpenRouterConfig) {}
async extract(
transcript: string,
catalog: ExtractionCatalog,
localeHint: string,
signal?: AbortSignal,
): Promise<ExtractionResult> {
const response = await fetch(`${this.config.baseUrl}/chat/completions`, {
method: 'POST',
signal,
headers: {
Authorization: `Bearer ${this.config.apiKey}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: this.config.model,
temperature: 0,
// Only route to endpoints that actually honour the JSON schema — otherwise OpenRouter may
// pick a provider that treats it as a hint and returns prose, failing intermittently.
provider: { require_parameters: true },
messages: buildExtractionPrompt(transcript, catalog, localeHint),
response_format: {
type: 'json_schema',
json_schema: {
name: 'voice_intent',
strict: true,
schema: VOICE_INTENT_JSON_SCHEMA,
},
},
}),
});
if (!response.ok) await readError(response, 'extraction');
const body = (await response.json()) as ChatResponse;
const content = body.choices?.[0]?.message?.content;
if (typeof content !== 'string' || !content.trim()) {
throw new VoiceProviderError(
'extraction',
'OpenRouter returned no content',
);
}
let wire: WireVoiceIntent;
try {
wire = JSON.parse(content) as WireVoiceIntent;
} catch {
// Schema-constrained output should be valid JSON; if it is not, the resolvers can do
// nothing with it, so fail here rather than pass rubbish downstream.
throw new VoiceProviderError(
'extraction',
'OpenRouter returned unparseable JSON',
);
}
return {
intent: toVoiceIntent(wire),
costUsd: numberOrNull(body.usage?.cost),
};
}
}

265
backend/src/modules/voice/tooth-intent.resolver.spec.ts Normal file
View File

@@ -0,0 +1,265 @@
import {
resolveToothIntent,
resolveToothIntents,
} from './tooth-intent.resolver';
import type { ToothIntent } from './voice.types';
const positional = (
arch: 'upper' | 'lower',
side: 'patient_right' | 'patient_left',
position: number,
spoken = 'x',
): ToothIntent => ({ kind: 'positional', arch, side, position, spoken });
const explicit = (fdi: string, spoken = 'x'): ToothIntent => ({
kind: 'explicit',
fdi,
spoken,
});
describe('resolveToothIntent', () => {
describe('positional intents', () => {
// "شش بالا راست" — upper right six — must be 16, not 26. A mirrored quadrant is a
// valid code for the wrong tooth and reaches the lab unnoticed.
it('resolves each quadrant from the patient perspective', () => {
expect(resolveToothIntent(positional('upper', 'patient_right', 6))).toBe(
'16',
);
expect(resolveToothIntent(positional('upper', 'patient_left', 6))).toBe(
'26',
);
expect(resolveToothIntent(positional('lower', 'patient_left', 6))).toBe(
'36',
);
expect(resolveToothIntent(positional('lower', 'patient_right', 6))).toBe(
'46',
);
});
it('returns null for an out-of-range position instead of clamping', () => {
expect(
resolveToothIntent(positional('upper', 'patient_right', 9)),
).toBeNull();
expect(
resolveToothIntent(positional('upper', 'patient_right', 0)),
).toBeNull();
});
it('returns null for a malformed arch or side', () => {
expect(
resolveToothIntent(
positional('sideways' as 'upper', 'patient_right', 6),
),
).toBeNull();
expect(
resolveToothIntent(
positional('upper', 'viewer_right' as 'patient_right', 6),
),
).toBeNull();
});
});
describe('explicit intents', () => {
it('accepts a permanent FDI code', () => {
expect(resolveToothIntent(explicit('14'))).toBe('14');
expect(resolveToothIntent(explicit('48'))).toBe('48');
});
it('rejects deciduous codes rather than snapping to a permanent tooth', () => {
expect(resolveToothIntent(explicit('51'))).toBeNull();
expect(resolveToothIntent(explicit('85'))).toBeNull();
});
it('rejects nonsense', () => {
for (const value of ['', '1', '99', '140']) {
expect(resolveToothIntent(explicit(value))).toBeNull();
}
});
});
it('returns null for a missing or unknown intent shape', () => {
expect(resolveToothIntent(undefined as unknown as ToothIntent)).toBeNull();
expect(
resolveToothIntent({ kind: 'guess' } as unknown as ToothIntent),
).toBeNull();
});
});
describe('resolveToothIntents', () => {
it('resolves a mixed list and sorts the result', () => {
const result = resolveToothIntents([
positional('upper', 'patient_right', 5, 'پنج بالا راست'),
explicit('14', 'یک چهار'),
]);
expect(result.teeth).toEqual(['14', '15']);
expect(result.unresolved).toEqual([]);
});
it('collapses a tooth named twice', () => {
const result = resolveToothIntents([
explicit('14', 'چهارده'),
positional('upper', 'patient_right', 4, 'چهار بالا راست'),
]);
expect(result.teeth).toEqual(['14']);
});
it('reports what it could not understand instead of dropping it', () => {
const result = resolveToothIntents([
explicit('14', 'یک چهار'),
explicit('51', 'دندان شیری'),
positional('upper', 'patient_right', 9, 'نه بالا راست'),
]);
expect(result.teeth).toEqual(['14']);
expect(result.unresolved).toEqual([
{ spoken: 'دندان شیری', reason: 'not_permanent_tooth' },
{ spoken: 'نه بالا راست', reason: 'position_out_of_range' },
]);
});
it('does not repeat an identical unresolved item', () => {
const result = resolveToothIntents([
explicit('51', 'شیری'),
explicit('51', 'شیری'),
]);
expect(result.unresolved).toHaveLength(1);
});
it('survives a non-array where the model should have sent a list', () => {
// The model can return an object or a number here; that must degrade, not 500.
for (const bad of [undefined, null, 5, 'teeth', { fdi: '14' }]) {
expect(resolveToothIntents(bad as unknown as ToothIntent[])).toEqual({
teeth: [],
unresolved: [],
});
}
});
it('reads a spoken number as its FDI code, digits in any script', () => {
// The product rule: the number the clinician says IS the tooth. 26 = quadrant 2
// (patient's upper left) + position 6 = first molar.
for (const raw of ['26', ' 26 ', '2 6', '\u06F2\u06F6', '\u0662\u0666']) {
expect(resolveToothIntents([explicit(raw, 'x')]).teeth).toEqual(['26']);
}
});
it('trims an explicit code, matching normalizeTeeth', () => {
expect(resolveToothIntents([explicit(' 14 ', 'x')]).teeth).toEqual(['14']);
});
it('distinguishes a deciduous tooth from nonsense in the reason it reports', () => {
// '51' really is a (primary) tooth the chart cannot show; '99' is not a tooth at all.
expect(
resolveToothIntents([explicit('51', 'shiri')]).unresolved[0].reason,
).toBe('not_permanent_tooth');
for (const junk of ['99', '19', '140', '', 'ab']) {
expect(
resolveToothIntents([explicit(junk, `j-${junk}`)]).unresolved[0].reason,
).toBe('malformed');
}
});
it('says the quadrant is missing rather than blaming the words', () => {
// Regression: "ترمیم برای دندون دو" reported "could not be read", sending the
// clinician to look for a transcription fault. Position 2 was understood fine —
// what is missing is the quadrant, and four teeth carry position 2.
const bare = {
kind: 'positional',
arch: null,
side: null,
position: 2,
spoken: 'دندون دو',
} as unknown as ToothIntent;
const result = resolveToothIntents([bare]);
expect(result.teeth).toEqual([]);
expect(result.unresolved).toEqual([
{
spoken: 'دندون دو',
reason: 'tooth_missing_quadrant',
// Every reading of "position 2", for the clinician to pick from.
candidates: ['12', '22', '32', '42'],
},
]);
});
it('narrows the candidates by whatever the clinician did say', () => {
const half = (arch: string | null, side: string | null) =>
resolveToothIntents([
{
kind: 'positional',
arch,
side,
position: 2,
spoken: 'دو',
} as unknown as ToothIntent,
]).unresolved[0].candidates;
expect(half('upper', null)).toEqual(['12', '22']);
expect(half('lower', null)).toEqual(['32', '42']);
// Quadrant 1 is the patient's upper right, 4 the lower right.
expect(half(null, 'patient_right')).toEqual(['12', '42']);
expect(half(null, 'patient_left')).toEqual(['22', '32']);
});
it('offers no candidates for a reason a choice cannot settle', () => {
// Nothing to choose between when the position itself was wrong, or the tooth is
// deciduous — offering chips there would invent options.
for (const intent of [
positional('upper', 'patient_right', 9, 'نه'),
explicit('51', 'شیری'),
]) {
expect(
resolveToothIntents([intent]).unresolved[0].candidates,
).toBeUndefined();
}
});
it('reports a missing quadrant for a half-specified tooth too', () => {
// "دو بالا" narrows it to 12 or 22 — still not one tooth, and still not our guess.
for (const half of [
{ arch: 'upper', side: null },
{ arch: null, side: 'patient_right' },
]) {
const result = resolveToothIntents([
{
kind: 'positional',
...half,
position: 2,
spoken: 'دو',
} as unknown as ToothIntent,
]);
expect(result.unresolved[0].reason).toBe('tooth_missing_quadrant');
}
});
it('still calls an out-of-range position out of range when the quadrant is missing', () => {
// Position wins: "nine" is wrong however completely it was said.
const result = resolveToothIntents([
{
kind: 'positional',
arch: null,
side: null,
position: 9,
spoken: 'نه',
} as unknown as ToothIntent,
]);
expect(result.unresolved[0].reason).toBe('position_out_of_range');
});
it('keeps unresolved items separate when the model omits the spoken span', () => {
// Without `spoken` these are indistinguishable; collapsing them would hide a lost tooth.
const result = resolveToothIntents([
{ kind: 'explicit', fdi: '51' } as ToothIntent,
{ kind: 'explicit', fdi: '52' } as ToothIntent,
]);
expect(result.unresolved).toHaveLength(2);
});
it('handles an empty or missing list', () => {
expect(resolveToothIntents([])).toEqual({ teeth: [], unresolved: [] });
expect(resolveToothIntents(undefined as unknown as ToothIntent[])).toEqual({
teeth: [],
unresolved: [],
});
});
});

145
backend/src/modules/voice/tooth-intent.resolver.ts Normal file
View File

@@ -0,0 +1,145 @@
import {
isFdiTooth,
normalizeFdiCode,
toFdi,
type Arch,
type PatientSide,
} from '../../common/fdi';
import type { ToothIntent, UnresolvedItem } from './voice.types';
const ARCHES: readonly Arch[] = ['upper', 'lower'];
const SIDES: readonly PatientSide[] = ['patient_right', 'patient_left'];
export type ToothResolution = {
/** Unique FDI codes, sorted (matching normalizeTeeth's ordering). */
teeth: string[];
unresolved: UnresolvedItem[];
};
/** Everything here parses untrusted model output, so nothing may throw. */
function normalizedFdi(intent: ToothIntent): string {
// The same normalisation the wire layer used to pick this branch, so the two agree.
return normalizeFdiCode((intent as { fdi?: unknown }).fdi);
}
/**
* Never guesses and never clamps: position 9, a deciduous tooth or a malformed intent all
* resolve to null, so the caller surfaces "not understood" rather than silently selecting a
* neighbouring tooth.
*/
export function resolveToothIntent(intent: ToothIntent): string | null {
if (!intent || typeof intent !== 'object') return null;
if (intent.kind === 'explicit') {
const fdi = normalizedFdi(intent);
return isFdiTooth(fdi) ? fdi : null;
}
if (intent.kind === 'positional') {
if (intent.arch !== 'upper' && intent.arch !== 'lower') return null;
if (intent.side !== 'patient_right' && intent.side !== 'patient_left')
return null;
return toFdi(intent.arch, intent.side, intent.position);
}
return null;
}
function unresolvedReason(intent: ToothIntent): UnresolvedItem['reason'] {
if (!intent || typeof intent !== 'object') return 'malformed';
if (intent.kind === 'explicit') {
const fdi = normalizedFdi(intent);
// Quadrants 1-4 are permanent and would already have resolved, so a well-formed
// quadrant+position reaching here is quadrant 5-8: deciduous. Anything else is noise.
return /^[1-8][1-8]$/.test(fdi) ? 'not_permanent_tooth' : 'malformed';
}
if (intent.kind === 'positional') {
const positionBad =
!Number.isInteger(intent.position) ||
intent.position < 1 ||
intent.position > 8;
if (positionBad) return 'position_out_of_range';
// The position was understood; the quadrant was never said. "دندون دو" names four
// teeth, so "could not be read" would send the clinician after the wrong fault.
const archMissing = intent.arch !== 'upper' && intent.arch !== 'lower';
const sideMissing =
intent.side !== 'patient_right' && intent.side !== 'patient_left';
return archMissing || sideMissing ? 'tooth_missing_quadrant' : 'malformed';
}
return 'malformed';
}
/**
* The teeth still consistent with what *was* heard — "دو" leaves four, "دو بالا" two. Not
* a guess: the full set of readings, for the clinician to choose from.
*/
function quadrantCandidates(intent: ToothIntent): string[] {
if (intent.kind !== 'positional') return [];
const arches =
intent.arch === 'upper' || intent.arch === 'lower' ? [intent.arch] : ARCHES;
const sides =
intent.side === 'patient_right' || intent.side === 'patient_left'
? [intent.side]
: SIDES;
const codes: string[] = [];
for (const arch of arches) {
for (const side of sides) {
const fdi = toFdi(arch, side, intent.position);
if (fdi) codes.push(fdi);
}
}
return codes.sort();
}
function spokenOf(intent: ToothIntent): string {
const spoken = (intent as { spoken?: unknown })?.spoken;
return typeof spoken === 'string' && spoken.trim() ? spoken.trim() : '';
}
/**
* Duplicates collapse; anything unresolvable is reported rather than dropped, so the sheet
* can show which words were not understood.
*/
export function resolveToothIntents(
intents: readonly ToothIntent[],
): ToothResolution {
const teeth = new Set<string>();
const unresolved: UnresolvedItem[] = [];
const seenUnresolved = new Set<string>();
// Not `intents ?? []`: a model may return an object or a number here, and a
// non-iterable must degrade like any other malformed shape rather than throw.
const list: readonly ToothIntent[] = Array.isArray(intents)
? (intents as readonly ToothIntent[])
: [];
for (const intent of list) {
const fdi = resolveToothIntent(intent);
if (fdi) {
teeth.add(fdi);
continue;
}
const reason = unresolvedReason(intent);
const spoken = spokenOf(intent);
const candidates =
reason === 'tooth_missing_quadrant' ? quadrantCandidates(intent) : [];
// Only dedupe what we can tell apart: without `spoken`, two lost references collapse
// into one blank row and a tooth vanishes. Candidates are part of the identity.
if (spoken) {
const key = `${spoken}::${reason}::${candidates.join(',')}`;
if (seenUnresolved.has(key)) continue;
seenUnresolved.add(key);
}
unresolved.push(
candidates.length > 0
? { spoken, reason, candidates }
: { spoken, reason },
);
}
return { teeth: [...teeth].sort(), unresolved };
}

29
backend/src/modules/voice/voice-throttler.guard.ts Normal file
View File

@@ -0,0 +1,29 @@
import { HttpStatus, Injectable } from '@nestjs/common';
import { ThrottlerGuard } from '@nestjs/throttler';
import { AppException, ErrorCode } from '../../common/errors';
/**
* Rate limits voice extraction per user, not per IP: the default tracker keys on `req.ip`,
* which behind nginx means the whole deployment shares one bucket unless `trust proxy` is
* set, and one clinic could then lock out every other.
*/
@Injectable()
export class VoiceThrottlerGuard extends ThrottlerGuard {
protected getTracker(req: Record<string, unknown>): Promise<string> {
const user = req?.user as { id?: unknown } | undefined;
if (typeof user?.id === 'string' && user.id) {
return Promise.resolve(`voice:user:${user.id}`);
}
// Unauthenticated requests never reach here, but fall back rather than share a bucket.
const ip = typeof req?.ip === 'string' ? req.ip : 'unknown';
return Promise.resolve(`voice:ip:${ip}`);
}
/** Without this, ThrottlerException surfaces as INTERNAL_ERROR — there is no 429 fallback. */
protected throwThrottlingException(): Promise<void> {
throw new AppException(
ErrorCode.VOICE_RATE_LIMITED,
HttpStatus.TOO_MANY_REQUESTS,
);
}
}

68
backend/src/modules/voice/voice.controller.ts Normal file
View File

@@ -0,0 +1,68 @@
import {
Body,
Controller,
Get,
Post,
Req,
Res,
UseGuards,
} from '@nestjs/common';
import { ApiBearerAuth, ApiOperation, ApiTags } from '@nestjs/swagger';
import type { Request, Response } from 'express';
import { ClinicOrgGuard } from '../../common/guards/clinic-org.guard';
import { JwtAuthGuard } from '../auth/guards/jwt-auth.guard';
import { ExtractVoiceDto } from './dto/voice.dto';
import { VoiceThrottlerGuard } from './voice-throttler.guard';
import { VoiceService } from './voice.service';
type VoiceRequestUser = { id: string; organizationId?: string };
@ApiTags('voice')
@ApiBearerAuth('JWT-auth')
@UseGuards(JwtAuthGuard, ClinicOrgGuard)
@Controller('voice')
export class VoiceController {
constructor(private readonly voiceService: VoiceService) {}
@Get('availability')
@ApiOperation({
summary: 'Whether voice entry is configured, and for which locales',
})
getAvailability() {
// The frontend cannot learn this from NEXT_PUBLIC_* — those are baked in at build
// time, so enabling a locale would otherwise require rebuilding the image.
return { success: true, data: this.voiceService.getAvailability() };
}
@Post('extract')
// Guarded here rather than globally, and configured by VOICE_THROTTLE_* rather than
// hardcoded. Availability is deliberately left unthrottled — it is cheap and the
// frontend calls it on load.
@UseGuards(VoiceThrottlerGuard)
@ApiOperation({
summary:
'Transcribe a recording and extract treatment detail intents (TAB_TREATMENT_EDIT)',
})
async extract(
@Req() req: Request & { user: VoiceRequestUser },
@Res({ passthrough: true }) res: Response,
@Body() dto: ExtractVoiceDto,
) {
// Cancelling in the browser closes the connection; propagate it as an abort so the vendor
// call stops rather than settling unseen. It is metered per minute.
const aborter = new AbortController();
res.on('close', () => {
if (!res.writableFinished) aborter.abort();
});
// dto.locale, not req.user.language: the client sends the locale the microphone was
// actually offered in, so the ASR hint, catalog labels and week start all match it.
const data = await this.voiceService.extract(
req.user,
dto,
dto.locale,
aborter.signal,
);
return { success: true, data };
}
}

12
backend/src/modules/voice/voice.module.ts Normal file
View File

@@ -0,0 +1,12 @@
import { Module } from '@nestjs/common';
import { PrismaService } from '../../../prisma/prisma.service';
import { ProsthesisCatalogModule } from '../prosthesis-catalog/prosthesis-catalog.module';
import { VoiceController } from './voice.controller';
import { VoiceService } from './voice.service';
@Module({
imports: [ProsthesisCatalogModule],
controllers: [VoiceController],
providers: [VoiceService, PrismaService],
})
export class VoiceModule {}

66
backend/src/modules/voice/voice.providers.ts Normal file
View File

@@ -0,0 +1,66 @@
import type { VoiceIntent } from './voice.types';
/**
* ASR and extraction are separate, independently swappable roles — they will not come from
* the same vendor for every locale. Both resolve per locale from `config.voice.profiles`.
*/
export type AudioInput = {
/** Raw base64, no data: prefix. */
data: string;
/** Container as the recorder produced it: webm, mp4, m4a, wav, … */
format: string;
};
export type AsrUsage = {
seconds: number | null;
costUsd: number | null;
};
export type AsrResult = {
text: string;
usage: AsrUsage;
};
export interface AsrProvider {
/** `localeHint` is ISO-639-1 and materially improves accuracy — always pass it. */
transcribe(
audio: AudioInput,
localeHint: string,
signal?: AbortSignal,
): Promise<AsrResult>;
}
export type ExtractionCatalog = {
/** Catalog codes with their labels in the actor's locale, so the model matches spoken words. */
treatmentTypes: { code: string; label: string }[];
prosthesisTypes: { code: string; label: string }[];
/** The clinic's linked labs — a closed choice list. */
labs: { id: string; name: string }[];
};
export type ExtractionResult = {
intent: VoiceIntent;
costUsd: number | null;
};
export interface ExtractionProvider {
extract(
transcript: string,
catalog: ExtractionCatalog,
localeHint: string,
signal?: AbortSignal,
): Promise<ExtractionResult>;
}
/** Raised when a vendor call fails; the service maps this onto the staged error codes. */
export class VoiceProviderError extends Error {
constructor(
readonly stage: 'asr' | 'extraction',
message: string,
readonly status?: number,
) {
super(message);
this.name = 'VoiceProviderError';
}
}

344
backend/src/modules/voice/voice.service.ts Normal file
View File

@@ -0,0 +1,344 @@
import { HttpStatus, Injectable, Logger } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { LinkStatus } from '@prisma/client';
import { PrismaService } from '../../../prisma/prisma.service';
import { AppException, ErrorCode } from '../../common/errors';
import {
civilDateInZone,
isValidIanaTimeZone,
} from '../../common/zoned-civil-time';
import type { VoiceConfig, VoiceProfile } from '../../configs/configurations';
import { hasEffectivePermission } from '../../common/membership-permissions';
import { normalizeCatalogLocale } from '../catalog/catalog-label.service';
import { ProsthesisCatalogService } from '../prosthesis-catalog/prosthesis-catalog.service';
import { TreatmentCatalogService } from '../treatment-catalog/treatment-catalog.service';
import { weekStartForLocale } from './due-date.resolver';
import {
resolveVoiceIntent,
type ResolvedExtraction,
} from './extraction.resolver';
import {
OpenRouterAsrProvider,
OpenRouterExtractionProvider,
} from './openrouter.provider';
import {
VoiceProviderError,
type AsrProvider,
type ExtractionCatalog,
type ExtractionProvider,
} from './voice.providers';
import type { ExtractVoiceDto } from './dto/voice.dto';
export type VoiceAvailability = {
enabled: boolean;
locales: string[];
maxRecordingMs: number | null;
};
export type VoiceExtractionResponse = ResolvedExtraction & {
transcript: string;
};
@Injectable()
export class VoiceService {
private readonly logger = new Logger(VoiceService.name);
constructor(
private readonly prisma: PrismaService,
private readonly config: ConfigService,
private readonly treatmentCatalog: TreatmentCatalogService,
private readonly prosthesisCatalog: ProsthesisCatalogService,
) {}
private get voiceConfig(): VoiceConfig {
return this.config.get<VoiceConfig>('voice')!;
}
/**
* What the frontend needs to decide whether to render the microphone. v1 is ungated beyond
* a configured locale profile; the Plan.features design is deferred, not dropped.
*/
getAvailability(): VoiceAvailability {
const voice = this.voiceConfig;
const hasKey = Boolean(voice.openRouter.apiKey);
const locales = hasKey ? Object.keys(voice.profiles) : [];
return {
enabled: locales.length > 0,
locales,
maxRecordingMs: voice.maxRecordingMs,
};
}
async extract(
user: { id: string; organizationId?: string },
dto: ExtractVoiceDto,
locale: string,
signal?: AbortSignal,
): Promise<VoiceExtractionResponse> {
const startedAt = Date.now();
const organizationId = this.assertOrganization(user);
await this.assertCanEditTreatment(user.id, organizationId);
const catalogLocale = normalizeCatalogLocale(locale);
const profile = this.resolveProfile(catalogLocale);
this.assertWithinCap(dto.durationMs);
const timeZone = isValidIanaTimeZone(dto.timeZone) ? dto.timeZone : 'UTC';
const todayIso = civilDateInZone(new Date(), timeZone);
const { asr, extraction } = this.buildProviders(profile);
// Stage 1 — audio never touches disk and is not retained beyond this call.
let transcript: string;
let asrCost: number | null = null;
let asrSeconds: number | null = null;
try {
const result = await asr.transcribe(
{ data: dto.audio, format: dto.format },
catalogLocale,
signal,
);
transcript = result.text;
asrCost = result.usage.costUsd;
asrSeconds = result.usage.seconds;
} catch (error) {
throw this.toAppException(error, 'asr');
}
// durationMs is client-reported, so not enforcement. usage.seconds is the vendor's own
// measurement — a client under-reporting to slip past the cap is caught here, after the
// ASR spend but before the more expensive extraction call.
if (asrSeconds != null) {
this.assertWithinCap(asrSeconds * 1000);
}
if (!transcript.trim()) {
throw new AppException(
ErrorCode.VOICE_NOTHING_RECOGNIZED,
HttpStatus.UNPROCESSABLE_ENTITY,
);
}
// Stage 2 — structure it. On failure the transcript still goes back to the client so
// the words the clinician already paid for are not lost (transcript salvage).
let resolved: ResolvedExtraction;
let llmCost: number | null = null;
try {
// Inside the try: the transcript is already paid for, so a catalog/DB failure here
// must still salvage it rather than becoming a generic 500 that throws it away.
const catalog = await this.buildCatalog(organizationId, catalogLocale);
const result = await extraction.extract(
transcript,
catalog,
catalogLocale,
signal,
);
llmCost = result.costUsd;
resolved = resolveVoiceIntent(result.intent, {
todayIso,
weekStartJs: weekStartForLocale(catalogLocale),
treatmentTypeCodes: new Set(catalog.treatmentTypes.map((t) => t.code)),
prosthesisTypeCodes: new Set(
catalog.prosthesisTypes.map((t) => t.code),
),
linkedLabIds: new Set(catalog.labs.map((l) => l.id)),
});
} catch (error) {
throw this.toAppException(error, 'extraction', transcript);
}
this.logTelemetry({
locale: catalogLocale,
durationMs: dto.durationMs,
elapsedMs: Date.now() - startedAt,
asrCost,
llmCost,
resolved,
});
return { ...resolved, transcript };
}
private assertOrganization(user: { organizationId?: string }): string {
if (!user?.organizationId) {
throw new AppException(
ErrorCode.AUTH_ORG_NOT_SELECTED,
HttpStatus.BAD_REQUEST,
);
}
return user.organizationId;
}
private async assertCanEditTreatment(userId: string, organizationId: string) {
const membership = await this.prisma.membership.findFirst({
where: {
userId,
organizationId,
OR: [{ isOwner: true }, { isActive: true }],
},
include: {
permissions: { include: { permission: true } },
organization: { include: { type: true, plan: true } },
},
});
if (!membership) {
throw new AppException(
ErrorCode.PERMISSION_NOT_MEMBER,
HttpStatus.FORBIDDEN,
);
}
if (!hasEffectivePermission(membership, 'TAB_TREATMENT_EDIT')) {
throw new AppException(
ErrorCode.PERMISSION_EDIT_TREATMENTS,
HttpStatus.FORBIDDEN,
);
}
}
private resolveProfile(locale: string): VoiceProfile {
const voice = this.voiceConfig;
const profile = voice.profiles[locale];
if (!profile || !voice.openRouter.apiKey) {
throw new AppException(
ErrorCode.VOICE_NOT_AVAILABLE,
HttpStatus.FORBIDDEN,
);
}
return profile;
}
/**
* The client auto-stops at maxMs and only then measures, so a capped recording always
* reports slightly over. Without this tolerance every auto-stopped recording — the exact
* case the cap exists for — would be rejected as too long.
*/
private static readonly CAP_TOLERANCE_MS = 2_000;
private assertWithinCap(durationMs: number) {
const max = this.voiceConfig.maxRecordingMs;
if (max != null && durationMs > max + VoiceService.CAP_TOLERANCE_MS) {
throw new AppException(
ErrorCode.VOICE_CLIP_TOO_LONG,
HttpStatus.PAYLOAD_TOO_LARGE,
);
}
}
private buildProviders(profile: VoiceProfile): {
asr: AsrProvider;
extraction: ExtractionProvider;
} {
const { apiKey, baseUrl } = this.voiceConfig.openRouter;
const base = { apiKey: apiKey!, baseUrl };
return {
asr: new OpenRouterAsrProvider({ ...base, model: profile.asr.model }),
extraction: new OpenRouterExtractionProvider({
...base,
model: profile.llm.model,
}),
};
}
/** Codes with labels in the actor's locale, plus the clinic's linked labs. */
private async buildCatalog(
organizationId: string,
locale: string,
): Promise<ExtractionCatalog> {
const [treatmentTypes, prosthesisTypes, labs] = await Promise.all([
this.treatmentCatalog.list(locale, null),
this.prosthesisCatalog.list(locale),
this.listLinkedLabs(organizationId),
]);
return {
treatmentTypes: treatmentTypes
.filter((entry) => entry.availableInTreatment)
.map((entry) => ({ code: entry.code, label: entry.label })),
prosthesisTypes: prosthesisTypes.map((entry) => ({
code: entry.code,
label: entry.label,
})),
labs,
};
}
private async listLinkedLabs(
organizationId: string,
): Promise<{ id: string; name: string }[]> {
const [linksA, linksB] = await Promise.all([
this.prisma.organizationLink.findMany({
where: { organizationAId: organizationId, status: LinkStatus.ACTIVE },
include: { organizationB: { select: { id: true, name: true } } },
}),
this.prisma.organizationLink.findMany({
where: { organizationBId: organizationId, status: LinkStatus.ACTIVE },
include: { organizationA: { select: { id: true, name: true } } },
}),
]);
return [
...linksA.map((l) => ({
id: l.organizationB.id,
name: l.organizationB.name,
})),
...linksB.map((l) => ({
id: l.organizationA.id,
name: l.organizationA.name,
})),
];
}
private toAppException(
error: unknown,
stage: 'asr' | 'extraction',
transcript?: string,
): AppException {
if (error instanceof Error && error.name === 'AbortError') {
// The clinician cancelled; not a failure worth a translated message.
return new AppException(ErrorCode.BAD_REQUEST, HttpStatus.BAD_REQUEST);
}
if (error instanceof VoiceProviderError) {
this.logger.warn(`voice ${stage} failed: ${error.message}`);
} else {
this.logger.error(`voice ${stage} failed unexpectedly`, error as Error);
}
const code =
stage === 'asr'
? ErrorCode.VOICE_ASR_FAILED
: ErrorCode.VOICE_EXTRACT_FAILED;
return new AppException(
code,
HttpStatus.BAD_GATEWAY,
transcript ? { transcript } : undefined,
);
}
/** Structured and patient-free: never the transcript, never audio, never a patient id. */
private logTelemetry(input: {
locale: string;
durationMs: number;
elapsedMs: number;
asrCost: number | null;
llmCost: number | null;
resolved: ResolvedExtraction;
}) {
const { resolved } = input;
this.logger.log(
JSON.stringify({
event: 'voice.extract',
locale: input.locale,
clipMs: input.durationMs,
elapsedMs: input.elapsedMs,
costUsd: (input.asrCost ?? 0) + (input.llmCost ?? 0),
resolvedFields: {
treatmentType: resolved.treatmentType != null,
teeth: resolved.teeth.length,
comment: resolved.comment != null,
prosthesisComplete: resolved.prosthesis?.complete ?? null,
lab: resolved.labId != null,
dueDate: resolved.dueDate != null,
},
unresolvedCount: resolved.unresolved.length,
}),
);
}
}

85
backend/src/modules/voice/voice.types.ts Normal file
View File

@@ -0,0 +1,85 @@
import type { Arch, PatientSide } from '../../common/fdi';
/**
* What the extraction model is allowed to return.
*
* The model emits *intents*, never resolved values: no FDI codes, no ISO dates. Pure,
* unit-tested resolvers turn intents into domain values, so the two highest-consequence
* mappings — quadrant mirroring and Jalali conversion — are testable rather than hopeful.
*/
/** A single spoken tooth reference. `spoken` is the transcript span, echoed back to the user. */
export type ToothIntent =
| { kind: 'explicit'; fdi: string; spoken: string }
| {
kind: 'positional';
arch: Arch;
side: PatientSide;
/** 1 = central incisor … 8 = third molar. */
position: number;
spoken: string;
};
/** A spoken deadline. The model never does calendar arithmetic. */
export type DueIntent =
| { kind: 'weekday'; weekday: Weekday; which: 'this' | 'next' }
| { kind: 'offset'; unit: 'day' | 'week' | 'month'; amount: number }
| { kind: 'jalali'; jy: number; jm: number; jd: number }
| { kind: 'gregorian'; y: number; m: number; d: number };
export const WEEKDAYS = [
'saturday',
'sunday',
'monday',
'tuesday',
'wednesday',
'thursday',
'friday',
] as const;
export type Weekday = (typeof WEEKDAYS)[number];
/** Two teeth defining an inclusive connected (bridge) span. */
export type ConnectedSpanIntent = { from: ToothIntent; to: ToothIntent };
export type ProsthesisIntent = {
/** Catalog code applied to every tooth unless overridden. */
defaultType: string | null;
overrides: { tooth: ToothIntent; type: string }[];
};
export type VoiceIntent = {
treatmentType: string | null;
teeth: ToothIntent[];
connectedSpans: ConnectedSpanIntent[];
comment: string | null;
prosthesis: ProsthesisIntent | null;
/** Must be one of the linked-lab ids supplied in the prompt, or null. */
labId: string | null;
/** False when the spoken name only approximately matched — the UI then requires an explicit tick. */
labMatchExact: boolean;
due: DueIntent | null;
};
/** Why a spoken item could not be turned into a domain value. Shown to the user. */
export type UnresolvedReason =
| 'not_permanent_tooth'
| 'position_out_of_range'
/** A position was understood but no quadrant was spoken — four teeth match. */
| 'tooth_missing_quadrant'
| 'malformed'
| 'span_not_same_arch'
| 'unknown_catalog_code'
| 'tooth_not_selected'
| 'invalid_date';
export type UnresolvedItem = {
/** The transcript span that could not be resolved, so the user can see what was heard. */
spoken: string;
reason: UnresolvedReason;
/**
* FDI codes still consistent with what was heard — "دو" leaves four, "دو بالا" two. Only
* `tooth_missing_quadrant` carries them; the sheet offers them as chips.
*/
candidates?: string[];
};

802
docs/specs/voice-treatment-entry/spec.md Normal file
View File

@@ -0,0 +1,802 @@
# Voice treatment entry
**Status:** Implemented on `feat/voice-treatment-entry`, with one specified piece missing —
the transcript-salvage dialog (§9). First live test on 2026-08-21 sent the tooth path back
for revision — a spoken number is now read as its FDI code (§6).
Still blocked on the ASR spike (§11 item 1) before it is trustworthy in front of patients
**Area:** Treatment workspace (CLINIC orgs)
**Created:** 2026-08-20
Fill a `TreatmentDetail` — including its lab dispatch — by speaking, instead of by
tapping through the type dropdown, the FDI chart, the prosthesis wizard and the lab
picker.
---
## 1. Goal
A clinician on the Treatment tab taps a microphone, describes the treatment for the
already-selected patient in one utterance, and is shown a **review sheet** of what was
understood. Fields they tick are applied to the open detail chip. Nothing is written to
the form without confirmation.
### In scope
One recording produces **exactly one** `TreatmentDetail`, and may fill every field of it:
| Field | Source |
|---|---|
| `treatmentType` | catalog code, matched against locale labels |
| `teeth` | FDI codes, via tooth-intent resolver |
| `toothSelectionGroups` | connected (bridge) / single spans |
| `comment` | cleaned dictated notes |
| lab: `prosthesisTypeCode` per tooth | default type + per-tooth overrides |
| lab: `destinationOrganizationId` | matched against the clinic's linked labs |
| lab: `dueDate` | via due-date intent resolver |
### Out of scope (v1)
- Multiple detail chips from one recording.
- `attachmentIds` — files cannot be dictated.
- Editing an existing detail by voice ("no, make that 15"). Confirming a recording always
creates a **new** detail (see §2).
- Creating the treatment or selecting the patient by voice. A patient is already
selected; voice only fills the form.
- Lab-side (`LAB` org) usage. Clinic only.
---
## 2. User flow and UI integration
### The control: Add detail, split
The `Add detail` button gains a second segment holding the mic. The halves read as
siblings — both end in a new detail — but they are **independent actions**:
- **Add half — unchanged.** Same `onAddDetail`, same seeding, same `setEntryStep`. It
gains a neighbour and nothing else. Its logic is not modified, wrapped or made
conditional.
- **Mic half** — starts a recording. Nothing is created until confirm (below).
The `Add detail` `<Button>` in `TreatmentDetailsEditor` becomes a segmented control built
exactly like the **detail chip's trash affordance** in the same file
(`TreatmentDetailsEditor.tsx`, the chip `<div>` + label `<button>` + remove `<button>`):
- an `inline-flex items-stretch overflow-hidden rounded-[var(--radius-md)]` wrapper;
- two raw `<button>` children divided by `border-s`, each with its own `disabled`,
`title`, `aria-label` and `focus-visible:ring-inset`;
- styled `primary` (`bg-primary text-white`) to preserve the button's current look;
- `w-full sm:w-auto` on the wrapper with the Add half `flex-1`, reproducing today's
`fullWidth` + `sm:w-auto shrink-0` behaviour.
The shared `Button` component is **not** reusable for the halves: it hardcodes
`rounded-[var(--radius-md)]` on each instance and owns auto-pending state, both of which
fight a segmented control. This is precisely why the chip pattern uses raw `<button>`s,
and this control follows it.
**Side:** the mic is the second flex child with `border-s` — the **logical end**, exactly
like the trash. Visually right in `en`/`nl`, visually left in `fa`. No physical
left/right anywhere, per the repo's RTL rule.
```
en / nl (LTR) fa (RTL)
[ Add detail │ 🎤 ] [ 🎤 │ افزودن ]
[ ترمیم 14,15 │ 🗑 ] [ 🗑 │ ترمیم 14,15 ]
↑ same side as the chip's trash, in both directions
```
### Flow
```
[editable day]
│
▼ tap 🎤 ← mic swaps to ■ (red); Add half disabled
recording ─────────── inline bar below the header row:
│ ● 0:12 / 2:00 ▁▃▇▅▂▆█▄▁ [لغو]
│ auto-stops at the 2:00 cap
▼ tap ■
processing ────────── ● transcribing… ○ extracting [cancel]
│
▼
review sheet
☑ Type ترمیم
☑ Teeth [mini FDI chart] 14 15
☑ Notes حساسیت به سرما
☐ Lab لابراتوار سینا ⚠ similar name
│
▼ [Apply n fields]
detail created or filled → normal save flow
```
**No layout shift.** The segmented control never changes size; only the mic's icon and
colour change. The timer and level meter live in a full-width bar inserted between the
header row and the chip strip — the header is `sm:justify-between`, so growing the button
mid-recording would shove the row on every start and every stop.
**Duration is capped at 2 minutes** (`maxMs`, configurable, v1 default `120_000`). The
timer shows `elapsed / 2:00` and the recorder auto-stops at the cap. `maxMs: null` means
uncapped and remains supported, but is not the v1 default.
Two minutes is generous against the longest realistic utterance — a full prosthesis
dictation with type, several teeth, a bridge, prosthesis type, lab and due date — while
bounding worst-case spend. It also sits comfortably inside the vendor limits (§4): 2
minutes of `webm/opus` is well under 1 MB against a 25 MB ceiling, and transcribes in a
few seconds against a 60s processing timeout.
Cost per recording is therefore bounded at **$0.012** of ASR (2 min × $0.006) plus
~$0.0008 of extraction — about **1.3¢ worst case**, against ~0.3¢ for a typical 20s
utterance.
### What confirm does
The detail is created **on confirm**, never on tap. Tapping the mic starts a recording and
nothing else, so cancelling, a vendor failure, a rate-limit, or navigating away leaves the
chip strip untouched — there is no orphan state to clean up.
Confirm **always appends a new detail**. One unconditional rule, no dependence on
invisible state: append `newDetail()` seeded with `defaultTreatmentTypeForAppointment`,
make it active, `setEntryStep('treatment')`, then apply the ticked rows on top — so an
extracted type overrides the seed, and unticking the type row leaves the seeded default in
place.
This reuses `newDetail()` and the same state transitions, but it is a **separate code
path**. `onAddDetail` is not called and not changed.
`setEntryStep('treatment')` matters: `showChrome` is always on, so the control is visible
during the **Lab** wizard step too. Confirming there returns to the treatment step.
**Confirm also saves.** The new detail is persisted immediately (`persistDraft({force:true})`),
and when the result carries a lab, a due date or a prosthesis map the lab case is saved with it
(`persistLabCases`). Not politeness — the autosave effect watches `details` only, so a lab draft
left in component state alone loses the destination lab, the due date and the whole prosthesis
map on the next reload. The detail survives, which is what makes that loss look like a
successful save.
One guard on it: `persistDraft` returns a **preview** treatment instead of saving when any
detail still lacks a treatment type — the blank chip the workspace opens with is enough — and a
preview's detail id falls back to the client id. Confirm therefore checks *what came back*, not
the precondition, and skips the lab-case save when it did not get a real id; posting a lab case
against an id the server has never seen fails the whole save. Checking the result rather than
the condition keeps this true for every early return `persistDraft` has.
> Accepted consequences:
> - Tapping Add and then 🎤 leaves behind the blank chip that Add created. It carries the
> usual trash affordance.
> - That same blank chip blocks confirm's immediate lab-case save until it is given a type
> or removed; the lab rows stay in local state until the ordinary Lab-step save.
> - Dictating into an existing detail is not supported in v1 — voice always makes a new
> one.
### Render policy
Three different reasons for "no", rendered differently:
| Reason | Condition | Render |
|---|---|---|
| **Technical** | no voice profile for the locale; no `MediaRecorder` | segment **absent** — the control is byte-for-byte today's plain Add button |
| **Commercial** | `Plan.features.voiceTreatmentEntry` false | segment **absent** — ⚠ *not enforced in v1*, see §8 |
| **Contextual** | `!canEdit \|\| disabled` — past day, read-only load, no `TAB_TREATMENT_EDIT` | both segments render and **disable together**, like the chip's trash (`disabled:opacity-40 disabled:cursor-not-allowed`) |
"This feature isn't yours" and "not right now" are different statements. Absence avoids a
permanently dead control; disabling avoids the button resizing as the day strip moves.
`TAB_TREATMENT_EDIT` is resolved via `common/membership-permissions.ts`, never by reading
`membership.permissions` directly.
### Component API
`TreatmentDetailsEditor` gains exactly **one** optional prop. All behaviour —
`MediaRecorder`, the API call, error state — lives in `lib/voice/useVoiceCapture.ts`
(hooks belong in `lib/` per `AGENTS.md`) and is owned by `TreatmentWorkspace`. The editor
stays presentational and renders both the segment and the recording bar with its own
classes, keeping the segmented styling beside the chip pattern it mirrors.
```ts
/** Omit when voice is unavailable — the Add button then renders unsplit. */
voice?: {
phase: 'idle' | 'recording' | 'processing';
elapsedMs: number;
level: number; // 0..1, for the meter
maxMs: number | null; // v1: 120_000 (2 min). null = uncapped, supported but not default
onStart: () => void;
onStop: () => void;
onCancel: () => void;
};
```
`voice === undefined` **is** the absent state above — availability is expressed by
presence rather than a separate flag, so the two cannot disagree.
### Accessibility
- Mic segment carries both `title` and `aria-label`, like the chip's trash, and its label
changes with `phase`.
- Announce **phase transitions** via a `role="status" aria-live="polite"` region — the
same idiom as the existing autosave status line. Do not put `aria-live` on the ticking
timer.
- Focus ring is `focus-visible:ring-inset` tinted primary when idle, red while recording.
---
## 3. Architecture
Two sequential stages, both server-side. The vendor API key never reaches the browser.
```
browser ──audio(base64)──► POST /treatments/voice-extract
│
┌─────────┴─────────┐
│ 1. AsrProvider │ audio + locale hint → transcript
└─────────┬─────────┘
┌─────────┴─────────┐
│ 2. ExtractionProv │ transcript + catalog + ctx → VoiceIntent
└─────────┬─────────┘
┌─────────┴─────────┐
│ 3. resolvers │ intents → FDI codes, ISO date
│ (pure, tested) │
└─────────┬─────────┘
▼
VoiceExtractionResult (resolved values + the intents
that produced them, for display)
```
**Why intents and not final values:** the model never emits an FDI code and never does
calendar arithmetic. It emits what it *heard*; deterministic, unit-tested code decides
what that means. This is what makes the two highest-consequence mappings — quadrant
mirroring and Jalali conversion — testable instead of hopeful.
### Endpoint
`POST /treatments/voice-extract`
- Guards: `JwtAuthGuard` + `ClinicOrgGuard`.
- Service-level check of `TAB_TREATMENT_EDIT`. The plan flag is **not** checked in v1
(§8).
- Per-user throttle via `@nestjs/throttler` (present in `package.json`, currently wired
nowhere in `src/` — this is its first use, so the module must be registered in
`app.module.ts`). Configurable; v1 default **6 requests / 60s per user**
(`VOICE_THROTTLE_LIMIT`, `VOICE_THROTTLE_TTL`). A human cannot approach that — a
recording plus processing takes ten seconds at minimum — so it is purely an abuse and
runaway-loop guard, which is doing more work than usual given v1 is ungated and
uncapped (open item 14).
- Body: base64 audio + declared format + `clientTodayIso` + IANA `timeZone` +
`treatmentDetailId`/`clientId` for context.
`clientTodayIso` and `timeZone` come from the client per the existing house rule
(`AGENTS.md`): never derive the clinic's civil day from `Date#getDay()` on the UTC
server. Reuse `common/zoned-civil-time.ts`.
---
## 4. Provider registry
ASR and extraction are **separate swappable roles**. They will not come from the same
vendor for every locale.
```ts
interface AsrProvider {
transcribe(audio: AudioInput, localeHint: string): Promise<{ text: string; usage: AsrUsage }>;
}
interface ExtractionProvider {
extract(transcript: string, catalog: CatalogPrompt, ctx: ExtractionContext): Promise<VoiceIntent>;
}
```
Resolved through a registry keyed by locale.
At launch **all three locales use the same profile** — OpenRouter with
`openai/whisper-1` for ASR. The per-locale indirection is kept anyway, because the
locale is the axis along which this is most likely to diverge: Persian ASR is the
weakest link (§11), and swapping only `fa` to a Persian-specialist vendor must not be a
code change.
```
VOICE_PROFILE_FA = openrouter:openai/whisper-1 | openrouter:<llm-model>
VOICE_PROFILE_EN = openrouter:openai/whisper-1 | openrouter:<llm-model>
VOICE_PROFILE_NL = openrouter:openai/whisper-1 | openrouter:<llm-model>
```
The `language` hint is the profile's locale as ISO-639-1 — `fa`, `en`, `nl` — not a
constant.
Reachability is a property of **where you deploy**, not of the code. The same image
serves an Iran-hosted instance with a domestic `fa` profile and a Europe-hosted instance
with an OpenRouter profile; only config differs. A locale with no configured profile has
no microphone button at all (§2).
> The frontend must learn which locales are enabled **from the API**, not from a
> `NEXT_PUBLIC_*` var — those are baked in at build time, so an env-var approach would
> make enabling a locale require rebuilding and repushing the frontend image.
### v1 provider: OpenRouter
Both roles, one API key.
**ASR** — `POST https://openrouter.ai/api/v1/audio/transcriptions`
```json
{
"model": "openai/whisper-1",
"input_audio": { "data": "<base64>", "format": "webm" },
"language": "fa"
}
```
- Accepted formats: WAV, MP3, FLAC, M4A, OGG, **WebM**, **AAC**.
Chrome/Android `webm/opus` and Safari/iPad `mp4/aac` both go through unmodified —
**no transcode dependency is required**.
- Limits: 25 MB; 60s upstream *processing* timeout. The 2-minute recording cap (§2) sits
comfortably inside both.
- Request shape **verified** against OpenRouter's STT docs: base64 JSON `input_audio` is
the documented primary path (multipart `file` is the OpenAI-compatible alternative), and
both default model slugs exist on the live models API.
- The cap is enforced twice: the client's reported `durationMs`, and again against the
vendor's own `usage.seconds` — the client's figure is a claim, not enforcement. The
server allows a 2s tolerance, because the client measures length *after* the recorder
stops and a recording that runs to the cap always reports slightly over it.
- Response: `{ text, usage: { seconds, total_tokens, cost } }`.
- Price: `openai/whisper-1` is **$0.006/minute, billed to the nearest second** → $0.002 for
a typical 20s utterance, **$0.012 at the 2-minute cap**. OpenRouter forwards this model
directly to OpenAI.
**Extraction** — OpenRouter chat completions with a JSON schema constraining the output
to `VoiceIntent`. Model id is config (`VOICE_LLM_MODEL`).
**Chosen: `google/gemini-3.7-flash`.**
Cost is deliberately *not* the selection axis. The call is ~1,250 input tokens (system
prompt + catalog labels + JSON schema + linked labs + transcript) and ~200 output, so the
whole candidate field spans about one cent per recording:
| Model | in / out per M tokens | ≈ per call |
|---|---|---|
| `google/gemini-3.7-flash` | $0.375 / $1.875 | $0.0008 |
| `qwen/qwen3.8-27b` | $0.45 / $3.20 | $0.0012 |
| `qwen/qwen3.8-max` | $2 / $6 | $0.0037 |
| `openai/gpt-5.6-terra` | $2 / $12 | $0.0049 |
| `anthropic/claude-opus-5` | $5 / $25 | $0.0113 |
Select for Persian comprehension and reliable constrained JSON instead. Gemini Flash wins
on the reasoning that this task is **not** reasoning-heavy — read a short sentence, pick
codes from a supplied closed list, emit small JSON — and it is simultaneously the cheapest
and lowest-latency candidate, which matters on a pipeline already at 3–10s.
**Escalation path if Persian comprehension proves weak** on real transcripts (item 1):
`qwen/qwen3.8-max` (leads current multilingual rankings, ~4.6× the cost) and then
`anthropic/claude-opus-5` as the accuracy ceiling. Both are a config change —
`VOICE_LLM_MODEL` — not a code change. Worth knowing when judging that ranking evidence:
it is marked provisional and was measured on the previous Qwen generation.
---
## 5. Extraction contract
The model returns intents only. Illustrative shape:
```ts
type VoiceIntent = {
treatmentType: string | null; // catalog code, from the supplied closed list
teeth: ToothIntent[];
connectedSpans: { from: ToothIntent; to: ToothIntent }[];
comment: string | null;
prosthesis: {
defaultType: string | null; // catalog code
overrides: { tooth: ToothIntent; type: string }[];
} | null;
labId: string | null; // must be one of the supplied linked-lab ids
labMatchExact: boolean;
due: DueIntent | null;
};
type ToothIntent =
| { kind: 'explicit'; fdi: string; spoken: string }
| { kind: 'positional'; arch: 'upper' | 'lower';
side: 'patient_right' | 'patient_left'; position: number; spoken: string };
type DueIntent =
| { kind: 'weekday'; weekday: string; which: 'this' | 'next' }
| { kind: 'offset'; unit: 'day' | 'week' | 'month'; amount: number }
| { kind: 'jalali'; jy: number; jm: number; jd: number }
| { kind: 'gregorian'; y: number; m: number; d: number };
```
Every code-valued field is constrained to a **closed list supplied in the prompt**:
- Treatment types and prosthesis types come from `CatalogLabelService` in the actor's
locale, so the model sees "پروتز" and "زیرکونیا مونولیتیک" as the spoken forms of
`prosthesis` and `monolithic_zirconia` rather than being asked to translate. Catalog
entities store a stable `code` and no label — never hardcode a label.
- Lab candidates are the clinic's linked labs only (`OrganizationLink`), passed as
`{ id, name }`. The model may return one of those ids or `null`, nothing else.
Every unresolved or rejected item is reported, never silently dropped.
---
## 6. Resolvers
Both live in `backend/src/`, pure and Jest-covered. The frontend has **no test runner**
(no jest/vitest, zero spec files) — putting them there would forfeit the testability
that justified this whole design.
### `resolveToothIntent()`
- Owns the patient-right convention in exactly one place: `upper + patient_right` →
quadrant 1, `upper + patient_left` → 2, `lower + patient_left` → 3,
`lower + patient_right` → 4. This is the mirroring bug, and it becomes a unit test.
- Rejects out-of-range positions rather than clamping. Position 9 is unresolved, never 8.
- Permanent dentition only — FDI 11–18/21–28/31–38/41–48, matching
`FDI_UPPER_LEFT_TO_RIGHT` / `FDI_LOWER_LEFT_TO_RIGHT`. Deciduous references
("دندان شیری") must resolve to **unresolved**, never snap to a permanent tooth.
- **Locale-neutral by construction.** `ToothIntent` carries `arch`/`side`/`position`, not
words, so the resolver needs no per-locale branches. The locale-specific part is the
*prompt*: each enabled locale needs its own spoken tooth vocabulary (`شش بالا راست`,
`upper right six`, `rechtsboven zes`).
- **A spoken tooth number is an FDI code, in every locale.** This is how clinicians
actually dictate — "بیست و شش" is tooth 26 — so the prompt *teaches* the notation
(first digit = quadrant from the patient's own point of view, second = position from
the midline) rather than refusing it. `arch`/`side`/`position` is the reading of a
tooth that was **described** instead of numbered, where a single digit is a position
and the quadrant comes from words. Revised after the first live test; the original
design had this backwards and made the descriptive form the only supported path.
- **A single digit alone is never resolved.** "دندون دو" names four teeth. It is reported
as `tooth_missing_quadrant` **with the candidate codes attached** — narrowed by whatever
*was* said, so "دو بالا" offers two — and the review sheet turns them into chips. The
clinician chooses; the resolver still never guesses.
- **Digits arrive in three scripts.** `normalizeFdiCode` (`common/fdi.ts`) folds Persian
and Arabic-Indic digits to ASCII and strips the spaces of a digit-by-digit dictation
before anything is matched, at both the wire branch choice and the final validation.
### `resolveDueDate()`
- Takes `clientTodayIso` + IANA `timeZone`; reuses `common/zoned-civil-time.ts`.
- Week start is **per locale**, because "next Thursday" is week-relative: `fa` starts
Saturday, `en` and `nl` start Monday. Hardcoding Saturday put an en/nl clinician's
deadline a week out. One place (`weekStartForLocale`), tested in both directions.
- `'this'` is occurrence-anchored (soonest strictly-future, never resolves into the past);
`'next'` is week-anchored. A missing qualifier is read as `'this'` — a bare weekday
carries none, and failing would discard a real spoken deadline.
- Jalali conversion is arithmetic, not inference: port `jalaliToGregorian` and
`toLatinDigits` from `frontend/src/lib/i18n/persianCalendar.ts` into
`backend/src/common/jalali.ts` with a spec. It is dependency-free integer math
(~60 relevant lines) and the calendar does not change, so the duplication is stable.
### Group / prosthesis rules
- Connected spans validate through the shipped helpers — `areArchNeighbors`, `sameArch`,
`teethBetweenInclusive`. **Never a 1-tooth connected group.** Anything invalid degrades
to singles and is flagged on the review sheet.
- Prosthesis: expand `defaultType` across all teeth, then apply per-tooth overrides.
- **All-or-nothing.** `assertCompleteToothProsthesisMap` requires every tooth on a
`prosthesis` detail to carry a `prosthesisTypeCode` or the send throws
`TREATMENT_TOOTH_PROSTHESIS_INCOMPLETE`. So if even one tooth ends untyped, the
prosthesis row is marked incomplete and stays unticked — the unshippable state surfaces
at review, where it is cheap, not at dispatch minutes later on another screen.
- **Apply order is teeth → groups → prosthesis**, so `pruneToothProsthesisForGroups`
behaves.
---
## 7. Review sheet
**Presentation:** a **modal** on desktop; a **full-screen overlay** on mobile.
> ⚠ On mobile this must be an overlay rendered in place — **not** a Next.js route. A real
> navigation unmounts `TreatmentWorkspace` and destroys the in-progress draft. This is the
> same constraint the realtime soft-refresh already lives under: never remount the
> treatment form, never clear a draft.
- Renders the transcript, then one row per extracted field in the app's own vocabulary:
translated catalog labels, and a mini FDI chart for the teeth rather than a list of
numbers.
- Each row has a checkbox. Ticked rows apply; nothing else is touched. Confirm is also
what creates the new detail — see §2.
- Rows default to ticked **except**:
- the lab row when `labMatchExact` is false — shipping to a lab always requires a
deliberate tick;
- any row carrying an unresolved item or an incomplete prosthesis map.
- Unresolved items are shown with what was heard ("دندان شیری — بازشناسی نشد"), so the
clinician can see what the system did not understand.
- **The sheet is a contract: confirm fills exactly what it previewed — no more.** Any
per-detail convenience that would top the case up afterwards has to be suppressed for a
voice-created case, because a default that quietly adds a prosthesis type to a tooth the
sheet never mentioned turns the confirmation step into a lie about what it was going to
do — which is the whole reason the step exists.
> This branch carried an exemption for one such default, the dispatch panel's
> remembered-prosthesis auto-fill. `origin/master` deleted that feature outright
> (`f52ad6b`), so the exemption went with it in the rebase and nothing enforces this rule
> in code today. It is a constraint on whatever gets added next, not a description of
> something that exists.
- An item that carries `candidates` renders them as **tappable chips** — the one place the
sheet is interactive. Picking one folds the tooth into the result (`withChosenTeeth`) and
ticks the teeth row, so an under-specified tooth is one tap from resolved instead of a
dead end. Everything the sheet renders comes from that folded result, not the raw one.
- RTL-safe: logical `text-start` / `text-end` only, never `text-left`/`text-right`.
Dates via `lib/i18n/format.ts`.
---
## 8. Gating and configuration
Gate = `TAB_TREATMENT_EDIT` ∧ `canEditTreatmentForDay` ∧ configured locale profile ∧
`Plan.features.voiceTreatmentEntry`. How each failing condition *renders* is in §2.
`Plan.features` is a `Json` column that already exists on the `Plan` model and is seeded
as `{}` for all five plans — and is **read nowhere in `backend/src/`**. Voice is its first
consumer, so:
- No migration is needed.
- The read should go through a small generic helper, since this establishes the pattern
for every future flag.
- Voice is metered vendor spend, which is why the gate is commercial (plan) rather than a
new `TAB_*` permission — a clinician who can already edit the form gains no capability
from voice, only speed.
The per-user throttle is a separate, non-commercial abuse control.
### v1 ships ungated
**The plan flag is designed but not enforced in v1** — voice is open to every clinic user
who can edit treatments, in every configured locale. `Plan.features.voiceTreatmentEntry`
and the availability API stay documented here as the intended gate, deferred rather than
dropped, so turning them on later is additive.
Consequence to accept deliberately: with no plan gate, the per-user throttle and the 2-minute
recording cap are the **only** controls on metered vendor spend. See open item 14.
---
## 9. Errors
Per the three-layer contract: a code in `common/errors/error-codes.ts`, the throw site,
and an `errors.X` key in **all three** of `frontend/messages/{en,fa,nl}.json`. Never a raw
English Nest exception for a user-facing failure.
| Code | When |
|---|---|
| `VOICE_MIC_DENIED` | microphone permission actually refused, or no input device — **client-side only**: needs the `errors.X` key in all three message files, but no `ErrorCode` entry and no throw site. Reserved for a real permission failure: see the note below |
| `VOICE_CLIP_TOO_LONG` | over `maxMs` (server-side re-check), over vendor limits, or a request body past the DTO's size cap |
| `VOICE_UNSUPPORTED_FORMAT` | **the browser cannot record at all** — no `MediaRecorder`, no container both it and the API accept, or a recorder that throws after permission was granted; and server-side, a `format` outside `VOICE_AUDIO_FORMATS` |
| `VOICE_ASR_FAILED` | transcription stage failed |
| `VOICE_EXTRACT_FAILED` | transcript obtained, structuring failed |
| `VOICE_NOTHING_RECOGNIZED` | empty or unusable transcript |
| `VOICE_NOT_AVAILABLE` | no profile for locale (v1); plan flag off, once enforced |
| `VOICE_RATE_LIMITED` | throttle |
**Two of these are raised by DTO validation, not by a throw site.**
`validationExceptionFactory` returns a constraint's `message` verbatim when the message is
itself a known `ErrorCode`, so the voice DTO names its own failures:
`@MaxLength(…, { message: ErrorCode.VOICE_CLIP_TOO_LONG })` and
`@IsIn(…, { message: ErrorCode.VOICE_UNSUPPORTED_FORMAT })`. Left to the shared constraint map
they fall through to `VALIDATION_FIELD_REQUIRED` and `VALIDATION_LANGUAGE_INVALID` — an
oversized recording telling the clinician a field is missing, and an unsupported container
telling them their language is invalid. Any new voice constraint should name its code the same
way.
**`VOICE_MIC_DENIED` is only for a real permission failure.** Three client paths used to report
it for something else entirely — no `MediaRecorder`, no acceptable container, and a recorder
that throws after permission was already granted. All three are "this browser cannot record"
and now report `VOICE_UNSUPPORTED_FORMAT`; blaming the microphone sends the clinician hunting
in site settings for a permission nothing ever asked for.
**Transcript salvage — specified, NOT built.** The backend half exists: `VOICE_EXTRACT_FAILED`
carries `details.transcript` and `HttpExceptionFilter` forwards it. The client half was
never written — `onError` only resolves a message through `getUserFacingError`, which never
reads `details`, so the transcript is shipped in an error body and dropped. Either build the
dialog below or stop returning the transcript; shipping dictation to the client and
discarding it is the worst of both.
When ASR succeeded and only extraction failed, the response still
carries the transcript and the failure dialog offers *"افزودن به یادداشت"*. That action
**creates a new detail with only `comment` set to the transcript** — everything else left
at `newDetail()` defaults. The words were captured and paid for; only the structure was
lost.
This keeps the feature's one invariant intact: **voice never writes into an existing
detail.** Dictating into an already-filled detail is a separate, later feature with its
own voice-to-text control scoped to that field (§1, out of scope).
It also does not bypass the confirmation rule — the dialog shows the transcript, and the
dentist taps to accept it. That review matters, because a raw transcript carries ASR
errors and may contain the patient's spoken name, and `comment` is persisted (§10).
---
## 10. Data handling
- Audio is held **in memory for the request only**. Never written to disk, never a Prisma
row. Note this is deliberately unlike treatment attachments, which do persist to
`backend/uploads/treatments`.
- The transcript goes to the browser for the review sheet and dies with it.
- The `comment` field persists a cleaned version of what was said — that is legitimate
clinical record-keeping and is the only durable trace.
- Telemetry is **structured and patient-free**: clip duration, which fields resolved,
unresolved count, vendor latency, `usage.cost`, outcome (applied / discarded / failed),
locale. Never the transcript, never audio, never a patient identifier.
The **destination is an open question** — this repo has no metrics infrastructure yet
(open item 9). Emit it as structured log lines in the interim so the fields exist and
can be routed later without changing call sites.
**Cancelling aborts the in-flight vendor call** via `AbortController`, rather than letting
it settle and discarding the result. Note this reduces spend but does not eliminate it:
work already performed upstream may still be billed.
Recordings are clinical descriptions of identifiable patients leaving the server for a
third party. Confirm the provider's retention and training policy in writing before
enabling this for real clinics.
---
## 11. Open items
1. **Persian ASR accuracy on tooth numbers is unmeasured, and it decides the feature.**
Everything downstream assumes a usable transcript; no design choice above compensates
for a bad one. Run this before writing feature code.
*Why tooth numbers specifically, not general accuracy:* a transcript can score well on
WER and still be useless here, because the errors land on the digits.
`چهار` (4) / `چهارده` (14) / `چهل` (40) differ by one syllable. FDI spoken as
`یک چهار` may return as `۱۴`, `14`, or `یک چهار`. Persian and Latin digit scripts mix
within one transcript. Jargon is loanwords (`زیرکونیا`, `پرسلن فیوزد تو متال`,
`اینله`/`آنله`) and clinicians code-switch into English mid-sentence. Suction and
handpiece run in the background.
*Protocol:* ~25 utterances from a dentist reading a script covering explicit FDI,
quadrant-relative phrasing, bridges, prosthesis types, due dates and notes — recorded
on the real device, ideally once quiet and once with the operatory running. POST each
to `/api/v1/audio/transcriptions` with `language: "fa"`. Score **per-tooth-reference
accuracy** (of every tooth spoken, how many survive recoverably?) and jargon
recognition separately. Notes accuracy barely matters. ~6 minutes of audio ≈ **$0.04**.
*Decision it drives:* if tooth accuracy holds, build. If not, the fix is not prompt
tuning — it is pointing the `fa` ASR slot at a Persian-specialist vendor while `en`/`nl`
keep `whisper-1`. That is a config change precisely because of decision 23.
*Unverified lead for that fallback:* recent Persian-ASR benchmark work reports
Qwen3-Omni as the strongest open Persian ASR. Not confirmed as available on
OpenRouter's transcription endpoint — check before relying on it.
*Byproduct:* the recordings become the fixture corpus for item 2.
2. ~~**`VOICE_LLM_MODEL`**~~ — **resolved:** `google/gemini-3.7-flash` (§4). Still worth
validating on real whisper output during item 1, since clean transcripts flatter every
model; the escalation path if Persian comprehension disappoints is documented in §4 and
is a config change.
3. ~~**Production reachability**~~ — **resolved:** confirmed reachable from the Iranian
production host. The per-locale registry is retained regardless (decision 23), so the
`fa` profile can still be repointed at a domestic vendor if item 1 goes badly.
4. ~~**Provider retention/training policy**~~ — **resolved:** restricted via OpenRouter's
account-level privacy/data-policy settings. Note this is an *account* setting, not a
per-request one: re-verify it if the API key or the OpenRouter account changes, and
remember `whisper-1` is forwarded to OpenAI, so the effective policy is OpenRouter's
plus that provider's.
5. ~~**English tooth numbering**~~ — **resolved:** a bare two-digit number is read as
**FDI in all three locales**. FDI is what the product is built on and what clinicians
dictate. Known trade-off, accepted: a clinician trained under Universal numbering says
"tooth 14" and means a different tooth, so an `en` clinic needs either training or a
later per-org notation setting. Revisit if a US clinic is onboarded.
6. **`nl` and `en` have no spike data.** The Persian spike (item 1) should be repeated per
locale before that locale's mic is enabled for real users — same protocol, same
scoring, different speaker.
7. ~~**Transcript salvage has no target**~~ — **resolved:** salvage creates a **new
detail** with only `comment` set to the transcript (§9). Preserves the invariant that
voice never writes into an existing detail; dictating into a filled detail becomes a
separate later feature with its own field-scoped control.
8. ~~**Throttle limits**~~ — **resolved:** configurable, v1 default **6 requests / 60s per
user** (§3). Unreachable by a human; a pure abuse guard.
9. **Telemetry has no sink.** §10 defines exactly what to record but not where it goes —
this repo has no metrics or analytics infrastructure yet. Deliberately deferred until
it does. Interim: structured log lines, so the fields exist and can be routed later
without touching call sites.
10. ~~**Review sheet modality**~~ — **resolved:** modal on desktop, full-screen **overlay**
on mobile (§7). Not a route — navigating would unmount `TreatmentWorkspace` and destroy
the draft.
11. ~~**Cancel and in-flight requests**~~ — **resolved:** abort via `AbortController` to
limit spend (§10). Upstream work already performed may still be billed.
12. ~~**Recording duration cap**~~ — **resolved:** **2 minutes**, configurable via `maxMs`
(§2). Still worth timing a realistic worst-case prosthesis dictation during item 1 to
confirm 2 minutes is comfortable rather than tight.
13. ~~**Availability API**~~ — **resolved for v1:** voice ships **open to everyone** with a
configured locale profile. No plan check, no availability endpoint. The `Plan.features`
design in §8 is deferred, not dropped.
### New
14. **Cost exposure is bounded but ungated in v1.** With the 2-minute cap (12) and the
6/60s throttle (8), worst case is 12 audio-minutes per user-minute ≈ **$0.072/min**, or
~$4.30 per hour of sustained abuse by one user — bounded, not free. Normal use is far
below this: a clinician doing 60 recordings a day at ~20s each costs about **$0.18/day**.
What remains open is that there is **no per-organization limit at all**, because v1
ships ungated (13). Worth deciding what to watch and at what number to react.
Mitigations, already designed and each a config or flag change: enforce the
`Plan.features` gate, lower `maxMs` or the throttle, or add an org-level monthly minute
budget.
---
## 12. Verification
- `cd backend && npm test` — new suites for `resolveToothIntent` (quadrant mapping in all
four quadrants, out-of-range rejection, deciduous → unresolved), `resolveDueDate`
(per-locale week start, "this" vs "next" weekday, Jalali leap year, month-end), the
Jalali port, prosthesis expansion + completeness, and connected-span validation.
- `cd backend && npm run build` — cross-cutting backend gate.
- `cd frontend && npx tsc --noEmit` — frontend gate.
- Manual: fa locale, editable day, prosthesis detail with a bridge, dispatch to a linked
lab. Then specifically:
- **past day** → both segments disabled, control still split (not absent);
- **locale with no profile** → control renders unsplit, identical to today;
- **fa vs en** → mic sits at the logical end in both, on the same side as the chip's
trash;
- **cancel mid-recording** → chip strip unchanged, no orphan detail;
- **confirm** → always appends a new chip, whatever the active detail contains;
- **Add half** → behaves exactly as it did before this change;
- **tap 🎤 during the Lab wizard step** → confirm returns to the treatment step;
- no layout shift in the header row on record start, stop, or the 2:00 auto-stop;
- **hold past 2:00** → auto-stops and proceeds to processing, not an error;
- **cancel during processing** → the vendor request is actually aborted;
- **review sheet on mobile** → full-screen overlay; closing it leaves the draft intact;
- **confirm with a lab, a due date or a prosthesis map, then reload** → all three are still
there. They live on the lab case, which the autosave effect does not watch, so this is
the check that catches a lab draft left unsaved in component state;
- **record straight after opening a visit**, while the blank chip is still untyped, and
confirm with a lab ticked → no error toast: confirm detects the preview treatment and
skips the lab-case save rather than posting an id the server has never seen;
- **dictate two different prosthesis types** ("۱۲ روکش PFM، ۱۳ روکش PFZ") → the form shows
both, and the bulk «اعمال برای همه دندان‌ها» select stays on its placeholder. Nothing may
rewrite a per-tooth type the sheet already showed.
---
## 13. Decision log
Settled in a grilling session on 2026-08-20.
| # | Question | Decision |
|---|---|---|
| 1 | Scope | Everything including lab dispatch |
| 2 | AI supply chain | Domestic provider originally; OpenRouter for v1, registry keeps both open |
| 3 | Apply model | Review sheet, then apply |
| 4 | Speech → FDI | LLM emits intent, code resolves. A spoken number **is** the FDI code (revised 2026-08-21, §6) |
| 5 | Cardinality | One detail per recording |
| 6 | Lab destination | Closed list of linked labs, explicit confirm, unticked when inexact |
| 7 | Due date | Intent + deterministic resolver |
| 8 | Resolver location | Backend, Jalali math ported |
| 9 | Prosthesis | Default type + overrides, all-or-nothing |
| 10 | Retention | Discard audio and transcript, non-PHI telemetry only |
| 11 | Capture | Tap to start/stop, hard cap (see 27) |
| 12 | Failure UX | Stage-aware codes, transcript salvage |
| 13 | Locales | Provider registry per locale; all three locales enabled |
| 14 | Reachability | Registry now, slots filled per deployment |
| 23 | ASR model | `openai/whisper-1` for **every** locale; registry kept so `fa` can diverge |
| 24 | Extraction model | **`google/gemini-3.7-flash`**; escalation path documented in §4 |
| 25 | Salvage target | Creates a new detail with only `comment` set — voice never writes into an existing detail |
| 26 | Throttle | Configurable; v1 default 6 requests / 60s per user |
| 27 | Duration cap | **2 minutes**, configurable via `maxMs` |
| 28 | Review sheet | Modal on desktop, full-screen overlay (not a route) on mobile; candidate chips are its only interactive part |
| 29 | Cancel | Aborts the in-flight vendor call |
| 30 | v1 gating | Open to everyone; `Plan.features` gate deferred, not dropped |
Added while getting the first live recordings working (2026-08-21):
| # | Question | Decision |
|---|---|---|
| 31 | Tooth numbering | A spoken number **is** its FDI code, in all three locales. A lone digit stays unresolved and offers its candidate teeth as chips (§6, §7) |
| 32 | What confirm writes | Confirm persists the detail *and* its lab case, because autosave watches `details` only — but skips the lab-case save when it got a preview treatment back (§2) |
| 33 | Preview as contract | Applying a voice result fills exactly what the sheet showed. Per-detail conveniences that would add more are suppressed for that case (§7) |
| 15 | Gating | `Plan.features` flag — its first consumer |
UI placement settled in a second grilling session on 2026-08-20.
| # | Question | Decision |
|---|---|---|
| 16 | Mic action | Independent record action; the Add half's logic is untouched |
| 17 | Record UI | Mic segment toggles ▶/■; inline bar below the header row |
| 18 | Unavailable | Absent for technical + commercial; disabled for contextual |
| 19 | Component API | One optional `voice` object prop; `undefined` means absent |
| 20 | Side | Logical end (`border-s`), exactly like the chip's trash |
| 21 | Creation | On confirm, never on tap |
| 22 | Creation rule | Confirm **always** appends a new detail — no blank-reuse guard |

35
frontend/messages/en.json
View File

@@ -899,7 +899,30 @@
"toothAria": "FDI tooth {fdi}",
"toothSelectedSuffix": ", selected",
"sentToAt": "Sent to {orgName} at {datetime}",
"fallbackOrgName": "organization"
"fallbackOrgName": "organization",
"voiceStart": "Record treatment",
"voiceStop": "Stop recording",
"voiceCancel": "Cancel",
"voiceProcessing": "Reading the recording…",
"voiceReviewTitle": "Check what was understood",
"voiceNothingExtracted": "Nothing usable was picked up from that recording.",
"voiceProsthesisIncomplete": "No prosthesis type for {teeth} — the case cannot be sent until every tooth has one.",
"voiceLabInexact": "The spoken name only partly matched this lab. Confirm before sending.",
"voiceNotUnderstood": "Not understood",
"voicePickTooth": "Which tooth?",
"voiceDiscard": "Discard",
"voiceApply": "{count, plural, one {Apply # field} other {Apply # fields}}",
"voiceUnresolved": {
"not_permanent_tooth": "not a permanent tooth",
"position_out_of_range": "not a valid tooth position",
"tooth_missing_quadrant": "not a whole tooth number — say e.g. “twenty-six”",
"malformed": "could not be read",
"span_not_same_arch": "a bridge cannot span both jaws",
"unknown_catalog_code": "not in this clinic’s list",
"tooth_not_selected": "that tooth is not part of this detail",
"invalid_date": "not a usable date"
},
"voiceFailed": "Voice entry failed. Please try again."
},
"organizations": {
"loadingOrganization": "Loading organization...",
@@ -1240,6 +1263,14 @@
"LAB_CASE_START_INCOMPLETE": "Complete teeth and prosthesis types before starting this case.",
"LAB_CASE_CLIENT_REQUIRED": "Enter a clinic name or a patient name for this case.",
"BAD_REQUEST": "The request could not be processed.",
"INTERNAL_ERROR": "Something went wrong on our end. Please try again later."
"INTERNAL_ERROR": "Something went wrong on our end. Please try again later.",
"VOICE_MIC_DENIED": "Microphone access was blocked. Allow it in your browser settings and try again.",
"VOICE_NOT_AVAILABLE": "Voice entry is not available for this language yet.",
"VOICE_CLIP_TOO_LONG": "That recording is too long. Please keep it under two minutes.",
"VOICE_UNSUPPORTED_FORMAT": "That recording format is not supported on this device.",
"VOICE_ASR_FAILED": "Could not turn the recording into text. Please try again.",
"VOICE_EXTRACT_FAILED": "Could not read the treatment details from the recording.",
"VOICE_NOTHING_RECOGNIZED": "No speech was recognised. Check the microphone and try again.",
"VOICE_RATE_LIMITED": "Too many recordings in a short time. Please wait a moment."
}
}

37
frontend/messages/fa.json
View File

@@ -900,7 +900,30 @@
"toothAria": "دندان FDI {fdi}",
"toothSelectedSuffix": "، انتخاب شده",
"sentToAt": "ارسال به {orgName} در {datetime}",
"fallbackOrgName": "سازمان"
"fallbackOrgName": "سازمان",
"voiceStart": "ثبت گفتاری درمان",
"voiceStop": "توقف ضبط",
"voiceCancel": "لغو",
"voiceProcessing": "در حال پردازش گفتار…",
"voiceReviewTitle": "بررسی آنچه دریافت شد",
"voiceNothingExtracted": "از این ضبط چیز قابل استفاده‌ای برداشت نشد.",
"voiceProsthesisIncomplete": "برای {teeth} نوع پروتز مشخص نشده — تا زمانی که همه دندان‌ها نوع داشته باشند، کیس ارسال نمی‌شود.",
"voiceLabInexact": "نام گفته‌شده فقط تا حدی با این لابراتوار مطابقت داشت. پیش از ارسال تأیید کنید.",
"voiceNotUnderstood": "شناسایی نشد",
"voicePickTooth": "کدام دندان؟",
"voiceDiscard": "انصراف",
"voiceApply": "{count, plural, one {اعمال # مورد} other {اعمال # مورد}}",
"voiceUnresolved": {
"not_permanent_tooth": "دندان دائمی نیست",
"position_out_of_range": "شماره دندان معتبر نیست",
"tooth_missing_quadrant": "شماره کامل دندان نیست — مثلاً «بیست و شش»",
"malformed": "قابل خواندن نبود",
"span_not_same_arch": "بریج نمی‌تواند بین دو فک باشد",
"unknown_catalog_code": "در فهرست این مطب نیست",
"tooth_not_selected": "این دندان بخشی از این مورد نیست",
"invalid_date": "تاریخ قابل استفاده نیست"
},
"voiceFailed": "ثبت گفتاری انجام نشد. لطفاً دوباره تلاش کنید."
},
"organizations": {
"loadingOrganization": "در حال بارگذاری سازمان...",
@@ -1241,6 +1264,14 @@
"LAB_CASE_START_INCOMPLETE": "قبل از شروع پرونده، دندان‌ها و نوع پروتز را کامل کنید.",
"LAB_CASE_CLIENT_REQUIRED": "نام کلینیک یا نام بیمار را وارد کنید.",
"BAD_REQUEST": "درخواست قابل پردازش نبود.",
"INTERNAL_ERROR": "مشکلی در سرور رخ داد. لطفاً بعداً تلاش کنید."
"INTERNAL_ERROR": "مشکلی در سرور رخ داد. لطفاً بعداً تلاش کنید.",
"VOICE_MIC_DENIED": "دسترسی به میکروفون مسدود شده است. در تنظیمات مرورگر اجازه دهید و دوباره تلاش کنید.",
"VOICE_NOT_AVAILABLE": "ثبت گفتاری هنوز برای این زبان در دسترس نیست.",
"VOICE_CLIP_TOO_LONG": "مدت ضبط بیش از حد است. لطفاً کمتر از دو دقیقه صحبت کنید.",
"VOICE_UNSUPPORTED_FORMAT": "قالب این ضبط پشتیبانی نمی‌شود.",
"VOICE_ASR_FAILED": "تبدیل گفتار به متن انجام نشد. لطفاً دوباره تلاش کنید.",
"VOICE_EXTRACT_FAILED": "اطلاعات درمان از روی گفتار استخراج نشد.",
"VOICE_NOTHING_RECOGNIZED": "گفتاری شناسایی نشد. میکروفون را بررسی کنید و دوباره تلاش کنید.",
"VOICE_RATE_LIMITED": "تعداد ضبط‌ها در بازه کوتاه زیاد بود. کمی صبر کنید."
}
}
}

37
frontend/messages/nl.json
View File

@@ -899,7 +899,30 @@
"toothAria": "FDI-tand {fdi}",
"toothSelectedSuffix": ", geselecteerd",
"sentToAt": "Verzonden naar {orgName} op {datetime}",
"fallbackOrgName": "organisatie"
"fallbackOrgName": "organisatie",
"voiceStart": "Behandeling inspreken",
"voiceStop": "Opname stoppen",
"voiceCancel": "Annuleren",
"voiceProcessing": "Opname wordt gelezen…",
"voiceReviewTitle": "Controleer wat is begrepen",
"voiceNothingExtracted": "Uit deze opname is niets bruikbaars opgepikt.",
"voiceProsthesisIncomplete": "Geen prothesetype voor {teeth} — de casus kan pas worden verstuurd als elk element er een heeft.",
"voiceLabInexact": "De uitgesproken naam kwam slechts deels overeen met dit lab. Bevestig voor verzending.",
"voiceNotUnderstood": "Niet begrepen",
"voicePickTooth": "Welk element?",
"voiceDiscard": "Verwerpen",
"voiceApply": "{count, plural, one {# veld toepassen} other {# velden toepassen}}",
"voiceUnresolved": {
"not_permanent_tooth": "geen blijvend element",
"position_out_of_range": "geen geldige elementpositie",
"tooth_missing_quadrant": "geen volledig elementnummer — bijv. “zesentwintig”",
"malformed": "kon niet worden gelezen",
"span_not_same_arch": "een brug kan niet over beide kaken lopen",
"unknown_catalog_code": "staat niet in de lijst van deze praktijk",
"tooth_not_selected": "dat element hoort niet bij dit onderdeel",
"invalid_date": "geen bruikbare datum"
},
"voiceFailed": "Spraakinvoer is mislukt. Probeer het opnieuw."
},
"organizations": {
"loadingOrganization": "Organisatie laden...",
@@ -1240,6 +1263,14 @@
"LAB_CASE_START_INCOMPLETE": "Vul tanden en prothesetypes in voordat u deze case start.",
"LAB_CASE_CLIENT_REQUIRED": "Voer een klinieknaam of een patiëntnaam in voor deze case.",
"BAD_REQUEST": "Het verzoek kon niet worden verwerkt.",
"INTERNAL_ERROR": "Er is iets misgegaan aan onze kant. Probeer het later opnieuw."
"INTERNAL_ERROR": "Er is iets misgegaan aan onze kant. Probeer het later opnieuw.",
"VOICE_MIC_DENIED": "Microfoontoegang is geblokkeerd. Sta dit toe in uw browserinstellingen en probeer opnieuw.",
"VOICE_NOT_AVAILABLE": "Spraakinvoer is nog niet beschikbaar voor deze taal.",
"VOICE_CLIP_TOO_LONG": "Die opname is te lang. Houd het onder twee minuten.",
"VOICE_UNSUPPORTED_FORMAT": "Dit opnameformaat wordt niet ondersteund.",
"VOICE_ASR_FAILED": "De opname kon niet naar tekst worden omgezet. Probeer het opnieuw.",
"VOICE_EXTRACT_FAILED": "De behandelgegevens konden niet uit de opname worden gelezen.",
"VOICE_NOTHING_RECOGNIZED": "Er is geen spraak herkend. Controleer de microfoon en probeer opnieuw.",
"VOICE_RATE_LIMITED": "Te veel opnames in korte tijd. Wacht even."
}
}
}

95
frontend/src/components/treatment/voiceReviewRows.ts Normal file
View File

@@ -0,0 +1,95 @@
import { groupsFromFlatTeeth } from '@/components/treatment/toothSelectionGroups';
import type { FdiToothId } from '@/types/treatment';
import type {
VoiceApplySelection,
VoiceExtractionResult,
VoiceProsthesisResult,
} from '@/types/voice';
/** Which rows the review sheet renders at all — a row with nothing extracted is noise. */
export function voiceRowAvailability(result: VoiceExtractionResult) {
return {
treatmentType: result.treatmentType != null,
teeth: result.teeth.length > 0,
comment: Boolean(result.comment?.trim()),
prosthesis: result.prosthesis != null,
lab: result.labId != null,
dueDate: result.dueDate != null,
};
}
/**
* Everything available ticks itself, with two exceptions: an inexactly-matched lab, because
* it is the one extracted value whose error leaves the building; and an incomplete
* prosthesis map, which cannot ship at all and would just move the failure to dispatch.
*/
export function initialVoiceSelection(result: VoiceExtractionResult): VoiceApplySelection {
const available = voiceRowAvailability(result);
return {
treatmentType: available.treatmentType,
teeth: available.teeth,
comment: available.comment,
prosthesis: available.prosthesis && result.prosthesis?.complete === true,
lab: available.lab && result.labMatchExact,
dueDate: available.dueDate,
};
}
/**
* Intersected with availability rather than counting ticks: a row can be ticked and then lose
* its content, and "Apply 1 item" that applies nothing is worse than a wrong number.
*/
export function countSelected(
selection: VoiceApplySelection,
available: Record<keyof VoiceApplySelection, boolean>,
): number {
return (Object.keys(selection) as (keyof VoiceApplySelection)[]).filter(
(key) => selection[key] && available[key],
).length;
}
/** Mirrors the backend's rule: every selected tooth needs a code, or the case cannot ship. */
function recheckProsthesis(
prosthesis: VoiceProsthesisResult,
teeth: readonly FdiToothId[],
): VoiceProsthesisResult {
const missingTeeth = teeth.filter((tooth) => !prosthesis.byTooth[tooth]);
return { ...prosthesis, missingTeeth, complete: missingTeeth.length === 0 };
}
/**
* Fold the candidate picks into the result, so nothing downstream has to know chips exist.
*
* Union rather than toggle: a candidate can coincidentally be a tooth the recording already
* produced ("۱۲ و دو"), and tapping it must not deselect that one.
*/
export function withChosenTeeth(
result: VoiceExtractionResult,
chosen: readonly FdiToothId[],
): VoiceExtractionResult {
if (chosen.length === 0) return result;
const teeth = [...new Set([...result.teeth, ...chosen])].sort() as FdiToothId[];
return {
...result,
teeth,
toothSelectionGroups: groupsFromFlatTeeth(teeth, result.toothSelectionGroups),
prosthesis: result.prosthesis ? recheckProsthesis(result.prosthesis, teeth) : null,
};
}
/** Teeth that are part of a bridge, for the read-only chart's connection marks. */
export function connectedTeethFromResult(result: VoiceExtractionResult): Set<FdiToothId> {
const connected = new Set<FdiToothId>();
for (const group of result.toothSelectionGroups) {
if (group.kind !== 'connected') continue;
for (const tooth of group.teeth) connected.add(tooth);
}
return connected;
}
/** A recording that produced nothing should say so, not show an empty form of checkboxes. */
export function hasAnythingToApply(result: VoiceExtractionResult): boolean {
return Object.values(voiceRowAvailability(result)).some(Boolean);
}

118
frontend/src/components/ui/treatment/TreatmentDetailsEditor.tsx
View File

@@ -2,12 +2,14 @@
import { useEffect, useRef, type ReactNode, type RefObject } from 'react';
import { useTranslations } from 'next-intl';
import { Trash2 } from 'lucide-react';
import { Mic, Square, Trash2 } from 'lucide-react';
import { Button } from '@/components/ui/shared/Button';
import { Dropdown } from '@/components/ui/shared/Dropdown';
import { formatDetailChipLabel } from '@/components/treatment/detailChipLabel';
import { autosaveStatusClass, labBlockedBannerClass } from '@/components/treatment/treatmentStatusStyles';
import { TreatmentDetailAttachmentsStrip } from '@/components/ui/treatment/TreatmentDetailAttachmentsStrip';
import { VoiceRecordingBar } from '@/components/ui/treatment/VoiceRecordingBar';
import type { VoiceCaptureState } from '@/lib/voice/useVoiceCapture';
import type { TreatmentDetailDraft } from '@/types/treatment';
import type { TreatmentCatalogEntry } from '@/types/treatment-catalog';
import { treatmentTypeColor, treatmentTypeOptionStyle } from '@/components/shared/treatmentTypeDisplay';
@@ -42,6 +44,12 @@ interface TreatmentDetailsEditorProps {
stepper?: ReactNode;
/** Shown below type + chart + notes (e.g. Continue to lab). */
footer?: ReactNode;
/**
* Voice entry. Omit when unavailable — the Add button then renders unsplit, exactly as
* before this feature existed. Presence *is* the availability flag, so the two cannot
* disagree.
*/
voice?: VoiceCaptureState;
/** Dim the chart until a treatment type is chosen. */
chartLocked?: boolean;
chartLockMessage?: string;
@@ -65,6 +73,7 @@ export function TreatmentDetailsEditor({
onRemoveAttachment,
showChrome = true,
showFields = true,
voice,
chart,
stepper,
footer,
@@ -109,18 +118,31 @@ export function TreatmentDetailsEditor({
<div>
<h3 className="text-sm font-semibold text-text-primary">{t('detailsTitle')}</h3>
</div>
<Button
type="button"
variant="primary"
disabled={!canEdit || disabled}
onClick={onAddDetail}
fullWidth
className="sm:w-auto shrink-0"
>
{t('addDetail')}
</Button>
{voice ? (
<AddDetailWithVoice
addLabel={t('addDetail')}
startLabel={t('voiceStart')}
stopLabel={t('voiceStop')}
disabled={!canEdit || disabled}
onAddDetail={onAddDetail}
voice={voice}
/>
) : (
<Button
type="button"
variant="primary"
disabled={!canEdit || disabled}
onClick={onAddDetail}
fullWidth
className="sm:w-auto shrink-0"
>
{t('addDetail')}
</Button>
)}
</div>
{voice ? <VoiceRecordingBar voice={voice} /> : null}
<div className="flex flex-wrap gap-2">
{details.map((d, idx) => {
const detailLocked = isDetailLocked(d);
@@ -312,3 +334,77 @@ function NotesField({
</label>
);
}
/**
* "Add detail", split into two segments with the microphone at the logical end.
*
* Built like the detail chip's trash affordance in this same file — a wrapper holding two
* raw `<button>`s divided by `border-s` — rather than two shared `Button`s, which hardcode
* their own rounding and would fight a segmented control. `border-s` puts the microphone
* visually right in en/nl and left in fa, on the same side as the chip's trash in both.
*
* Add keeps its exact existing behaviour; the microphone is an independent action.
*/
function AddDetailWithVoice({
addLabel,
startLabel,
stopLabel,
disabled,
onAddDetail,
voice,
}: {
addLabel: string;
startLabel: string;
stopLabel: string;
disabled: boolean;
onAddDetail: () => void;
voice: VoiceCaptureState;
}) {
const isRecording = voice.phase === 'recording';
const isBusy = voice.phase !== 'idle';
const micLabel = isRecording ? stopLabel : startLabel;
return (
<div
className={`
inline-flex w-full items-stretch overflow-hidden rounded-[var(--radius-md)]
bg-primary text-white shrink-0 sm:w-auto
${disabled ? 'opacity-60' : ''}
`}
>
<button
type="button"
onClick={onAddDetail}
disabled={disabled || isBusy}
className="
flex-1 px-4 py-2 text-sm font-medium transition-all duration-200
hover:opacity-90 focus:outline-none focus-visible:ring-2 focus-visible:ring-inset
focus-visible:ring-white/60
disabled:cursor-not-allowed disabled:opacity-60 disabled:hover:opacity-60
"
>
{addLabel}
</button>
<button
type="button"
onClick={isRecording ? voice.onStop : voice.onStart}
disabled={disabled || voice.phase === 'processing'}
title={micLabel}
aria-label={micLabel}
className={`
inline-flex items-center justify-center border-s border-white/25 px-3
transition-all duration-200 focus:outline-none focus-visible:ring-2
focus-visible:ring-inset focus-visible:ring-white/60
disabled:cursor-not-allowed disabled:opacity-60
${isRecording ? 'bg-red-600 hover:bg-red-700' : 'hover:opacity-90'}
`}
>
{isRecording ? (
<Square className="h-4 w-4 fill-current" aria-hidden />
) : (
<Mic className="h-4 w-4" aria-hidden />
)}
</button>
</div>
);
}

163
frontend/src/components/ui/treatment/TreatmentWorkspace.tsx
View File

@@ -29,6 +29,14 @@ import {
import { appointmentsApi } from '@/lib/api/appointments';
import { treatmentCatalogApi } from '@/lib/api/treatment-catalog';
import { prosthesisCatalogApi } from '@/lib/api/prosthesis-catalog';
import { voiceApi } from '@/lib/api/voice';
import { useVoiceCapture } from '@/lib/voice/useVoiceCapture';
import { VoiceReviewSheet } from '@/components/ui/treatment/VoiceReviewSheet';
import type {
VoiceApplySelection,
VoiceAvailability,
VoiceExtractionResult,
} from '@/types/voice';
import { treatmentsApi } from '@/lib/api/treatments';
import { notificationsApi } from '@/lib/api/notifications';
import { pickAutoAppointment } from '@/components/shared/treatmentSelection';
@@ -464,6 +472,11 @@ export function TreatmentWorkspace({
const [showWholeTreatmentPlan, setShowWholeTreatmentPlan] = useState(false);
const [entryStep, setEntryStep] = useState<EntryStep>('treatment');
const [voiceAvailability, setVoiceAvailability] = useState<VoiceAvailability | null>(null);
const [voiceResult, setVoiceResult] = useState<VoiceExtractionResult | null>(null);
const isDetailLocked = useCallback(
(detail: TreatmentDetailDraft) =>
labCaseDrafts.some((lc) => lc.sentAt && lc.detailClientId === detail.clientId),
@@ -489,6 +502,21 @@ export function TreatmentWorkspace({
[appointments, selectedAppointmentId],
);
const voice = useVoiceCapture({
// The locale the clinician is actually reading and speaking in. Sent explicitly so
// the server's ASR hint, catalog labels and week start match what the microphone was
// offered for — req.user.language can drift from the URL locale.
locale,
maxMs: voiceAvailability?.maxRecordingMs ?? null,
onExtracted: setVoiceResult,
onError: (error) => showError(getUserFacingError(error, tErrors, t('voiceFailed'))),
});
/** Absence is the unavailable state — the Add button then renders unsplit. */
const voiceForEditor =
voiceAvailability?.enabled && voiceAvailability.locales.includes(locale) ? voice : undefined;
const selectedStandalone = useMemo(
() => standaloneTreatments.find((t) => t.id === selectedStandaloneId) ?? null,
[standaloneTreatments, selectedStandaloneId],
@@ -927,12 +955,18 @@ export function TreatmentWorkspace({
let cancelled = false;
void (async () => {
try {
const [orgsResponse, catalogResponse, prosthesisResponse] = await Promise.all([
treatmentsApi.listLinkedOrganizations(),
treatmentCatalogApi.list(),
prosthesisCatalogApi.list(),
]);
const [orgsResponse, catalogResponse, prosthesisResponse, voiceResponse] =
await Promise.all([
treatmentsApi.listLinkedOrganizations(),
treatmentCatalogApi.list(),
prosthesisCatalogApi.list(),
// Voice availability comes from the API, not a NEXT_PUBLIC_* var: those are
// baked in at build time, so enabling a locale would need a frontend rebuild.
// A failure here must not take the whole treatment tab down with it.
voiceApi.availability().catch(() => null),
]);
if (cancelled) return;
setVoiceAvailability(voiceResponse?.data ?? null);
setOrgs(orgsResponse.data);
setTreatmentCatalog(catalogResponse.data);
setProsthesisCatalog(prosthesisResponse.data);
@@ -1935,6 +1969,114 @@ export function TreatmentWorkspace({
],
);
/**
* Voice entry.
*
* Confirm always appends a NEW detail — it never edits an existing one, and never
* touches onAddDetail. Nothing is created until this runs, so cancelling or a failed
* recording leaves the chip strip untouched.
*/
const applyVoiceResult = useCallback(
(result: VoiceExtractionResult, selection: VoiceApplySelection) => {
const detail = newDetail(
defaultTreatmentTypeForAppointment(selectedAppointment?.purpose, treatmentCatalog),
);
// Ticked rows land on top of the seeded defaults, so unticking the type row leaves
// the appointment-purpose default rather than a blank.
if (selection.treatmentType && result.treatmentType) {
detail.treatmentType = result.treatmentType;
}
if (selection.teeth) {
detail.teeth = [...result.teeth];
detail.toothSelectionGroups = result.toothSelectionGroups.map((group) => ({
...group,
teeth: [...group.teeth],
}));
}
if (selection.comment && result.comment) {
detail.comment = result.comment;
}
const nextDetails = [...detailsRef.current, detail];
setDetails(nextDetails);
// persistDraft reads detailsRef, and setDetails has not rendered yet.
detailsRef.current = nextDetails;
setActiveDetailId(detail.clientId);
setEntryStep('treatment');
// Lab-side rows ride on a lab case draft keyed by the detail's *client* id, so a
// brand-new unsaved detail can still carry one; it is persisted after the detail is.
const wantsLabDraft =
(selection.prosthesis && result.prosthesis) ||
(selection.lab && result.labId) ||
(selection.dueDate && result.dueDate);
if (wantsLabDraft) {
const draft = newLabCaseDraft();
draft.detailClientId = detail.clientId;
if (selection.lab && result.labId) {
draft.destinationOrganizationId = result.labId;
}
if (selection.dueDate && result.dueDate) {
draft.dueDate = result.dueDate;
}
if (selection.prosthesis && result.prosthesis) {
// byTooth keys are plain strings; the group's teeth are FdiToothId.
const groupOf = (tooth: string) =>
result.toothSelectionGroups.find((group) =>
(group.teeth as readonly string[]).includes(tooth),
)?.groupId ?? '';
// Only teeth that actually landed on the detail. Unticking "teeth" while
// leaving "prosthesis" ticked would otherwise attach prosthesis rows for teeth
// the treatment does not contain — nothing downstream filters them, and they
// would reach task generation as work for teeth nobody is treating.
const detailTeeth = new Set<string>(detail.teeth);
draft.toothProsthesis = Object.entries(result.prosthesis.byTooth)
.filter(([tooth]) => detailTeeth.has(tooth))
.map(([tooth, prosthesisTypeCode]) => ({
detailClientId: detail.clientId,
tooth,
prosthesisTypeCode,
selectionGroupId: groupOf(tooth),
}));
}
const updatedLabCases = [...labCaseDrafts, draft];
setLabCaseDrafts(updatedLabCases);
// Autosave only watches `details`, so a lab draft left in state alone loses the
// lab, the due date and the prosthesis map on reload — silently, because the
// detail itself survives.
void (async () => {
try {
const saved = await persistDraft({ force: true });
// persistDraft returns a *preview* when the details are not persistable — one
// blank detail is enough — and a preview's detail id falls back to the client
// id. Check what came back, not the precondition, so this holds for every early
// return persistDraft has.
const savedDetail = saved.details.find((d) => d.clientId === detail.clientId);
if (!savedDetail?.id || savedDetail.id === detail.clientId) return;
await persistLabCases(saved, updatedLabCases);
} catch (error: unknown) {
showError(getUserFacingError(error, tErrors, t('errorSaveLabShipments')));
}
})();
}
setVoiceResult(null);
},
[
labCaseDrafts,
persistDraft,
persistLabCases,
selectedAppointment?.purpose,
showError,
t,
tErrors,
treatmentCatalog,
],
);
const handleRemoveDetail = useCallback(
(detailClientId: string) => {
if (!canEditTreatmentForDay) return;
@@ -2478,6 +2620,7 @@ export function TreatmentWorkspace({
}}
showChrome
showFields={entryStep === 'treatment'}
voice={voiceForEditor}
chartLocked={
entryStep === 'treatment' && !activeTypeSelected && !showWholeTreatmentPlan
}
@@ -2725,6 +2868,16 @@ export function TreatmentWorkspace({
)}
</div>
</div>
{voiceResult ? (
<VoiceReviewSheet
result={voiceResult}
treatmentCatalog={treatmentCatalog}
prosthesisCatalog={prosthesisCatalog}
labs={orgs}
onApply={(selection, applied) => applyVoiceResult(applied, selection)}
onDiscard={() => setVoiceResult(null)}
/>
) : null}
</div>
);
}

90
frontend/src/components/ui/treatment/VoiceRecordingBar.tsx Normal file
View File

@@ -0,0 +1,90 @@
'use client';
import { useTranslations } from 'next-intl';
import { Loader2, X } from 'lucide-react';
import type { VoiceCaptureState } from '@/lib/voice/useVoiceCapture';
const METER_BARS = 9;
function formatElapsed(ms: number): string {
const totalSeconds = Math.floor(Math.max(0, ms) / 1000);
const minutes = Math.floor(totalSeconds / 60);
const seconds = totalSeconds % 60;
return `${minutes}:${String(seconds).padStart(2, '0')}`;
}
/**
* Live recording / processing strip.
*
* Sits between the header row and the chip strip rather than inside the segmented control:
* the header is `sm:justify-between`, so growing the button mid-recording would shift the
* whole row.
*/
export function VoiceRecordingBar({ voice }: { voice: VoiceCaptureState }) {
const t = useTranslations('treatment');
if (voice.phase === 'idle') return null;
const isRecording = voice.phase === 'recording';
return (
<div
className="flex items-center gap-3 rounded-[var(--radius-md)] border border-border/70 bg-background-secondary/70 px-3 py-2"
role="status"
aria-live="polite"
>
{isRecording ? (
<>
<span className="h-2 w-2 shrink-0 animate-pulse rounded-full bg-red-500" aria-hidden />
<span className="shrink-0 text-sm tabular-nums text-text-primary">
{formatElapsed(voice.elapsedMs)}
{voice.maxMs != null ? (
<span className="text-text-muted"> / {formatElapsed(voice.maxMs)}</span>
) : null}
</span>
<LevelMeter level={voice.level} />
</>
) : (
<>
<Loader2 className="h-4 w-4 shrink-0 animate-spin text-text-secondary" aria-hidden />
<span className="text-sm text-text-secondary">{t('voiceProcessing')}</span>
</>
)}
<button
type="button"
onClick={voice.onCancel}
title={t('voiceCancel')}
aria-label={t('voiceCancel')}
className="ms-auto inline-flex shrink-0 items-center gap-1 rounded-[var(--radius-md)] px-2 py-1 text-xs text-text-secondary transition-colors hover:bg-red-500/15 hover:text-red-600 focus:outline-none focus-visible:ring-2 focus-visible:ring-red-500/40"
>
<X className="h-3.5 w-3.5" aria-hidden />
{t('voiceCancel')}
</button>
</div>
);
}
/** Proves the microphone is actually hearing something — silence looks identical otherwise. */
function LevelMeter({ level }: { level: number }) {
return (
<span className="flex h-4 flex-1 items-end gap-0.5" aria-hidden>
{Array.from({ length: METER_BARS }, (_, index) => {
// Bars light up left to right as the level rises, with a floor so the meter never
// looks dead while a quiet voice is still being captured.
const threshold = (index + 1) / METER_BARS;
const active = level >= threshold * 0.9;
const height = active ? 30 + threshold * 70 : 20;
return (
<span
key={index}
className={`w-1 rounded-sm transition-all duration-75 ${
active ? 'bg-primary' : 'bg-border'
}`}
style={{ height: `${height}%` }}
/>
);
})}
</span>
);
}

306
frontend/src/components/ui/treatment/VoiceReviewSheet.tsx Normal file
View File

@@ -0,0 +1,306 @@
'use client';
import { useMemo, useState } from 'react';
import { useTranslations } from 'next-intl';
import { AlertTriangle } from 'lucide-react';
import { Button } from '@/components/ui/shared/Button';
import { Checkbox } from '@/components/ui/shared/Checkbox';
import {
ResponsiveDialogOverlay,
ResponsiveDialogPanel,
} from '@/components/ui/shared/ResponsiveDialog';
import { FdiToothChart } from '@/components/ui/treatment/FdiToothChart';
import {
connectedTeethFromResult,
countSelected,
hasAnythingToApply,
initialVoiceSelection,
voiceRowAvailability,
withChosenTeeth,
} from '@/components/treatment/voiceReviewRows';
import { useLocale } from 'next-intl';
import { useAppFormatters } from '@/lib/hooks/useAppFormatters';
import type { TreatmentCatalogEntry } from '@/types/treatment-catalog';
import type { ProsthesisCatalogEntry } from '@/types/treatment-catalog';
import type { FdiToothId, LinkedOrganizationOption } from '@/types/treatment';
import type { VoiceApplySelection, VoiceExtractionResult } from '@/types/voice';
interface VoiceReviewSheetProps {
result: VoiceExtractionResult;
treatmentCatalog: TreatmentCatalogEntry[];
prosthesisCatalog: ProsthesisCatalogEntry[];
labs: LinkedOrganizationOption[];
/** The result is handed back because the sheet may have added teeth the model missed. */
onApply: (selection: VoiceApplySelection, result: VoiceExtractionResult) => void;
onDiscard: () => void;
}
/**
* Confirmation step between the model's output and the form.
*
* Modal on desktop, bottom sheet on mobile — an overlay and not a route, because navigating
* would unmount TreatmentWorkspace and destroy the in-progress draft.
*/
export function VoiceReviewSheet({
result,
treatmentCatalog,
prosthesisCatalog,
labs,
onApply,
onDiscard,
}: VoiceReviewSheetProps) {
const t = useTranslations('treatment');
const locale = useLocale();
const { formatDate } = useAppFormatters();
const [selection, setSelection] = useState<VoiceApplySelection>(() =>
initialVoiceSelection(result),
);
const [chosen, setChosen] = useState<FdiToothId[]>([]);
// Everything below renders from `effective`, never from `result` — a tooth picked from
// the candidate chips has to reach the rows, the chart and the apply count alike.
const effective = useMemo(() => withChosenTeeth(result, chosen), [result, chosen]);
const available = useMemo(() => voiceRowAvailability(effective), [effective]);
const connectedTeeth = useMemo(() => connectedTeethFromResult(effective), [effective]);
const selectedTeeth = useMemo(() => new Set(effective.teeth), [effective.teeth]);
const nothingToApply = !hasAnythingToApply(effective);
const selectedCount = countSelected(selection, available);
const pickCandidate = (tooth: FdiToothId) => {
const nextChosen = chosen.includes(tooth)
? chosen.filter((t) => t !== tooth)
: [...chosen, tooth];
setChosen(nextChosen);
setSelection((prev) => ({
...prev,
// The teeth row starts unticked whenever the recording produced no teeth of its own,
// and a picked tooth that is not ticked applies nothing.
teeth: true,
// A picked tooth has no prosthesis type, so the map is no longer shippable — leaving the
// row ticked would apply a map dispatch rejects. Only ever unticks; re-ticking is the
// clinician's call.
prosthesis:
prev.prosthesis &&
withChosenTeeth(result, nextChosen).prosthesis?.complete !== false,
}));
};
const labelFor = (code: string | null, catalog: { code: string; label: string }[]) =>
catalog.find((entry) => entry.code === code)?.label ?? code ?? '';
const toggle = (key: keyof VoiceApplySelection) => (checked: boolean) =>
setSelection((prev) => ({ ...prev, [key]: checked }));
return (
<ResponsiveDialogOverlay onBackdropClick={onDiscard}>
<ResponsiveDialogPanel
role="dialog"
aria-modal="true"
aria-labelledby="voice-review-title"
maxWidthClass="sm:max-w-xl"
>
<h2 id="voice-review-title" className="text-base font-semibold text-text-primary">
{t('voiceReviewTitle')}
</h2>
<p className="mt-2 rounded-[var(--radius-md)] bg-background-card/60 px-3 py-2 text-sm text-text-secondary">
{effective.transcript}
</p>
{nothingToApply ? (
<p className="mt-4 text-sm text-text-secondary">{t('voiceNothingExtracted')}</p>
) : (
<div className="mt-4 space-y-3">
{available.treatmentType ? (
<Row
label={t('treatmentType')}
checked={selection.treatmentType}
onChange={toggle('treatmentType')}
>
<span className="text-sm text-text-primary">
{labelFor(effective.treatmentType, treatmentCatalog)}
</span>
</Row>
) : null}
{available.teeth ? (
<Row label={t('entryStepTeeth')} checked={selection.teeth} onChange={toggle('teeth')}>
<div className="mt-1">
<FdiToothChart
readOnly
compact
scale={0.55}
selected={selectedTeeth}
connectedTeeth={connectedTeeth}
/>
</div>
</Row>
) : null}
{available.comment ? (
<Row
label={t('comments')}
checked={selection.comment}
onChange={toggle('comment')}
>
<span className="text-sm whitespace-pre-wrap text-text-primary">
{effective.comment}
</span>
</Row>
) : null}
{available.prosthesis && effective.prosthesis ? (
<Row
label={t('prosthesisColType')}
checked={selection.prosthesis}
onChange={toggle('prosthesis')}
warning={
effective.prosthesis.complete
? undefined
: t('voiceProsthesisIncomplete', {
teeth: formatToothList(effective.prosthesis.missingTeeth, locale),
})
}
>
<span className="text-sm text-text-primary">
{Object.entries(effective.prosthesis.byTooth)
.map(
([tooth, code]) => `${tooth}: ${labelFor(code, prosthesisCatalog)}`,
)
.join(' · ')}
</span>
</Row>
) : null}
{available.lab ? (
<Row
label={t('entryStepLab')}
checked={selection.lab}
onChange={toggle('lab')}
warning={effective.labMatchExact ? undefined : t('voiceLabInexact')}
>
<span className="text-sm text-text-primary">
{labs.find((lab) => lab.id === effective.labId)?.name ?? effective.labId}
</span>
</Row>
) : null}
{available.dueDate && effective.dueDate ? (
<Row
label={t('dueDateLabel')}
checked={selection.dueDate}
onChange={toggle('dueDate')}
>
<span className="text-sm text-text-primary">
{formatDate(civilDateToLocalDate(effective.dueDate))}
</span>
</Row>
) : null}
</div>
)}
{effective.unresolved.length > 0 ? (
<div className="mt-4 rounded-[var(--radius-md)] border border-amber-500/40 bg-amber-500/10 px-3 py-2">
<p className="text-xs font-medium text-amber-700 dark:text-amber-400">
{t('voiceNotUnderstood')}
</p>
<ul className="mt-1 space-y-0.5">
{effective.unresolved.map((item, index) => (
<li key={`${item.spoken}-${index}`} className="text-xs text-text-secondary">
{item.spoken ? `“${item.spoken}” — ` : ''}
{t(`voiceUnresolved.${item.reason}`)}
{item.candidates && item.candidates.length > 0 ? (
<span className="mt-1 flex flex-wrap items-center gap-1">
<span className="text-text-muted">{t('voicePickTooth')}</span>
{item.candidates.map((tooth) => {
const picked = chosen.includes(tooth as FdiToothId);
return (
<button
key={tooth}
type="button"
aria-pressed={picked}
aria-label={t('toothAria', { fdi: tooth })}
onClick={() => pickCandidate(tooth as FdiToothId)}
className={`rounded-full border px-2 py-0.5 text-xs transition-colors ${
picked
? 'border-transparent bg-primary text-white'
: 'border-border text-text-primary hover:border-border-strong'
}`}
>
{tooth}
</button>
);
})}
</span>
) : null}
</li>
))}
</ul>
</div>
) : null}
<div className="mt-5 flex flex-col gap-2 sm:flex-row sm:justify-end">
<Button type="button" variant="secondary" onClick={onDiscard} fullWidth className="sm:w-auto">
{t('voiceDiscard')}
</Button>
<Button
type="button"
variant="primary"
disabled={selectedCount === 0}
onClick={() => onApply(selection, effective)}
fullWidth
className="sm:w-auto"
>
{t('voiceApply', { count: selectedCount })}
</Button>
</div>
</ResponsiveDialogPanel>
</ResponsiveDialogOverlay>
);
}
function Row({
label,
checked,
onChange,
warning,
children,
}: {
label: string;
checked: boolean;
onChange: (checked: boolean) => void;
warning?: string;
children: React.ReactNode;
}) {
return (
<div className="rounded-[var(--radius-md)] border border-border/70 px-3 py-2">
<Checkbox checked={checked} onChange={onChange} label={label} />
<div className="mt-1 ps-7 min-w-0">{children}</div>
{warning ? (
<p className="mt-1 ps-7 flex items-start gap-1 text-xs text-amber-700 dark:text-amber-400">
<AlertTriangle className="mt-0.5 h-3 w-3 shrink-0" aria-hidden />
{warning}
</p>
) : null}
</div>
);
}
/**
* `new Date('2025-10-17')` parses a civil date as UTC midnight, which renders as the 16th
* west of Greenwich. Build it from its parts so it means the same day everywhere.
*/
function civilDateToLocalDate(iso: string): Date {
const [year, month, day] = iso.split('-').map(Number);
return new Date(year, (month ?? 1) - 1, day ?? 1);
}
/** Locale-aware list separator — the Arabic comma is not correct in en or nl. */
function formatToothList(teeth: readonly string[], locale: string): string {
try {
return new Intl.ListFormat(locale, { style: 'short', type: 'unit' }).format([...teeth]);
} catch {
return teeth.join(', ');
}
}

29
frontend/src/lib/api/voice.ts Normal file
View File

@@ -0,0 +1,29 @@
import { apiClient } from './client';
import type { VoiceAvailability, VoiceExtractionResult } from '@/types/voice';
export interface ExtractVoicePayload {
/** Base64 audio, no data: prefix. */
audio: string;
format: string;
/** IANA zone — the server derives "today" from it for relative deadlines. */
timeZone: string;
durationMs: number;
/** Locale the clinician is speaking; the server uses it rather than the stored one. */
locale: string;
}
export const voiceApi = {
availability: async (): Promise<{ success: boolean; data: VoiceAvailability }> => {
const response = await apiClient.get('/voice/availability');
return response.data;
},
extract: async (
payload: ExtractVoicePayload,
signal?: AbortSignal,
): Promise<{ success: boolean; data: VoiceExtractionResult }> => {
// Forwarded so cancelling closes the connection; the controller turns that into an abort.
const response = await apiClient.post('/voice/extract', payload, { signal });
return response.data;
},
};

60
frontend/src/lib/voice/audioFormat.ts Normal file
View File

@@ -0,0 +1,60 @@
/** Containers the backend accepts, in the order we prefer to record them. */
const PREFERRED_MIME_TYPES = [
'audio/webm;codecs=opus',
'audio/webm',
'audio/mp4',
'audio/aac',
'audio/ogg;codecs=opus',
'audio/ogg',
] as const;
/**
* Pick a container this browser can record AND the backend accepts. Chrome and Android give
* webm/opus, Safari and iPad mp4/aac; both go to the vendor unmodified, so there is no
* transcode step and the list is an intersection, not a preference.
*/
export function pickRecordingMimeType(): string | null {
if (typeof MediaRecorder === 'undefined') return null;
if (typeof MediaRecorder.isTypeSupported !== 'function') {
// Safari <14.1 shipped MediaRecorder without the feature check; let it choose.
return '';
}
for (const type of PREFERRED_MIME_TYPES) {
if (MediaRecorder.isTypeSupported(type)) return type;
}
return null;
}
/** `audio/webm;codecs=opus` → `webm`, which is what the API's `format` field wants. */
export function mimeTypeToFormat(mimeType: string): string {
const base = mimeType.split(';')[0]?.trim().toLowerCase() ?? '';
const subtype = base.startsWith('audio/') ? base.slice('audio/'.length) : base;
// Safari/iOS records `audio/mp4`, but the transcription endpoint's documented container
// list names m4a, not mp4. Same container; send the name the vendor documents, so iPad
// recordings do not fail while Chrome's webm works.
if (subtype === 'x-m4a' || subtype === 'm4a' || subtype === 'mp4') return 'm4a';
if (subtype === 'mpeg') return 'mp3';
return subtype || 'webm';
}
/** Blob → base64 without the `data:` prefix, which the API does not want. */
export async function blobToBase64(blob: Blob): Promise<string> {
const buffer = await blob.arrayBuffer();
let binary = '';
const bytes = new Uint8Array(buffer);
// Chunked to avoid blowing the argument limit on a two-minute recording.
const chunkSize = 0x8000;
for (let i = 0; i < bytes.length; i += chunkSize) {
binary += String.fromCharCode(...bytes.subarray(i, i + chunkSize));
}
return btoa(binary);
}
export function isMediaRecorderSupported(): boolean {
return (
typeof window !== 'undefined' &&
typeof MediaRecorder !== 'undefined' &&
typeof navigator !== 'undefined' &&
Boolean(navigator.mediaDevices?.getUserMedia)
);
}

300
frontend/src/lib/voice/useVoiceCapture.ts Normal file
View File

@@ -0,0 +1,300 @@
'use client';
import { useCallback, useEffect, useRef, useState } from 'react';
import { voiceApi } from '@/lib/api/voice';
import type { ApiError } from '@/types/api';
import type { VoiceExtractionResult, VoicePhase } from '@/types/voice';
import {
blobToBase64,
isMediaRecorderSupported,
mimeTypeToFormat,
pickRecordingMimeType,
} from './audioFormat';
export interface UseVoiceCaptureOptions {
/** The locale the clinician is speaking, sent so the server does not have to guess. */
locale: string;
/** null means uncapped; otherwise the recorder auto-stops here. */
maxMs: number | null;
onExtracted: (result: VoiceExtractionResult) => void;
onError: (error: unknown) => void;
}
export interface VoiceCaptureState {
phase: VoicePhase;
elapsedMs: number;
/** 0..1, for the level meter — proves the microphone is actually hearing something. */
level: number;
maxMs: number | null;
onStart: () => void;
onStop: () => void;
onCancel: () => void;
}
const LEVEL_POLL_MS = 100;
/**
* Client-side failures must be ApiError-shaped or getUserFacingError cannot resolve them
* and every one renders the generic fallback, leaving errors.VOICE_MIC_DENIED dead.
*/
function clientError(code: string): ApiError {
return { statusCode: 0, code };
}
/**
* Microphone capture for treatment voice entry.
*
* Lives in lib/ rather than in the editor: TreatmentDetailsEditor stays presentational
* and receives only a `voice` prop, so MediaRecorder and the API call never enter ui/.
*/
export function useVoiceCapture({
locale,
maxMs,
onExtracted,
onError,
}: UseVoiceCaptureOptions): VoiceCaptureState {
const [phase, setPhase] = useState<VoicePhase>('idle');
const [elapsedMs, setElapsedMs] = useState(0);
const [level, setLevel] = useState(0);
const recorderRef = useRef<MediaRecorder | null>(null);
const streamRef = useRef<MediaStream | null>(null);
const chunksRef = useRef<Blob[]>([]);
const startedAtRef = useRef(0);
const timerRef = useRef<ReturnType<typeof setInterval> | null>(null);
const audioContextRef = useRef<AudioContext | null>(null);
const abortRef = useRef<AbortController | null>(null);
/** Set when the user cancels, so the recorder's stop handler discards instead of sending. */
const cancelledRef = useRef(false);
/** getUserMedia is async; without this a permission granted after unmount leaks the mic. */
const mountedRef = useRef(true);
/**
* Set synchronously on click: `phase` only becomes 'recording' once getUserMedia resolves,
* so a second click during the permission prompt would orphan the first stream.
*/
const startingRef = useRef(false);
const teardown = useCallback(() => {
if (timerRef.current) {
clearInterval(timerRef.current);
timerRef.current = null;
}
streamRef.current?.getTracks().forEach((track) => track.stop());
streamRef.current = null;
void audioContextRef.current?.close().catch(() => undefined);
audioContextRef.current = null;
recorderRef.current = null;
setLevel(0);
}, []);
// Releasing the microphone on unmount matters: the browser shows a recording indicator
// for as long as the track is live, and an orphaned one looks like the app is listening.
useEffect(() => {
// Re-armed on every mount: React StrictMode runs mount → unmount → mount in dev, and
// a ref that is only ever set false would leave the hook permanently "unmounted".
mountedRef.current = true;
return () => {
mountedRef.current = false;
cancelledRef.current = true;
abortRef.current?.abort();
try {
recorderRef.current?.stop();
} catch {
// already stopped
}
teardown();
};
}, [teardown]);
const send = useCallback(
async (blob: Blob, mimeType: string, durationMs: number) => {
setPhase('processing');
const controller = new AbortController();
abortRef.current = controller;
try {
const audio = await blobToBase64(blob);
const response = await voiceApi.extract(
{
audio,
format: mimeTypeToFormat(mimeType),
timeZone: Intl.DateTimeFormat().resolvedOptions().timeZone,
durationMs,
locale,
},
controller.signal,
);
if (cancelledRef.current) return;
onExtracted(response.data);
} catch (error) {
if (cancelledRef.current || controller.signal.aborted) return;
onError(error);
} finally {
abortRef.current = null;
setPhase('idle');
setElapsedMs(0);
}
},
[locale, onExtracted, onError],
);
const stop = useCallback(() => {
// No recorder means nothing will fire `onstop`, so nothing else moves the phase.
if (!recorderRef.current) {
teardown();
setPhase('idle');
return;
}
try {
recorderRef.current.stop();
} catch {
teardown();
setPhase('idle');
}
}, [teardown]);
const onStart = useCallback(() => {
if (phase !== 'idle' || startingRef.current) return;
if (!isMediaRecorderSupported()) {
// VOICE_UNSUPPORTED_FORMAT, not MIC_DENIED: nothing asked for a permission yet, and
// blaming the microphone sends the clinician into site settings for no reason. Same
// for the two paths below.
onError(clientError('VOICE_UNSUPPORTED_FORMAT'));
return;
}
cancelledRef.current = false;
startingRef.current = true;
void (async () => {
try {
let stream: MediaStream;
try {
stream = await navigator.mediaDevices.getUserMedia({ audio: true });
} catch {
// Permission refused, or no input device. Never a server round-trip.
onError(clientError('VOICE_MIC_DENIED'));
return;
}
if (!mountedRef.current) {
// Permission resolved after the component went away — release it immediately
// rather than leaving the browser's recording indicator lit.
stream.getTracks().forEach((track) => track.stop());
return;
}
const mimeType = pickRecordingMimeType();
if (mimeType === null) {
stream.getTracks().forEach((track) => track.stop());
onError(clientError('VOICE_UNSUPPORTED_FORMAT'));
return;
}
streamRef.current = stream;
chunksRef.current = [];
const recorder = new MediaRecorder(stream, mimeType ? { mimeType } : undefined);
recorderRef.current = recorder;
recorder.ondataavailable = (event) => {
if (event.data.size > 0) chunksRef.current.push(event.data);
};
recorder.onstop = () => {
const durationMs = Date.now() - startedAtRef.current;
const blob = new Blob(chunksRef.current, { type: recorder.mimeType || mimeType });
teardown();
if (cancelledRef.current || blob.size === 0) {
setPhase('idle');
setElapsedMs(0);
return;
}
// Prefer what the recorder actually produced, then the blob's own type. Old
// Safari accepts no mimeType hint, and defaulting to webm would mislabel its
// mp4/aac clips as something they are not.
void send(blob, recorder.mimeType || blob.type || mimeType || 'audio/webm', durationMs);
};
attachLevelMeter(stream, audioContextRef, setLevel);
startedAtRef.current = Date.now();
recorder.start();
setPhase('recording');
setElapsedMs(0);
timerRef.current = setInterval(() => {
const elapsed = Date.now() - startedAtRef.current;
setElapsedMs(elapsed);
// Auto-stop proceeds to processing with what was captured; discarding two
// minutes of dictation because a timer expired would be the worst failure.
if (maxMs != null && elapsed >= maxMs) stop();
}, LEVEL_POLL_MS);
} catch {
// `new MediaRecorder()` and `recorder.start()` both throw on some browsers, and by
// then the stream is live — without this the mic indicator stays lit until unmount.
teardown();
setPhase('idle');
onError(clientError('VOICE_UNSUPPORTED_FORMAT'));
} finally {
startingRef.current = false;
}
})();
}, [maxMs, onError, phase, send, stop, teardown]);
const onCancel = useCallback(() => {
cancelledRef.current = true;
// Aborting closes the connection, which aborts the vendor call server-side.
abortRef.current?.abort();
try {
recorderRef.current?.stop();
} catch {
// already stopped
}
teardown();
setPhase('idle');
setElapsedMs(0);
}, [teardown]);
return { phase, elapsedMs, level, maxMs, onStart, onStop: stop, onCancel };
}
/** Drives the level meter from the live stream; failure here must not stop recording. */
function attachLevelMeter(
stream: MediaStream,
contextRef: React.MutableRefObject<AudioContext | null>,
setLevel: (value: number) => void,
) {
try {
const AudioContextCtor =
window.AudioContext ?? (window as { webkitAudioContext?: typeof AudioContext }).webkitAudioContext;
if (!AudioContextCtor) return;
const context = new AudioContextCtor();
contextRef.current = context;
const source = context.createMediaStreamSource(stream);
const analyser = context.createAnalyser();
analyser.fftSize = 512;
source.connect(analyser);
const data = new Uint8Array(analyser.frequencyBinCount);
// Sample every frame so a transient is not missed, publish at LEVEL_POLL_MS. The hook
// lives in TreatmentWorkspace, so an unthrottled setLevel is ~7,200 whole-tree renders
// across a two-minute recording.
let peakSinceEmit = 0;
let lastEmit = 0;
const tick = (now: number) => {
if (contextRef.current !== context || context.state === 'closed') return;
analyser.getByteTimeDomainData(data);
let peak = 0;
for (const sample of data) peak = Math.max(peak, Math.abs(sample - 128));
peakSinceEmit = Math.max(peakSinceEmit, peak);
if (now - lastEmit >= LEVEL_POLL_MS) {
lastEmit = now;
setLevel(Math.min(1, peakSinceEmit / 128));
peakSinceEmit = 0;
}
requestAnimationFrame(tick);
};
requestAnimationFrame(tick);
} catch {
// A missing or blocked AudioContext costs the meter, not the recording.
}
}

64
frontend/src/types/voice.ts Normal file
View File

@@ -0,0 +1,64 @@
import type { FdiToothId, ToothSelectionGroup } from '@/types/treatment';
/** Mirrors the backend's ResolvedExtraction — values already resolved, plus what was not. */
export type VoiceUnresolvedReason =
| 'not_permanent_tooth'
| 'position_out_of_range'
| 'tooth_missing_quadrant'
| 'malformed'
| 'span_not_same_arch'
| 'unknown_catalog_code'
| 'tooth_not_selected'
| 'invalid_date';
export interface VoiceUnresolvedItem {
/** The transcript span that could not be resolved, so the clinician sees what was heard. */
spoken: string;
reason: VoiceUnresolvedReason;
/**
* FDI codes still consistent with what was heard, when a choice would settle it — the
* review sheet offers them as chips. Only `tooth_missing_quadrant` carries these.
*/
candidates?: string[];
}
export interface VoiceProsthesisResult {
byTooth: Record<string, string>;
/** False means the case cannot ship — every tooth needs a prosthesis type. */
complete: boolean;
missingTeeth: FdiToothId[];
}
export interface VoiceExtractionResult {
transcript: string;
treatmentType: string | null;
teeth: FdiToothId[];
toothSelectionGroups: ToothSelectionGroup[];
comment: string | null;
prosthesis: VoiceProsthesisResult | null;
labId: string | null;
/** When false, the lab row must not tick itself — the name only approximately matched. */
labMatchExact: boolean;
dueDate: string | null;
unresolved: VoiceUnresolvedItem[];
}
export interface VoiceAvailability {
enabled: boolean;
locales: string[];
/** null means uncapped. */
maxRecordingMs: number | null;
}
/** Which review rows the clinician ticked. */
export interface VoiceApplySelection {
treatmentType: boolean;
teeth: boolean;
comment: boolean;
prosthesis: boolean;
lab: boolean;
dueDate: boolean;
}
export type VoicePhase = 'idle' | 'recording' | 'processing';