admin/dyolink
0
0
Fork 0
Code Issues 24 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

feat(frontend): split Add detail into a segmented control with voice

Browse Source 
The microphone becomes the second segment of the Add detail button, built like
the detail chip's trash affordance in the same file — an overflow-hidden rounded
wrapper holding two raw <button>s divided by border-s — rather than two shared
Buttons, which each hardcode their own rounding and would fight a segmented
control. border-s puts the mic at the logical end: visually right in en/nl,
visually left in fa, on the same side as the chip's trash in both directions.

The two halves share a wrapper and nothing else. Add keeps its exact behaviour.

The control never changes size while recording; the timer and level meter live
in a bar between the header row and the chip strip, because the header is
sm:justify-between and growing the button would shove the row on every start and
stop. The meter exists to prove the microphone is actually hearing something —
silence and a dead mic look identical otherwise.

Voice reaches the editor as one optional `voice` prop, so its absence *is* the
unavailable state and the two cannot disagree.

Fixes from review of this commit:

- mountedRef was set false on unmount and never re-armed, so under StrictMode
  the hook was permanently "unmounted" in dev and recording silently never
  started.
- onStart guarded only on `phase`, which does not change until getUserMedia
  resolves; a second click during the permission prompt orphaned the first
  MediaStream, leaving the mic indicator lit.
- Week start is now per locale. "Next Thursday" is week-relative, and hardcoding
  Saturday put an en/nl clinician's deadline a week out.
- A missing `which` on a weekday intent is read as "this" rather than failing —
  a bare weekday carries no qualifier, and rejecting it discarded a real
  deadline.
- durationMs is client-reported and so is a claim, not enforcement; the cap is
  now also checked against the vendor's own usage.seconds.
- Blob type falls back to the recorder's actual mimeType before webm, so old
  Safari's mp4/aac clips are not mislabelled.

Two review findings were rejected as incorrect, both re-verified against live
sources: google/gemini-3.7-flash does exist on OpenRouter (1M context,
$0.375/$1.875 per M), and base64 JSON input_audio is the documented primary
path for /audio/transcriptions, with multipart as the OpenAI-compatible
alternative. The spec's stale "unverified" note is corrected, and the provider
now has unit tests covering the request shape, usage parsing, and that a vendor
error body never reaches the thrown message.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Amin Mousavi
2026-08-20 19:47:10 +03:30
parent 93c6513df6
commit c07f550e00
13 changed files with 672 additions and 146 deletions
Download Patch File Download Diff File Expand all files Collapse all files

170
backend/src/modules/voice/due-date.resolver.spec.ts
View File

@@ -1,86 +1,121 @@
import { resolveDueDate } from './due-date.resolver';
import { resolveDueDate, weekStartForLocale } from './due-date.resolver';
import type { DueIntent } from './voice.types';
// 2025-10-11 is a Saturday — the first day of the Iranian week.
const FA_WEEK = 6; // Saturday
const EU_WEEK = 1; // Monday
const SATURDAY = '2025-10-11';
const THURSDAY = '2025-10-16';
describe('resolveDueDate', () => {
describe('weekday intents', () => {
it('resolves "this <weekday>" to the coming occurrence in the same week', () => {
it('resolves "this <weekday>" to the coming occurrence', () => {
const result = resolveDueDate(
{ kind: 'weekday', weekday: 'thursday', which: 'this' },
SATURDAY,
FA_WEEK,
);
expect(result.dueDate).toBe(THURSDAY); // Sat -> Thu is 5 days in a Saturday-start week
expect(result.dueDate).toBe(THURSDAY); // Sat -> Thu is 5 days
});
it('resolves "next <weekday>" to the following week', () => {
it('resolves "next <weekday>" to that weekday in the following week', () => {
const result = resolveDueDate(
{ kind: 'weekday', weekday: 'thursday', which: 'next' },
SATURDAY,
FA_WEEK,
);
expect(result.dueDate).toBe('2025-10-23');
});
it('anchors "next" to the week, not to "this" plus seven', () => {
// Said on Thursday 2025-10-16: next week runs Sat 10-18 .. Fri 10-24, so its
// Said on Thursday 2025-10-16, the Iranian week runs Sat 10-18 .. Fri 10-24, so its
// Thursday is 10-23. Adding a week to "this Thursday" (already 10-23) would
// overshoot to 10-30 — a lab case a week late.
expect(
resolveDueDate(
{ kind: 'weekday', weekday: 'thursday', which: 'next' },
THURSDAY,
FA_WEEK,
).dueDate,
).toBe('2025-10-23');
});
it('lets "this" and "next" coincide when they name the same day', () => {
// On a Thursday, "the coming Saturday" and "Saturday next week" are both 10-18.
for (const which of ['this', 'next'] as const) {
expect(
resolveDueDate(
{ kind: 'weekday', weekday: 'saturday', which },
THURSDAY,
FA_WEEK,
).dueDate,
).toBe('2025-10-18');
}
});
it('reads "by Thursday" said on a Thursday as the next one, not today', () => {
// A deadline of today is almost never what was meant.
expect(
resolveDueDate(
{ kind: 'weekday', weekday: 'saturday', which: 'this' },
{ kind: 'weekday', weekday: 'thursday', which: 'this' },
THURSDAY,
FA_WEEK,
).dueDate,
).toBe('2025-10-18');
expect(
resolveDueDate(
{ kind: 'weekday', weekday: 'saturday', which: 'next' },
THURSDAY,
).dueDate,
).toBe('2025-10-18');
).toBe('2025-10-23');
});
it('never resolves a weekday into the past', () => {
// Sunday already passed in the week containing Thursday 10-16.
for (const which of ['this', 'next'] as const) {
const result = resolveDueDate(
{ kind: 'weekday', weekday: 'sunday', which },
THURSDAY,
FA_WEEK,
);
expect(result.dueDate).not.toBeNull();
expect(result.dueDate! > THURSDAY).toBe(true);
}
});
it('reads "by Thursday" said on a Thursday as the next one, not today', () => {
// A deadline of today is almost never what was meant.
const result = resolveDueDate(
{ kind: 'weekday', weekday: 'thursday', which: 'this' },
THURSDAY,
it('anchors "next" to a Monday week for en and nl', () => {
// The same sentence means a different day depending on where the week starts.
// Said on Saturday 10-11: the Monday-start week is 10-13..10-19, Thursday = 10-16.
// The Saturday-start week is 10-18..10-24, Thursday = 10-23.
const intent = {
kind: 'weekday',
weekday: 'thursday',
which: 'next',
} as const;
expect(resolveDueDate(intent, SATURDAY, EU_WEEK).dueDate).toBe(
'2025-10-16',
);
expect(resolveDueDate(intent, SATURDAY, FA_WEEK).dueDate).toBe(
'2025-10-23',
);
expect(result.dueDate).toBe('2025-10-23');
});
it('resolves the Saturday that starts the next week', () => {
const result = resolveDueDate(
{ kind: 'weekday', weekday: 'saturday', which: 'this' },
SATURDAY,
);
expect(result.dueDate).toBe('2025-10-18');
it('maps each locale to its week start', () => {
expect(weekStartForLocale('fa')).toBe(FA_WEEK);
expect(weekStartForLocale('en')).toBe(EU_WEEK);
expect(weekStartForLocale('nl')).toBe(EU_WEEK);
expect(weekStartForLocale('unknown')).toBe(EU_WEEK);
});
it('rejects an unknown weekday or qualifier', () => {
it('treats a missing qualifier as "this" rather than failing', () => {
// A bare weekday carries no qualifier; failing would discard a real spoken deadline.
expect(
resolveDueDate(
{
kind: 'weekday',
weekday: 'thursday',
which: null,
} as unknown as DueIntent,
SATURDAY,
FA_WEEK,
).dueDate,
).toBe(THURSDAY);
});
it('rejects an unknown weekday', () => {
expect(
resolveDueDate(
{
@@ -89,16 +124,7 @@ describe('resolveDueDate', () => {
which: 'this',
} as unknown as DueIntent,
SATURDAY,
).dueDate,
).toBeNull();
expect(
resolveDueDate(
{
kind: 'weekday',
weekday: 'thursday',
which: 'soon',
} as unknown as DueIntent,
SATURDAY,
FA_WEEK,
).dueDate,
).toBeNull();
});
@@ -107,16 +133,25 @@ describe('resolveDueDate', () => {
describe('offset intents', () => {
it('adds days, weeks and months', () => {
expect(
resolveDueDate({ kind: 'offset', unit: 'day', amount: 1 }, SATURDAY)
.dueDate,
resolveDueDate(
{ kind: 'offset', unit: 'day', amount: 1 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBe('2025-10-12');
expect(
resolveDueDate({ kind: 'offset', unit: 'week', amount: 1 }, SATURDAY)
.dueDate,
resolveDueDate(
{ kind: 'offset', unit: 'week', amount: 1 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBe('2025-10-18');
expect(
resolveDueDate({ kind: 'offset', unit: 'month', amount: 1 }, SATURDAY)
.dueDate,
resolveDueDate(
{ kind: 'offset', unit: 'month', amount: 1 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBe('2025-11-11');
});
@@ -153,8 +188,11 @@ describe('resolveDueDate', () => {
it('rejects negative, fractional and absurd amounts', () => {
for (const amount of [-1, 1.5, 10_000, Number.NaN]) {
expect(
resolveDueDate({ kind: 'offset', unit: 'day', amount }, SATURDAY)
.dueDate,
resolveDueDate(
{ kind: 'offset', unit: 'day', amount },
SATURDAY,
FA_WEEK,
).dueDate,
).toBeNull();
}
});
@@ -163,8 +201,11 @@ describe('resolveDueDate', () => {
describe('jalali intents', () => {
it('converts by arithmetic, not inference', () => {
expect(
resolveDueDate({ kind: 'jalali', jy: 1404, jm: 7, jd: 25 }, SATURDAY)
.dueDate,
resolveDueDate(
{ kind: 'jalali', jy: 1404, jm: 7, jd: 25 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBe('2025-10-17');
});
@@ -190,16 +231,25 @@ describe('resolveDueDate', () => {
describe('gregorian intents', () => {
it('accepts a real date and rejects an impossible one', () => {
expect(
resolveDueDate({ kind: 'gregorian', y: 2025, m: 10, d: 17 }, SATURDAY)
.dueDate,
resolveDueDate(
{ kind: 'gregorian', y: 2025, m: 10, d: 17 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBe('2025-10-17');
expect(
resolveDueDate({ kind: 'gregorian', y: 2025, m: 2, d: 30 }, SATURDAY)
.dueDate,
resolveDueDate(
{ kind: 'gregorian', y: 2025, m: 2, d: 30 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBeNull();
expect(
resolveDueDate({ kind: 'gregorian', y: 2025, m: 13, d: 1 }, SATURDAY)
.dueDate,
resolveDueDate(
{ kind: 'gregorian', y: 2025, m: 13, d: 1 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBeNull();
});
});
@@ -216,15 +266,21 @@ describe('resolveDueDate', () => {
it('treats a date decades away as unresolved', () => {
expect(
resolveDueDate({ kind: 'gregorian', y: 2099, m: 1, d: 1 }, SATURDAY)
.dueDate,
resolveDueDate(
{ kind: 'gregorian', y: 2099, m: 1, d: 1 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBeNull();
});
it('accepts today itself via a zero-day offset', () => {
expect(
resolveDueDate({ kind: 'offset', unit: 'day', amount: 0 }, SATURDAY)
.dueDate,
resolveDueDate(
{ kind: 'offset', unit: 'day', amount: 0 },
SATURDAY,
FA_WEEK,
).dueDate,
).toBe(SATURDAY);
});

42
backend/src/modules/voice/due-date.resolver.ts
View File

@@ -96,12 +96,28 @@ function describe(intent: DueIntent): string {
}
}
/** The Iranian week starts Saturday. */
const WEEK_START_JS = WEEKDAY_TO_JS.saturday;
/**
* Which weekday starts the week, per locale.
*
* "Next Thursday" is week-relative, so this changes the answer: the Iranian week starts
* Saturday, the Dutch and (European) English week starts Monday. Hardcoding Saturday
* would put an en/nl clinician's deadline a week out.
*/
const WEEK_START_BY_LOCALE: Record<string, number> = {
fa: WEEKDAY_TO_JS.saturday,
en: WEEKDAY_TO_JS.monday,
nl: WEEKDAY_TO_JS.monday,
};
/** Most recent Saturday, counting today if today is Saturday. */
function startOfWeek(iso: string): string {
const back = (civilDateJsWeekday(iso) - WEEK_START_JS + 7) % 7;
const DEFAULT_WEEK_START = WEEKDAY_TO_JS.monday;
export function weekStartForLocale(locale: string): number {
return WEEK_START_BY_LOCALE[locale] ?? DEFAULT_WEEK_START;
}
/** Most recent week-start day, counting today if today is that day. */
function startOfWeek(iso: string, weekStartJs: number): string {
const back = (civilDateJsWeekday(iso) - weekStartJs + 7) % 7;
return addDays(iso, -back);
}
@@ -119,20 +135,25 @@ function startOfWeek(iso: string): string {
function resolveWeekday(
intent: Extract<DueIntent, { kind: 'weekday' }>,
todayIso: string,
weekStartJs: number,
) {
const targetJs = WEEKDAY_TO_JS[intent.weekday];
if (targetJs === undefined) return null;
if (intent.which === 'this') {
// A bare weekday ("پنجشنبه") carries no qualifier, and the model may leave `which`
// null. Treat that as 'this' rather than failing an utterance that named a real day.
const which = intent.which === 'next' ? 'next' : 'this';
if (which === 'this') {
const todayJs = civilDateJsWeekday(todayIso);
let delta = (targetJs - todayJs + 7) % 7;
if (delta === 0) delta = 7;
return addDays(todayIso, delta);
}
if (intent.which === 'next') {
const offsetInWeek = (targetJs - WEEK_START_JS + 7) % 7;
return addDays(startOfWeek(todayIso), 7 + offsetInWeek);
if (which === 'next') {
const offsetInWeek = (targetJs - weekStartJs + 7) % 7;
return addDays(startOfWeek(todayIso, weekStartJs), 7 + offsetInWeek);
}
return null;
@@ -156,6 +177,7 @@ function resolveOffset(
export function resolveDueDate(
intent: DueIntent | null | undefined,
todayIso: string,
weekStartJs: number = DEFAULT_WEEK_START,
): DueResolution {
// Absent is not an error — most utterances carry no deadline. Anything else that is not
// an intent object is a deadline we failed to understand, and must be flagged rather
@@ -179,7 +201,7 @@ export function resolveDueDate(
let resolved: string | null = null;
switch (intent.kind) {
case 'weekday':
resolved = resolveWeekday(intent, todayIso);
resolved = resolveWeekday(intent, todayIso, weekStartJs);
break;
case 'offset':
resolved = resolveOffset(intent, todayIso);

2
backend/src/modules/voice/extraction.resolver.spec.ts
View File

@@ -14,6 +14,8 @@ const tooth = (fdi: string, spoken = fdi): ToothIntent => ({
const CTX: ResolveContext = {
todayIso: '2025-10-11',
weekStartJs: 6, // Saturday — the fa week
treatmentTypeCodes: new Set(['restoration', 'prosthesis', 'extraction']),
prosthesisTypeCodes: new Set(['monolithic_zirconia', 'pfm_crown']),
linkedLabIds: new Set(['lab-sina', 'lab-mehr']),

4
backend/src/modules/voice/extraction.resolver.ts
View File

@@ -48,6 +48,8 @@ export type ResolvedExtraction = {
export type ResolveContext = {
todayIso: string;
/** JS weekday index the clinician's week starts on — see weekStartForLocale. */
weekStartJs: number;
treatmentTypeCodes: ReadonlySet<string>;
prosthesisTypeCodes: ReadonlySet<string>;
linkedLabIds: ReadonlySet<string>;
@@ -272,7 +274,7 @@ export function resolveVoiceIntent(
);
unresolved.push(...prosthesisResult.unresolved);
const due = resolveDueDate(intent?.due, ctx.todayIso);
const due = resolveDueDate(intent?.due, ctx.todayIso, ctx.weekStartJs);
if (due.unresolved) unresolved.push(due.unresolved);
const comment =

197
backend/src/modules/voice/openrouter.provider.spec.ts Normal file
View File

@@ -0,0 +1,197 @@
import {
OpenRouterAsrProvider,
OpenRouterExtractionProvider,
} from './openrouter.provider';
import { VoiceProviderError } from './voice.providers';
const CONFIG = {
apiKey: 'test-key',
baseUrl: 'https://openrouter.test/api/v1',
model: 'm',
};
const CATALOG = {
treatmentTypes: [{ code: 'prosthesis', label: 'پروتز' }],
prosthesisTypes: [{ code: 'pfm_crown', label: 'روکش پی‌اف‌ام' }],
labs: [{ id: 'lab-1', name: 'لابراتوار سینا' }],
};
type ChatBody = {
temperature: number;
provider: { require_parameters: boolean };
response_format: { type: string; json_schema: { strict: boolean } };
messages: { role: string; content: string }[];
};
function parseBody(spy: jest.Mock): ChatBody {
const init = (spy.mock.calls[0] as [string, RequestInit])[1];
return JSON.parse(init.body as string) as ChatBody;
}
function mockFetch(response: {
ok: boolean;
status?: number;
body?: unknown;
text?: string;
}) {
const spy = jest.fn().mockResolvedValue({
ok: response.ok,
status: response.status ?? (response.ok ? 200 : 500),
json: () => Promise.resolve(response.body),
text: () => Promise.resolve(response.text ?? ''),
});
global.fetch = spy;
return spy;
}
afterEach(() => {
jest.restoreAllMocks();
});
describe('OpenRouterAsrProvider', () => {
it('posts base64 JSON with the language hint, per the documented STT contract', async () => {
const spy = mockFetch({
ok: true,
body: { text: ' سلام ', usage: { seconds: 12, cost: 0.002 } },
});
const result = await new OpenRouterAsrProvider(CONFIG).transcribe(
{ data: 'BASE64', format: 'webm' },
'fa',
);
const [url, init] = spy.mock.calls[0] as [string, RequestInit];
expect(url).toBe('https://openrouter.test/api/v1/audio/transcriptions');
expect(init.headers).toMatchObject({ Authorization: 'Bearer test-key' });
expect(JSON.parse(init.body as string)).toEqual({
model: 'm',
input_audio: { data: 'BASE64', format: 'webm' },
language: 'fa',
});
expect(result.text).toBe('سلام');
expect(result.usage).toEqual({ seconds: 12, costUsd: 0.002 });
});
it('reports missing usage as null rather than zero', async () => {
mockFetch({ ok: true, body: { text: 'x' } });
const result = await new OpenRouterAsrProvider(CONFIG).transcribe(
{ data: 'B', format: 'webm' },
'en',
);
expect(result.usage).toEqual({ seconds: null, costUsd: null });
});
it('raises a staged error without leaking the vendor body', async () => {
// A 4xx can echo the request back, transcript included.
mockFetch({
ok: false,
status: 400,
text: 'transcript: patient name here',
});
const provider = new OpenRouterAsrProvider(CONFIG);
await expect(
provider.transcribe({ data: 'B', format: 'webm' }, 'fa'),
).rejects.toMatchObject({
name: 'VoiceProviderError',
stage: 'asr',
status: 400,
});
await expect(
provider.transcribe({ data: 'B', format: 'webm' }, 'fa'),
).rejects.not.toThrow(/patient name/);
});
});
describe('OpenRouterExtractionProvider', () => {
const wireContent = JSON.stringify({
treatmentType: 'prosthesis',
teeth: [
{ spoken: 'یک چهار', fdi: '14', arch: null, side: null, position: null },
],
connectedSpans: [],
comment: null,
prosthesisDefaultType: 'pfm_crown',
prosthesisOverrides: [],
labId: 'lab-1',
labMatchExact: true,
due: {
kind: 'none',
weekday: null,
which: null,
unit: null,
amount: null,
jy: null,
jm: null,
jd: null,
y: null,
m: null,
d: null,
},
});
it('constrains output with a strict JSON schema and a schema-honouring provider', async () => {
const spy = mockFetch({
ok: true,
body: {
choices: [{ message: { content: wireContent } }],
usage: { cost: 0.0008 },
},
});
const result = await new OpenRouterExtractionProvider(CONFIG).extract(
'روی دندان ۱۴ روکش',
CATALOG,
'fa',
);
const body = parseBody(spy);
expect(body.response_format.type).toBe('json_schema');
expect(body.response_format.json_schema.strict).toBe(true);
// Without require_parameters OpenRouter may route to a provider that treats the
// schema as a hint and returns prose, failing parsing intermittently.
expect(body.provider).toEqual({ require_parameters: true });
expect(body.temperature).toBe(0);
expect(result.intent.treatmentType).toBe('prosthesis');
expect(result.intent.teeth[0]).toEqual({
kind: 'explicit',
fdi: '14',
spoken: 'یک چهار',
});
expect(result.intent.due).toBeNull();
expect(result.costUsd).toBe(0.0008);
});
it('sends the catalog codes and lab ids the model is allowed to choose from', async () => {
const spy = mockFetch({
ok: true,
body: { choices: [{ message: { content: wireContent } }] },
});
await new OpenRouterExtractionProvider(CONFIG).extract('x', CATALOG, 'fa');
const body = parseBody(spy);
const system = body.messages[0].content;
expect(system).toContain('prosthesis');
expect(system).toContain('pfm_crown');
expect(system).toContain('lab-1');
expect(body.messages[1]).toEqual({ role: 'user', content: 'x' });
});
it('fails loudly on unparseable content rather than passing rubbish downstream', async () => {
mockFetch({
ok: true,
body: { choices: [{ message: { content: 'I think tooth 14?' } }] },
});
await expect(
new OpenRouterExtractionProvider(CONFIG).extract('x', CATALOG, 'fa'),
).rejects.toBeInstanceOf(VoiceProviderError);
});
it('fails when the model returns no content at all', async () => {
mockFetch({ ok: true, body: { choices: [] } });
await expect(
new OpenRouterExtractionProvider(CONFIG).extract('x', CATALOG, 'fa'),
).rejects.toMatchObject({ stage: 'extraction' });
});
});

12
backend/src/modules/voice/voice.service.ts
View File

@@ -12,6 +12,7 @@ import { hasEffectivePermission } from '../../common/membership-permissions';
import { normalizeCatalogLocale } from '../catalog/catalog-label.service';
import { ProsthesisCatalogService } from '../prosthesis-catalog/prosthesis-catalog.service';
import { TreatmentCatalogService } from '../treatment-catalog/treatment-catalog.service';
import { weekStartForLocale } from './due-date.resolver';
import {
resolveVoiceIntent,
type ResolvedExtraction,
@@ -92,6 +93,7 @@ export class VoiceService {
// Stage 1 — audio never touches disk and is not retained beyond this call.
let transcript: string;
let asrCost: number | null = null;
let asrSeconds: number | null = null;
try {
const result = await asr.transcribe(
{ data: dto.audio, format: dto.format },
@@ -100,10 +102,19 @@ export class VoiceService {
);
transcript = result.text;
asrCost = result.usage.costUsd;
asrSeconds = result.usage.seconds;
} catch (error) {
throw this.toAppException(error, 'asr');
}
// durationMs is client-reported and therefore not enforcement. usage.seconds is the
// vendor's own measurement of the audio it decoded, so a client under-reporting length
// to slip past the cap is caught here — after the ASR spend, but before the extraction
// call, and visibly in telemetry.
if (asrSeconds != null) {
this.assertWithinCap(asrSeconds * 1000);
}
if (!transcript.trim()) {
throw new AppException(
ErrorCode.VOICE_NOTHING_RECOGNIZED,
@@ -126,6 +137,7 @@ export class VoiceService {
llmCost = result.costUsd;
resolved = resolveVoiceIntent(result.intent, {
todayIso,
weekStartJs: weekStartForLocale(catalogLocale),
treatmentTypeCodes: new Set(catalog.treatmentTypes.map((t) => t.code)),
prosthesisTypeCodes: new Set(
catalog.prosthesisTypes.map((t) => t.code),