Tuning concept search: what the cosine scores actually said

Our fee-schedule lookup tool now answers plain language. Type "knee replacement" and you get the CPT codes for it, instead of having to know that the code is 27447. The plumbing is ordinary: embed a corpus of 594 clean-room code descriptions with Voyage AI, store the vectors in pgvector, rank by cosine similarity at query time.

Shipping that took an afternoon. Making it good took measuring it, and the measurements contradicted three things we believed.

What the search does

Two lanes run per query and merge code-first:

• Code lane — prefix match on HCPCS. Typing 9921 returns 99213, 99214, and their siblings.
• Concept lane — cosine search over the embedded descriptions, for anything that reads like words rather than a code.

A concept hit scoring below a floor gets dropped rather than padding the list with noise. That floor is where the trouble started.

Assumption 1: 0.45 was a reasonable floor

We shipped MIN_SCORE = 0.45 with a comment justifying it — concept hits below this are noise for a corpus this dense in domain vocabulary. That sentence describes a hypothesis. Nobody had checked it against a score.

Checking it took one tinker loop that skipped the filter and printed raw similarities — the same pgvector cosine query the concept lane runs, minus the MIN_SCORE cutoff:

>>> $vector = json_encode(app(VoyageEmbeddingClient::class)->embedQuery('sore throat'));
>>> DB::select(
...     'select hcpcs, 1 - (embedding <=> ?::vector) as score
...      from code_embeddings order by embedding <=> ?::vector limit 5',
...     [$vector, $vector],
... );

Every top match is correct. Every one below 0.45 was being thrown away. "Sore throat" returned nothing at all, even though two strep-test codes in the corpus name that symptom outright.

Two things fall out of this. First, a one-word query scores lower than the same word inside a phrase, against the identical target document — "laceration" loses 0.077 to "laceration repair" while pointing at the same code. Less context means a weaker match, so any floor tuned on phrases quietly discards single words.

Second, the typo scored higher than the correct spelling (0.434 vs 0.426). Voyage already handles misspellings. We had been about to solve a problem the model solved for us, and the real defect was the number sitting in the constant.

We moved the floor to 0.30 after a second sweep across 23 queries covering patient language, coder jargon, and abbreviations. Everything above 0.30 stayed on topic. The constant now carries the scores that justify it, so the next person to touch it inherits evidence instead of another hypothesis.

Assumption 2: a lower floor fixes short queries

It fixes some. The same sweep found queries that no floor rescues:

"Stitches" appears verbatim in the wound-repair descriptions. The word is right there, and cosine similarity still ranks it at 0.27, because a short colloquial token carries little signal against paragraphs of careful clinical prose. Dropping the floor to 0.27 to catch it would admit genuine noise everywhere else.

"CBC" fails differently. That abbreviation appears nowhere in the corpus. The description spells out "Complete Blood Count with Differential" — correct, professional, and not what anyone types.

So the two failures need two fixes.

Lexical search handles the first. A Postgres tsvector lane now runs beside the vector lane over the same fields, and a literal keyword match floors the score at 0.5:

private const LEXICAL_FLOOR = 0.5;

// ...

foreach ($this->lexicalMatches($query, $limit) as $hcpcs) {
    $scores[$hcpcs] = max($scores[$hcpcs] ?? 0.0, self::LEXICAL_FLOOR);
}

lexicalMatches() runs a prefix-matched tsquery — each typed word gets a trailing :*, since this fires on every keystroke and websearch_to_tsquery has no prefix mode — against the same fields the embedding sees:

$rows = DB::select(
    "select hcpcs
     from code_descriptions
     where to_tsvector('english',
             category_label || ' ' || lead || ' ' ||
             coalesce(when_used, '') || ' ' || coalesce(patient_examples, '') || ' ' ||
             coalesce(includes, '') || ' ' || coalesce(aka, ''))
           @@ to_tsquery('english', ?)
     order by ts_rank(...) desc
     limit ?",
    [$tsQuery, $tsQuery, $limit],
);

Vector similarity ranks documents by meaning; exact word matching is what it gives up to do that. Running both recovers "stitches" immediately, with no new content.

Synonyms handle the second. Each code gained an aka field — lay terms and abbreviations, folded into both the embedding input:

$description->aka === null ? null : "Also searched as: {$description->aka}.",

and the full-text index above, via that same coalesce(aka, ''). A real entry, from the hand-curated synonym table that seeds it:

'85025' => ['cbc', 'complete blood count', 'blood count test'],

Writing 594 of those by hand was impractical, so 15 subagents wrote them in parallel, each seeing only its own batch and each bound by the same rule that governs the rest of the corpus: derive every term from the code's existing clean-room prose, never from AMA descriptor text. That produced 1,817 terms across 592 codes. "CBC" and "EKG" now resolve on the first keystroke.

Assumption 3: bad results mean bad search

Two queries in the sweep returned nothing useful, and neither was a search problem.

"Cast removal" scored 0.15 against noise. Cast application and removal codes are simply absent from our 594-code corpus. That is a content gap, and search tuning cannot close it.

"Modifier 25" scored 0.25 against noise. Modifiers are not procedure codes at all. The corpus indexes procedures, so the right destination is the glossary, not this index.

Both got logged as content work. Neither got a threshold adjustment. Grinding on relevance for a query whose answer the corpus does not contain wastes the afternoon and teaches you nothing.

Watching it work

Relevance tuning never finishes, so the loop needs to stay open. Two small additions keep it open.

Click position now rides in the analytics event name:

trackEvent('Search Result Click: ' + r.kind + ' (rank ' + (i + 1) + ')');

producing events like Search Result Click: concept (rank 3). Fathom, our analytics provider, supports no custom event properties, only names, so encoding rank there keeps the distribution readable straight off the dashboard. Cardinality stays bounded at two kinds times eight ranks. Clicks landing consistently at rank 5 will tell us which terms still need synonyms, and the data will pick the next batch better than we can guess it.

Cost has a hard ceiling. Query embeddings cache for seven days keyed by model and text, and the endpoint truncates queries at 120 characters. Neither bounds a distributed attack, where thousands of IPs each send unique uncached queries and sail past a per-IP rate limit. A global daily counter now stops new embedding calls past a fixed budget:

private function reserveEmbedBudget(): bool
{
    $key = 'voyage:embed-budget:'.now()->toDateString();

    Cache::add($key, 0, now()->endOfDay());

    return Cache::increment($key) <= self::DAILY_EMBED_BUDGET;
}

Cache::add only sets the counter if it's missing, so concurrent requests racing to create the key don't reset an in-flight count. At 20,000 embeds a day, the pricier Voyage model, and the longest query the route accepts, the worst case runs about seven cents — off by an order of magnitude and it still stays under a dollar.

The pattern

Every fix here came from running real queries against the real corpus and reading the numbers.

The floor was wrong because nobody had compared it to a score. The lexical gap hid because "stitches" scoring 0.27 looks fine until you notice the word is sitting in the document. The content gaps looked like search bugs until we checked whether the answers existed.

None of this needed special tooling — a tinker one-liner, two dozen queries chosen to sound like actual users, and a willingness to let measurements overrule the comment someone wrote at ship time.