llm-schema-research

LLM and schema research review

Type: Finding
Status: Active supporting research
Canonical for: Narrative interpretation of adjacent LLM tool/schema studies
Depends on: Evidence ledger, which owns claim-level evidence strength

Strongest adjacent evidence

BiasBusters

Microsoft Research’s controlled study varies tool features, metadata and pretraining exposure across seven models. It reports semantic alignment between a query and tool metadata as the strongest predictor of selection, shows that description perturbations shift selection, and finds that repeated pretraining exposure amplifies bias. Empirical adjacent evidence. BiasBusters

Transfer: an enum label and description are part of the evidence an agent uses. A locally unusual meaning may have to overcome a learned prior. Limit: tools are not design variants, and tool calls usually have clearer executable success criteria.

ToolScope

ToolScope studies large toolsets with overlapping names and descriptions. It reports degraded selection under ambiguity and improvements from context-aware filtering. Empirical adjacent evidence. ToolScope

Transfer: confusable alternatives should be retrieved with discriminating descriptions, not dumped into an undifferentiated list. Limit: filtering tools is not the same as resolving visual hierarchy in a composition.

ToolExpNet

ToolExpNet targets similar tools and multi-tool dependencies, using trial experience and a relationship network to improve decisions. Empirical adjacent evidence. ACL Findings paper

Transfer: alternatives and relationships are first-class decision evidence. Limit: the method does not evaluate design-system schemas.

ToolTweak

ToolTweak shows that jointly changing tool names and descriptions can bias selection among functionally equivalent tools. Because both channels change together, it does not isolate a name-only effect. Empirical adjacent evidence from a preprint/under-review study, not settled design-system evidence. arXiv preprint

What is established

• Model selection is sensitive to semantic cues in names and descriptions.
• Overlapping or weakly differentiated options increase selection difficulty.
• Learned familiarity and conventional associations can bias choices.
• Descriptions and contextual filtering can change outcomes; name-only interpretation is not the only available mechanism.

What is not established

• The effect size for component or variant selection.
• Whether secondary causes more errors than an intent label in an actual design task.
• Whether renaming outperforms adding structured metadata.
• Whether richer schemas help consistently once token cost, retrieval error and invalid combinations are included.
• Whether current frontier models differ materially on this problem.

Strength and limits of the analogy

The analogy is strong when an agent maps a task to one value in a finite, documented enum. It is weaker when the “correct” design choice depends on visual neighbors, workflow stage, product policy, platform conventions or user research. Design correctness can also be plural and negotiated; many tool benchmarks assume one executable answer.

Therefore the adjacent literature justifies the controlled evaluation in this package. It does not validate a naming prescription.

Prior conflict

Inferred from adjacent evidence: a common label with a locally exceptional meaning creates two burdens: retrieve the local definition and suppress the pretrained association. A novel domain label removes the familiar conflict but introduces a grounding burden. Neither global convention nor local precision wins automatically. Explicit definitions, examples, counterexamples and scoped retrieval are mechanisms for resolving both.