Deep research synthesis (2023–2026 empirical and adjacent literature) on the three research questions. Sources are drawn primarily from arXiv papers, controlled benchmarks, engineering studies, and practitioner reports on LLM tool/function calling, schema design, and constrained generation. Citations reference the search results.
1. LLM enum-value and parameter semantics
Research question
How do property names, enum-value names, and attached descriptions independently affect an AI agent’s ability to select the correct value from a component schema? (Distinguish tool selection from parameter/enum-value selection.)
Evidence table of relevant experiments
| Study / Source | Year | Models | Key manipulation | Main finding | Maps to design-system variant selection? |
|---|---|---|---|---|---|
| PA-Tool (Lee et al.) | 2025/26 | SLMs ≤8B (Llama-3.x, Qwen2.5, etc.) on MetaTool & RoTBench | Rename tools/params to pretraining-aligned names (peakedness signal); descriptions held constant | Up to +17% tool selection; schema-misalignment (hallucinated names) ↓80%. Tool-name alignment mainly helps selection; param-name alignment helps parameter ID. Naming > description enhancement for SLMs. | High – direct analog of variant/property name choice under fixed descriptions. |
| Semantic Method Naming (Hood) | 2026 | Claude Sonnet 4.6, Grok-3, GPT-4o, Llama-3.2-3B; 7,200 trials | Agentic intent names (BOOK/FIND/QUERY) vs REST/CRUD; description-swap ablation | Agentic names +10–29 pp over CRUD for frontier models (absent at ~3B). Description-swap: giving CRUD endpoints agentic descriptions collapses 39–43 pp; reverse hurts only 1–15 pp. Name itself encodes intent that survives doc noise. | High – property/enum names as primary signal. |
| RoTBench noise / PA-Tool analysis | 2024–26 | Llama-3.x etc. | Slight/Medium/Heavy name perturbations (descriptions intact) | Even mild name noise causes large drops (e.g., 58%→37%). Medium (meaningless names) worst; Heavy (shuffled real names) partially recovers. Names are primary cues for SLMs. | High for enum/property labels. |
| Tool description edits (arXiv 2505.18135) | 2025 | GPT-4.1, Qwen2.5-7B + 17 models | Append persuasive phrases to descriptions | Edited tools receive 2–7×+ more usage when competing with identical originals. Descriptions are highly manipulable. | Medium – shows description power, but for tool (not enum) selection. |
| Practitioner reports (CallSphere, DEV, Adaline, etc.) | 2025–26 | Production agents | Rename tools/params; add/remove enums; “when/not” in descriptions | Renames alone moved accuracy 71%→94% in one case. Enums strongly reduce invented values/casing variants. Parameter names carry independent semantic weight. | High for enum + property naming. |
| Constrained generation / JSON Schema enum | 2024–26 | Various (incl. SLMs) | Hard enum constraints via structured outputs | Validity →100%; format errors eliminated. Mixed or negative effects on semantic correctness for small models (“constraint tax”). | Medium – enforces selection after generation. |
| Fuzzy enum matching / casing studies | 2025–26 | Multi-provider | Casing, separators, abbreviations | Models frequently emit near-misses (north_america vs NorthAmerica). | High for invented/variant values. |
Findings by strength of evidence
Demonstrated
- Names (tool, parameter, and by extension enum values) are a primary selection cue, especially for smaller models. Changing names while holding descriptions fixed produces large, reliable gains or losses (PA-Tool, Hood description-swap, RoTBench noise).
- Schema misalignment (plausible but non-schema names) is a dominant failure mode for SLMs; aligning names to pretraining distributions reduces it dramatically.
- Descriptions are powerful and can override or amplify name effects, but for SLMs name alignment often outperforms pure description enhancement.
- Familiar-but-conflicting or opaque names both hurt; opaque/random names are especially damaging; familiar-but-wrong names can partially recover via residual priors.
- Enum constraints (JSON Schema
enum) almost eliminate out-of-vocabulary / invented values and casing variants when enforced (structured outputs / constrained decoding). - Capability threshold exists: strong name effects appear at frontier scale; near-absent or weaker at ~3B.
Adjacent
- Property-name semantics (
variant,tone,intent,queryvssearch_term) influence interpretation independently of description (practitioner consensus + parameter-name ablation in PA-Tool). - Position bias and name bias (certain strings preferred) interact with selection.
- Constrained decoding guarantees syntactic validity but can impose a “constraint tax” on semantic accuracy for small models.
Unestablished (gaps relative to the brief)
- Direct, controlled experiments that hold both name and description identical while varying only enum value labels (vs. inventing new values).
- Systematic comparison of familiar-conflicting vs. opaque identifiers specifically for enum values inside a fixed property.
- How property name (
variantvstonevsintent) modulates the same set of enum values across model families/sizes. - Interaction of all three factors (property name × enum name × description) in a full factorial design on design-system-like schemas.
Mapping to design-system variant selection
Closest existing work is PA-Tool + Hood-style name/description ablations. Design-system “variant” selection is essentially parameter/enum selection under a fixed component (tool). The same mechanisms (pretraining name priors, description override power, constraint enforcement) transfer directly. Gaps remain on combinatorial property×value interactions and design-token-style opaque identifiers.
Candidate conditions & controls for a future Button evaluation
- Factors: property name (
variant/tone/intent/ opaque ID); enum-value names (semantic / familiar-conflicting / opaque / random); description quality (identical / minimal / detailed “use when”); model size/family; presence of hard enum constraint. - Controls: fixed component description; fixed task distribution (positive + near-miss + negative cases); order randomization; temperature=0 + multiple samples; measure both selection accuracy and invented-value rate.
- Metrics: exact match to gold value; rate of schema-violating inventions; confidence calibration; recovery under description-only vs name-only vs both.
- Do not assume richer descriptions always dominate names (evidence is mixed by scale).
2. Decision-boundary metadata and retrieval
Research question
What form and delivery method of semantic guidance most reliably helps an AI agent distinguish when a design-system component or variant should and should not be used?
Comparison of metadata formats
| Format | Strengths | Weaknesses | Evidence strength |
|---|---|---|---|
| Concise definition only | Low token cost; always fits | Insufficient for boundary cases; high wrong-tool rate | Weak – practitioners report frequent ambiguity |
| Definition + “use when” | Clear positive trigger | Still misses near-misses | Medium – widely recommended |
| + “avoid when” / “do not use for” + named alternatives | Strongest practical signal; reduces overlap errors | Negative instructions can be fragile (“pink elephant” effect in some settings) | Strong practitioner + some controlled description-edit results |
| Positive examples + counterexamples | Helps draw boundaries | Token-heavy; ordering/truncation sensitive | Adjacent (few direct tool-selection RCTs) |
| Contrastive / distinguishing criteria | Explicit differentiation | Requires careful authoring | Medium |
Structured metadata (useWhen/avoidWhen fields) vs equivalent prose | Potentially more reliable parsing | Unproven superiority; risk of schema bloat | Unestablished for selection accuracy |
| Retrieval granularity (per-component / per-value / per-pattern / per-task) | Can keep context lean | Missing/stale/conflicting units cause failures | Adjacent (tool-menu filtering, RAG literature) |
Evidence on examples, counterexamples, and selection boundaries
- Explicit “when to use / when not to use” (naming competing tools) is repeatedly cited as the highest-leverage description change for reducing wrong-tool selection.
- Pure negative instructions (“never…”, “don’t…”) are brittle; positive reframing or explicit alternatives perform better in several reports.
- Description edits alone can multiply usage 2–7× (or collapse accuracy 39+ pp when mismatched), showing boundaries are highly sensitive to wording.
- Tool-menu / retrieval filtering (exposing only relevant tools) can raise success from ~32% to ~86% while cutting tokens dramatically (ToolMenuBench-style results).
- Partial guidance (truncation, ordering, missing negative examples) degrades performance; models attend to the correct tool yet still pick wrong in some failure analyses (“looking is not picking”).
- Strong metadata can partially compensate for ambiguous names, but evidence is stronger that good names + good boundaries together outperform either alone.
Recommended experimental conditions (not production schemas)
- Compare: definition-only vs +useWhen vs +avoidWhen+alternatives vs +examples/counterexamples vs structured fields, holding total tokens roughly constant.
- Retrieval units: whole-component doc vs property-value slice vs task-pattern vs full catalog.
- Truncation / ordering ablations; conflicting guidance; stale versions.
- Measure: correct selection, false-positive use, false-negative omission, recovery after partial context.
- Test whether metadata can rescue deliberately ambiguous or conflicting names.
Retrieval-unit options & tradeoffs
- Per-component: simple, but over-exposes variants.
- Per-value / per-variant: precise boundaries, higher retrieval cost and risk of missing relational rules.
- Per-pattern / task: good for common flows, brittle for novel combinations.
- Hybrid + negative-guidance retrieval: promising but unproven at scale.
Failure cases
Missing, conflicting, stale, or excessive guidance all degrade selection. Overly long catalogs cause attention and positional failures even when the gold item is attended to. Negative-only guidance can backfire.
Key caveat: richer metadata is not always better; token budget, ordering, and conflict risk matter.
3. Constraint models for agent-consumable design systems
Research question
How should design-system rules, dependencies, and invalid combinations be represented so agents can generate valid decisions and tools can verify them?
Comparison matrix of constraint approaches
| Approach | Expressiveness for DS rules (requires / conflicts / only-inside / exactly-one / context-dependent) | Generation-time guidance | Post-generation verification | Complexity & governance risk | LLM agent fit |
|---|---|---|---|---|---|
| JSON Schema (if/then/else, allOf, oneOf, enum, dependencies) | Good for structural + simple conditionals; limited relational/context | Via constrained decoding / structured outputs | Excellent, fast | Moderate; schema explosion with many variants | High – already native to tool calling |
| SHACL / ShEx / knowledge-graph shapes | Excellent for graph relations, cardinality, complex paths | Weak (hard to surface to LLM) | Excellent | High learning curve; versioning non-trivial | Medium – better for backend validation |
| Feature models / product-line (requires, excludes, cardinality) | Native fit for combinatorial variants | Can be compiled to guidance text or SAT | Strong (solvers) | Combinatorial explosion; feature interactions | Medium – needs compilation step |
| Type systems / discriminated unions | Strong for “exactly one”, mutually exclusive | Via typed stubs or schema | Compile-time / static | Good versioning if in code | High if agents emit typed code |
| Policy-as-code (OPA/Rego etc.) | Flexible rules + exceptions | Possible via retrieval | Strong | Governance-friendly but another language | Medium |
| Planning preconditions/effects | Natural for sequential/contextual rules | Can be shown as text | Planner verification | High if full planner | Lower for pure selection |
| Design linting / static analysis | Post-hoc, explanatory | Limited | Good for CI | Familiar to designers/devs | Complementary |
| Configuration solvers | Handles large valid-combination spaces | Indirect | Optimal | Scalability & explanation hard | Backend only |
Representative design-system rules in viable models
- JSON Schema:
ifproperty = X then require Y;oneOffor exclusive variants;enum+dependencies. - Feature model:
variantA requires size=large;icon conflicts with text-only. - Discriminated union:
type: "primary" | "secondary"with different required fields. - SHACL: property paths + cardinality + class constraints for “only valid inside Container”.
Generation-time guidance vs post-generation validation
- Show to agent (pre): high-level requires/conflicts, use/avoid boundaries, small enums – improves selection without overwhelming.
- Enforce deterministically after: full combinatorial validity, context-dependent rules, “final confirmation only”, complex exclusions. Structured outputs / constrained decoding guarantee syntactic & simple enum validity; richer rules need a separate validator.
- Evidence of “constraint tax”: hard schemas can raise validity to 100% while lowering semantic correctness, especially on small models. Prefer “reason free, constrain late” or soft guidance + hard post-check.
- Combinatorial growth: valid combinations explode; pure enumeration in the prompt fails. Solvers or incremental validation are required.
Complexity and governance risks
- Schema bloat and versioning conflicts.
- Explanation of why a combination is invalid (needed for agent repair loops).
- Exceptions and contextual overrides hard to keep consistent.
- Dual maintenance (human-readable docs + machine constraints) drifts.
Candidate evaluation conditions
- Express the same rule set in JSON Schema vs feature-model vs typed unions; measure agent generation validity, repair cycles, and human authoring effort.
- Ablate: full constraints in prompt vs summary + post-validation vs hybrid.
- Stress combinatorial scale and missing/conflicting rules.
- Measure both generation accuracy and verifiability (false accepts / false rejects).
Cross-cutting observations
- Names and descriptions are not interchangeable. For selection (tool or value), names often dominate for smaller models; descriptions dominate or amplify for frontier models and boundary decisions.
- Hard constraints are necessary but not sufficient. They stop invalid outputs; they do not guarantee the correct valid choice and can hurt small models.
- Retrieval and filtering of the right guidance/schema subset is as important as the content of that guidance.
- Most direct empirical work is on tools/APIs, not design-system variants. Transfer is strong on mechanisms (naming priors, description boundaries, enum constraints, post-validation) but specific factorial experiments on
variant/tone/etc. remain sparse → the proposed Button evaluation would fill a genuine gap.
This synthesis stays within the brief’s scope: evidence tables, demonstrated/adjacent/unestablished distinctions, mapping assessments, and candidate experimental conditions only. No PDS names, API changes, or production constraint language are recommended.