How the score is computed
Evaluation modes. Every audit runs in exactly one of two labeled modes. live_model: selection is performed by an external language model at temperature 0, and results describe behavioral evaluation under the displayed model and test configuration. synthetic: selection is performed by a deterministic heuristic simulator — no external language model is used, and results estimate likely tool ambiguity; they do not represent measured production model accuracy. The mode is shown on every report and export, and a misconfigured live-model audit fails loudly instead of silently degrading to synthetic.
Ground-truth provenance. Every test case carries a source label. human_fixture: intent and expected tool authored by a person (all built-in demo audits, including the public example, score against these static fixtures). user_defined: provided by the user. ai_generated: machine-generated and machine-validated — the validation pass is performed by the same system family that evaluates selection, so these results are directional rather than independently verified. Independent validation is on the roadmap.
A known circularity in synthetic before/after. In synthetic mode the optimizer and the selection heuristic are the same system, and the heuristic explicitly rewards the usage-boundary sentences the optimizer writes. The synthetic before/after therefore estimates how much disambiguating structure a rewrite adds; it is not evidence that a real model improves. Live-model before/after reruns the identical suite under the same external model, which is the comparison to trust.
ToolChoice separates two kinds of evidence and never lets them blur. Deterministic findings come from a static rule engine — the same input produces the same findings, byte for byte. Behavioral evidence comes from tool-selection runs — a model (or the deterministic simulator) receives your tool definitions exactly as an MCP client would, plus one realistic user request, and must answer in structured JSON: selected tool, arguments, confidence, reason.
Test intents are generated per tool, then pass a separate validation step. Tests whose expected answer cannot be defended with at least 0.7 confidence are marked exploratory and shown, but excluded from the score — genuinely ambiguous requests should inform you, not punish you. Negative tests check that a capability your server does not have yields NO_TOOL rather than a misfire.
Selection runs at temperature 0 for repeatability. Model name, settings, token usage and latency are recorded per run. The final score is the fixed weighted sum below — no language model invents or adjusts any number.
Before/after comparisons rerun the identical suite against the optimized definitions. Improvements are claimed only when the second evaluation supports them; regressions are reported just as prominently.
ToolChoice never executes submitted tools, never connects to private networks, redacts detected secrets before storage, and deletes anonymous audits after 7 days. Reports describe simulated or measured selection behavior — never verified production incidents.
Score dimensions
| dimension | weight | measured as |
|---|---|---|
| Tool selection accuracy | 35% | The share of scored intents for which the model selects the expected tool (or correctly selects NO_TOOL). |
| Tool distinguishability | 20% | Static penalties for near-duplicate names and overlapping descriptions, plus behavioral penalties for observed confusion rates. |
| Description quality | 15% | A checklist per tool: states an action, a when-to-use condition, a when-not-to-use boundary, an expected result. Verbosity is never rewarded — descriptions over 600 characters forfeit the conciseness point. |
| Parameter clarity | 15% | Per parameter: description present, non-generic name, declared type, documented enums, documented identifier and date formats. |
| Argument generation reliability | 10% | The share of selected-tool runs whose generated arguments are schema-valid: required coverage, no hallucinated keys, correct types and enum membership. |
| Safety and side-effect clarity | 5% | The share of destructive or consequential tools whose descriptions state their effect explicitly. |