public audit report · 2026-07-16

CRM Server (example)

6 tools · automated analysis by ToolChoice · input schemas are hidden on public pages

This is the built-in example report, generated live by the evaluation pipeline from the deliberately ambiguous CRM demo server. Nothing here is hand-written — rerun the demo and you'll get the same result.

SYNTHETIC EVALUATION

Synthetic heuristic simulation. No external language model was used. Results estimate likely tool ambiguity and do not represent measured production model accuracy.

Executive summary

56/100

Behavioral performance

simulated routing + arguments

47/100

Definition quality

static schema + safety

51/100

Overall readiness

weighted aggregate

The simulator routed the correct tool in 6 of 10 scored scenarios and produced valid arguments 4/10 of the time. The main observed failures were a boundary request that selected update_customer instead of get_customer, a boundary request that selected update_customer instead of create_customer. Overall readiness is held back by static definition-quality and safety findings, not by behavioral performance.

View 4 observed failures →6 potential ambiguity pairs →View 7 recommended fixes →View rewrite validation →

6tools analyzed

6/10correct selections

4observed failures

6potential ambiguity pairs

score breakdown · deterministic weighted sum

Selection accuracy

6035%

Argument reliability

4010%

Distinguishability

3920%

Description quality

3315%

Parameter clarity

5515%

Safety clarity

1005%

Observed failures

Failures that actually occurred during behavioral evaluation

expected get_customerselected update_customertool confusion

“I already know the customer ID — it's cus_6398. Pull up the complete customer record.”

Reasoning: get_customer, update_customer, create_customer, search_customers all appear to handle customer requests and none states when to use which; picked the most detailed description.

expected create_customerselected update_customerprohibited alternative

“We just signed a brand-new customer, Northwind — they are not in the system yet. Add them.”

Reasoning: create_customer, update_customer all appear to handle customer requests and none states when to use which; picked the most detailed description.

expected get_customerselected search_customerstool confusion

“Retrieve the full customer profile for cus_1111.”

Reasoning: create_customer, get_customer, search_customers, update_customer all appear to handle customer requests and none states when to use which; picked the most detailed description.

expected NO_TOOLselected update_customerfalse trigger

“Merge customer cus_1200 and cus_1201 into a single record.”

Reasoning: update_customer, create_customer, get_customer, search_customers all appear to handle customer requests and none states when to use which; picked the most detailed description.

Behavioral test results

10 scored + 0 exploratory · temperature 0

expected get_customerselected update_customertool confusion

“I already know the customer ID — it's cus_6398. Pull up the complete customer record.”

FAIL

boundaryHUMAN FIXTUREconfidence 55%

args: missing fields

Reasoning: get_customer, update_customer, create_customer, search_customers all appear to handle customer requests and none states when to use which; picked the most detailed description.

expected create_customerselected update_customerprohibited alternative

“We just signed a brand-new customer, Northwind — they are not in the system yet. Add them.”

FAIL

boundaryHUMAN FIXTUREconfidence 55%

args: missing id, fields

Reasoning: create_customer, update_customer all appear to handle customer requests and none states when to use which; picked the most detailed description.

expected get_customerselected search_customerstool confusion

“Retrieve the full customer profile for cus_1111.”

FAIL

positiveHUMAN FIXTUREconfidence 55%

Reasoning: create_customer, get_customer, search_customers, update_customer all appear to handle customer requests and none states when to use which; picked the most detailed description.

expected NO_TOOLselected update_customerfalse trigger

“Merge customer cus_1200 and cus_1201 into a single record.”

FAIL

negativeHUMAN FIXTUREconfidence 55%

args: missing fields

Reasoning: update_customer, create_customer, get_customer, search_customers all appear to handle customer requests and none states when to use which; picked the most detailed description.

“Find the customer whose email address is deniz@acme.io. I don't have an ID.”

PASS

boundaryHUMAN FIXTUREselected search_customers (55%)

“Add a new customer for Globex — they just signed up.”

PASS

positiveHUMAN FIXTUREselected create_customer (80%)

args: missing email

“Change the billing address on customer cus_1222 to their new office.”

PASS

positiveHUMAN FIXTUREselected update_customer (55%)

args: missing fields

“Show me all contacts we have on file.”

PASS

positiveHUMAN FIXTUREselected list_contacts (55%)

“Find the contact whose email address is priya@northwind.dev.”

PASS

positiveHUMAN FIXTUREselected search_contacts (83%)

args: missing query

“Find the customer with email alex@example.com.”

PASS

argumentHUMAN FIXTUREselected search_customers (55%)

Observed confusion

4 misroutes during behavioral evaluation

expected ↓ / selected →	get_customer	search_customers	create_customer	update_customer	list_contacts	search_contacts	NO_TOOL
get_customer	·	1	·	1	·	·	·
search_customers	·	2	·	·	·	·	·
create_customer	·	·	1	1	·	·	·
update_customer	·	·	·	1	·	·	·
list_contacts	·	·	·	·	1	·	·
search_contacts	·	·	·	·	·	1	·
NO_TOOL	·	·	·	1	·	·	·

Diagonal cells (green) are correct selections; off-diagonal cells (red, bordered) are misroutes. Exact counts shown in every cell.

get_customer vs update_customer33% confused

Likely cause: the names are lexically close; neither description says when it should not be used.

· Rename get_customer to get_customer_by_id
· Add a "when to use" sentence to both tools
· Add a "do not use for …" sentence naming the sibling capability

create_customer vs update_customer33% confused

Likely cause: the names are lexically close; neither description says when it should not be used.

· Add a "when to use" sentence to both tools
· Add a "do not use for …" sentence naming the sibling capability

get_customer vs search_customers25% confused

Likely cause: the names are lexically close; neither description says when it should not be used.

· Rename get_customer to get_customer_by_id
· Add a "when to use" sentence to both tools
· Add a "do not use for …" sentence naming the sibling capability

Potential ambiguity

Risks inferred from static analysis — not observed failures

criticalTC003_OVERLAPPING_TOOLSOBSERVED IN TESTSTools have overlapping definitions

Observed failure: a scored intent was routed to the wrong tool in this pair. These two tools cover related actions on the same object and neither description states a usage boundary, so a model may route intents to the wrong one. This is a potential ambiguity, not an observed failure.

"get_customer" vs "search_customers" · get/search on "customer" · description overlap 33% · no usage boundaries

Fix: Add a "use this when …" sentence to each tool naming the distinguishing condition.

criticalTC014_SEARCH_GET_UNDISTINGUISHEDSTATIC ONLYSearch and retrieval tools are not distinguished

A get-by-id tool and a search tool cover the same object, and at least one of them does not state its boundary. This is the most common tool-selection failure pattern.

"get_customer" and "search_customers" both operate on "customer"

Fix: Make the get tool say the identifier must already be known ("Do not use to search"), and the search tool say it finds records by attributes.

criticalTC003_OVERLAPPING_TOOLSOBSERVED IN TESTSTools have overlapping definitions

"create_customer" vs "update_customer" · create/update on "customer" · description overlap 40% · no usage boundaries

Fix: Add a "use this when …" sentence to each tool naming the distinguishing condition.

warningTC007_MISSING_USAGE_BOUNDARYSTATIC ONLYSimilar tools do not explain their boundaries

Sibling operations on the same object do not explain when each should be used.

"create_customer" and "update_customer"

Fix: Add one "Use this when…" sentence to each tool that names the distinguishing condition.

warningTC003_OVERLAPPING_TOOLSSTATIC ONLYTools have overlapping definitions

These two tools cover related actions on the same object and neither description states a usage boundary, so a model may route intents to the wrong one. This is a potential ambiguity, not an observed failure.

"list_contacts" vs "search_contacts" · list/search on "contact" · description overlap 33% · no usage boundaries

Fix: Add a "use this when …" sentence to each tool naming the distinguishing condition.

warningTC007_MISSING_USAGE_BOUNDARYSTATIC ONLYSimilar tools do not explain their boundaries

Sibling operations on the same object do not explain when each should be used.

"list_contacts" and "search_contacts"

Fix: Add one "Use this when…" sentence to each tool that names the distinguishing condition.

Argument reliability

4/10 generated argument sets were schema-valid (required coverage, no hallucinated keys, correct types and enums).

update_customer: missing fields
update_customer: missing id, fields
create_customer: missing email
update_customer: missing fields
search_contacts: missing query
update_customer: missing fields

Tool-by-tool analysis

get_customer0/2 selected correctly · 5 findings

Gets customer data.

warningTC008_SHORT_DESCRIPTION Description is extremely short — Expand to 1–3 sentences covering action, required context and usage boundary. Do not pad with filler.
warningTC004_UNDOCUMENTED_PARAMETER Required parameter has no description — Describe "id": what it is, its format, and one example value.
infoTC010_GENERIC_PARAMETER_NAME Parameter name is generic — Rename to a scoped name such as "customer_id" or add an explicit description.
warningTC006_AMBIGUOUS_IDENTIFIER Identifier format is unclear — Document the format, e.g. "Customer ID with the cus_ prefix, e.g. cus_9f3k2".
infoTC016_UNDOCUMENTED_OUTPUT Tool output is undocumented — Add an outputSchema, or a final sentence: "Returns the full customer record."

search_customers2/2 selected correctly · 1 finding

Gets customer information from the CRM.

infoTC016_UNDOCUMENTED_OUTPUT Tool output is undocumented — Add an outputSchema, or a final sentence: "Returns the full customer record."

create_customer1/2 selected correctly · 4 findings

Creates a customer.

warningTC008_SHORT_DESCRIPTION Description is extremely short — Expand to 1–3 sentences covering action, required context and usage boundary. Do not pad with filler.
warningTC009_DESCRIPTION_ECHOES_NAME Description merely repeats the tool name — State what the tool does, which inputs it needs, and when it should be preferred over its siblings.
warningTC004_UNDOCUMENTED_PARAMETER Required parameter has no description — Describe "email": what it is, its format, and one example value.
infoTC016_UNDOCUMENTED_OUTPUT Tool output is undocumented — Add an outputSchema, or a final sentence: "Returns the full customer record."

update_customer1/1 selected correctly · 5 findings

Creates or updates customer records in the system.

warningTC004_UNDOCUMENTED_PARAMETER Required parameter has no description — Describe "id": what it is, its format, and one example value.
infoTC010_GENERIC_PARAMETER_NAME Parameter name is generic — Rename to a scoped name such as "customer_id" or add an explicit description.
warningTC006_AMBIGUOUS_IDENTIFIER Identifier format is unclear — Document the format, e.g. "Customer ID with the cus_ prefix, e.g. cus_9f3k2".
warningTC004_UNDOCUMENTED_PARAMETER Required parameter has no description — Describe "fields": what it is, its format, and one example value.
infoTC016_UNDOCUMENTED_OUTPUT Tool output is undocumented — Add an outputSchema, or a final sentence: "Returns the full customer record."

list_contacts1/1 selected correctly · 4 findings

Contact list.

warningTC008_SHORT_DESCRIPTION Description is extremely short — Expand to 1–3 sentences covering action, required context and usage boundary. Do not pad with filler.
warningTC009_DESCRIPTION_ECHOES_NAME Description merely repeats the tool name — State what the tool does, which inputs it needs, and when it should be preferred over its siblings.
warningTC006_AMBIGUOUS_IDENTIFIER Identifier format is unclear — Document the format, e.g. "Customer ID with the cus_ prefix, e.g. cus_9f3k2".
infoTC016_UNDOCUMENTED_OUTPUT Tool output is undocumented — Add an outputSchema, or a final sentence: "Returns the full customer record."

search_contacts1/1 selected correctly · 3 findings

Find contacts.

warningTC008_SHORT_DESCRIPTION Description is extremely short — Expand to 1–3 sentences covering action, required context and usage boundary. Do not pad with filler.
warningTC004_UNDOCUMENTED_PARAMETER Required parameter has no description — Describe "query": what it is, its format, and one example value.
infoTC016_UNDOCUMENTED_OUTPUT Tool output is undocumented — Add an outputSchema, or a final sentence: "Returns the full customer record."

Static schema findings

Deterministic rules — same input, same findings, every run

warningTC008_SHORT_DESCRIPTIONget_customerDescription is extremely short — Expand to 1–3 sentences covering action, required context and usage boundary. Do not pad with filler.

warningTC004_UNDOCUMENTED_PARAMETERget_customerRequired parameter has no description — Describe "id": what it is, its format, and one example value.

infoTC010_GENERIC_PARAMETER_NAMEget_customerParameter name is generic — Rename to a scoped name such as "customer_id" or add an explicit description.

warningTC006_AMBIGUOUS_IDENTIFIERget_customerIdentifier format is unclear — Document the format, e.g. "Customer ID with the cus_ prefix, e.g. cus_9f3k2".

infoTC016_UNDOCUMENTED_OUTPUTget_customerTool output is undocumented — Add an outputSchema, or a final sentence: "Returns the full customer record."

infoTC016_UNDOCUMENTED_OUTPUTsearch_customersTool output is undocumented — Add an outputSchema, or a final sentence: "Returns the full customer record."

warningTC008_SHORT_DESCRIPTIONcreate_customerDescription is extremely short — Expand to 1–3 sentences covering action, required context and usage boundary. Do not pad with filler.

warningTC009_DESCRIPTION_ECHOES_NAMEcreate_customerDescription merely repeats the tool name — State what the tool does, which inputs it needs, and when it should be preferred over its siblings.

warningTC004_UNDOCUMENTED_PARAMETERcreate_customerRequired parameter has no description — Describe "email": what it is, its format, and one example value.

infoTC016_UNDOCUMENTED_OUTPUTcreate_customerTool output is undocumented — Add an outputSchema, or a final sentence: "Returns the full customer record."

warningTC004_UNDOCUMENTED_PARAMETERupdate_customerRequired parameter has no description — Describe "id": what it is, its format, and one example value.

infoTC010_GENERIC_PARAMETER_NAMEupdate_customerParameter name is generic — Rename to a scoped name such as "customer_id" or add an explicit description.

warningTC006_AMBIGUOUS_IDENTIFIERupdate_customerIdentifier format is unclear — Document the format, e.g. "Customer ID with the cus_ prefix, e.g. cus_9f3k2".

warningTC004_UNDOCUMENTED_PARAMETERupdate_customerRequired parameter has no description — Describe "fields": what it is, its format, and one example value.

infoTC016_UNDOCUMENTED_OUTPUTupdate_customerTool output is undocumented — Add an outputSchema, or a final sentence: "Returns the full customer record."

warningTC008_SHORT_DESCRIPTIONlist_contactsDescription is extremely short — Expand to 1–3 sentences covering action, required context and usage boundary. Do not pad with filler.

warningTC009_DESCRIPTION_ECHOES_NAMElist_contactsDescription merely repeats the tool name — State what the tool does, which inputs it needs, and when it should be preferred over its siblings.

warningTC006_AMBIGUOUS_IDENTIFIERlist_contactsIdentifier format is unclear — Document the format, e.g. "Customer ID with the cus_ prefix, e.g. cus_9f3k2".

infoTC016_UNDOCUMENTED_OUTPUTlist_contactsTool output is undocumented — Add an outputSchema, or a final sentence: "Returns the full customer record."

warningTC008_SHORT_DESCRIPTIONsearch_contactsDescription is extremely short — Expand to 1–3 sentences covering action, required context and usage boundary. Do not pad with filler.

warningTC004_UNDOCUMENTED_PARAMETERsearch_contactsRequired parameter has no description — Describe "query": what it is, its format, and one example value.

infoTC016_UNDOCUMENTED_OUTPUTsearch_contactsTool output is undocumented — Add an outputSchema, or a final sentence: "Returns the full customer record."

Recommended improvements

Evidence → problem → change → expected effect

get_customernameconfidence: high

evidence: a boundary test involving get_customer misrouted during evaluation

before

get_customer

after

get_customer_by_id

Why: The original description was too short to disambiguate. It shares an object with search_customers, create_customer, update_customer, so explicit boundaries were added. The rename encodes the required-identifier constraint in the name itself.

Validation: applied as a rewrite candidate and re-evaluated on the identical test suite (see Rewrite validation). Only adopted if it passes the gate.

get_customerdescriptionconfidence: high

evidence: a boundary test involving get_customer misrouted during evaluation

before

Gets customer data.

after

Retrieve one customer by its unique id. Use this tool when the customer id is already known. Do not use it to search by name, email, or other attributes — use search_customers for that.

Validation: applied as a rewrite candidate and re-evaluated on the identical test suite (see Rewrite validation). Only adopted if it passes the gate.

search_customersdescriptionconfidence: high

evidence: a boundary test involving search_customers misrouted during evaluation

before

Gets customer information from the CRM.

after

Search customer by attributes such as name, email, or company and return matching results. Use this tool when the identifier is not known. Do not use it when the customer ID is already known — use get_customer for that.

Why: The original description was too short to disambiguate. It shares an object with get_customer, create_customer, update_customer, so explicit boundaries were added.

Validation: applied as a rewrite candidate and re-evaluated on the identical test suite (see Rewrite validation). Only adopted if it passes the gate.

create_customerdescriptionconfidence: high

evidence: a boundary test involving create_customer misrouted during evaluation

before

Creates a customer.

after

Create a new customer record. The customer must not already exist. Do not use it to change an existing customer — use the update tool instead.

Why: The original description was too short to disambiguate. It shares an object with get_customer, search_customers, update_customer, so explicit boundaries were added.

Validation: applied as a rewrite candidate and re-evaluated on the identical test suite (see Rewrite validation). Only adopted if it passes the gate.

update_customerdescriptionconfidence: high

evidence: a boundary test involving update_customer misrouted during evaluation

before

Creates or updates customer records in the system.

after

Update fields on an existing customer identified by id. Do not use it to create a new customer.

Why: It shares an object with get_customer, search_customers, create_customer, so explicit boundaries were added.

Validation: applied as a rewrite candidate and re-evaluated on the identical test suite (see Rewrite validation). Only adopted if it passes the gate.

list_contactsdescriptionconfidence: medium

before

Contact list.

after

List contact with optional filters and pagination. Use this tool to browse or enumerate; use a search tool for attribute matching.

Why: The original description was too short to disambiguate. It shares an object with search_contacts, so explicit boundaries were added.

Validation: applied as a rewrite candidate and re-evaluated on the identical test suite (see Rewrite validation). Only adopted if it passes the gate.

search_contactsdescriptionconfidence: medium

before

Find contacts.

after

Search contact by attributes such as name, email, or company and return matching results. Use this tool when the identifier is not known.

Why: The original description was too short to disambiguate. It shares an object with list_contacts, so explicit boundaries were added.

Validation: applied as a rewrite candidate and re-evaluated on the identical test suite (see Rewrite validation). Only adopted if it passes the gate.

Rewrite validation

Each candidate re-evaluated on the identical heuristic engine and test suite

NO VALIDATED IMPROVEMENT

No rewrite candidate passed validation on the identical test suite. Every candidate either lowered selection accuracy or argument reliability, introduced a new behavioral failure, or failed to improve overall readiness. Your original definitions are unchanged, and the downloadable JSON contains the originals — no regressing rewrite is shipped as “optimized”.

2 distinct rewrite candidates were generated and each was evaluated on the identical test suite.

Renames + descriptionsBREAKING CHANGE · 1 renamedRejected — regression detected

1 tool rename (breaking):

get_customer → get_customer_by_id

Consumers, prompts, allowlists, tests, analytics, and integrations that reference the old names may require updates.

overall 51 → 69selection 60 → 80arguments 40 → 50distinguishability 39 → 55

· 1 test that passed on the baseline now fail.
· 3 tests that failed on the baseline now pass.

intent	expected	baseline	candidate	change
I already know the customer ID — it's cus_6398. Pull up the complete customer record.	get_customer	update_customer	get_customer_by_id	▲ improvement
We just signed a brand-new customer, Northwind — they are not in the system yet. Add them.	create_customer	update_customer	create_customer	▲ improvement
Retrieve the full customer profile for cus_1111.	get_customer	search_customers	get_customer_by_id	▲ improvement
Change the billing address on customer cus_1222 to their new office.	update_customer	update_customer	create_customer	▼ regression
Merge customer cus_1200 and cus_1201 into a single record.	NO_TOOL	update_customer	create_customer	↔ lateral

Descriptions onlyRejected — regression detected

overall 51 → 69selection 60 → 80arguments 40 → 50distinguishability 39 → 55

· 1 test that passed on the baseline now fail.
· 3 tests that failed on the baseline now pass.

intent	expected	baseline	candidate	change
I already know the customer ID — it's cus_6398. Pull up the complete customer record.	get_customer	update_customer	get_customer	▲ improvement
We just signed a brand-new customer, Northwind — they are not in the system yet. Add them.	create_customer	update_customer	create_customer	▲ improvement
Retrieve the full customer profile for cus_1111.	get_customer	search_customers	get_customer	▲ improvement
Change the billing address on customer cus_1222 to their new office.	update_customer	update_customer	create_customer	▼ regression
Merge customer cus_1200 and cus_1201 into a single record.	NO_TOOL	update_customer	create_customer	↔ lateral

Synthetic-mode caveat: the rewrites and the selection heuristic come from the same system. Treat validated gains as clarity added, not measured model improvement — rerun in live-model mode to measure real improvement.

Methodology

How this score is computed — modes, weights, ground truth, and what “synthetic” means

This report was produced in synthetic mode. No external language model was called; a deterministic heuristic simulator estimated likely tool ambiguity. Results do not represent measured production model accuracy.

Behavioral performance (selection accuracy, argument reliability) measures what the simulator actually did. Definition quality (distinguishability, description quality, parameter clarity, safety) measures the static tool schemas. Overall readinessis the deterministic weighted sum: selection 35%, distinguishability 20%, description 15%, parameters 15%, arguments 10%, safety 5%. No language model invents any number.

Observed confusion (the matrix) is misroutes that actually happened during evaluation. Potential ambiguity is inferred from static analysis and is never labeled an observed failure. Tools that merely share an object noun (e.g. get/create/delete customer) are not flagged as overlapping — only genuinely confusable verb pairs without usage boundaries are, and severity rises to critical only when a real misroute is observed.

Only high-confidence tests (≥ 0.7) are scored; the rest are exploratory. Before/after reruns the identical suite — improvements are only claimed when the second evaluation supports them, and regressions are shown. Results depend on the selected model, prompt format, tool set, and client behavior; they do not guarantee identical production performance.

mode: synthetic · model: toolchoice-simulator-v1 · tokens: 8,238 in / 1,800 out · est. cost $0.0517 · evaluated 2026-07-16

This report was generated automatically and describes simulated tool-selection behavior, not verified production incidents. Server owners can request corrections or removal at hello@toolchoice.dev.