Empirical Analysis of Agentic Retrieval: Knowledge Graphs vs. Relational Schemas in CRM Workflows

Written by: Santhosh Kumar Manavasi Lakshminarayanan, Damon Lin, Sanjay Sriram, and Shriram Sridharan

Benchmarking demonstrates that a 27B parameter open-weight model utilizing a knowledge graph representation outperforms frontier models querying standard relational tables. This performance gap is attributed to data representation rather than model capability. While the Model Context Protocol (MCP) facilitates basic queries (e.g., deal amount, close date), it fails on complex relational tasks such as champion identification or sentiment analysis. These data points often reside in unstructured sources like email and transcripts. Connecting multiple MCP servers to an LLM incurs a "schema tax," where token consumption is dominated by schema ingestion rather than reasoning. Transitioning to a revenue stack - syncing CRM data into a data warehouse for SQL querying - partially addresses this but requires frequent schema migrations and backfills for new fields. Conversely, a knowledge graph serves as an abstraction layer where relationships are stored as edges, allowing for dynamic expansion without altering the underlying table structure.

This study benchmarks the cost and accuracy of knowledge graph retrieval versus raw table scans across multiple model families. Production revenue workflow analysis shows that the relational setup achieved 8.9% accuracy with a mean consumption of 103.6K tokens. In contrast, the knowledge graph setup achieved 99.9% accuracy with 5K tokens, representing a 20x cost reduction and 11x accuracy improvement.

Introduction

Standard agent implementations typically sync Salesforce data into a warehouse (e.g., Snowflake) alongside secondary datasets. LLMs then generate SQL queries based on the provided schema. This approach is effective for primary keys and native joins, such as close dates or opportunity ownership.

Retrieval degradation occurs during cross-source mapping. For instance, email and calendar data lack native account identifiers. Heuristic matching via email domains is unreliable for organizations with subsidiaries, sub-brands, or recent mergers, leading to inconsistent entity resolution.

The graph layer

Static relational schemas cannot accommodate the dynamic nature of enterprise relationships. The knowledge graph setup abstracts these relationships into edges. It preserves the expressive power of the original schema while allowing the insertion of new relationship types as graph rows without schema modification.

The implementation involves two phases: entity mapping and relationship materialization. Entity mapping resolves records from disparate sources (CRM, email, calendar) to a canonical entity. Relationship materialization pre-computes edges at build time. In this study, the graph utilizes six entity types and eighteen edge types, converting relationship identification from an inference task into a graph traversal.

What was already known

Whether a graph representation helps an LLM query enterprise data was first measured by Sequeda, Allemang and Jacob in 2023, on an insurance schema: GPT-4 answering zero-shot over SQL scored 16%, and the same model over a knowledge graph representation of the same database scored 54% [1]. We follow their setup deliberately, virtualizing an ontology over the relational schema and giving each setup a zero-shot prompt carrying its own native schema description.

Recent advancements in agentic loops and native model reasoning require a re-evaluation of these initial findings.

This experiment evaluates 31 configurations across eight model families, incorporating various reasoning and effort settings. The benchmark utilizes production customer queries to ensure real-world relevance.

The experiment

The relational setup and the knowledge graph setup both access the same Snowflake data warehouse. The relational setup utilizes an eleven-table Salesforce schema queried via SQL. The knowledge graph setup uses a virtual ontology served by Ontop [2], where the model generates SPARQL [3] against an OWL vocabulary [4], which is then translated into SQL.

The setups differ by three specific edges: meetingWithAccount, emailWithAccount, and hasChampion. While production environments contain hundreds of such edges, this study focuses on these three to measure the baseline delta. 

The ten questions come from our own revenue workflows and run against our own production CRM, with ground truth hand-verified against the system of record. Five are answerable through foreign keys and five need one of the three edges. We call these keyed and unkeyed questions.

Unkeyed questions were designed to require non-verbatim retrieval. Q7 and Q8 evaluate domain resolution: one account uses a common suffix, and another utilizes unrelated domains. The experiment includes 3,100 runs (31 configurations, five iterations per question). Grading was performed by an independent LLM judge applied consistently to both setups.

Experimental Findings

On keyed questions, both setups averaged 99% accuracy. The knowledge graph setup incurred a 20% token overhead, attributed to ontology description and SPARQL verbosity. This represents the baseline cost of the abstraction layer when query requirements do not exceed the relational schema's capabilities.

On unkeyed questions the delta is very large in terms of both accuracy and token spend.

Performance divergence was consistent across all model families. The highest accuracy for the relational setup was 20% (GPT-5.5), while the lowest for the knowledge graph setup was 98% (DeepSeek).

That 8.9% has structure, because the five unkeyed questions make two different claims.

In questions Q7-Q9, the relational setup failed across all 465 runs because the required account-domain mappings were absent from the metadata. The models were unable to infer secondary domains during query time.

Q6 and Q10 targeted the "champion" role, which is not natively stored in the relational schema. The relational setup reached 38.7% on Q6 due to an annotation coincidence in one contact record, but fell to 5.8% on Q10 where no such hint existed.

Cost analysis indicates that the knowledge graph setup is significantly more efficient. On unkeyed questions, it reduced token consumption by approximately 66%. For tasks requiring exhaustive scans (e.g., champion identification), the cost difference increased by orders of magnitude.

Token consumption is driven primarily by retrieval volume. In the relational setup, failed retrieval attempts lead to multi-turn re-queries, compounding context length without improving accuracy. The knowledge graph setup maintained a stable query count (mean 1.21) across all unkeyed questions.

The graph issues between 1.0 and 1.4 queries on every unkeyed question. Its average of 1.21 sits below its own keyed figure of 1.41, so following an added edge costs less than following a foreign key.

Q10 required identifies champions within a 302-contact account. The relational setup averaged 9.5 queries per run, frequently exceeding context limits. The knowledge graph setup resolved this in a single traversal.

Failure Mode Analysis

Resolution Errors. The relational setup often failed to resolve sub-brands or aliases, leading to significant undercounting or overcounting. For example, filtering on "cloud.com" incorrectly aggregates data from unrelated entities with similar domain suffixes.

Overcounting. Cloud Software Group receives mail at cloud.com. Filtering on it returns 1,491 emails against a ground truth of 883, the excess belonging to unrelated companies whose domains end in the same nine characters, chiefly jumpcloud.com and icloud.com.

Both at once. Blackhawk Network sends mail from blackhawknetwork.com and bhn.com. Search only the first and you undercount, returning 15 against a ground truth of at least 53. Find bhn.com and you overcount, because that domain is shared with a subsidiary.

Inference Latency/Accuracy Trade-off. Without explicit role mapping, agents attempted to infer relationships from transcripts. This method frequently confuses mentions of a role with the actual status of the entity.

The knowledge graph setup mitigates these failures via offline entity resolution. Continuous mapping of domains and aliases ensures that acquisitions and rebrands are reflected in the edge set. This provides a testable, persistent artifact that the model inherits during retrieval.

Stored edges convert complex inference tasks into deterministic traversals. This reduces the cognitive load on the model and ensures that acquisition-driven changes are handled at the data layer rather than the prompt layer.

What compute buys

The data suggests that increased compute (reasoning/effort settings) does not compensate for poor representation. A 27B model on the knowledge graph setup consistently outperformed frontier models on the relational setup, regardless of reasoning settings.

Expanding the effort budget for GPT-5.5 from minimum to maximum increased token usage by 11x (44k to 491k) while accuracy remained stagnant. Frontier models showed no correlation between effort spend and accuracy on unkeyed tasks.

High-effort settings occasionally resulted in performance degradation. Increased search depth into transcripts led models to prioritize narrative mentions over structural signals, resulting in false positives. The knowledge graph setup avoids this by providing resolved "hasChampion" edges, requiring no deliberation during retrieval.

In summary, the knowledge graph setup enables high accuracy at the lowest compute setting. The relational setup fails to reach parity even at maximum effort and cost.

What a prompt cannot carry

It is worth being precise about what the agent is missing, because "it needs more business context" points at a different fix: write the context down, in the system prompt or a skill file, and let the model reason from it. Our setup tests that, and it is not enough.

Both setups received identical business context. Even with instructions to resolve meetings via attendee domains, the relational setup failed. This highlights the limitation of type-level knowledge (provided in prompts) versus instance-level facts (stored in the graph).

Instance-level facts (e.g., specific role holders or subsidiary mappings) are high-cardinality and dynamic. Encoding these into a system prompt is non-scalable and untestable. The knowledge graph setup manages these facts as data, whereas the relational setup requires the model to perform ad-hoc resolution on every run.

So the two layers do different jobs and you want both. The ontology says a champion relationship exists and what it connects. The edge says who it is. Reasoning operates on whatever the second layer supplies, which is why more of it changes nothing when that layer is empty.

Auto-logged activities

Automated activity logging tools often use the same heuristic domain matching as the relational setup, inheriting its failure modes. Moving the heuristic from query time to capture time does not improve resolution accuracy. Entity resolution requires multi-signal processing offline, where results can be validated before ingestion.

Entity resolution is a different operation. It decides which real company a domain, an alias, or a subsidiary belongs to, using more signal than a substring, and it runs once where it can be checked. And the activity link is only the floor. The relationships a revenue team asks about sit higher: who is the champion on this deal, who is the detractor, who is the economic buyer, which subsidiaries roll up to which parent, where a champion landed after a job change. No domain rule produces any of those. In a graph they are edges like any other.

Learnings

Representation defines query expressivity. The accuracy delta between 8.9% and 99.9% is a direct result of the representation layer. Relational schemas limit the model to pre-existing joins, whereas the knowledge graph setup enables retrieval of dynamic relationships.

Compute cannot substitute for data structure. Reasoning is effective for navigating reachable data but cannot recover unrecorded relationships. Increased token expenditure in frontier models yielded no accuracy gains on tasks requiring resolved instances.

Offline resolution ensures traversal reliability. Pre-computing edges transforms complex inference into simple traversal. This shifts the computational burden to the offline build phase, where accuracy can be systematically tested and maintained.

Abstraction layers enable scalable growth. Knowledge graphs allow for the continuous addition of new relationship types as data rows, avoiding the cost and complexity of relational schema migrations.

Silent failures pose operational risks. The relational setup produced plausible but incorrect answers with high confidence. Without structural validation, these errors remain undetected in production workflows.

This study identifies the data layer as the primary bottleneck in enterprise agent development. While multiple MCP servers provide connectivity, the lack of a unified representation layer prevents reliable cross-source retrieval.

Effective agentic retrieval requires a representation layer that grows independently of the underlying schema. Correctness is a property of the data representation; in the absence of structure, agents are forced to rely on inconsistent narrative evidence.

References

[1] Juan Sequeda, Dean Allemang and Bryon Jacob. A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases. GRADES-NDA '24, Santiago, Chile, 2024. https://doi.org/10.1145/3661304.3661901 (preprint: https://arxiv.org/abs/2311.07509)

[2] Diego Calvanese, Benjamin Cogrel, Sarah Komla-Ebri, Roman Kontchakov, Davide Lanti, Martin Rezk, Mariano Rodriguez-Muro and Guohui Xiao. Ontop: Answering SPARQL Queries over Relational Databases. Semantic Web Journal 8(3):471–487, 2017. https://doi.org/10.3233/SW-160217

[3] W3C. SPARQL 1.1 Query Language. W3C Recommendation, 21 March 2013. https://www.w3.org/TR/sparql11-query/

[4] W3C. OWL 2 Web Ontology Language Document Overview. W3C Recommendation, 11 December 2012. https://www.w3.org/TR/owl2-overview/

[5] Anthropic. Model Context Protocol. https://modelcontextprotocol.io