Introducing ask-web: our in-house web search infrastructure

By: Sarunyu Thongjarast + Shriram Sridharan

Rox Revenue agents answer hundreds of millions of web search questions a week from the public web. Commercial search vendors or frontier labs don’t work for us because of scale, cost or domain-fit. So, we went in-house. Rox’s in-house web search agent, ask-web, sits on the cost-per-accuracy pareto frontier of the hyper-parameter grid when compared to frontier labs and commercial search agent providers delivering 91.3% accuracy at 1.03 cents per query. This is not just a research by-product but has been in production serving dozens of enterprise customers for the past 6 months. We dive deep into the architecture, benchmark results and a few learnings in this post. 

Introduction

Rox builds Revenue agents. Revenue agents automate or augment the end to end revenue life cycle from inbound, outbound, deal management to renewals. These agents continuously monitor both public and private signals to act on. Public signals come from the web, examples being an RFP posted on a government website, a funding announcement, an acquisition, an expansion into a new market. Agents and human sellers use these signals to decide downstream revenue creation activity that can either help with outbound or deal-management. An example would be a company hiring a new RevOps person along with AI transformation for sales in their 10K is a signal for the outbound agent in Rox to reach out to a contact in the company to book a meeting. Similarly, a champion leaving / c-suite change is an important signal for the deal-management agent to surface a risk in the deal. 

Each enterprise customer of Rox has hundreds of thousands of accounts they handle and require prioritization across all of those accounts for their sellers. The prioritization is constant and happens weekly, sometimes daily. Each signal in each account of each customer could span into multiple web search queries. At scale across these deployments, we need to answer hundreds of millions of web search questions a week from the public web. 

Why we went in-house

We started with commercial search vendors, including every provider named in this paper. Three constraints pushed us toward an in-house system:

• Scale. Our query volume was very high relative to external vendor rate limits.

• Cost. External search was expensive at our volume.

• Domain fit. We had to tune and evaluate individual stages of the pipeline for the revenue domain, which closed vendor pipelines do not allow.

So we built our own system using open-weight models and our own architecture. We still use the commercial vendors as fallback when needed.

What we built: ask-web

ask-web is a focused agent for web questions. Its pipeline uses only raw web SERP and raw LLM calls, without tools. This gives us freedom in designing the architecture and lets us optimize for the parameters that matter most in production: low cost, high accuracy, and high throughput. Our queries are narrow and homogeneous: one question shape, one evidence pattern, repeated millions of times. Nothing about them needs an agentic loop that decides at runtime how many searches to run or when to stop reading.

ask-web is a static, single-pass pipeline with five stages:

1. Query expansion. An LLM expands the question into N Google search queries.

2. Search. Each subquery goes to a raw SERP API, returning M candidate URLs with titles and snippets.

3. URL ranking. A second LLM selects the N URLs most likely to answer the question.

4. Crawl. The selected pages are crawled and each is trimmed to X tokens.

5. Synthesis. A third LLM reads the crawl results and outputs the verdict.

Each stage gets the cheapest model that does the job. A small closed model expands queries. Self-hosted open-weight models handle ranking and synthesis, which is where most of the tokens are. Fixed stages also mean fixed request shapes, so traffic piles up into clean batches and we run our own GPUs at full utilization instead of paying someone per call. This is also where the burst problem resolves itself: the workload is latency-insensitive, so bursts do not need to be answered instantly, they need to be absorbed at high throughput, and saturated batches on owned capacity do exactly that.

The benchmark

We built a benchmark of 612 binary company-activity questions from real production prompts, spanning 36 companies and 19 prompt templates. Ground truth is strict. Six frontier providers answer each question independently, three judges review every response, and only unanimous verdicts survive. Of 684 candidate questions, 72 came back inconclusive and were dropped. One hedge worth stating plainly: the providers we benchmark against also served as answer candidates during adjudication, which if anything biases ground truth in their favor, not ours. This paper covers only this binary suite. Rox maintains other revenue-specific evals that require more sophisticated reasoning; those are outside the scope of this paper. 

Results

Rox ask-web reached 91.3% accuracy at an average cost of 1.03 cents per query. Several providers beat us on accuracy, and one of them is close on cost as well. We picked this operating point anyway. The deciding constraints were the ones from the beginning: sustained throughput under burst load, which no external rate limit accommodates at our volume, and control over a pipeline we can tune stage by stage. Within those constraints, ask-web is the configuration that survived a year of variants.

What a query actually costs

The 1.03 cents has no list price behind it. Two of the three LLM stages run on GPUs from our inference partners which bill by the hour, not by the token. To price them honestly, we saturated a single pinned GPU with 64 concurrent requests across 15 different input/output shape mixes, measured what it processed in fixed 180-second windows, and fit a cost model to the results.

A linear per-token model does not fit. Each output token attends to everything before it, so

GPU time grows roughly quadratically with sequence length. The model we fit is fit with non-negative least squares. 

For our synthesis model, the dominant quadratic term is the in.out attention term, and the fitted surface curves up sharply for long-prompt, long- generation requests. The quadratic fit has a 6.2% mean absolute error across the measured windows; a linear-only fit misses by 25.4%. No flat per-token rate describes a self-hosted model. These fitted per-token prices are what the 1.03 cents is built from. The same surface is what tells us a configuration change is affordable before we ship it, because a change that lengthens prompts moves cost along the steep axis of the surface, not along a flat rate card.

Hyper-parameter search to beat our config

The chosen configuration uses 5 subqueries, 5 crawled pages, and 5,000 tokens per page. We perturbed it in every direction we could. A grid over the three hyperparameters gives 26 variants around the chosen configuration. The chosen point sits on the cost-accuracy Pareto frontier: the grid contains more accurate points, but only at higher cost. 

Note: 5,5,5k = 5 subqueries, 5 crawled pages, and 5,000 tokens per page

Nine model swaps across the three LLM stages fare no better. Every swap either reduces accuracy, raises cost, or is statistically indistinguishable from the chosen configuration. The one swap that looked cheaper on paper saved 0.07 cents with no significant accuracy difference, and a cheaper open-weight synthesizer kept failing during evaluation and needed reruns to produce valid output, so it would not have survived production.

Learnings

The strongest available model does not make the best production agent. To estimate the pipeline's ceiling, we replaced each of the three LLM components, one at a time, with the strongest model available. The largest improvement came from the final synthesis step: accuracy rose from 91.3% to 94.1%, and cost rose from 1.03¢ to 15.31¢ per query. Trace analysis showed why. The stronger model forms long chains of thought to infer the answer from the crawled text, and it uses prior knowledge when the crawl is incomplete. Both abilities require a larger model and more reasoning tokens. For our workload, a 2.8 point gain does not justify 15x the cost.

A modular pipeline gives more control on a narrow, repeated task. We use a small model for query expansion, self-host open-weight models for the heavy stages, and tune how much web content each query consumes. That gives our agent a predictable behavior from a fixed workflow, competitive accuracy on company-activity research, low per-query cost at volume, capacity that scales with GPUs, and clear upgrade points as better open-weight models become available.

We will continue to invest in making ask-web better in terms of both accuracy and cost over time. At the same time, the commercial vendors are also pushing the frontier on cost-accuracy and we will learn/use them as needed in our product. We care about delivering the best end to end revenue agent at the lowest cost for our customers and search infrastructure is one piece of the puzzle.