OmegaUse-OfficeVal measures whether LLM agents can automate office workflows at a cost below human labor rates. The benchmark pairs 100 long-horizon office-suite tasks (Excel, Word, PowerPoint) with two economic signals: human labor time and task price proxy. This lets you compare LLM inference costs directly against the cost of hiring a human to do the same work.
The paper (arXiv 2607.27155v1) comes from a team at Baidu and introduces a critical constraint missing from most agent evals: can your agent complete a workflow for less than it would cost to pay someone $25/hour to do it manually?
Why Economic Grounding Matters
Agent benchmarks typically report success rates. You see “GPT-4 completed 72% of tasks” and assume that’s good enough to deploy. But if each successful task burns $8 in API calls and the human equivalent costs $3, you have an agent that works but loses money on every invocation.
OmegaUse-OfficeVal surfaces this gap by tracking:
- Human labor time: Average 2.32 hours per task in the benchmark, derived from practitioner estimates.
- Task price proxy: A dollar value representing what you’d pay a contractor or employee to complete the same workflow.
- LLM inference cost: Token usage across all API calls, tool invocations, and retries.
The benchmark asks whether the agent finished cheaply enough to justify replacing a human.
Task Structure and Scope
The 100 tasks come from real office-suite requests, anonymized and adapted for evaluation. Each task spans multiple steps and often crosses application boundaries:
- Generate a financial report in Excel, then summarize findings in a Word document.
- Extract data from a PowerPoint deck, pivot it in Excel, and create a new slide deck with updated charts.
- Merge data from three spreadsheets, apply conditional formatting, and export a PDF summary.
Tasks average 2.32 hours of human labor. This is not “write a formula in cell B2.” This is “reconcile quarterly sales data across three regions, flag anomalies, and produce a board-ready presentation.”
Long-Horizon Orchestration Challenges
Office-suite workflows expose orchestration problems that short evals miss:
- Cross-application state: An agent must open Excel, read a range, close Excel, open Word, paste formatted data, then return to Excel to update a pivot table. State leaks between tools.
- Retry loops: A formula fails silently. The agent must detect the error, backtrack, and try a different approach. Each retry adds cost.
- Tool call overhead: Opening an application, navigating menus, and invoking macros all require separate API calls. A 10-step workflow might require 40+ tool invocations.
The benchmark tracks all of this. You see whether the agent succeeded and how many tokens it burned getting there.
Cost Instrumentation
The paper measures economic viability by comparing LLM inference costs against human labor baselines. The instrumentation tracks three cost components:
-
Token usage: Every API call logs prompt tokens and completion tokens. The benchmark multiplies these by model-specific pricing (for example, GPT-4 input tokens at $0.03 per 1K tokens, output tokens at $0.06 per 1K tokens).
-
Tool invocation overhead: Each call to open an application, execute a macro, or navigate a menu interface counts as a separate operation. The benchmark logs the tool name, duration, and timestamp.
-
Retry cost accumulation: When an agent detects an error and retries a step, all tokens and tool calls from the failed attempt still count toward the total cost.
The economic viability calculation is straightforward: if the sum of all LLM costs for a task is less than the human labor cost (hours × hourly rate), the agent is economically viable for that task. The benchmark also reports a cost efficiency ratio (LLM cost divided by human cost) to show how much cheaper the agent is relative to a human worker.
This approach exposes a critical deployment constraint. If a task costs $58 in human labor (2.32 hours at $25/hour) and your agent completes it for $4, you have a 93% cost reduction. If the agent costs $60, you have an agent that works but costs more than the human.
Verification and Grading
The benchmark uses code-based verifiers built from fine-grained rubrics. Instead of asking a human to eyeball the output, the verifier checks:
- Does the Excel file contain the expected pivot table structure?
- Are the formulas correct and referencing the right cells?
- Does the Word document include all required sections?
- Are the PowerPoint charts populated with the correct data series?
This is harder than it sounds. Office file formats are complex, and verifying “correct” output often requires parsing XML, checking relationships between objects, and validating formatting rules.
The rubrics break each task into sub-goals. A task might have 15 checkpoints. The agent gets partial credit for completing 12 of them, and the verifier reports which 3 failed.
Benchmark Results
The paper evaluates several frontier LLMs against a human baseline. Key findings:
- All LLMs are cheaper and faster than humans: Even the most expensive model costs less per task than paying someone $25/hour.
- None match human deliverable quality: Success rates range from 40% to 65%, depending on the model and task complexity.
- Cost efficiency varies widely: Some models complete tasks at 10% of human cost but with 50% success rate. Others hit 60% success but cost 40% as much as a human.
The gap is not “agents are bad at office tasks.” The gap is “agents that succeed are cheap, but most agents fail too often to deploy.”
Trade-Offs and Failure Modes
| Dimension | High-Cost Model | Low-Cost Model | Human Worker |
|---|---|---|---|
| Success rate | 60-65% | 40-50% | ~95% |
| Cost per task | $2-4 | $0.50-1.00 | $58 (2.32 hrs × $25/hr) |
| Retry tolerance | Can afford 3-4 retries | Budget exhausted after 1 retry | N/A |
| Failure mode | Hallucinates formulas, misreads rubric | Skips steps, loses context | Requires clarification |
| Observability | Full token log, tool trace | Same | None (black box) |
The economic constraint changes how you think about retries. If a task costs $1 and you can retry twice before hitting the human cost threshold, you have room to build a self-correcting loop. If the task costs $3 and the human baseline is $4, you get one shot.
Deployment Shape
If you want to deploy an office-suite agent using this benchmark as a guide, the architecture looks like this:
- Task router: Classify incoming requests by complexity. Simple tasks (single-step Excel edits) go to a cheap model. Complex tasks (multi-document workflows) go to a more capable model or get routed to a human.
- Cost budget enforcer: Set a per-task budget based on the human labor baseline. If the agent exceeds 80% of the budget without completing the task, abort and escalate.
- Verifier in the loop: Run the code-based verifier after each major step. If the verifier detects an error, trigger a retry with a more explicit prompt.
- Observability pipeline: Log every API call, tool invocation, and token count. Feed this data into a cost dashboard so you can track economic viability across task types.
The key insight is that you cannot treat all tasks the same. A $1 task and a $10 task require different orchestration strategies.
Security Boundaries
Office-suite agents touch sensitive data. The benchmark does not address this directly, but deployment requires:
- Sandboxed execution: Run the agent in an isolated environment. Do not let it access the network or write to arbitrary file paths.
- Data redaction: Strip PII from input documents before passing them to the LLM. The verifier should check for leakage in the output.
- Audit logs: Every tool call and file modification must be logged. If the agent corrupts a spreadsheet, you need to know which API call caused it.
The long-horizon nature of these tasks makes security harder. An agent might read a file in step 3, store a reference in its context, and leak that reference in step 12. You need a state inspector that tracks what data the agent has touched.
Technical Verdict
Use OmegaUse-OfficeVal if your cost-per-task constraint is below $50 and tasks require 30+ minutes of human labor. The benchmark tells you whether your agent can complete a workflow for less than it costs to hire a human. It is most useful for teams building automation products where cost-per-task is a first-class constraint. If you are selling an agent that replaces a $25/hour worker, you need to know whether your inference costs stay below $25/hour. This benchmark gives you that number.
The long-horizon task structure also exposes orchestration bottlenecks that short evals miss. You see where agents lose context, where retries pile up, and where tool call overhead dominates cost.
Avoid this benchmark if your workflows are single-step (for example, “format this cell”) or operate outside office-suite applications. The benchmark is overkill for simple workflows. It also does not yet cover collaboration workflows (multiple agents working on the same document) or real-time constraints (complete this task in under 5 minutes).
For single-agent, cost-constrained office automation, this is the most realistic eval available. Use it when economic viability is a deployment gate. Skip it if you only measure success rate or work outside Excel, Word, and PowerPoint domains.