Introduction to Agentic CPU Testing
Agentic AI has set off a firestorm in the world of CPUs over the past six months, yet it is an area of performance that still lacks solid, repeatable benchmarks to evaluate where the latest hardware falls. An internal set of tests is currently in development to evaluate at least one aspect of contentious agentic loops end-to-end, isolating CPU and memory performance from the accelerator to understand how these components contribute to the agentic loop.
This work remains a progress in the making, aimed at understanding the problem, the testing methodology, and what has been learned about agentic loops along the way. While the test has been prototyped and the central design settled, additional refinement is required before the resulting numbers can withstand broader scrutiny.
Given that this area of computing is bleeding-edge and rapidly evolving, blind spots undoubtedly exist. The testing is structured to examine how agentic performance scales across the CPU as sessions increase, determining an ideal throughput for each chip that accommodates as many agent sessions as possible while remaining interactive.
The evaluation also addresses the shortfalls of this approach, along with concepts for tackling multi-turn agentic flows. These tests are not meant to replace existing testing of local AI, but rather to supplement it as the hardware characteristics become better defined.
Defining Agentic AI Loops
AI agents generally imply autonomous workers driven by an LLM to handle assigned tasks. However, a more technical definition is necessary to benchmark agentic AI performance effectively.
Two key components are vital. First are functions. With agentic AI, the LLM is given access to tools—such as Python functions in popular frameworks like LangGraph. These tools might read or write a file, query a database, run code tests, or execute other defined operations.
Tool calls are written with LangGraph inside Python, identifying the function as a tool the LLM can call using specific decorators and arguments specifying string returns.
This is followed by a description readable by the LLM, prompting it to run build or test commands using utilities like make or pytest. The function is then wrapped and passed back to an overarching trace function to illustrate tool call mechanics.
Beyond coding tools, agents can read, write, and edit files, or search code using regular expressions. Because tasks frequently require calling a tool multiple times or utilizing different tools sequentially, agentic AI operates as a continuous loop.
For a Python project, an LLM prompt to debug a script might trigger a file-listing tool, followed by a file-reading or code-search tool to isolate troubleshooting areas, before executing edit and run commands iteratively.
This constitutes the agentic loop. The LLM receives a prompt, evaluates the requirement, calls necessary tools, processes the returned information, re-evaluates, and potentially initiates further calls.
Starting with Concurrency
Development of the test took place on an Nvidia DGX Spark system featuring 128 GB of unified memory, a Blackwell GPU, and an Arm-based Grace CPU. The model utilized is secondary as long as tool calling is supported and tests can be re-recorded with newer models.
The primary focus is the distribution of time spent rather than specific model tasks. Each result sequence consists of tasks executed against the model, where the majority of time is consumed by the GPU. Even demanding CPU tasks represent a small computational fraction when paired with a powerful GPU.
A closer examination of specific task sets reveals frequent CPU execution windows, though these operations complete so rapidly that they barely register in overall timelines.
Task spacing is uneven. Document processing agents, for instance, expend nearly all agent time scanning and extracting PDF data at the start, followed by pure GPU inference.
While a more powerful GPU accelerates inference, severe CPU-GPU imbalances are required for single-loop CPU bottlenecks to emerge. However, agents spawn sub-agents and run multiple sandboxed sessions concurrently, making resource contention a genuine concern.
Consequently, the benchmark prioritizes concurrency over artificial hardware imbalances. Sweeping across the chip with concurrent sessions handling task loops isolates CPU contention as tool execution, web browsing, and compilation workloads accumulate.
How the Benchmark Works
Custom LangGraph agents execute a sequence of 30 tasks against a llama.cpp server running a specific model. A proxy server handles requests to record execution traces.
During live recording, the proxy passes calls to the server, writing LLM requests and responses to a master JSON file while isolating tool calls into separate JSON logs. Correct responses are generated, hashed, and stored.
Recording enables the live model to be bypassed during testing. Agents operate normally against the proxy, which immediately returns responses from captured traces while maintaining live tool calls.
Incoming requests are verified by session label and hash against the master JSON. Mismatches invalidate the run, while verified responses allow agent sessions to execute necessary tool calls.
To prevent benchmark drift, recorded responses are always returned to the agent regardless of live tool call accuracy, with true mismatches logged and flagged as invalid.
This architecture permits straightforward expansion, allowing new Python personas using LangGraph or alternative frameworks like CrewAI, along with re-recorded traces for updated models.
Testing scales from a single worker process up to the CPU thread count with three repeats per level. Offset start points across worker processes create imbalanced contention as variable-duration tasks overlap.
Measuring Throughput and Performance
The core metric is sessions per second, tracking completed end-to-end tasks at each concurrency level to observe throughput divergence as workloads increase.
Two primary headline scores emerge from the sweep: peak throughput, indicating maximum sessions per second and corresponding concurrency, and ideal throughput, marking the threshold where the slowest 5% of sessions exceed 1.5 times the baseline duration.
By eliminating GPU delay during concurrency sweeps, the test isolates CPU throughput limits, though results must still be interpreted within a realistic system context.
GPU bottlenecks can cause agent queuing, degrading overall processor utilization. The benchmark bridges the gap between theoretical napkin math and complex end-to-end system evaluations.
Live inference sweeps evaluate system-level performance by recording token generation through the llama.cpp server, accounting for prefill, decode, cache performance, and queue delays.
Both metrics offer valuable insights: live sweeps reflect realistic local AI system behavior, while frozen replays isolate maximum CPU throughput.
Initial Agent Personas
Six distinct agent personas are defined via LangGraph Python scripts. Each persona executes unique tasks against local document corpora, such as a research persona querying a 50,000-article Wikipedia database.
Most personas follow a structure comprising a planning stage, a main agent loop, and a final report. Analyst personas split questions into structured multi-step outputs processed in parallel before writer aggregation.
The coder persona differs by addressing software bugs across sample projects, confirming fixes via test execution, and passing changes through a review step.
Current testing evaluates single-turn tasks where agents perform multiple LLM loops and tool calls to resolve a query without retaining previous context across separate actions.
While multi-turn agentic workflows are common, scripting them locally introduces divergence risks and potential context window overflows during complex interactions.
Representative examples illustrate underlying mechanics without exhaustively detailing every individual task.
Task variations include document verification with changing error types and support queries addressing varied error codes, policies, and multi-part knowledge base synthesis.
Limitations of the Current Approach
Current testing is limited to two hardware platforms: a 64-core Threadripper workstation with 128GB of DDR5-6000 memory and an Nvidia DGX Spark system.
Expanding to diverse local AI setups—including Mac Studios and GPU-heavy workstations—will require test adjustments and re-recordings as CPU loads shift with GPU performance and model quantization.
API performance represents another variable. While cloud frontier models can be integrated easily, tools execute locally unless cloud sandboxes are used.
Initial development largely deferred hybrid configurations combining weaker host PCs with cloud APIs due to token costs during setup.
Sandboxing via Docker introduces minimal overhead below two percent, though future exploration of alternative local agent isolation methods remains necessary.
Future Outlook
Given the rapid evolution of agentic AI definitions, hardware, and models, creating a permanently definitive benchmark is impractical. This framework serves as an extensible starting point.
Immediate next steps involve gathering empirical performance figures, including testing free-threaded Python versions to evaluate Global Interpreter Lock impacts within single sessions.
Future expansions will incorporate alternative frameworks such as CrewAI alongside automated single-click agent implementations.
