Intel’s new performance tool casts doubt on benchmark scores

A popular benchmark tool, Geekbench, says it will issue a warning when Intel’s new “Arrow Lake Refresh” desktop chips enable Intel’s new IBOT feature. Why? Because the benchmark vendor can’t be sure that scores reported with it can be considered trustworthy.

Intel’s new Core Ultra 200S Plus desktop processors (also known as the Arrow Lake Refresh generation) aren’t too different than their 2024 “Arrow Lake” predecessors. But one of the key additions is the Intel Binary Optimization Tool (IBOT), which rearranges code in certain applications for more efficient execution, improving performance. IBOT can also run on the Core Ultra Series 3 chips (aka “Panther Lake”).

The problem is, IBOT isn’t well documented… and Geekbench’s developer, Primate Labs, issued a short blog post warning as much.

Essentially, if you test your Core Ultra 200S Plus PC or processor using the Geekbench tool, it will issue a warning as a “hopefully temporary” workaround: “This benchmark result may be invalid due to binary modification tools that can run on this system.”

IBOT’s “black box” needs to be opened, Primate Labs says

The Geekbench warning touches on the big issue that sits at the heart of all benchmarking endeavors. When you use a laptop or a desktop PC, you don’t really have a quantitative measure of how fast or slow it’s running, so you can’t exactly compare performance against other machines. That’s what a benchmark does: it issues a specific number that you can use as a comparative reference.

So, what Primate Labs is saying is two things. Firstly, the benchmark lab isn’t sure how IBOT works, but they’re pretty sure it modifies the benchmark code itself. That means that a Core Ultra 200S Plus chip is essentially running different code than what an AMD processor would run, at least according to Primate Labs.

“Since the tool modifies the benchmark, and it is unclear to both Primate Labs and the general public how these changes occur, results generated with the tool are not comparable to results generated without it,” Primate Labs said.

Secondly, the Geekbench tool can’t see whether or not IBOT has been enabled, which also adds an additional level of uncertainty. Could a Geekbench score be better than what was reported by the tool? Primate Labs, for now, has no way of knowing.

Geekbench, a respected benchmark, is on the right track: Intel shipped the Core Ultra 200S chips without really defining what IBOT is or what it does, prompting us to virtually sit down with Intel executives and ask about it. Unfortunately, we had that conversation before Primate Labs issued this statement. We’ve contacted Intel for comment, and we’ll report back what was said. As Primate Labs indicated, hopefully this is just a temporary miscommunication.

Benchmarking a game, however, should be a bit more straightforward. Our Core Ultra 200S Plus review tests recorded scores with both IBOT on and IBOT off for the small number of games supported by IBOT technology, and you should see the same results. But for the small number of synthetic benchmarks that might benefit from IBOT, it’s possible that, for now, some results may remain in doubt.

Still, the story of the Core Ultra 200S remains relatively unchanged: it’s slightly better than the original Arrow Lake, still behind AMD’s Ryzen 9000X3D chips, but significantly cheaper.