Why John Stewart Bell has been haunting quantum mechanics for decades

Some people think they have a poltergeist in their attic, some say they’ve seen ghosts on dark nights – I have John Stewart Bell. The physicist’s research and his tremendous legacy have been haunting me for years.

I guess I shouldn’t be surprised. Do you ever think about how much of what we experience as reality is actually, objectively, unambiguously real? I have to, or I couldn’t write about the nature of space and time, and the intricate goings-on in the quantum realm. Bell loved pondering these things too, and his work forever changed how we understand them.

He was born in Belfast in 1928 and was, by all accounts, an exceptionally inquisitive and bright child. He latched onto physics early, landing his first gig as a lab technician when he was 16. He was trained in both theoretical and experimental physics and built much of his career in the world of particle accelerators, where he worked on calculations so complex that we now relegate them to supercomputers. But what really kept Bell up at night were the cracks he could see in the foundations of quantum theory.

Today, this is an established field of physics and many of its practitioners have been featured in the pages of New Scientist – contemporary physics isn’t unfriendly to those who ask questions that sit at the border of physics, mathematics and philosophy. However, when Bell was coming up as a researcher, physicists were still taken by the debates between quantum theory’s first wave of greats – people like Niels Bohr and Albert Einstein – and either considered them settled or thought that what was left was a matter of philosophy rather than physics.

So, Bell only worked on them after hours, almost as a hobby. That changed in 1963 when he and his wife, also an accomplished physicist, took a sabbatical from their accelerator work and Bell used that time to parlay his hobby into a pair of seminal papers. Though they were received without fanfare and were largely overlooked for years, their importance cannot be overstated.

Bell took one line of this philosophical questioning and turned it into something that could be investigated in a lab. It centred on the idea of “hidden variables” in quantum mechanics.

As it was developed by Bohr and his colleagues in the 1920s and 30s, quantum mechanics is no friend to certainty or determinism. Infamously, you can say very little that’s definitive about a quantum object until you interact with it. You can predict what properties it might have upon measurement, but only probabilistically. For example, you may know that an electron has a 98 per cent chance of having a certain amount of energy when you measure it, and a 2 per cent chance of having some other energy, but which one it will actually be is completely random.

How does nature decide which energy to randomly serve up to you? One explanation is that it’s not actually randomness at play here, but that some properties – some variables – are hidden from researchers. If they could just pin down what these hidden variables are, physicists could bring absolute predictability to quantum theory.

Bell devised a test that would eliminate a large swath of hidden-variable theories from competition to replace, or at least amend, quantum theory. This test calls for two experimenters, typically nicknamed Alice and Bob. Pairs of entangled particles are produced repeatedly, then one particle in each pair is sent to Alice, while its partner particle goes to Bob at a faraway lab. Upon receiving their particles, Alice and Bob each independently choose to measure a particular property. For instance, Alice might measure her particle’s spin.

Concurrently, Bob is also making measurements, and choosing how to do them, but Alice and Bob don’t communicate during the experiment. At the end, they plug their respective data into an equation that Bell derived in 1964. This “inequality” equation tests the data for correlations between Alice’s and Bob’s measurements. Even without quantum effects, some correlations may arise by chance. But Bell determined a level of correlation that demonstrates that something else is going on: the particles are correlated in a way that only exists in quantum physics and cannot exist if there are local hidden variables.

In this way, Bell’s test does more than diagnose quantum theory as a better description of our reality than these deterministic, hidden-variable theories – it also zeroes in on the odd property of “non-locality” as something that seems to be a bizarre feature of our reality. Non-locality means that quantum objects can maintain a connection, and that their behaviours can stay inextricably correlated, regardless of how far apart they are. Einstein was a huge critic of this, in part because it was uncomfortably close to instantaneous communication between objects, which is strictly forbidden by his theory of special relativity.

Bell was something of an acolyte of Einstein’s, but the vagaries of physical reality led him to ultimately prove his idol wrong. His test pointed a firm finger towards our world being quantum, something that researchers are still wrestling with today, especially when it comes to the seemingly unbridgeable chasm between quantum theory and our best understanding of gravity as developed by Einstein.

I couldn’t find any mention of Bell actually working on experimental implementations of his test himself, and it long proved to be technologically difficult. While the first such experiment was completed in 1972, it took until 2015 for a test free of loopholes – as rigorous as possible – to finally put the last nail in the coffin of local hidden-variable theories. In 2022, physicists Alain Aspect, John F. Clauser and Anton Zeilinger were jointly awarded the Nobel prize in physics for their decades of work on these experiments.

So why am I still seeing John Stewart Bell everywhere I turn? Have I been subjected to some quantum curse?

The short answer is that his work, and all the experiments that tested it, opened almost as many questions about quantum physics and the nature of physical reality as they set out to answer. For instance, while many physicists agree that our world simply is non-local, some are still trying to figure out exactly which physical mechanism underlies non-locality. Others are working on developing new hidden-variable theories that cannot be stymied by Bell’s test. Yet others are painstakingly unravelling any and all mathematical assumptions that Bell made in his papers from the 1960s. All of them seem to believe that finding some new angle on Bell’s work, or some overlooked intricacy within it, could be a skeleton key for pushing interpretations of quantum theory beyond its current state and perhaps even constructing an elusive theory of everything.

The ripple effects from Bell’s work are everywhere in quantum physics. In fact, we got better at entangling particles simply by trying to do Bell tests over the past 50 years. But that’s just the start. A few weeks ago, I spent lots of time speaking with physicists who found a way to leverage Bell’s work to devise quantum tests for whether free will can be partial, i.e. whether our freedom of choice can be cosmically constrained in some cases but not others. Then, I got on the phone with a different team of researchers, presumably to discuss gravity and the nature of space and time, but ended up talking about Bell yet again. These physicists were inspired by his approach and wanted to devise a test similar to his but for gravitational properties of reality, rather than quantum ones.

This too, I think, is part of why I can’t escape Bell – his ability to turn philosophical issues into tangible tests of reality reflects the allure at the core of physics. The promise of physics is that it can help us chip away at the world’s most confounding mysteries through experiments, and Bell’s test is an incredibly elegant embodiment of that promise.

If I have to be haunted by something, I honestly couldn’t ask for a better ghost.