On Dirac: Glory From Perserverance
A physicist, who found triumph amid defeat.
Paul Adrien Maurice Dirac, was one of the fathers of quantum mechanics, and mathematically, balanced upon a ”dizzying path between genius and madness”.
This is Dirac, from the perspective of Einstein, in August of 1926, in a correspondence between him and another star of the field, Paul Ehrenfest, who linked quantum mechanics and statistical mechanics, as well as generated the theory of phase transitions.
Other contemporaries of his would also remark on his incredible reticence, often refusing to speak, or speaking sparsely when he did- Einstein was not the only one would venture as far as labeling him mad for some of his peculiarities.
Known for his mathematical precision, and insistence on mathematical beauty, Dirac was uniquely equipped to approach the problems of the day, due to the skills he picked up as a student in Bristol, and during his continued education. He was educated in technical drawing, the technique engineers used to develop three dimensional representations of devices, without distortion, as well as geometrical drawing, dealing with idealistic shapes, such as cones, and spheres.
The First World War, was also influential, in a rather direct way: due to the number of young men participating in the event, students such as himself, that displayed high aptitude, were filed into higher level classes. One powerful benefit of this was that it gave Dirac his first experience in working in a laboratory, interacting with chemicals, and learning about them on an atomic level, via the recent discoveries of Ernest Rutherford.
Another boon to his education was a teacher who, after furnishing him with a set of complex calculations, surprised at the speed with which he completed them, pointed him in the direction of Riemannian Geometry, which helped equip him to move through the mathematics of Einstein's relativity theories with gusto, as an undergrad engineering student, at a time when physicists proper were so intimidated by the notation involved that they made excuses not to bother with it- Dirac showed no such fear, hesitation or aversion.
He also received a thorough education in electrical engineering, built up from the theoretical foundations in electricity, and magnetism. In the associated laboratories he would deal with hydraulics, observe the varying strengths of different materials, how they responded to pressure, and learn about the technical machinations of industry: combustion engines, steam turbines, pumps, etc.
To round it out, we have Dirac's reliance on projective geometry, which he was introduced to formally, while earning a Mathematics degree, having not been able to assemble the funding necessary to continue his schooling at Cambridge. I point out that this was his formal introduction to the technique, but in fact, it was the same geometry underlying the technical drawing he'd learned in his earlier schooling.
Of course, this is to say nothing of his sheer giftedness, often writing down the solutions to problems in their finished form, or helping professors correct a mistake amidst two chalkboards of equations and calculations.
When we put all of this together, it certainly sounds like the making of one of the most accomplished scientists of the modern world, leaving not much surprise about the fact that the equation that bears his name, gained him a Nobel prize. He provided the relativistic equation of motion describing the wave function of the electron, and out from this fell a few astounding ideas: antimatter, positrons, knowledge about quarks.
Oh, he's also regarded as the father of Quantum Field Theory, and the mathematical notation he introduced, eponymously named, is ubiquitous in Physics and is integral to Quantum Computing.
All of this taken into account, one could come to believe that the pathway for this gifted and well trained individual to stardom was walked rather straightforwardly, however, that would be as far from the truth as Dirac was from being a loudmouth.
You see, Dirac, being a "first-rate man" in the making, in the sense of the Cambridge dons, was under no illusion at this time about the position he found himself in. This was an era ripe with the fruit of science, offering up a bounty of opportunities to make ones name, one that was henceforth unprecedented. This was an era when longstanding beliefs about the nature of reality, put forth by the prior titans of the field, were falling left and right, as a consequence of Max Planck's insatiable curiosity and experimental dexterity- his analyses of black body radiation were the stick that dealt the first fatal blow to the Quantum piñata.
Dirac had something to prove (Maynard Keynes himself recommended him for a research program which funded Dirac for three years), as well as the capability and opportunity, however, what he was missing, was a properly difficult problem that would distinguish him from from the pack, and cement his standing amongst the premiere scientists of the day.
While he did accomplish this, stunningly at that (his work is considered one of the greatest triumphs in theoretical physics- discovering anti-matter through pure reason), his first attempt was stymied, as was his second, third, and, well, allow me to explain.
Generalizing Heisenberg
In 1925, Dirac received a set of proofs, from a paper by Heisenberg, wherein he was challenging some of Bohr's conceptions of the atom, chiefly the location of the electrons surrounding it, and the time it took them to orbit the nucleus.
As far as Heisenberg was concerned, this was a mistake, as he wasn't convinced that any experimenter would ever be able to measure these quantities. However, he knew that would be extremely intensive, perhaps even unwieldy, to describe the movement of an electron in three dimensions, so he settled on one.
This is actually a relatively common technique in Physics, proving things in one dimension, and then generalizing them to higher dimensional spaces, such as 2D and 3D.
While working through the calculations, Heisenberg encountered an oddity: there seemed to be no way to mention a real number describing the location of an electron, but rather what was required was a matrix, the elements of which are pairs of numbers that correspond to properties of the electron, namely its energy levels, while also representing the probability that the electron will jump between two of the energy levels. Because electrons shed radiation as light, during these jumps, the amount of light shed, can be used to calculate the numbers of the matrix. What this means is that Heisenberg's calculations forced him to start speaking probabilistically about events, while giving him a set of mathematical tools that can be used to describe atomic and sub-atomic quantities based solely on the results of measurement.
The standout oddity of this paper, however, for Dirac, was the opportunity to work with non-commutative numbers: numbers that, when multiplied, do not return the same value as they would, had they have been swapped with each other: , is not equivalent to here. Dirac had experience with these numbers, having encountered them while learning Grassmann algebra, and projective geometry during his education.
This was an extreme diversion from mathematical common sense, and for Dirac, presented a key opportunity to begin laying claim to something within the nascent field of quantum mechanics. What Dirac set out to do here, was a favorite pastime of his in mathematical physics: take theories that had not been successfully described relativistically, and try to do so himself. One of the major hangups of the theory, that Dirac was more comfortable with than Heisenberg, was that it introduced non-commutative numbers where, as representative of certain physical quantities: they were common in the projective geometry and Grassmann algebra. To him, these were the keys to making Heisenberg's theory behave appropriately under relativistic conditions.
During one of his meandering walks for which he was also known, he had the idea to inquire about the significance of the difference between the two non-commuting numbers: in short,
Dirac soon realized that this question was rather similar to a mathematical structure, a Poisson bracket, which he remembered related to the Hamiltonian method of describing motion. The Hamiltonian, is a mathematical operator that describes the sum total of energies in a system, both kinetic and potential. He was confident that by using this construct to root out the answer to his question, he could bring some clarity to Heisenbergs theory, the presence of non-commutative properties in it, and relate this all to the expectation of classical physical theory- and he was correct. Upon researching the mathematical construct, he realized he could apply it to the theory, using quantities that correspond to classical numerical quantities, including an equation that I would like you to remember for later on in the article:
This was an extreme departure from typical perspectives of these quantities, and what the equation presented: these everyday properties are now no longer representative of their real world corollaries, but rather entirely symbolic, abstract and mathematical in nature. The next step, for Dirac, however, was to make sure that his new mathematical conception still played by the rules of physics. He did this by showing that his theory had validity, by displaying that it obeyed the law of conservation of energy, and agreed with the recent work of Bohr, who had laid out the nature of atomic electrons, and how they pass between higher and lower energy levels, releasing a quantum of light in the process.
Dirac proceeded then to write a paper, entitled "The Fundamental Equations of Quantum Mechanics", and shipped it off to print, having been given the title of a Fellow within the UK's academy of science, the Royal Society- which shortened the time between submission greatly, from 3 months, to 3 weeks. Heisenberg of course, read this paper, and was deeply enthusiastic about it, calling the work beautiful, and ensuring Dirac of the accuracy of his theory. Of course, Heisenberg could only do this, because the results (the relationship between position and momentum) had already been found by none other than Max Born, the man who coined the term 'Quantum Mechanics'.
Dirac had been handed his first defeat in his attempts of theoretic achievement, having been narrowly beat to print by some of the best physicists of the day, working in tandem, Heisenberg, Pascual Jordan, and Max Born, in Göttingen- but he kept on, knowing that quantum theory was nascent, and that there was still plenty of opportunity for scientific glory.
Generalizing Schrödinger
While Dirac was working on his PhD thesis, attempting to present a compact piece of work outlining his mathematical perspective on quantum mechanics, he realized that the boundaries of the new physical theory were greater than he had previously thought: Schrödinger's equation was traveling around the scientific community, but it was a quantum theory that bore no resemblance to the work of Heisenberg, which was highly predicated on matrices.
Schrödinger's equation describes quanta of matter, according to the waves that were associated with them (photons and electrons, can behave in a wave-like as well as particle-like manner). His achievement here was a stellar generalization of the work of Louis De Broglie, whose wave theory of matter was only applicable to matter that wasn't under the influence of external forces.
As opposed to Heisenberg's theory, which was largely predicated on non-commutative numbers, Schrödinger's theory was based on the equations of waves, which were much more familiar to physicists, being that it was the foundation of undergraduate physics study at the time. In addition to this, Schrödinger's theory provided a much needed and appreciated tool in the understanding of quantum mechanics: it allowed for the visualization of quantum objects and quantum phenomena, in a way that had been assumed moot for the theory, due to it's bizarre implications about the nature of observable quantities.
Schrödinger's theory allowed for the energy levels of the atom to be thought of as the waves that can travel along a piece of rope, where multiple wavelengths can be present on the same rope. While the visualization lined up neatly with the pre-existing mathematics and knowledge of atomic / particle physics, no one, including Schrödinger, was quite certain of what they meant. Nonetheless, the physics community accepted them, because they were easier to work with than the matrices introduced by Heisenberg's theory.
Interestingly, at first Dirac was rather non-plussed by Schrödinger's theory, until being urged by Heisenberg to take it seriously, even though he himself was not the biggest fan of it, and it was a competing theory- honorable, in my opinion. Shortly after familiarizing himself with the work, he realized that he in fact could have derived Schrödinger's equation from his theory, if he hadn't been so focused on generating connections between Quantum and Classical mechanics.
Nonetheless, Dirac still struck out to make a name for himself, because the first version of Schrödinger's equation was only applicable to physical quantities that didn't change over time, providing Dirac an opportunity to generalize the equation, to deal with circumstances that fluctuated, such as the rotation of an atom under the influence of a magnetic field. Unfortunately, again, Dirac would be beat to this discovery, as **Schrödinger also wrote down the same generalized equation **(roughly 4 weeks after the first version), leaving him empty handed once again, having had a treasure unearthed from right below his nose.
Firmly, Fermi.
Having gained a new appreciation for the work of Schrödinger, and narrowly missed out on the opportunity to attach his name to a blockbuster equation, Dirac set out to tackle a problem that had been buzzing in his mind, and the greater mind of the scientific community for some time: a universal law describing how elements and similar substances interacted with each other.
There was reason to believe that Quantum Mechanics, could offer up a solution to this issue, but at the time, in 1926, the theory was not evolved enough to be applicable to atoms with more than one electron. Dirac had learned at Bishop Road, that chemicals were comprised of atoms, and now that physicists had expanded greatly our understanding of it, as well as some of its constituent parts, particularly grappling with the electron, it may have appeared that discovering the rules underlying the relationship between groups of electrons, energy levels of the atom, and subsequently a deeper understanding of chemistry, was within reach.
Particularly, Wolfgang Pauli's exclusion principle, was key in elucidating the processes interwoven in atomic nuclei: atoms have a number of "energy levels", that can be at most occupied by two electrons, meaning additional electrons are forced into higher-energy quantum states, one of the aforementioned energy levels of the atom. It is this very principle that gives us the variety of the chemical elements of our world, among other things, as atoms have varying numbers of available energy levels, which can only contain two electrons, and this makes it impossible for atoms to behave identically, which is what would occur if the exclusion principle wasn't an aspect of nature.
Dirac, at this point in time, was certainly aware of the implications of this principle, but wasn't satisfied with the level of understanding that it gave, knowing that the theory couldn't describe the chemical processes he observed at Bishop Road, nor could it be applied to atoms with more than one electron (a fact that stymied many physicists, in many problems, as all but the simplest elements have more, with hydrogen having one, and helium having two). He was considering, what the relationship between the waves of Schrödinger's theory and heavy atoms, while mulling over the point Heisenberg had made, that quantum mechanical theories should be architected in terms of the observable quantities that come about as a result of some measurement of a quantum system.
Particularly, he presented himself with the question, a minor gedankenexperiment, if you will, concerned with whether or not an experimenter would be able to ascertain a difference between the Schrödinger waves given off by two electrons in an atom, if those electrons swapped places, and concluded that, in fact, the waves would be identical, because the light released from the atoms, would be identical, and that light, of course, is our observable quantity here.
In a few pages of mathematics, Dirac algebraically laid out how groups of electrons share energy as they fill in the available energy states of an atom- this very same mathematical formalism is alive and well to this very day, in use by researchers who focus on semi-conductors, metals, and the flows of heat and electricity through them, which is determined by the electron content of the material- of course, the industrial applications of his work were never of any real significance to Dirac, who was satisfied to have developed the understanding that the wave describing a group of electrons change signs if two electrons move places, but for photons, there's no change in the nature of the Schrödinger wave that describes them.
Having no doubt that the work he'd done was of a rare quality, and having been toying with this idea for a long time no doubt, since his first introduction to quantum theory in Bristol, in lectures by Arthur Tyndall, (however at the time, Dirac couldn't be quite persuaded to focus on that over his first love in physics, relativity theory, particularly special relativity), which introduced him to the problem of black body radiation, the phenomena that required Planck to quantize some values, particularly energy, was as of yet unexplained in terms of quantum mechanics- but Dirac had found a way to do just that very thing. To his dismay however, he had yet again been beaten to print, and this time, he wasn't even close: Enrico Fermi, who would go on to be the associate director of the Manhattan project, had released a paper with a different mathematical technique for explaining the phenomena of energy transfer between groups of electrons, roughly 8 months earlier, which delivered the same results as Dirac's paper.
Roughly a month after the release of his paper, Fermi reached out to him, perhaps a bit sourly, wanting to notify him, that he had already solved the problem, assuming that Dirac probably had not seen it, given that this was an unusually active and productive watershed era in theoretical physics.
That being said, today, the descriptions of groups of electrons are associated with Dirac, as well as Fermi, often referred to as Fermi-Dirac statistics, however this name was, at the time, not a thing at all, and thus, was probably heralded more as the work of Fermi. Another interesting anecdote about this is that Pascual Jordan is noted to have developed these same statistics independently a year earlier than Dirac, or Fermi, but the results weren't published in a timely manner, because his adviser at the time, Max Born, left it at the bottom of one of his suitcases, forgotten for a number of months, during which the year changed and the Fermi & Dirac papers were released.
Passed by Pascual, A Hint of Heisenberg
Following this, in late 1926, Dirac found himself splitting a year of research between Göttingen and Copenhagen, utilizing scholarship funding he'd been awarded in the prior year, finding himself in the tutelage of Bohr in the latter city, and working with Heisenberg and his colleagues in the former. It was an environment that seemed to be rather opposed to his nature, being rather social and informal, the intention of Bohr at play, leading to games of ping pong and trips to the cinema.
It was quite obvious to researchers, experimentalists, chemists and the like, that the quantum theory was in full swing, and being built by them- here was a once in a lifetime to make a discovery that could justify etching ones name with Maxwell, Newton, and more recently, Planck and Einstein.
At the time, one of the biggest issues presenting itself to physicists of the time, was the meaning of the symbols in their equations. Max Born had found a way to interpret Schrödinger's waves usefully, but this required an appalling giving of quarter in the logical intuition regarding the phenomenon he was studying (particles): that the future state of any particle can always be predicted. This knowledge falls out of the nature of classical mechanical physics allowing you to predict the state of some system at any point in the future, if you have the accurate initial conditions of that system, and know the dynamical laws under which the system can evolve.
Particularly, Born was considering how an electron would scatter upon impacting a target, realizing that one can not ascertain before hand where the electron will be deflected to, but rather one can calculate the probabilities of the electron deflecting by an arbitrary angle. Going one step further, he insisted that should a Schrödinger wave describe an electron, one can calculate the probability of detecting it in any tiny region, and that the wave itself has no physical existence, but rather solely a mathematical existence that allowed for calculating the likelihood of some phenomenon in the future, a sharp blow to the determinism put forth by Newton.
Thinking intensely in a library in Copenhagen, Dirac was looking for the translation key between the theories of Schrödinger, and the waves that it produced, and the work of Heisenberg, whose theory produced the same results, yet with an entirely different notation. Impressively, Dirac managed to perceive the rules that would allow for that very translation, and providing a higher level of clarity into the Schrödinger equation than had previously been available. In his theoretical conception, these waves were the necessary quantities for transforming a description of some quantum, based on the energy values, to a description based on the possible values of the position of the particle. Not only that, but it provided support for Born's statement that the wave does not predict future values of some phenomena, but rather the probability that any of a set of possible values would be observed.
It was within this realization, that something extremely important became obvious to Dirac: the amount of information any experimenter can have about the behavior of some quantum is limited, stating that "...one cannot answer any question on the quantum theory which refers to the numerical values for both...", referring to the position and momentum of a quantum. He also noted, inversely, that one could only expect to be able to suitably answer questions that only contained one of these values.
This, in almost excruciatingly plain language, is a description of none other than the famed uncertainty principle (also referred to as the indeterminacy relation)- it's hard to imagine how much closer to claiming the discovery of this principle an individual could have gotten, without realizing it.
This is also why I wanted you to remember that equation from earlier: , is just a painfully short jump from the uncertainty principle: .
Truly, Dirac was in high spirits as a result of this work, having been particularly pleased at solving the problem of showing the synthesis of the theories at play, especially given that it was the result of one of his favorite techniques for solving problems in quantum mechanics, by drawing out subtle and sneaky classical analogues to the quantum phenomenon. Having been a long time fan of Hamilton, he was quite aware of the ability of Transformation theory to relate disparate representations of the same event, and used this to advance the work of both Heisenberg and Schrödinger.
To top it all off, not only in this process did the uncertainty principle sneak by him, but again, the process of printing stymied his claim to theoretical fame once more: Pascual Jordan, had solved the exact same problem, with the news reaching Dirac in the autumn of 1926, and while Dirac's paper probably contained the more elegant formation of the relationship between these two descriptions of quantum behavior, he wasn't able to plant the first flag in that ground. Dirac would yet again, have to suss out an opportunity to cement himself, though by this time, there was no shortage of top tier admirers of his ability, even if they didn't quite understand it, one such admirer was Einstein, and it was at this point in Dirac's career that Einstein uttered the quote about his balancing on a "dizzying path between genius and madness".
Tables turn, Electrons spin
Whilst working in Copenhagen, Dirac began to build the concept that now represents our understanding of nature at the most fundamental level: quantum fields. While fields are not new in physics, having been presented by Maxwell's electrodynamics, which posits a magnetic and electric field, as well as Einstein's theories, which built into the universe a gravitational field, all of which bear one of the key properties of classical phenomena: continuity.
As per the nature of continuity, there should be fluid variation between values, no 'jumps', so to speak, which is in direct contradiction of the results of quantum theoretic calculations, that put forth a discrete, comminuted picture of nature at the tiniest scales. Dirac's goal now was to resolve the contradiction between the two, which was haranguing physicists, who were looking for a univocal representation of the inner workings of nature. Perhaps he was aware of the potential of answering this question, perhaps he was just following his natural inclination to search for the deep truths of physics, but he was about to rewire our understanding of physics forevermore.
The first thread Dirac pulled on, was an old one, the thread that allowed him to generalize the Schrödinger equation, which was taking quantities that were assumed to be regular numbers, and treating them as if they were non-commutative. From this launch point, Dirac was able to mathematically define the creation and destruction of photons, from their creation throughout the universe in various events, from the large thermonuclear processes in stars, and the ejection of high energy photons, often referred to as gamma rays, from atoms, to the destruction of these particles by the human retina, or plants, during photosynthesis, wherein the energy of photons becomes a source of sustenance (as opposed to a source of information, by the human eye). The issue at hand was that classical electrodynamics provided no way to speak about things coming into existence out of nowhere, or disappearing out of existence. Dirac found a way to describe these processes, associating these processes with operators, mathematical structures that can be applied to quantum states to yield observable quantities, as quantum states are not directly observable themselves.
Extending his novel conception, Dirac ventured further, realizing that these operators, for the creation and annihilation of quanta, corresponded to excitations, and de-excitations of the quantum field underlying them, at particular points in space and time. Dirac was aware he had done something magnificent, having put an end to the consternation over the contradictions between the wave and particle descriptions of particles. The next step for him, was to test his theory against the facts presented by earlier, tried and true theories, and he chose the work of Einstein- a bold stick to pick from the pile of things by which to measure one's work.
The theory Einstein generated roughly a decade earlier, allowed for the calculation of the rate of emission of photons from atomic electrons, as well as the rate of absorption, as well as putting forth the concept of stimulated emission which allowed for the excitation of an atom, from its ground state, to a higher energy state, via the application of a photon, the energy of which must equivalent to the difference in the energy of the ground state, and the excited state. Due to the fact that atomic electrons do not stay in high energy states forever, the effect of the application of a properly energetic photon, is that a photon will be ejected from the atom as it leaves this state, as well as the incident photon that caused the excitation in the first place.
Dirac's theory, produced the same results as the formulae set forth by Einstein, as well as being a more general and elegant way of speaking about these processes, a remarkable feat: he had one upped the "uncrowned king of physics". Not stopping here, he went on to extend his work from not just photons, to electrons, and of course, stemming from his love of relativity theory, he desired to put forth a relativistic, quantum field theoretic description of electrons. Having come across Bohr at the Solvay conference of 1927, he was asked what he was working on by the senior Dane, a physicist of great renown himself, he mentioned his relativistic endeavors in understanding the electron, only to be rebuffed by Bohr, who stated that another physicist, Oskar Klein, had already solved this problem- however this time, unlike any other time before, what Dirac had up his sleeve was not only advanced far beyond the current understanding of the electron, it was beyond anything anyone could have expected.
Dirac, had found, and laid claim to something that no one else could. This time, he would not be thwarted. At this point in time, the electron was still being conceptualized as a point particle, despite the addition of the wave description, the particle description still had utility. It was this particle description, however, that lead to an issue which Dirac could not let go: the theory put forth by Oskar Klein, allowed, at times, for an electron to have a probability of being located in some region of space, that was less than zero.
Realizing that the solution to this would be hard to stumble upon, what he chose to do, was to decrease the size of the solution space, giving himself constraints based on the characteristics he felt the correct equation should have. The first two constraints he set forth for himself, were that the equation would have to reproduce the results of relativity theory correctly, presenting space and time on equal footing, and that it would have to agree with his transformation theory, which linked the work of Schrödinger and Heisenberg. The third constraint was that the description of a slow electron, compared to the speed of light, had to fall in line with the predictions made by quantum theory in general.
While this was helpful in narrowing down the options, it still left a practically infinite number of equations that could be written down, forcing Dirac to rely solely on his intuition going forward, to target the proper equation. Moving forward, he expected that the equation would be simple (elegant), featuring the momentum and energy of the electron itself, as opposed to squares, or roots of these values. Remembering that he and Wolfgang Pauli had independently found ways to describe electron spin, using matrices, a structure that he knew very well could have a prominent role in the equation he was searching for.
In his dedication, and persistence, trying out equation after equation, Dirac hit upon the one that would revolutionize physics in a way that would require all of his peers to reconsider what they thought about nature at the fundamental level. He had found an equation that eschewed using Schrödinger waves, and presented an entirely new type of wave, that had four sections, coupled together. While Dirac had put forth an elegant piece of mathematics, he wasn't in the clear just yet- it had to conform to the results experimentalists had set forward about the magnetic field and the spin of the electron- it wasn't enough for the mathematics to be beautiful, it had to relate to the physical world as it was known to behave. Setting out a few pages of mathematics, he showed that his new theory described the spin, mass, and magnetic field of a particle, which was identical to an electron.
He had done it.
On New Years day, 1928, Dirac, via Fowler, sent to the royal society, a paper entitled "The Quantum Theory of the Electron, clearing up some minor details a month later in a second paper. Dirac sent a communique to Born, only referencing the paper in post-script, who considered it 'an absolute wonder', who then showed it to his colleagues Jordan and Wigner, who were also working on the problem, absolutely blindsided that Dirac had found the solution on his own- a soloist endeavor with symphonic results. After many an attempt to procure for himself placement amongst the greats, he'd beaten all of his contemporaries to the treasure, and solved a master level problem in quantum theory.
Trouble in Paradise
While the general consensus on what is now called the Dirac equation, in its early days, was that it was a work of theoretical genius, there was, as Einstein said of his own theory, an issue of low grade wood, plaguing the fine marble: according to the new theory, which was in alignment with special relativity, the mathematics predicted that an electron could have negative energy levels, due to the general equation for the energy of some particle specifying the square of the energy itself. The complication is simply that, the square root of the value didn't have to be positive, due to the fact that squaring a negative number yields a positive one.
Usually, such absurdity could be ignored, or hand waived away, but in the context of quantum theory, it begged explanation, as it allowed for a positively energetic electron to jump into the negative energy state- but no such behavior had been observed experimentally. This particularly irked Heisenberg, who found the idea of negative energy levels to be truly absurd, and would go on to be one of the main critics of the equation, pushing Dirac to provide a solution, or relegate it to the pile of "almost were" equations in physics.
Adding to the issue, was the fact that the Dirac equation only agreed with experimental results if the electron possessed these 'negative energy states', which mean that the success of the equation rested solely upon Dirac's ability to explain this prediction of his theory. Despite the growing concern, and the oddity of the theoretic predictions, Dirac was not swayed, and marched forward sternly, right into the answer he needed. It struck him that if one was to consider all of the electrons in the universe, filling in energy states, common sense dictated that the negative energy states would be occupied first, meaning that there were no available negative energy states for a positive energy electron to jump into, a subtle application of Pauli's exclusion principle, which indicated a negative energy state could only hold one electron.
From this, he extrapolated the idea that the universe is awash with these negative energy electrons, a sea, if you will, of constant density, and unfaltering uniformity, meaning that only a departure from this consistency could be observed experimentally. It became clear to him that there would, however, be occasionally vacant energy states of the negative energy electrons, which would be filled by ordinary electrons, resulting in the emission of a small bit of radiation, meaning one could be detected if a positive energy electron jumped into it. Additionally, according to this conception, these vacancies, which were coming to be known as 'holes', indicate the absence of a negative energy electron, however, this very absence, indicates the presence of positive charge- leading him to the conclusion that this hole, is in fact itself, a proton.
As confident as Dirac was in his ideas, there was a growing choir of critique regarding the negative energy electrons, and the sheer audacity of assuming that an unobservable type of electron was not only extremely abundant, but was more abundant than normal electrons. Dirac found himself against the ropes, with nothing left to do but the seemingly impossible. A particularly intense moment in this situation was a talk entitled 'The Proton', attended by Delbrück and Landau, who were looking to see whether or not he had made any progress regarding the 'hole theory', and upon realizing that he as of yet, had not, Landau sent a single word telegram to Bohr regarding the talk and it's contents, referring to it simply as 'crap'.
Amidst his work, Dirac found a fruitful aside, in his attempts to make sense of electrical charge, realizing that quantum theory in fact allows for magnetic poles that don't come in pairs, but in fact exist singularly, now known as magnetic monopoles. After the experimenter Millikan demonstrated that charge necessarily exists only in discrete integer multiples of the electrons charge. Working with electrodynamics and quantum mechanics, he'd extended the range of the theory to include a new particle, without introducing any contradictions, finding a way to relate the smallest possible electric charge, with the smallest possible magnetic charge.
Dirac was able to put together two useful inferences from this work: a monopole, would likely be found sporting a quantized magnetic field, and that the strength of the force connecting two monopoles was roughly five thousand times the force holding electrons around atomic nuclei, which could be the reason no monopole had been observed. The second was that, the existence of a monopole, would provide an explanation to the quantized nature of electrical charge.
Here, Dirac made a grand leap into the area of theoretical predictions, being lambasted by Oppenheimer and Weyl that the hole must necessarily have the same mass as the electron.
Having no other option, he posited that the hole might actually be representative of the existence of a heretofore undiscovered particle: identical to electron except for in the value of its charge, he presented the idea of the anti-electron. He also mapped this idea onto the proton, moving away from the more familiar idea that the proton and electron had a deeper relationship than being constituents of the atom, he suggested the proton has its own own negative energy states. In a display of theoretical prescience, boldness and creativity, he'd laid eyes on the the antiproton, and positron.
On the tail of this discovery, Dirac laid out his personal ethos for doing Physics, urging his peers to eschew the scientific tradition of experiment in physics, in favor of seeking more powerful and expressive mathematical foundations for their theories. Having presented these ideas to the scientific community, he suggested that they be regarded not as mathematical constructs, but as physical phenomena, and proffered the possibility that high energy experimental physics, would reveal these very particles, however he was skeptical about the time it would take for experimental physics to be able to generate the required amount of energy to bring photons to high enough speeds for the collision to produce the desired results.
Perhaps a result of Dirac's monomaniacal focus and insouciance towards experimentation, he managed to miss the fact that there was in fact a source of energy high enough that could have been used to prove his prediction: cosmic rays, sporting energy several orders of magnitude higher than the particles ejected from atoms, the subject of a number of articles in the New York Times. Truly an example of the beauty of the collaboration between experimentalists and theoreticians, while Dirac was quietly waiting for the day that experimental physics, as he knew it, would be able to generate the energies necessary, an up and coming experimentalist, Carl Anderson, looking to put himself on the map, was preparing to prove him right.
Anderson was working with cloud chambers, which make visible the tracks of electrically charged particles as they travel through a cloud of water vapor, in the presence of a magnetic field. The density of the droplets in a given track, along with the deflection of the track, allowed for the calculation of the charge and the momentum, of the particle. While analyzing the tracks of present in some cloud chamber images, Anderson stumbled upon an anomalous set of tracks, which indicated the track of an electron, albeit one with a negative charge, based on the direction of the deflection present in the tracks, along with a positively charged particle that appeared in tandem, as a result of a cosmic ray striking a nucleus in the cloud chamber.
Presenting these to Millikan, and Blackett, resulted in their realizing that something of importance, in fact, did reside in these photos, and they realized this was an opportunity for an experimental talent to make contributions to a field, as well as impose some additional order, and clarity upon the situation.
Another boon to Dirac's theory, albeit a tad bit indirect, was that elsewhere in the Cavendish lab, another experimentalist, was hunting down the opportunity of a lifetime: James Chadwick was hot on the heals of finding proof of Rutherford's neutron, a particle he had predicted some time before, but that had yet to be provably observed in nature. He seized on the disbelief of two French experimenters, Frédéric Joliot and Iréne Curie, who had been firing helium nuclei at beryllium targets, they found that particles were ejected, which had no charge, and roughly the same mass of the proton- but assumed they were photons.
Chadwick wasn't satisfied with this, believing that in fact, that those particles were neutrons, and set out to attain proof of this. After three weeks of intensive experimentation and calculation, poring over radioactive samples and configuring apparatuses, Chadwick was convinced, as well as Rutherford, the lead experimenter he was working under, that his results conclusively indicated the existence of the neutron, shipping off a paper to Nature magazine, and providing deep support for the idea that there were other subatomic particles yet to be discovered. This discovery had major implications for its cousin, the anti-electron, as well as Pauli's neutrino: they very well might not be mathematical fictions, but true constituents of nature.
Shortly thereafter, perhaps a year or so, still mulling over his anti-electron and hole theory, on the coast of Crimea, Dirac became the target of fate: Carl Anderson, locked away in a Caltech lab, working through his vacation, was cycling through photo after photo of cloud chamber tracks, the large majority of them leaving him empty handed, until one of them presented a powerful anomaly. He found himself looking at a small track, almost as thin as a hair, short, with the density that was indicative of the presence of an electron, however, the curvature of the track, as a result of the particle being deflected by the magnetic field, was the inverse of the directional deflection displayed by electrons, indicating that the particle was, in fact, positively charged.
Shaken and invigorated, Anderson did not let himself fall into sloppiness, under the weight of excitement and potential recognition, and proceeded to thoroughly check his equipment, particularly the alignment of the magnets, to make sure that he himself, nor others, had not spoiled the experiment with human error, or mischief, which was not entirely uncommon even in such a serious scientific community and environment. Desiring certainty, Anderson looked for more examples of the type of track he had observed, but was only able to gather two similar tracks, neither of which were of the same quality as the first. Nonetheless, he rather conservatively presented a paper, "The Apparent Existence of Easily Deflectable Positives",
Despite the potentially undiscovered particle matching the requirements of Dirac's anti-electron, Anderson wasn't actually aware of what he had done, being too caught up in assisting Millikan with explaining the nature of cosmic rays, though he'd attended several lectures by Oppenheimer on Dirac's hole theory. Not only that, but Oppenheimer himself missed the connection between the work of his peers, having written to his brother that he was 'worrying about Andersons positive electrons'.
Following this, Dirac himself, while at a lecture presented by Blackett and Occhialini, two experimental physicists who were starting to generate a very large amount of excitement in Cavendish for their new way of getting cosmic rays to 'photograph' themselves, by setting the cloud chamber to be triggered by geiger counters on the top and bottom of the chamber, devices that were responsive to the cosmic rays as they entered the chamber. When they presented their findings, Dirac was in the audience, but in his caution, he still did not press the fact that his theoretical prediction was correct. Additionally, Dirac's peers were not entirely comfortable with his hole theory, and lacked the conviction in its ability to predict particles necessary to realize what was unfolding before their eyes.
All of this aside, Blackett and Ochiallini, were confident that they had spied Dirac's anti-electron, and sat down with him, doing meticulous calculations, bouncing between the predictions of Dirac's theory, and the contents of their cloud chamber photographs, featuring both positive and negatively charged particle tracks, eventually realizing there was no other explanation except that they had found proof of the anti-electron.
Thus began falling over the necessary dominoes.
Blackett gave a speech about the rather sensationally named, cosmic rays, showing picture after picture to an insatiable public, mentioning that the contents of these agreed thoroughly with the predictions of hole theory. The ball was in the court of the physics community, with Rutherford and Bohr remaining suspicious, having not been convinced of the theory, and thus the subsequent prediction, however the fuel for this was drying up rapidly. However, by the end of 1933, there was no room for physicists to maneuver, and the community accepted the creation of positron-electron pairs, as a result of cosmic rays striking atomic nuclei, and by 1934, it was solidified, due to the increasing number of cloud chamber tracks that matched the original ones observed by Anderson: positrons had been detected in labs all over the world, roughly thirty thousand, in '34. Experimentalists had even gone on to create these particle pairs out of the vacuum in labs, no longer requiring cosmic rays, having found a way to bring about the same phenomena with radioactive materials.
Eyes on the Prize
Having laid out the theory that predicted the positron, and having experimentalists seduce agreement from nature, Dirac was about to receive the honor of a lifetime, and was notified by the grapevine, receiving a call from Stockholm. He was slated to receive a Nobel, with Schrödinger, for their discoveries and contributions to theoretical physics. While certainly not one for the fanfare he knew was coming, so much so that he considered a refusal, until Heisenberg pointed out that would only increase the publicity, Dirac nonetheless resolved to accept the prize and deliver the necessary speech, and became the youngest theoretical physicist to receive the prestigious award, at the age of thirty one.
There's something to be said, for me, about the fact that, despite being dealt a number of heavy handed defeats, by people who, while certainly having been lionized greatly by time, where absolutely mountainous in their stature when they were his colleagues. I would not venture so far as to say nature rewarded Dirac for his patience, and fortitude, but I do think the denouement is poetic, cinematic even, and most importantly, deserved.
Having faltered certainly in his beliefs, ideas, and perspectives, at times, he never wavered in his focus, on his path forward into the unknown of science, aided by precision and reason, toward his eponymous equation.