On November 25th, 1915, almost exactly 100 years ago, Einstein presented the final form of his equations (defining the general theory of relativity) to the Prussian Academy of Sciences. Papers were more or less ready but they only appeared in 1916.
Institute for Advanced Study has organized an event, GR at 100, and this lecture by the IAS director and my once co-author (and an independent co-father of our matrix string theory) Robbert Dijkgraaf previously gave this October 2015 talk which is the only one whose video I can find.
The video mentions the Baby Einstein, Newton, the Solvay conferences, some classical thought experiments, principles, and effects we associate with the general theory of relativity, as well some newest results such as the holographic principle at the end. Robbert's presentation is poetic so some wise quotes by Einstein are included.
If we ignore the "partially" independent efforts by David Hilbert etc., general relativity was a result of a pure scientific enterprise by a solitary genius. When he discovered special relativity (and other things) during his miraculous year of 1905, Einstein wasn't satisfied. Newton's laws of gravity were incompatible with special relativity (for example because Newton's gravity was thought to act immediately i.e. faster than light – but more mathematically because such an instantaneous action simply violates the Lorentz symmetry) and after a decade of struggles, he saw the light at the end of the tunnel (his own words) and completed the new picture of gravity in terms of a curved spacetime.
Needless to say, from the beginning, he was trying to describe gravity e.g. in terms of a scalar (modern jargon: Klein-Gordon) field that was meant to "be" the gravitational potential. Similar theories never quite work. After all, we know that the gravitational potential is pretty much a component of a symmetric tensor and replacing it with a scalar is similar to attempts to reduce the stress-energy tensor to a scalar.
Sometimes around 1911 or 1912, at the German University in Prague, Einstein finally appreciated the importance of the equivalence principle. The physical phenomena in a freely falling frame are locally indistinguishable from the physical phenomena in the absence of all gravitational fields because all objects accelerate with the same acceleration in a given gravitational field (equivalently, it's because the inertial mass and the gravitational mass are equal, or proportional to each other with a universal conversion factor). That's great because there has to exist a non-linear transformation of coordinates that locally makes all the effects of the gravitational field disappear.
Einstein realized that this principle was very constraining – capable of instantly killing an overwhelming fraction of his candidate theories – and it has also directed him towards theories that are invariant under arbitrary, general coordinate transformations (or "diffeomorphisms"). The laws of physics in special relativity are only invariant under the Lorentz or Poincaré transformations – the "special" changes of the observer – which is why the original relativity was renamed to "special relativity". Similarly, one needs the more general, nonlinear coordinate transformations and the equivalence of all observers, including the accelerating ones, which is why the "new" relativity became known as the "general theory of relativity".
Once the equivalence principle became a core of Einstein's thinking, the path towards the final equations was wide open. Einstein had to learn the Riemannian geometry – which has been available in the mathematics books since the 19th century – and go through almost all the possible technical mistakes you can think of. For example, before he found the final form of the equation, he had equations where the Einstein tensor was replaced by the "simpler" Ricci tensor. Einstein was no Einstein, either. Well, he was one but not in the normal sense. ;-) The Ricci-not-Einstein equations had to be wrong because the stress-energy tensor was conserved (the divergence equals zero) but if one imposed the same conservation on the Ricci tensor (which was assumed to be equal), it would imply that the Ricci scalar had to be constant (probably zero), which would also mean that the stress-energy tensor would have to be traceless (which it usually isn't) etc. OK, it's some technical detail, you may say. You get instantly the Einstein tensor in the equations if you derive them as the variation of the Einstein-Hilbert action.
His confirmation of the previously observed Mercury's perihelion precession and the successful prediction of the bending light during the 1919 solar eclipse were the early experimental tests that distinguished general relativity from Newton's theory and the theory was quickly accepted. Just to be sure, Einstein didn't need experiments to be sure that Newton's theory of gravity had to lose in the experimental match: it was incompatible with the empirically verified principle of relativity (as implemented in special relativity).
Karl Schwarzschild found the first black hole solution of the theory already in 1917. But the world needed to wait for more than a decade to appreciate the general relativity's implications for cosmology. The Universe can't be static, we finally learned. Instead, it's expanding ("shrinking" was a priori allowed as well, but wrong) and the rate of expansion has to change with time. The big bang theory more or less follows from general relativity.
In the 1960s, people found new experiments that verified the gravitational red shift and other predictions. In the same decade, people began to actually trust the existence of the black holes (John Wheeler coined the new, better name for what had been known as "frozen stars" in that decade). In the 1970s, Bekenstein and Hawking started the research into the thermodynamic and quantum properties of black holes and string theory was identified as a consistent of quantum theory of gravity. The full realistic versions of string theory only began to be studied in the 1980s but that's already a little bit different history.
These days, some of the motivations and "modes of thinking" that Einstein exploited to converge to his right equations seem totally obsolete.
We know that the cosmological constant term that Einstein originally wanted to "erase" as an "ugly modification" is very natural and should be expected to be present. It is indeed nonzero, as observed in the late 1990s, which is why the expansion of the Universe keeps on accelerating. In fact, we are stunned that its numerical magnitude is so tiny. While the anthropic explanation is plausible, it may be wrong in which case we need a new perspective that is "in between" the modern view that implies that the cosmological constant term should be large; and Einstein's old view that it's ugly and should be erased altogether.\[
R_{\mu \nu} - {1 \over 2}R \, g_{\mu \nu} + \Lambda g_{\mu \nu}= {8 \pi G \over c^4} T_{\mu \nu}
\] Equally importantly, we understand Einstein's equations just as the equations of an "effective field theory". There may be corrections such as terms of the form \(\ell^2 R_{\mu\lambda}R_\nu{}^\lambda\) on the left hand side – the higher-derivative terms – and the only reason why they may be ignored in practice is that the coefficient \(\ell^2\) is dimensionful and small (the units of squared length) and its effect on physical predictions becomes increasingly negligible at increasingly long length scales, i.e. at \(L\gg \ell\). The terms in Einstein's equations are the leading terms. The cosmological constant is "even more leading" than the Einstein tensor but because the cosmological constant (the coefficient of the term) is so tiny, it only matters at cosmological scales.
We know that the correct theory doesn't have to be constructed out of the fundamental metric tensor field at all. The effective field theory may look like Einstein's equations even if the exact theory that is valid at shorter (and all) length scales is a qualitatively different beast.
Also, we have kind of returned to the historical status of general relativity which is a "theory of gravity compatible with [special] relativity". While general relativity may be viewed as a "more general theory" and the special relativity is just its limit for the cases when gravity is negligible, we may also agree with many particle physicists and view general relativity as a particular theory of spin-two tensor fields that are added on top of a special relativistic Minkowski spacetime, with the right interactions that guarantee the diffeomorphism (local) symmetry and therefore the equivalence principle (and the decoupling of the unphysical excitations of the spin-two field).
General relativity or something with the same local symmetry is the only effective theory that locally preserves the Lorentz invariance and contains spin-two excitations. Spin-two excitations (gravitational waves and, in the quantum theory, gravitons) are needed if the stress-energy tensor is supposed to interact with itself at a distance.
The equations of general relativity are considered "beautiful" although these days, we may describe its advantages and uniqueness in terms of less emotional and more "provable" words. The beauty really comes from the uniqueness of interactions of spin-two fields that are compatible with the special relativity.
Thanks for those great contributions, Einstein.
General relativity has led to lots of solutions and mysteries, wormholes and time machines. Most of those are probably forbidden by the "full" list of laws of physics although they are "possible" as mere geometries. In particular, it seems likely that the time machines can never arise and only non-traversable wormholes are allowed thanks to the energy condition (and these non-traversable wormholes may also be described via the quantum entanglement in quantum gravity).
Some theorems in GR have been proven – e.g. the singularity theorems started by Hawking and Penrose that imply that the birth of a black hole is inevitably under some conditions. Also, Penrose promoted his Cosmic Censorship Conjecture. I believe it's right to say that this conjecture has been largely debunked, especially in higher-dimensional spacetimes where full-fledged counterexamples seem to exist. But even in the limited realm of \(D=4\) dynamics, the precise wording has to be weakened a lot (generic situations have to be required etc.) for it to be defensible and even this weakened statement seems unjustified.
Penrose's conjecture de facto claims that naked singularities can never arise because classical GR would be desperate – it wouldn't know what to predict about the matter coming from the naked singularity because the answers depend on the extreme physical effects near the singularity, i.e. depend on quantum gravity. But there's really nothing wrong about the dependence of something on quantum gravity! The full dynamics is ultimately described by a theory of quantum gravity and if the classical limit is insufficient to figure out what happens, even approximately, then it's insufficient. But it's no contradiction in any sense.