The discovery of the Heartbleed bug sent service providers scrambling to patch their versions of OpenSSL and customers to change their compromised passwords. The affect was so widespread that Heartbleed is widely considered as the worst security bug ever to hit the Internet.
As security expert Bruce Schneier wrote, "'Catastrophic' is the right word. On the scale of 1 to 10, this is an 11."
Almost as devastating, however, is the blow Heartbleed has dealt to the image of free and open source software (FOSS). In the self-mythology of FOSS, bugs like Heartbleed aren't supposed to happen when the source code is freely available and being worked with daily.
Or, as Eric Raymond famously said, "given enough eyeballs, all bugs are shallow."
Tired of FOSS's continual claims of superior security, some Windows and OS X users welcome the idea that Heartbleed has punctured FOSS pretensions. But is that what has happened? To what extent does Heartbleed challenge or re-affirm FOSS' belief that it represents a superior method of software development?
The Original Statement
Raymond made his famous statement in his 1999 book The Cathedral and the Bazaar. A comparison of proprietary and FOSS methods of software development, the book summarizes the beliefs of many FOSS developers – then and now – about why their work habits are supposed to produce higher quality software with fewer bugs.
Implicit in the description is not only the idea that peer review can substitute for software testing, but also that no special effort is needed to detect bugs. Simply by going about their business as developers, FOSS project members are likely to notice bugs so that they can be repaired.
This claim has not gone unchallenged. It is a statement of belief, not the conclusion of a scientific study, a rationalization of the fact that peer review in FOSS has always been easier than software testing. Moreover, in Facts and Fallacies about Software Engineering, Robert L. Glass claims that no correlation exists between the number of bugs reported and the number of reviewers.
Yet despite the claim's weaknesses, it remains one of FOSS's major assertions of superiority. Heartbleed seems an exception that at least challenges the widely believed rule, or maybe even overturns it completely.
The Problems with Eyeballs
At first glance, Raymond's statement seems to survive any challenge from Heartbleed. Unproved or not, the statement is conditional; it is only true if enough eyes are constantly on the code. However, as the idea is examined, the flaws and unstated assumptions start to reveal themselves.
Robin Seggelmann, the OpenSSL developer who claims responsibility for Heartbleed, says that both he and a reviewer missed the bug. He concludes that more reviewers are needed to avoid a repetition of the incident -- that there were not enough eyes in this case.
Another conclusion that might be drawn from Seggelmann's account is that depending on developers to review their own work is not a good idea. Unless considerable time passes between the writing of the code and the review, the developers are probably too close to the code to be likely to observe the flaws in it.
However, the weakness of Seggelmann's perspective is that the argument is circular: if Heartbleed was undiscovered, then there must not have been enough eyes on the code. The proof is in the discovery or the failure to discover, which is not exactly a useful argument.
A more useful analysis has been offered by Theo de Raadt, the founder of OpenBSD and OpenSSH. De Raadt notes that malloc, a memory allocation library, was long ago patched to prevent Heartbleed-type exploitations. However, at the same time, OpenSSL added "a wrapper around malloc & free so that the library will cache memory on its own, and not free it to the protective malloc" -- all in the name of improving performance on some systems.