Software bugs are an unfortunate reality of complex systems. Even with rigorous testing, some bugs inevitably slip through the cracks into production. This begs the question – how many bugs are lurking in our software, still undiscovered? Let’s try to estimate.
The number of known bugs
The number of reported and fixed bugs gives us a lower bound estimate. Open source projects that use public bug trackers give us real data to work with. For example, over its lifetime the Linux kernel has had:
| Total reported bugs | 100,000+ |
| Total fixed bugs | 90,000+ |
Other major open source projects report similar numbers of known bugs:
| Mozilla Firefox | 750,000+ |
| Chromium | 950,000+ |
For proprietary software, we have to rely on rare disclosures from companies. Microsoft reportedly fixed:
| Bugs in Windows 7 | 36,000 |
| Bugs in Windows 8 | 44,000 |
| Bugs in Windows 10 | 50,000+ |
Estimating total bugs
Not all bugs are reported. To estimate the total number, we need to consider:
- Many bugs are never hit during testing
- Many found bugs are never reported by users
- Complex interactions lead to undiscovered bugs
Studies have shown only 25-30% of bugs are discovered during development. Additional user testing finds another 40-50%. The remaining 20-30% are undiscovered pre-release.
Post-release, users encounter bugs that went undetected. But not all get reported. According to Mozilla, only 0.15% of Firefox users submit bug reports. The true number of bugs encountered is much higher.
Finally, the interactions between components result in nonlinear effects. Completely new bugs emerge from complex systems. These are the hardest bugs to find with testing.
Given all this, total bugs likely outnumber identified bugs by at least a factor of 3x. Possibly as high as 5-10x for complex software.
A case study: healthcare.gov
Healthcare.gov provides a relevant case study. This complex government software project had a notoriously rocky launch in 2013. In the first month, only 6% of visitors could successfully enroll. What went wrong?
Post-mortems revealed the issues:
- Insufficient end-to-end testing
- Late design changes and scope creep
- Complexity of integrating many systems
All prime conditions for bugs. In the first months, over 500 bugs were fixed. Over its first year, likely thousands of total bugs were addressed through ongoing repairs.
Healthcare.gov illustrates how even rigorous efforts miss bugs at launch. And complexity multiplies them. It serves as a microcosm of the challenges all software projects face.
Estimating totals
Given the data points and ratios discussed, we can crudely estimate total bugs for sample projects:
| Linux kernel | 300,000 – 1,000,000 |
| Mozilla Firefox | 2,250,000 – 7,500,000 |
| Windows 10 | 150,000 – 500,000 |
These broad ranges account for uncertainties in underlying assumptions. But they illustrate the scale of undiscovered bugs.
Implications
What do these findings mean for software development?
- Striving for bug-free code is futile. Bugs will always exist.
- Testing and quality efforts should target the most critical flows.
- Monitoring and rapid patching are imperative after launch.
- Expectation setting with users and management is key.
While we cannot eliminate bugs, we can mitigate their impact through prudent practices. And acceptance of imperfection can lead to better systems.
Conclusion
Our analysis shows potentially millions of undiscovered bugs in existing software. From Linux to Windows and beyond, defects lurk everywhere. While disheartening, this reality highlights the inherent complexity of software. The humble bug reminds us that all systems are imperfect across technology, business, and life. Success comes not from perfection but from how we respond.