This is (I hope) a once-in-a-lifetime incident, so we have to be careful not to extrapolate it too hard. On the other hand, it's seriously big deal. Overall, it seems to strongly challenge a couple of important assumptions.
As you say, it's a strong challenge to the mantra of "never implement your own crypto". I think (think) it still holds for the crypto primitives. If you're reimplementing AES you're probably doing it wrong. But the protocols? I'm not so sure now. Common wisdom seemed to be that if you implement the protocols yourself you'd screw them up, and you should stick with tried-and-true existing implementations. Now it's apparent that "tried" doesn't have to imply "true". Something being used by millions of people for years doesn't prevent it from having a huge vulnerability for years. Are your odds better or worse rolling your own? I'm not so sure now.
Consider Apple's "goto fail" bug, for example. Among a lot of other stuff, they caught some criticism for reimplementing TLS instead of just using OpenSSL. Well, if they had used OpenSSL instead, it turns out that they would have been shipping an even more serious bug for even more time.
It's also interesting to me how it challenges the idea of encrypting stuff by default. For years, people have been saying that as much traffic as possible should be encrypted, even unimportant stuff that nobody cares about. By doing that, the idea goes, using encryption isn't suspicious and you force attackers to spread out their resources. If only a small amount of traffic is encrypted, attackers can focus just on that traffic. Accordingly, a lot of sites that didn't really need it enabled SSL or even required it, including my own. By doing this, a lot of them inadvertently made things much much worse. A site that's only accessible over HTTP is much better off than a site that's accessible over HTTPS but vulnerable to heartbleed. I don't think the general idea is wrong, but it certainly gives me pause, and I think more consideration has to be given to the increase in attack surface you take on when you enable encryption.
In any case, I really hope we see some new crypto projects come out of this, or more resources put into existing OpenSSL alternatives.
Let's surmise that the reason two distinct researchers (Mehta and Codenomicon) found this same bug in a short timeframe is that the recent Apple & GnuTLS bugs have caused many teams to begin a fresh review of long-ignored shared codebases.
If so, is this the first major bug discovered, with many more to come as they are flushed out by the new level of vigilance? Or, is it the only/last one, being revealed now because the deep dive has now wrapped up?
A few minutes' thought doesn't turn up a counterexample, so I concede the point.
I still wouldn't bet on it, though, as we haven't had a lifetime's experience with such widespread use of relatively homogeneous software. The failure of Enigma wasn't worldwide, but for those who used it, it was a security calamity.
If anything, the limited message I'll take away from this is that defense in depth is good. It's only the web's reliance upon OpenSSL that makes this particular bug so bad.
Well, the internet is only ~20 years old. I'm not strong enough in statistics to actually model this, but if something "this bad" happened once within 20 years, it might be reasonable to expect it would happen again within another 20. It's possible it is a every-hundred-years event, but unlikely. Perhaps even more often in the future, as software complexity continues to grow?
There are a surprising number of crypto-related implementation difficulties in all the language ecosystems (even if all you want to do is wrap OpenSSL) that have nothing to do with the "hard" parts of the algorithms.
Even though this hole is huge, the ratio of attack-surface-exposed to security-gained from SSL is still in general worth it; you'd be throwing the baby out with the bathwater to disable encryption entirely IMO
For comparison, would you avoid having a web server (or roll your own) because of CodeRed?
I would say that something like Code Red is an argument against putting up a web server when you don't need it and it provides no direct advantages to you, just out of some idea that increasing the overall amount of web traffic is good.
As you say, it's a strong challenge to the mantra of "never implement your own crypto". I think (think) it still holds for the crypto primitives. If you're reimplementing AES you're probably doing it wrong. But the protocols? I'm not so sure now. Common wisdom seemed to be that if you implement the protocols yourself you'd screw them up, and you should stick with tried-and-true existing implementations. Now it's apparent that "tried" doesn't have to imply "true". Something being used by millions of people for years doesn't prevent it from having a huge vulnerability for years. Are your odds better or worse rolling your own? I'm not so sure now.
Consider Apple's "goto fail" bug, for example. Among a lot of other stuff, they caught some criticism for reimplementing TLS instead of just using OpenSSL. Well, if they had used OpenSSL instead, it turns out that they would have been shipping an even more serious bug for even more time.
It's also interesting to me how it challenges the idea of encrypting stuff by default. For years, people have been saying that as much traffic as possible should be encrypted, even unimportant stuff that nobody cares about. By doing that, the idea goes, using encryption isn't suspicious and you force attackers to spread out their resources. If only a small amount of traffic is encrypted, attackers can focus just on that traffic. Accordingly, a lot of sites that didn't really need it enabled SSL or even required it, including my own. By doing this, a lot of them inadvertently made things much much worse. A site that's only accessible over HTTP is much better off than a site that's accessible over HTTPS but vulnerable to heartbleed. I don't think the general idea is wrong, but it certainly gives me pause, and I think more consideration has to be given to the increase in attack surface you take on when you enable encryption.
In any case, I really hope we see some new crypto projects come out of this, or more resources put into existing OpenSSL alternatives.