> We saw this with Heartbleed too: given sure confidence that there is a vulnerability in a particular diff, skilled security researchers can find it very quickly.

That reminds me of the old QA technique / tactic of only telling the developers where a bug was found. Sometimes you don't even have to find an issue, you can just pick a complicated module.

There was a study done decades ago where code was seeded with a number of bugs. Developers were told that there were a number of bugs. They found that many, but the overlap was not total. That is to say, they found bugs that the study authors did not seed.

Phew, I sure am glad that we instituted a company-wide policy of not putting any bugs in the code in the first place. Just makes life easier all around.

Well, I remember at the time reading this that if one picked an arbitrary number, and told the developers that there were that many, they would find that many even if none were inserted.

You laugh, but this is closer to most companies "quality policy" than otherwise.

True quality comes from good process and systems, nothing more, nothing less.

If you can dig up the citation, I'd love to read this. That's fascinating.

I was not successful, but a sibling comment to yours https://news.ycombinator.com/item?id=8458030 provides a very useful description of the phenomenon.

You can use overlap to estimate total number of bugs:

http://en.wikipedia.org/wiki/Lincoln_index

http://allendowney.blogspot.com/2014/07/last-year-my-occasio...

Pornin did this --- to Thai Duong, no less! --- with CRIME, too. That guy is a freak of nature.

I also ruefully remember how Halvar Flake did this to Dan Kaminsky's embargoed DNS vulnerability.

When Pornin disclosed CRIME I was telling myself: "Wait a minute, I saw this name somewhere." Then I remembered that it was Pornin's article [1] that helped me understand how zlib flush modes work.

I thought that was pretty funny.

[1] http://www.bolet.org/~pornin/deflate-flush-en.html

You guys are both my heroes.

I don't follow -- how is it an independent discovery if it was posted after complete details had already been released by Google?

Pornin's comments on SE predate the publication.

Oh my bad. What is the explanation of this part?

"given sure confidence that there is a vulnerability in a particular diff, skilled security researchers can find it very quickly."

Was there previously an announcement that a vulnerability existed, without details of what the vulnerability was?

There was a lot of Twitter traffic about SSL3 being bad. But TLS1 has very few differences from SSL3.

In Heartbleed, we knew that it was introduced with a particular OpenSSL version. That had 12 changes; 8 were DTLS. Two were clearly not relevant. One was heartbeat. Dan Franke took about fifteen minutes to go from that set of inputs to a sketch of heartbleed.

I now advocate QA by what I call "virtual tests": you don't have to write the tests, just the failure messages. Developers will do the rest of the work on their own.

Yes, there was: https://news.ycombinator.com/item?id=8452931

That announcement implied that the vulnerability was SSL 3.0 only. Knowing that, the obvious place to look at would be the differences between SSL 3.0 and its successor TLS 1.0.