Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
Re: [Dailydave] What is the state of vulnerability research?

From: security curmudgeon (jerichoattrition.org)
Date: Thu Feb 16 2006 - 11:15:14 CST

On most days I would agree with you, and come up with a reply along these
lines =) However, since I am a cynic AND involved in the same 'scene' and
core interests Christey is, I think I understand this fairly well.

: MITRE's Centers of Technical Excellence

: Our Centers of Technical Excellence are centralized areas in which we
: build knowledge of key technologies. They act as a resource for the

: And they bring back knowledge to the center about how technology is
: being used in the field and what the customers might need next.

: Why do I think Mitre should be coming out with answers, not questions?

Vulnerability research is straight forward. There isn't a lot of black
magic and secret arts when it comes to finding vulnerabilities. For the
most part, 99% of vulnerabilites are very well documented (even if the
'researcher' doesn't document it), easy to understand by others in the
field, and leave little to imagination. It has been years since we've seen
a truly new class of vulnerability surface. If I post details of an
overflow of *any kind* to this list, there are a hundred folks that can
digest what I post in seconds, then go to town on me for not going into
details, not looking at VectorX, FunctionY or Z.c =)

The other side of vulnerability disclosure is the human element. The
sociology and mindset behind what we do, and why we do it. This is the
angle that has interested me for years, and the type of book I will grab
before any 'technical' (generous term usually) security/hacking book. Not
only are there dozens of questions that can be asked of the researcher
about his mindset and ethical views, there are countless other people
involved in the process. Does the researcher have partners? Is he an
employee of a security company? What vendor is he dealing with? Which
vendor is it? How many people is he dealing with on the vendor side?

All of that will factor in to how a disclosure plays out, along with a
thousand other variables. Trying to understand that is not easy (if at all
possible). If Christey or MITRE said "we understand the state of vuln
research, here it is", you and I would both be flaming him until we pass
out at the keyboard. The fact that he *knows* he doesn't know is actually
an impressive quality. The fact he will actually *admit* that he doesn't
know, is staggering. The fact he will ask the community their thoughts
(hey, peer review?!) and factor it into his own research and answers..
wow! If you think Mitre should be coming out with answers, without asking
these questions, then I think you should step back and think about the
state of vulnerability research.

I don't say this as an insult or a derogatory comment at all. A couple
years ago I would have posted the same thing, without realizing just how
complex of an issue this really is. Since I started working with OSVDB.org
and running such a database, my appreciation and understanding of the
complexity of vulnerability research and disclosure has skyrocketed. That
skyrocket has now put me somewhere between "i don't have a fuckin clue"
and "i think i have an inkling of understanding here". I say this because
behind the scenes, after you read the bugtraq or full-disclosure post
about an issue, the real research frequently takes place. If you or anyone
else knew just how many mistakes, oversights and shortcomings occured in
the average disclosure, you may be surprised. If Christey or I had a
nickel for everytime we mailed the other about a flaw or oversight in the
other's database, we'd both be surrounded by a dozen hookers feeding us
tropical fruits and exotic drugs on our own private island.

The current state of vulnerability research SUCKS ASS. Yes, quote me on
that. I say that in the context of the overall research we see. I say that
based on a cynical and jaded perspective I have after spending countless
hours clarifying, correcting and further researching issues that have
already been 'researched'. I say this after dealing with dozens of vendors
who tell me that *I* am wrong because OSVDB has information culled from
other sources, and that say *I* am a moron for thinking X is vulnerable to
Y and Z is the impact. Doesn't matter that half the time (or more) the
vendor is wrong and sending me a snap judgement mail defending their
product. Doesn't matter that sometimes they are right, but in figuring
that out I find several other vulnerabilities in their product. Doesn't
matter that CVE, OSVDB, SecurityTracker and others spend a few days to
finally figure out and resolve discrepancies in the most boring and
simples of disclosures days before. Doesn't matter that one such vendor
had an office 1.4 miles from me and the only thing that stopped me from
pissing on their front door was 18 degree weather.

Think about it.. why would ANY 'vulnerability report' require that much
followup and analysis? One of the many things on my 'to blog about' list
is a subset of Christey's question. Why do people think they are actually
doing 'vulnerability research' by pasting a ' into a field and getting an
SQL error? Hello.. that doesn't mean you can inject SQL queries! Pasting
the standard <script> tags into a field and getting a popup doesn't mean
you have 'researched' an XSS vulnerability. Hell, even a lot of
researchers we all know and respect fairly well continually post "X is
vulnerable to an Overflow" without clarifying if it is an overflow that
allows code execution, or an overflow that is ONLY good for crashing an
application! If a researcher makes no claims either way, and makes it
clear that they did not fully test it, fine! It's all about qualifying
your research and explaining your process. I know you have as little free
time as I do, I know you can't fully test an application. As long as you
make that clear, then the rest of the world can take your research for
what it is and expand on it, or move forward knowing it hasn't been fully

Do you ever wonder about the cut and paste kiddies discovering XSS and SQL
injection vulnerabilities? Why they find a vuln in a product, then a week
later another person finds a different vuln in the same version of the
same product? How could the original person miss it? Easy, because they
didn't test it. Why didn't they? They wanted to find one vulnerability,
ANY vulnerability in that product, publish it and move on. It's about the
same mindset and skillset as the web defacers who would tag
somerandom-123-456.webhost.com domain with "0mg d4v3 w4s h343" and move
on. They didn't own that machine any more than these vulnerability
researchers 'audited' a product.

So.. next time you are reading Bugtraq or Full-Disclosure, stop and
examine the post for a few seconds. Did they include all the vital info to
make the issue clear? Did they forget to include the exact vendor name
and/or vendor URL (so many don't)? Did they say every version prior to the
current was vulnerable (even though the vulnerable function was added ONE
revision back)? Did they produce one error condition in one script and
nothing else? Did they forget to tell you that it only occurs when
register_globals is set to 'on', something that is NOT the default, and
hasn't been that way since Aug 29, 2000? Did they fail to mention that the
same issue was disclosed a year before, and that the vendor never fixed

The simple truth is, formal and public vulnerability research is
primitive. While there are a few (percentage wise) researchers who know
the score and do a good job with their work, a majority of researchers are
new to the field, undisciplined, or lacking some fundamental knowledge
needed to properly research and test software vulnerabilities. The
databases that track these disclosures are equally primitive. The only
reason they have evolved a few years beyond the researchers, is that they
are *aware* of just how lacking it all is, and want to see it change for
the better.