OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Subject: Re: Validating IDS Rulesets
From: Robert Graham (robert_david_grahamYAHOO.COM)
Date: Sat Oct 21 2000 - 01:53:39 CDT

>-----Original Message-----
>From: Joshua Krage
>Your database... a listing of signatures? Including network traces?
>Or keeping an archive of exploit code? Or something else?

Primarily network traces, as well as exploit code, organized according to
signature.

>If I'm understanding what you're saying, once you have entered a
>signature into the databsae, its integrity is protected by procedures
>and documentation. And I'll assume that you've dealt with the lower
>level data integrity issues (backups, checksums, etc.).

Something like that. For example, we make regular CDROM backups.

>So you're keeping extensive documentation about the signature and its
>modification history. But how are you validating the signature itself?
>What triggers your "rework of the signature"?

An example of a rework is the recent UTF8 encoding bug. Remember that our
technology roughly emulates the web server itself rather than simply looking
for patterns. It already does urlencoding %xx and directory canonicalization
(i.e. anti-whisker technology). To handle this new vulnerability, we had to
insert a UTF8 parsing module between the urldecode and the canonicalization.
Did this affect any of the existing HTTP signature decodes? Simple to test:
we run the old code through the existing tracefile library, recompile with
the change, and run it again. The results should be identical for existing
traces (and they were).

This is subject near and dear to our heart. Choosing to go the "protocol
decode" route is extremely dangerous. In security, a common rule is that
the more complex a system, the more likely vulnerabilities/bugs will be
found. I'm surprised this doesn't pop up more in protocol-decode vs.
network-grep debates.

In order to defend against this, we developed a test plan BEFORE we started
work on the product (actually, the test plan was the first document
written -- even before the business plan). The code has numerous
"design-for-test" features; sometimes it seems that half the code we write
is simply for the purpose of testing the other half (i.e. we've written
numerous client/server programs for the sole purpose of creating hostile
traffic designed to crash the IDS; we've successfully crashed two
competitor's IDSs using these programs). We also have strict coding
guidelines and code review procedures (at least within the sensor portion of
the code, user interface stuff is a little more relaxed, which burned us
once).

Take for example this last IIS UTF8 cononicalization problem. The
engineering staff had to:
1. research the vulnerability (much more thoroughly than has been posted)
2. develop test exploits, especially those that differ widely from the
posted information
3. capture cononical test tracefiles for this exploit (this resulted in 5
exploit tracefiles, 1 tracefile forcing boundary conditions on the decode,
and 1 false-positive tracefile).
4. run the existing product over the tracefile libraries
5. implement the new parser, being careful to step through each line of new
code in a source level debugger and force every code path
6. run the new product over the tracefile library and diff the output with
step 4
7. hand off to the general QA staff for blackbox testing
8. independent code review by other engineers and possible whitebox testing

Just because we got the fix out in a few hours shouldn't be misinterpreted
that we didn't put a lot of careful work into it. We understand that such
major vulnerability events happen on a regular basis (this specific
vulnerability is pretty important) and the entire team is prepared for them.
It is like a military drill where everyone knows exactly where their place
is and what they have to do. This barely impacted the regular developement
activities.

I'm not quite sure if this is more than what you wanted to hear, but you
made the mistake of asking :-)

Regards,
Robert Graham
CTO/Network ICE

PS: don't even get me started on performance testing

_________________________________________________________
Do You Yahoo!?
Get your free yahoo.com address at http://mail.yahoo.com