OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Subject: Validating IDS Rulesets
From: Joshua Krage (jkrageBUSER.NET)
Date: Tue Oct 17 2000 - 22:08:32 CDT

After experiencing a notable IDS vendor's lack of QA on a released update,
I started thinking... (which is generally dangerous for any bystanders. :)

How do you (as IDS vendor or IDS tester/architect) validate your chosen
IDSs' rulesets? Do you simply assume your vendor is infallible? I do
believe that this is the default answer for most organizations.

Lets say a vendor releases a new improved version of their software. And
lets also say that their data-entry clerk transposes a couple of digits
in multiple rulesets and the signatures will no longer function as
intended. How do you find out?

I'm not talking about theoretical methods here... I'm talking about
actual applied methods. For theoretical, we can say that we'll go
download all the goodies from packetstorm and run them past the IDS. But
when was the last time you ran, say, the Sendmail 'wiz' exploit past your
sensors? :)

Has anyone put together a few hundred megabytes of network traces that
can be tcpreplay'd past your IDS sensors and trigger every known alert?
Is this even worth it? :)

To kick a discussion off, my methods of the moment include:
  - lots of research (like this list)
  - review of the software (esp. signature files and such)
  - spot testing (run exploits past the sensor)
  - comparing reported events with another system (other IDS, tcpdump)

Thoughts?