OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Subject: Validating IDS Rulesets
From: stewart_watkissUK.IBM.COM
Date: Thu Oct 19 2000 - 04:30:38 CDT

I recently attended RAID (Recent Advances in Intrusion Detection) Workshop.
 This is a meeting of primarily researchers (academic & commercial) who are
involved in the development and advancement of IDS. I am not a researcher
but went along to get a feel of how IDS is developing and to get a greater
insight
into the technology.

See:
http://www.raid-symposium.org/raid2000/

One of the key things that was discussed is known as "1998 Lincoln Labs
Data".
This is freely available sample data of numerous attacks against a
hypothetical
military base. The data is often used when trying to evaluate the
effectiveness
of an IDS rule set.

Obviously a single set of data cannot be 100% inclusive of all possible
attacks,
but then it's down to the vendors to monitor CERT etc.

The underlying importance is how the vendor incorporates this into their
signature
rules. It is not really practical for "users", without large funds
available, to try and
fully test the rule set of a particular IDS product. Really it falls down
to trusting
the vendors to produce their own sets of tests to check against their
products.

For organisations that see attacks that are not detected by the "out of the
 box"
IDS products then most products allow you to define your own rules
(although
how comprehensive these are is down to the individual products). This comes
down to the tuning of your IDS product once purchased.

If you are interested in finding out more about the Lincoln Labs data then
I
suggest you try:

http://www.ll.mit.edu/IST/ideval/

However be prepared for a lot of work involved in trying to test an IDS
product.

A final point, is that I have been running data against a few IDS products
to
"get a feel" of how they alert on attacks. To do this I have just run a few
scanning
tools against a machine. This is however far from being a comprehensive
test
of the rule set.

Regards
Stewart

___________________________________
Stewart Watkiss - Security Specialist
EMEA Firewall & Security Services - Security Analysis
AT&T Global Network Services

Email: swatkissemea.att.com

Disclaimer: Views expressed are my personal opinions and do not necessarily
reflect the opinions and views of AT&T GNS UK or any other associated
companies.

---------------------- Forwarded by Stewart Watkiss/UK/IBM on 19/10/2000
09:49 ---------------------------

Joshua Krage <jkrageBUSER.NET> on 18/10/2000 04:08:32

Please respond to Joshua Krage <jkrageBUSER.NET>

To: FOCUS-IDSSECURITYFOCUS.COM
cc:
Subject: Validating IDS Rulesets

After experiencing a notable IDS vendor's lack of QA on a released update,
I started thinking... (which is generally dangerous for any bystanders. :)

How do you (as IDS vendor or IDS tester/architect) validate your chosen
IDSs' rulesets? Do you simply assume your vendor is infallible? I do
believe that this is the default answer for most organizations.

Lets say a vendor releases a new improved version of their software. And
lets also say that their data-entry clerk transposes a couple of digits
in multiple rulesets and the signatures will no longer function as
intended. How do you find out?

I'm not talking about theoretical methods here... I'm talking about
actual applied methods. For theoretical, we can say that we'll go
download all the goodies from packetstorm and run them past the IDS. But
when was the last time you ran, say, the Sendmail 'wiz' exploit past your
sensors? :)

Has anyone put together a few hundred megabytes of network traces that
can be tcpreplay'd past your IDS sensors and trigger every known alert?
Is this even worth it? :)

To kick a discussion off, my methods of the moment include:
  - lots of research (like this list)
  - review of the software (esp. signature files and such)
  - spot testing (run exploits past the sensor)
  - comparing reported events with another system (other IDS, tcpdump)

Thoughts?