OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Subject: Re: Gigabit IDS solutions
From: Dragos Ruiu (drKYX.NET)
Date: Wed Nov 15 2000 - 11:51:30 CST

Whoa.... I really would rather stay out of the 3-4 way
quasi-mudslinging going on with this thread but I have
been biting my tongue for so long now it's starting to hurt :-).
(That said you'll all wonder at the stupidity of my next
statements. :-)

My current project is a vendor independent evaluation
of IDSes (like the ones the fine folks at neohapsis do - whom
I also hope to compare notes with). All of the gentlemen
you see lobbing posts at each other here are participating.
(And as a snort kinda guy I'll certainly be including it too, but
I'm not going to play favorites to anyone there, if anything
I'll likely be tougher on Snort because I know all the good
ways to blow it up... :-) I will probably have several papers,
test software releases and articles resulting from this work
as it is rather a massive project that has been underway
for about a month now. I'm gonna shut up about this following
this post until I have the real results.... because I don't want
to steal my own thunder, but I wanted to note:

The performance of an IDS can be judged in many ways,
numbers on raw data rates (fairly meaningless IMHO), packet
rates (though I would argue that the raw packet counts you
get by firing semi-stupid smart-bits packets are at best only
a rough measure of real world performance), signature coverage,
packet rates on varying real world traffic mixes, clarity and detail
of reporting, multi-probe integration and scalability of reporting
(I agree with Ron, tough to measure), frequency of updates
(impossible to judge without long-term mileage), customization,
applicability in different usage scenarios (home/small business,
small-midsize enterprise, large enterprise... all with different
needs), remote management and security of such management
architectures, price, effectiveness of coverage and many, many,
other factors. For my tests the final test lineup is not even
complete. But as far as gigabit goes, I feel the entire IDS industry
has a long ways to go... particularly because the main bottlenecks
there still lie in the underlying computing architecture as opposed
to the IDS software itself. I believe, and will outline in detail
shortly that pathological synthetic traffic sequences exist that the
current systems can't even deal with at a 100Mbps rate, nevermind
gigabit... Though I am not trying to discourage anyone from deploying
IDSes on Gigabit links, because IMHO, by distributing traffic to
multiple systems in various ways I _do_ feel it can be feasible to
monitor a Gig link for security with 3-4+ boxes dedicated to the
function through some front-end switching.

And even if your particulalr IDS can claim that it handles 320 Mbps
per second (which I believe is the fastest boast I've heard from
vendors until now), or 50K pps, there is the little niggling detail of
exactly how accurate you are at that rate and exactly how much work
and checking is done at that rate. The dillemma with IDS is that you
can always make it run faster by checking less....

Overall, since this is such a complicated measurement, that even the vendors
can't agree amongst themselves as to how to run the "drag-race," they will
likely have a hard time convincing their users of these numbers and resolving
the ambiguity in the performance numbers.... (that's something that independent
evaluators _can_ address)

The bottom line for me after looking at this problem very closely is
that the maximum speed of an IDS is a damn hard thing to judge.
I took on a project where I have to do this... and my punt will be
to expose my testing methodology, publish the raw numbers and
let all the potential buyers of these IDS systems judge which factors
are most important to them in picking which system is right for them.
Because some of these IDSes are better suited to particular applications
and scenarios than others... and the knowledge level of the users and
the amount of dedicated security staff or hours available for monitoring
are also very important factors that have to be weighed in... Mostly
because some tools are better suited to novice users and others more
appropriate for experts or those with dedicated security staff.

In the near future, I will be providing some of my _opinions_ and my cut at
rankings and rate "drag-races" between products from these fine gentlemen
who are having heated data-sheet wars in this forum right now, but after
some initial looks at all their products, I must say that they each have some
merit in their own specific areas. It's still a very tough decision for those
purchasing such solutions, and requires some careful analysis of and
confidence in what you are trying to acheive in the first place..... As far
as Gigabit IDS goes.... I also have to ask.... what are you hoping to achieve
by monitoring your GigE backbone rather than the careful monitoring of the
choke-points in your network. (And if you _do_ have a Gigabit WAN uplink....
gee, can I co-lo some servers in your farm ? ;-)

And now that I've gotten that off my chest I go go back to biting my tongue,
watching the mud-slinging, and crafting test sequences... :-)

cheers,
--dr