"If NSS wanted to do it for free, we would be happy to participate." -

I am sure most vendors would - but then we would not be in business for very
long. This is old ground, but at the end of the day someone has to pay for
testing - the lab, the vendor or the end user. We do not believe that the
subscription model is viable these days - most end users do not want to pay
for such reports. As a lab, we do plenty for free (we have included Snort in
the latest round of testing as just one small example). But at the end of
the day, the vendor has to pay.

The "significant amount of money" Klaus mentions is significantly less than
a full page ad in a major trade publication, and I would think the results
would be read by far more people than would take notice of a magazine ad. I
would really rather take this off line, however, since I think that the
majority of people on here simply want to see the test results and are not
too worried about who paid for them as long as the testing is thorough and
accurate. Thorough and accurate testing, however, costs money.....

"Most network IDSes now have 100 mbit speeds...."

I am not sure I agree. Certainly most IDS's CLAIM 100Mbit speeds, but they
do not all achieve it. I do agree that the bar should be raised to Gigabit
speeds, however, and we intend to do that with our next round of testing in
2002. I wonder how many vendors will take up that challenge? And yes, it
WILL cost money (Gigabit infrastructure and test equipment does not come
cheaply....)

"We like to compare when all checks are turned on..."

I agree - we ALWAYS leave all signatures active for a worst case scenario.
If the product performs at a given level with all sigs active, it can only
improve when tuned for your own environment. There is little point testing
it any other way (although I admit there are many other ways it COULD be
tested - let's not go there...)

"Compare IDS vendors on breadth and coverage of IDS solution"

We try to do that - our evaluations are extensive and not just focussed on
performance alone

Compare IDS vendors on ability to offer remote round-the-clock IDS
monitoring service.

This is probably another type of "product test" altogether. To date, we have
covered only software solutions, but, as we have done with our PKI report,
we may well extend coverage to managed/outsourced IDS services.

Compare IDS products based on Signature Coverage and Accuracy.

We try to cover this

Compare IDS products on Frequency and Response Time for Signature Updates.

We cover this

"It is hard to determine the best IDS on performance alone. There are many
other attributes one should look at when deciding on an IDS, and developing
fair tests for them are always challenging. But IDS is not unique in having
these type challenges, if you look at other technical benchmarks and
standards. I'm sure as this industry evolves, these benchmarks and tests
will emerge as well."

I agree wholeheartedly - we are trying to evolve our testing methodologies
all the time. Edition 2 of our report (due November) is an "evolution" of
the Edition 1 testing. Next year there will be a radical overhaul of the
testing methodology and we will include Gigabit. We will remain with a pure
lab testing methodology, and leave the real-world stuff to Neohapsis who did
such a wonderful job with their massive undertaking earlier this year. I
believe there is room for both approaches, and both sets of results make
interesting reading.

We would love to have input to our testing methodology from ALL the vendors
in this market space (and any users who have been heavily involved in
testing) and would like to have as many of them participate as possible -
that way we can create as level a playing field as possible.

This is a very new (in relative terms) and very complex market. Even when
they have the expertise, end users can rarely afford the time or the
equipment to perform the type of testing we do. This is even true of some
vendors - as I have said before, this type of testing is not cheap to
perform when done properly.

Hopefully, our reports are just one more tool to enable interested parties
to cut through the marketing blurb and hype surrounding the IDS industry.

Regards

Bob

-----Original Message-----
From: Klaus, Chris (ISSAtlanta) [mailto:CKlausiss.net]
Sent: 20 October 2001 00:05
To: 'focus-idssecurityfocus.com'
Subject: RE: Realsecure

> -----Original Message-----
> From: Bob Walder [mailto:bwaldernss.co.uk]
> Sent: Wednesday, October 10, 2001 1:30 PM
> To: focus-idssecurityfocus.com
> Subject: RE: Realsecure
>
> Sorry to nit pick, but in our testing we found that
> RealSecure cannot handle
> anything like 100Mbps in terms of raw sniffing speed with
> small packets.
>
> We are re-doing our testing this year with additional participants and
> including a "real world" packet mix to try and give people an
> idea of how
> these things will perform in a "real" network (how long is a piece of
> string....). Unfortunately, ISS has declined to participate -
> read into that
> what you will!

NSS charges a significant amount of money to do performance testing per
product. We currently have two products in this space. With our company
goal to become profitable this last quarter and money was already spent
elsewhere, it became a purely financial decision.

In the future, we can always reevaluate to participate in performance
testing.

> Of course, one of the best performing products we found last year was
> NetworkICE

If NSS wanted to do it for free, we would be happy to participate. We
believe BlackIce technology would remain in the top performer class and our
current roadmap for integration leverages this core capability. Because
the NetworkICE technology is being used as the chassis for doing the
high-speed protocol analysis in the integrated version of RealSecure 7, you
will find significant performance capability in the single integrated
solution.

Some comments about testing in general (not directed at NSS).

Most network IDSes now have 100 mbit speeds. If performance testing is done
at 100 mbits, these tests do less to distinguish the capabilities of the
current IDS technologies. With current evolution of IDS, I believe the next
bar for performance testing should be set at 1G speeds.

One challenge with performance testing is many times it is done, where the
signature coverage is compromised by turning off most checks and only
looking for a small subset of attacks. We like to compare when all checks
are turned on.

Other standards and benchmarks I would like to see how IDS solutions
compare:

Compare IDS vendors on breadth and coverage of IDS solution:
desktop IDS
server IDS
network IDS
gigabit IDS
inline IDS

Compare IDS vendors on ability to offer remote round-the-clock IDS
monitoring service. We are finding many customers are looking to extend this
value-add service to their IDS, rather than have their entire security team
looking at IDS screens all the time, they can do more strategic activities,
and get an alert from the vendor only when it's serious. And does the IDS
vendor have an Emergency Response Services (ERS) team to help deal onsight
with real incidents?

Compare IDS products based on Signature Coverage and Accuracy. How many
signatures does the IDS have and how many are false positiving? With
protocol analysis, one issue we have ran into on some performance testing,
is that our signatures are starting to look at the return packets, to
determine if the attack was successful or not. If not, don't send an alarm.
We had some customers thought we were missing the attack, when infact, the
policy was configured to only alert on when it was successful. So they
have to set up a vulnerable server in that situation if you are doing those
tests.

Compare IDS products on Frequency and Response Time for Signature Updates.
How fast did the IDS vendor respond when Code Red was unleashed? How often
does the IDS really get updated? (like list the last 5 signature updates and
the date they each were released).

It is hard to determine the best IDS on performance alone. There are many
other attributes one should look at when deciding on an IDS, and developing
fair tests for them are always challenging. But IDS is not unique in having
these type challenges, if you look at other technical benchmarks and
standards. I'm sure as this industry evolves, these benchmarks and tests
will emerge as well.

***********************************************************************
Christopher W. Klaus
Founder and CTO
Internet Security Systems (ISS)
6303 Barfield Road
Atlanta, GA 30328
Phone: 404-236-4051 Fax: 404-236-2637
web http://www.iss.net
NASDAQ: ISSX

Internet Security Systems ~ The Power To Protect

• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]