OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
NFR Wizards Archive: Re: Penetration testing via shrinkware

Re: Penetration testing via shrinkware

Frederick M Avolio (fredavolio.com)
Mon, 21 Sep 1998 11:03:19 -0400

At 09:04 PM 9/20/98 -0400, Marcus J. Ranum wrote:
>John McDermott wrote:
>>[...] Just out of curiousity
>>does ICSA look at the source for certification?
>
>Last I saw, the ICSA "certification" was based on a firewall's
>ability to not have identified vulnerabilities in an ISS scan
>and a couple of other "advanced" tests.
>

[Disclosure: ICSA is a sometime client of mine. But they don't pay me to
respond to mailing list letters. :-)]

In this discussion, which I've read with interest, four "methods" or
approaches have come out (I think):

        A formal, thorough, evaluation process. This process includes design
        review and code review, and takes a very long time. By the time it
        is done, the system tested is not the system shipped. It is "downright
        damaging," according to Marcus (no disagreement from me).

        A process such as ICSA's: printed criteria that includes functional
        specifications in a given form and testing criteria that is public,
        extensible, and represents someone's (person or organization) opinion
        of what the minimum criteria are. The testing criteria specifies
        what will be tested, but not how. (They do more than run ISS.)

        Running canned software yourself.

        Full fledged penetration testing by professionals.

None of the first three are penetration tests, as I think has been stated a
few times. All are evaluation processes of one kind or another. Running
software yourself is not sufficient (as others pointed out), but certainly
useful for verification. If one says "We do not allow FTP," it is useful to
check even if one is sure.

With the ICSA "method," they (and others like them, I suppose) take the
stand: if we test everything and do a full fledged penetration test on
every product, it will take as long as an A1 criteria test and be of as
little use. We will test for what we know to be problems -- that's almost
the best anyone can do anyway (I agree with this practical approach, but I
also know the shortcomings). Turns out they find problems in over 50% of
the products tested and give vendors a chance to fix the problems and
inform their customers. I've liked this kind of testing mainly because it
does do something useful and does it in a timely manner.

The availability of source code is, I think, a red herring. When I was at
TIS, as Marcus pointed out, "source code availability" for the FWTK was
written into the proposal we sent to DARPA for the work. This was carried
over into the commercial product and became one of the 7 design tenets of
good security we then used.

Experience showed, after a number of years, that while many people talked
about the requirement for source code availability (we called it a "crystal
box", a phrase coined by the first TIS commercial firewall customer), and
we trumpeted it as a "must have" for security, few customers actually
reviewed the source code. Most just had it ("if we ever want to examine it
...", "well, I can't read this, but it is good to know others are looking
at it ..."). Those who did use it used it to add features or to fix bugs.

When TIS decided to stop shipping source code, I was not in favor of the
decision. In my opinion, it affected a "feeling" since most customers did
nothing with the source code. But by that time I was VP of Technology
Marketing, so "feelings" mattered a lot to me. :-)

I still think most people do not look at source code for products --
certainly not in any detail enough to catch security problems. I think what
happens is a bug is discovered, and the customer with source code fixes the
bug, and provides the bug fix to the vendor and other customers.
Experiences with the FWTK bear this out. This has the appearance of the
code being reviewed and refined because of source code availability, but
that's not what is going on.

Do I think source code *should* be reviewed (and so reviewable)? Of
course. Should people care about such things? Yes. However, most do not.

Fred

This archive was generated by hypermail 2.0b3 on Sat Jul 17 1999 - 07:11:47 CDT