Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email firstname.lastname@example.org
Re: Solutions, Results, and Comments - Was [ISA Server and SQL Injection]
From: David (davidclicksee.net)
Date: Wed Feb 23 2005 - 22:18:12 CST
2 cents on code reviews:
I find that there is a place(s) for code review in the release cycle itself.
A 3rd party reviewer is not going to be as familiar with software as an
inhouse developer and will be far more expensive. Why not have inhouse
developers check each other's code as part of the release cycle? Before
anything gets released another developer should look over the originator's
code. This could also be done by the developer manager as well. This is sort
of like having the fox gaurd the henhouse I admit but just as security
should be multi-layered so should commitment to quality if you expect to
have it. Then, semi-anually, take time to go back over the code and have
bug-hunt month or week or whatever. This (bug hunt week/month every so
oftern) is what we did at my old dev shop and it mostly worked as a
compromise between slamming out software as fast as humanly possible and
having software that works. Damn those project managers and their rigid
This may be unrealistic for most companies as the business side typically
demands on schedule releases without bugs in the first place and doesn't see
as high a value in 'skipping' releases to look for bugs that 'shouldn't be
there in the first place'.
Quality, price, speed. Which 2 do you want? Sales, marketing, and the board
seem to take price and speed every time. Developers will take quality and
speed every time but then they don't control the checkbook... IT on the
other hand will take quality and quality. "Screw speed and price! We want
security and flawlessness!"
----- Original Message -----
From: "Jeremiah Grossman" <jeremiahwhitehatsec.com>
To: "Mark Curphey" <markcurphey.com>
Sent: Wednesday, February 23, 2005 5:24 AM
Subject: Solutions, Results, and Comments - Was [ISA Server and SQL
> On Monday, February 21, 2005, at 12:06 PM, Mark Curphey wrote:
>> At Foundstone we have tested literally thousands of apps and the most
>> frequent and most insidious issues are rarely data validation issues
>> days. They were a few years back but not today. They are issues with
>> authorization systems, user management systems, business logic, data
>> protection in (storage and transport), broken protocols, badly
>> cryptography and such like. The bar has been raised.
>> A few people keep asking me where I think validation should be done. I
>> always say in the code. The argument sometimes comes back that that's not
>> always possible. While I completely disagree (every commercial company I
>> have worked for has a release cycle of every few weeks and consistently
>> changes the code base so either I have been living under a rock in major
>> financial services companies or the vendors marketing thought they had a
>> good sales angle)
>> The same is true of automated testing, especially pen testing. I
>> firmly believe that the most cost effective and efficient way to test
>> software is to use the code. To say that code review is too expensive is
>> misleading and outright wrong. It's a common misconception perpetuated by
>> the uneducated. I wrote an article about it at www.softwaremag.com. Pen
>> testing web apps is a Turing problem, the number of inputs has to equal
>> number of outputs and that's computationally unfeasible in a modern app.
>> low hanging fruit its great but for the type of issues being found now
>> bar has been raised I don't think its appropriate. Again at work we have
>> driven trucks past app that have been given a clean bill of health by the
>> scanners. And that's pretty frequent not an edge case BTW. If you look at
>> code analysis tools as well they are getting good at finding bugs like
>> overflows but flaws like people using poor PRNG or storing keys in config
>> files they are next to hopeless. Guess what, flaws are the most insidious
>> type of issue we find. Its not that these tools are of no value. We use
>> automated tools for code review and will likely buy or build something
>> web app pen tests this year. For the LHF it's a no brainer. The point is
>> automated solution is a subset of the problem space. At the moment I
>> say its less than 25% for the web app firewall space and about 20% for
>> pen test space (gut feeling).
> <the above has been snipped to highlight the key points>
> Per your "balanced" approach, I find conflict within several key points of
> your assertions.
> Web application software is not the same as other forms of software we are
> used to dealing with. They're rate of change, the types of attacks, and
> the scale of usage differs entirely. As such, so do they're security
> requirements. You said above "...every commercial company I have worked
> for has a release cycle of every few weeks and consistently changes the
> code..." on this we agree. Then you went onto say "...code review is too
> expensive is misleading and outright wrong." This is where we disagree.
> 1) Source Code Reviews ARE expensive
> While we could argue on how to define "expensive", we can all agree
> line-by-line analysis is time intensive and requires intelligent testers.
> As a result price quotes will go up into the several 10's of thousands
> range, if not higher depending on the scope of work. So we can use this as
> a basis for further discussion.
> The first time a company has a code review performed, this is where
> they'll get the most bug for the buck. Through an exhaustive process, a
> good code review can knock out a ton of security issues and prevent a lot
> of pain down the road. They also can find problems no scanner or black-box
> pen-tester will ever find, like those pesky backdoors and easter eggs.
> The draw back is the cost of third-party source code reviews will prevent
> them from being repeatable. This is important to understand if you
> subscribe to security is a process principal.
> If companies have a release cycle every few weeks (per your quote), having
> a third-party assessor constantly in-house is certainly going to extend
> way beyond anyones budget. However, if your shipping shrink wrapped
> software or a newly designed website where the code won't/can't be changed
> for the next year, source code review is the way to go. Your maximizing
> ROI. Pro and Con.
> 2) You compare Source Code Reviews and Security Assessments (Pen-Tests)
> and its like comparing apples to oranges. Both solutions should be used in
> different circumstances and maintain different ROI models.
> If your web application is frequently updated, this is where you might
> find security assessments (pen-tests) more cost effective. Security
> assessments are typically faster and less expensive than your typical
> source code review. Current assessment methodology will normally identify
> nearly all (if not all) of the easy issues and most of the hard ones.
> They'll also give you visibility into what a real world attacker would
> see. I find it rare for attackers to have access to your source code prior
> to targeting your website. Its going to be far more common for attackers
> to black-box you and go from there. Vulnerability Scanners and continuous
> assessment services help make make the process more repeatable. But
> neither Source Code Reviews or Security Assessments will ever claim to
> find everything.
> The point I'm trying to make here is that customers have several options
> available to them depending on they're needs. They have access to Source
> Code Reviews, Penetration Tests, Vulnerability Scanners, Application
> Firewalls, etc etc. Each with they're own set of pros and cons.
> To say your way is the one and only way is the part I find misleading and
> 3) Scoping the Web Application Security problem.
> We also assess the security of tons of web sites as well. As such, we
> have comparable results. You said your results show the most frequent and
> insidious issues today are "...authorization systems, user management
> systems, business logic, data protection in (storage and transport),
> broken protocols, badly implemented cryptography and such like". On the
> severity of these issues when they are identified, I believe our results
> would closely match. However, as far as the total number of
> vulnerabilities found by class, nothing tops basic SQL Injection and
> Cross-Site Scripting. The discrepancy might have something to do with our
> mutually different approached and/or methodologies.
> 4) Hypocritical discussion of "Thinly disguised marketing posts in public
> You began and ended your email harping on vendors for what you said are
> "Thinly disguised marketing posts in public lists". With a good moderator
> present, this is something many of us find annoying yet tolerable. You
> then proceeded to lace your message with the very same negatives issues
> you were accusing the other vendors of. Mentioning your company and
> offerings numerous times. Lets be honest, you have a product to sell no
> different than anyone else. Yours just happens to be called consulting.
> And if anyone were to read closely... you mirror your own comment. "I
> think x and BTW I happen to make a product that does exactly this."
> I'm going to skip over the application firewall topic because thats a
> whole other issue and I've written enough for now.
> Jeremiah Grossman