On Mon, 14 Jan 2002, The Owasp Project wrote:

> As you know we are going to capture a topic a week
> and compile the best knowledge into some of the
> testing Framework sections at www.owasp.org. So this
> weeks topic is Black Box vs White Box testing.
>
> What is black box and white box testing; when is it
> appropriate to use one or the other; should you do
> both; what can you find with one that you can't find
> with another; is one more skilled; does one cost
> more to do; does one take longer; which one produces
> better results etc
>
> Please share your experience, points of view or
> thoughts and well capture it for the testing
> Framework project.

I'm probably a bit late for this, but I've written up a little note
regarding black-box and white-box testing that some people might
find interesting. I may be completely out in left field with this
one, so don't hesitate to tell me where I've gone wrong. I dolled
it up a bit to look presentable. It's pasted below.

vertigo

########## Begin Technical Note ##########

        Technical Note: Black-Box Vs. White-Box Testing, An Introduction

Abstract

   In this note, I outline black-box and white-box testing and provide
some
suggestions on testing an application. The first two sections cover the
basic testing types, and the remainder provide tips and tricks that I have
learned in my brief development experience.

Contents
1. Overview
   1.1 Conflicting Definitions
   1.2 Tips and Tricks
2. Testing Methods
   2.1 Black-Box Testing
   2.2 White-Box Testing
   2.3 Coverage Metrics (This section is incomplete)
3. Tips and Tricks
   3.1 Boundary-Conditions and Edge-Cases
   3.2 The Intermittent Bug and Logging
   3.3 Built-In Testing
4. Conclusion
5. Notes
6. Bibliography

1. Overview

1.1 Conflicting Definitions
   There are conflicting definitions for black-box and white-box testing.
At the
foundation of the conflict lie the ideas of source-code visibility and
source-
code execution--or coverage. One set of definitions pairs the
invisibility of
the source-code to the testing entity (black-box) with the visibility of
such
code (white-box). The other set of definitions pairs functional testing
(black-
box) with structural testing (white-box).

1.2 Tricks
   Regardless of the method and its definition, there are tricks to
testing and
debugging software. Concentrating on edge-cases and boundary-conditions,
logging,
and built-in testing are some of the more useful tricks.

2. Testing Methods

2.1 Black-Box Testing
   Black-box, or functional, testing attempts to validate the output of
the sys-
tem being tested. If the system is a simple calculator, then the
functions that
must be tested are Add(), Subtract(), Multiply(), and Divide(). These are
tes-
ted with a set of inputs against a set of outputs that are known to be
correct.
If the correct output of Add(1,1) is 2, but our tests consistently show
the out-
put to be 3, then the calculator fails our test.
   There are many appropriate scenarios for this type of testing. The
first is
unit-testing of isolated components. For example, I have a utility class
whose
responsibility-for-doing is the replacement of every ASCII space character
in an
HTML document with an "&nbsp" string. The first clue to the
appropriatness of
black-box testing for this class is it has recognizable inputs and
outputs.(1)
Second, it is not included in a larger system and, if it were included,
would
cause no "side-effects" within that system.
   A second appropriate scenario for black-box testing would be
all-or-nothing
tests of a web-based application by an independent quality-assurance (QA)
team.
It is important that this team be isolated from the development team. A
test is
devised that involves user-registration, authentication, storage of some
per-
sonalization data, logging off, then authenticating again and retrieving
that
data. The inputs and outputs are identifiable, if not immediately
apparent.(2)
The output is also easily validated. This type of test is useful when
preparing
a release. Does the application do what it's supposed to? If so, then
it's put
into production, otherwise it goes back to development. (3)
   This scenario, however, exemplifies the importance of white-box
testing. What
happens if a tester enters incorrect or duplicate registration data? What
hap-
pens if the user fails to authenticate? These situations may be handled
quite
nicely by the application. On the other hand, your registration component
might
contain a remotely-exploitable buffer overflow that gives every
script-kiddie on
the planet root access to your servers. Nobody will ever know unless
every line
of authentication code is analyzed and executed. Our next method of
testing does
exactly that.

2.2 White-Box Testing
   The goal of white-box testing is to make every line of code execute at
least
once. The only way this is possible, unfortunately, is if every line of
code is
available to the testing entity. This is next to impossible in today's
develop-
ment environment. Pre-compiled libraries, third-party components and
massively
bloated APIs all add to the difficulty of ensuring every line is executed.
White-
box testing is, however, of utmost importance to the success of a software
development project. At the very least, try to make every line of your
own code
execute.
   White-box testing includes validation the output of individual
components,
but it also places more stress on the operation of the application as a
whole.
Often times it takes the expertise of developers, system administrators
(if this
is a web-based application), and perhaps even software(4) to create the
specific
situations necessary to execute every piece of code, and to interpret the
re-
sults. For large applications, critique from several classes of users,
developers,
and administrators should be garnered.
   White-box testing is applicable to any software development project
regardless
of scale. Even if the code is perfect from the perspective of the
developer,
there are always unexpected bugs. One scenario where white-box testing
could
have uncovered a particularly nasty bug in one of my development projects
invol-
ves an email notification component I wrote in Perl. It worked perfectly
in de-
velopment. It was small, brainless, and efficient. Unfortunately, the
network
latency between the production web-server and the production mail-server
was so
long, that the component frequently timed out. This condition was written
into
the API I was using for SMTP, but I had neglected to test for timeouts in
the
application.
   Another scenario in which white-box testing is appropriate is when
testing
exception-handling. Not all exceptions are thrown directly as a result of
user
input. Drive failures, running out of memory, stale file-handles, even
database
servers crashing are all within the realm of expected exceptions. Without
a
method to simulate the causes of these exceptions and execute the code
that han-
dles them, one can only wait and hope for the best. This kind of
simulation
depends on source-code visibility, as well as source-code coverage without
user-
input. Both of these dependencies are, by definition, not provided by the
black-
box test.

(Note: This section is incomplete.)
2.3 Coverage Metrics
   Source-code coverage is the primary goal of white-box testing. This
can be
measured in one of two ways: lines executed, and functions executed. If
one is working
from an object-oriented perspective, this could involve testing of the
object life-cycle
(instantiation, mutation, persistence, destruction, etc.). The Unix-based
'tcov'
package is also a useful tool.

3. Tips and Tricks

3.1 Boundary-Conditions and Edge-Cases
   The most effective method of breaking software is by attacking
boundary-conditions and edge
cases. A boundary-condition is a condition where some conceptual boundary
is reached: a
calendar changing days at midnight, or a user session timing-out. An
edge-case is similar
to a boundary-condition except it is, often times, a range limit. (Note,
these definitions
differ from their mathematical origins.)
   I will use Unix dates as an example. The range of valid dates in most
Unix systems is
Fri Dec 13 20:45:52 1901 UTC, or INT_MIN seconds before the Unix epoch, to
Tue Jan 19
03:14:07 2038, or INT_MAX seconds past the epoch. Logical "edges" are
reached at INT_MIN
and INT_MAX, these being the smallest and largest signed numbers
representable with 32 bits.
These are both edge-cases. Something interesting happens when our timer
is set to 0: the
Unix Epoch begins. This date, Thu Jan 1 00:00:00 1970 UTC is our
boundary-condition. There
is no logical edge, but interesting problems can arise. The fact that
this date is
represented by a 0 can lead to potentially fatal exceptions, and must be
handled with care.
  Another example of a boundary-condition involves databases. Consider a
table with an
integer field used as a primary key. This key is generated by the
application (not by the
database) by selecting the record with the greatest primary key, then
incrementing the
value of that record's primary key by 1, and usinT this value as the
primary key of the
record being inserted. The boundary-condition in this example is the time
between the
initial SELECT statement to the final INSERT statement. The reason being,
unless careful
locking rules are used, a new record could be inserted by a concurrent
process, and records
with duplicate primary keys would be inserted.(5)
  A second example of testing an edge-case would be determining the effect
of instantiating
a java.lang.Integer object with a java.lang.String object that holds a
value representing a
number outside the range [-2147483648,2147483647]. In the case of an
early build of an
application-server I once used (which will remain anonymous) this would
cause a fatal crash
if attempted in a Servlet. If it had not been for our own testing of this
edge-case, we
might never have discovered this problem, or worse, we might have
discovered it in production.

3.2 The Ittermitent Bug and Logging
   This is the worst and most time-consuming bug to fix. Most often, bugs
are reproduceable.
There are times, however, when one of the testers sees something strange
and it cannot be
reproduced. The bug may also occur so rarely that it is difficult to
diagnose the problem.
The key to tracking down these bugs is a logging system. Write everything
down. If that
involves writing to a file on your servxr, do it. If it involves sending
emails to everyone
in the company when a certain component fails, do it. I cannot stress the
importance of
logging more. It is worth the time or money to write your own logging
components or buy
someone else's. Logging is indispensable when attempting to fix this kind
of bug.

3.3 Built-in Testing
   One idea that is covered in John Lako's "Large Scale Software
Development in C++"
   In this note, I outline bl is the
idea of built-in testing. Developing a small testing component, or
driver, for each
class will save you time, money, and frustration. It may even help prove
that your code
is fine, and the system in which it is running is not. Building test
components will
increase development time marginally, but relieves stress in the long-run.
Although I have
used built-in testing on a fairly limited basis, I believe it should be
used as often
as possible and intend on using it in all future projects.
   The main advantage, I believe, to writing the testing component along
side of the
actual component is bug resolution time. If a bug is discovered a day
before a release
date, I would prefer not to mess about with the code, try to figure out
what I had
written 6 months earlier, and write a driver that will stress-test that
component. I want
that tool available so I can concentrate on why that piece is failing, and
not how it works.

4. Conclusion
   In closing, consider white-box testing an extension of black-box
testing. If the app
works, fulfills every user's hopes and dreams, and upper management likes
it, then a
successful black-box test is sufficient for a release. If you, as a
developer,
don't want your boss breathing down your neck wondering why something
didn't work or
why someone hacked root, then perform a white-box test. There will always
be bugs.
You will never solve all of them. Good luck.

5. Notes
1) One might consider replacing input/output with
starting-state/stopping-state
   in an object-oriented framework.
2) The identifiable input/output is the following:
   Step 1. Input: User registration information, Output: A confirmation of
regis-
           tration.
   Step 2. Input: Authentication credentials (username and password),
Output: A
           session-token (cookie) used as proof of authentication.
   Step 3. Input: Personalization data Output: Personalization
confirmation.
   Step 4. Input: Logout signal Output: Logout confirmation (invalidation
of
           session-token).
   Step 5. Input: Authentication credentials (username and password),
Output: A
           session-token (cookie).
   Step 6. Input: Request for personalization information, Output: Correct
per-
           sonalization information.
3) Ease and cost of distribution are key factors when preparing a release.
A web
   application release is trivial compared to, for example, embedded
systems. Web
   apps are typically released and patched frequently.
4) There is software which automates some of the testing process, but the
learning
   curve is often-times unwelcome and steep. See Rational Purify, OC
Systems Aprobe,
   Applied Microsystems CodeTEST. Mercury Interactive's WinRunner was
indispensable
   at my last company. The Unix tools tcov and lprof may also be helpful.
5) This practice is far more common than I ever imagined.

6. Bibliography

Cole, Oliver. "White-Box Testing." Dr. Dobb's Journal, March 2000.
http://www.ddj.com/documents/s=887/ddj0003a/0003a.htm

Rational Software. "Testing Object-Oriented Code"
http://www.rational.com/products/whitepapers/312.jsp

Lakos, John. Large-Scale C++ Software Design. Addison-Wesley Professional
Computing
Series, 1996.

Ian S. Graham, Alan Cameron Wills, Alan O'Callaghan. Object-Oriented
Methods:
Principles and Practice. Addison Wesley Longman, Inc., December 2000.

• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]