OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Subject: Re: small vs big boxes (was: hmmmm....)
From: Brad Knowles (blkskynet.be)
Date: Tue Sep 19 2000 - 17:26:37 CDT

At 6:10 PM -0400 2000/9/19, Wietse Venema wrote:

> Basically, the name of the game was doing rdist from a couple
> prototype directory trees with configuration stuff, one prototype
> tree for each system "variant".

        Right, we did that for a while. IIRC, we even used tools like
SCCS or RCS or CVS (I can't remember which) to maintain a version
history of the files that would be pushed out, so that we could
easily roll back to a previous version if we ever found the need.

> The hard part was maintaining the
> discipline to keep the prototype trees up to date rather than making
> changes to individual systems.

        Maybe that's what we were doing wrong. I dunno. I just know
that we had a hard time of making this work as well as we would have
liked.

> This was more than 5 years ago, so the tools may be out of date.

        The basic tools should still suffice, although there may be
better stuff now available.

> A lot of monitoring was cron driven, like looking for thresholds
> and changes in ps, df, and the stuff that everyone is supposed to
> watch for system health. I used a comp.sources.unix package called
> watcher that would occasionally send mail if something was going
> out of bounds.

        We're doing a lot of system historical performance monitoring for
these sorts of things using UCD SNMP, although we'd like to be able
to look at a lot more different things than is currently possible.
We'd also like that same SNMP system to tie into something like mon,
so that we can automatically get alerts sent to us via SMS if there
are serious problems.

        I had some basic stuff built on top of mon 0.38 a while back, but
it hung on the need to have a database to tell it who was on call for
what date (we have a rotating schedule), and who to notify at what
time of the day/shift. I could work out the latter part, but I never
found a good solution for the first part.

> I've been doing UNIX since 1985, so about the same years, although
> we have been working on different things. My sysadmin career lasted
> only for six years or so (aside from managing my own systems).

        Then obviously you're just a more intelligent guy than I am.
That's okay, I can accept that.

> Even a 1-unit height box can be RAIDed just fine.

        Yeah. Sun sells a three-disk unit in the Netra st D130, and
Maxtor just announced a NAS device supposedly capable of storing
~320GB in 1U.

> The problem is not MBTF of disks, it's maintaining the consistency
> of a group of identical systems.

        These are two different problems. One is a hardware issue (how
can you build reasonably reliable systems with good MTBF numbers?),
and the other is a people issue (how can you manage those machines in
a consistent fashion?).

> That is a people problem. Everyone
> needs some sort of configuration control and change management lest
> they end up in chaos. It just becomes more difficult when more
> people are involved.

        And, in my experience, it becomes more difficult when more
machines are involved. 

--
   These are my opinions -- not to be taken as official Skynet policy
======================================================================
Brad Knowles, <blkskynet.be>                || Belgacom Skynet SA/NV
Systems Architect, Mail/News/FTP/Proxy Admin || Rue Colonel Bourg, 124
Phone/Fax: +32-2-706.13.11/12.49             || B-1140 Brussels
http://www.skynet.be                         || Belgium

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.