From: Aaron Martinez (mlproficuous.com)
Date: Thu Feb 21 2008 - 22:36:15 CST

• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Giancarlo Razzolini wrote:
> Aaron escreveu:
>
>> I am trying to configure ifstated on an i386 4.2 Stable pair of openbsd
>> firewalls but having some issues on how to determine connectivity of a
>> backup/secondary wan interface.
>>
>> The carp states seem solid and preempt seems to work great. The only
>> thing I'm really worried about is an upstream link dying, carp staying
>> master and traffic getting blackholed. I want ifstated to simply change
>> the default route to the backup wan interface should connectivity out
>> the primary get interrupted and then switch back when primary
>> connectivity comes back. I'm just trying to get it figured out on one
>> machine first before I move to the second. I'm having trouble figuring
>> out if there is connectivity on the backup wan interface. I read some
>> posts that suggested using ping -I so that the pings go out the
>> appropriate interface, but this seems to not work, if i try to ping
>> anything other than the backup wan's gateway, it still goes out the
>> default route.. It is only able to ping the gateway address and with
>> the (-r) option the pinged host has to be on the directly connected
>> network.
>>
> snip...
>
> Now this is what i call a lot of info. :) Well, let me drop my
> experience with ifstated and ping, and multiple wan links. First of all,
> avoid pinging external address like you would avoid devil himself. I say
> this, because it's isn't as reliable as many people think. How to do
> then? My advice is, use snmp. Almost all (if not all of them) network
> devices, which are fcc compliant, must have support for snmp, at least
> version 1 of it. Install net-snmp on openbsd, and do a snmpwalk on the
> modem, router, etc. Most of them will come with snmp enabled and with
> the default communities "public" and "private". As you won't be
> changing anything, i recommend using the public comm. Try this:
>
> snmpwalk -v 1 -c public <ip of your router>
>
> This will give you a lot of info. There is a snmp MIB called IF. As you
> might guess, it refers to the interfaces of the device. This is the mib
> you will most certainly use. Take a look at the following output from
> one of my adsl devices:
>
> IF-MIB::ifDescr.1 = STRING: loopback (pseudo ethernet)
> IF-MIB::ifDescr.2 = STRING: ti
> IF-MIB::ifDescr.3 = STRING: Bridge
> IF-MIB::ifDescr.4 = STRING: Ethernet
> IF-MIB::ifDescr.5 = STRING: Ethernet over USB
> IF-MIB::ifDescr.6 = STRING: ATM
> IF-MIB::ifDescr.7 = STRING: RFC-2684B PPPoE Proxy
> IF-MIB::ifDescr.8 = STRING: PPPoE
> IF-MIB::ifType.1 = INTEGER: ethernetCsmacd(6)
> IF-MIB::ifType.2 = INTEGER: adsl(94)
> IF-MIB::ifType.3 = INTEGER: ethernetCsmacd(6)
> IF-MIB::ifType.4 = INTEGER: ethernetCsmacd(6)
> IF-MIB::ifType.5 = INTEGER: ethernetCsmacd(6)
> IF-MIB::ifType.6 = INTEGER: atm(37)
> IF-MIB::ifType.7 = INTEGER: ethernetCsmacd(6)
> IF-MIB::ifType.8 = INTEGER: ppp(23)
> .
> .
> .
> IF-MIB::ifOperStatus.1 = INTEGER: up(1)
> IF-MIB::ifOperStatus.2 = INTEGER: up(1)
> IF-MIB::ifOperStatus.3 = INTEGER: up(1)
> IF-MIB::ifOperStatus.4 = INTEGER: up(1)
> IF-MIB::ifOperStatus.5 = INTEGER: down(2)
> IF-MIB::ifOperStatus.6 = INTEGER: up(1)
> IF-MIB::ifOperStatus.7 = INTEGER: up(1)
> IF-MIB::ifOperStatus.8 = INTEGER: up(1)
>
> The ifDescr atributes tell you what kind of if is this. In this specific
> device i monitor the adsl, atm, and ppp if's. Any one of those that goes
> down, mean that your wan link is dead. The attribute that you will use
> to check it is the ifOperStatus. If some of the 3 if's i mentioned
> before is with it ifOperStatus down, instead of up, you certainly is
> with your wan link down. Then you can easily create a shell script to
> accomplish this task. As ifstated tests expect a 0 for success and 1 for
> error, your script only need to return this. Then you can call it
> directly from ifstated, beside this you can overcheck and se if there
> physical ethernet link. I do 3 checks in my ifstated: First the snmp
> check, to check for wan connectivity directly with the device, second i
> do ping the router to see if i can reach it (i know what i said about
> ping before, but this one is different, and can help you) and third i
> check the ethernet interface for connectivity. This way you can deal
> with the 3 cases: no wan connectivity, but if up and the router up (call
> your provider), no wan connectivity, and the router is down, but the if
> is up (most certainly your router is hang up, so take a look at it) and
> the third case, the if is down, can have 2 meanings: the router is
> totally down or the physical if is with problems. This way i can say you
> are all covered up. :) I also send nice mails to myself to inform of the
> three cases.
>
> Now to the failover part. It's not a good thing to change the router of
> the firewall itself. It will die the connections of clients instantly,
> what isn't a good thing. Instead, change the route of then using the
> route-to statement of pf, and let the new conn's migrate to the other
> wan link. Do this to avoid the connections dying when the primary link
> backs up. I had this problems, as i do have 3 wan connections. Well,
> there is much more to do, but the principle is here. I can help you with
> more examples if you want.
>
> My regards,
>
> --
> Giancarlo Razzolini
> Linux User 172199
> Red Hat Certified Engineer no:804006389722501
> Moleque Sem Conteudo Numero #002
> Slackware Current
> OpenBSD Stable
> Ubuntu 6.10 Edgy Eft
> Snike Tecnologia em Informatica
> 4386 2A6F FFD4 4D5F 5842 6EA0 7ABE BBAB 9C0E 6B85
>
> [demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]
>
>
This sounds really great.. and I've seen people suggest snmp as an
option in the past, but my snmp knowledge is somewhat limited. I'd very
much appreciate some guidance in this arena (off list if more
appropriate). One thing about the machines I've put together is, they
have 0 non-base packages installed. I am not sure of the requirements
of snmp, but i'm trying like the plague to keep this box minimal, if
nothing else I would like to avoid running into the libexpat issue that
will require me to install x-base.

I'd just like to point out as well, I did manage to get the ping going
out the correct interface and to use the proper route, with the
following route-to statements in my pf.conf, but i'm not exactly sure
why it's working since it doesn't seem like passing out on fxp1 and then
routing to fxp0 and vice versa would have much effect since if the
packet was passed out on fxp0, seems as if it'd be past the point of
being "routed"

pass out on fxp1 route-to ( fxp0 192.168.3.94 ) inet proto icmp from
fxp0 to any
pass out on fxp0 route-to ( fxp1 192.168.2.190 ) inet proto icmp from
fxp1 to any

I'm particularly interested in not changing the default route and using
the route-to as you suggested, as I ran into the immediate death of my
connection when the route changed.. and of course it wouldn't pick up
with the current download after the route change. I'd be very grateful
too if you can point me in the right direction to how i can prevent this.

Here is what i have as my ifstated.conf (mainly a modification of the
default with the addition of routing changes) and please note I wasn't
finished with it, but thought maybe if i was doing something blatantly
wrong, someone could point them out or me in the right direction.

# whether we have connectivity. Make sure the hosts are always up, or
# test multiple ip's, 'or'-ing the tests.
pnet = '( "ping -q -c 1 -w 1 -I 192.168.3.66 192.168.3.94 > /dev/null"
every 10 && \
   "ping -q -c 1 -w 1 -I 192.168.3.66 www.hotbot.com > /dev/null" every
10 || \
       "ping -q -c 1 -w 1 -I 192.168.3.66 192.168.3.94 > /dev/null"
every 10 && \
       "ping -q -c 1 -w 1 -I 192.168.3.66 www.yahoo.com > /dev/null"
every 10 )'

bnet = '( "ping -q -c 1 -w 1 -I 192.168.2.162 192.168.2.190 > /dev/null"
every 10 && \
   "ping -q -c 1 -w 1 -I 192.168.2.162 www.hotbot.com > /dev/null" every
10 || \
       "ping -q -c 1 -w 1 -I 192.168.2.162 192.168.2.190 > /dev/null"
every 10 && \
       "ping -q -c 1 -w 1 -I 192.168.2.162 www.yahoo.com > /dev/null"
every 10 )'

# The peer addresses below are the real ip addresses of the OTHER firewall
peer = '( "ping -q -c 1 -w 1 -I 192.168.3.66 192.168.3.67 > /dev/null"
every 10 && \
   "ping -q -c 1 -w 1 -I 192.168.2.162 192.168.2.163 > /dev/null" every 10)'

state auto {
       if $carp_up && $pnet
               set-state primary
       if $carp_down || !$pnet
               set-state backup
}

state primary {
       init {
               run "ifconfig carp0 advskew 10"
               run "ifconfig carp1 advskew 10"
               run "ifconfig carp2 advskew 10"
               run "ifconfig carp3 advskew 10"
               run "route delete default && route add default 192.168.3.94"
               run 'echo "using pnet route" | mailx -s "switch route"
<email address>'
       }
       if (! $pnet || ! $carp_up) && $bnet
               set-state demoted
}

state demoted {
       init {
               run "ifconfig carp0 advskew 254"
               run "ifconfig carp1 advskew 254"
               run "ifconfig carp2 advskew 254"
               run "ifconfig carp3 advskew 254"
               run "route delete default && route add default 192.168.2.190"
               run 'echo "using bnet route" | mailx -s "switch route"
<email address>'
       }
       if $pnet && $bnet
               set-state primary
        if $carp_sync && ! $pnet
                set-state broute
}

state promoted {
       init {
               run "ifconfig carp0 advskew 0"
               run "ifconfig carp1 advskew 0"
               run "ifconfig carp2 advskew 0"
               run "ifconfig carp3 advskew 0"
               run "route delete default && route add default 192.168.3.94"
               run 'echo "using pnet route" | mailx -s "which route"
<email address>'
       }
# if $peer || ! $net
       if ! $pnet && $bnet
               set-state backup
}

state backup {
       init {
               run "ifconfig carp0 advskew 100"
               run "ifconfig carp1 advskew 100"
               run "ifconfig carp2 advskew 100"
               run "ifconfig carp3 advskew 100"
               run "route delete default && route add default 192.168.2.190"
               run 'echo "using bnet route" | mailx -s "which route"
<email address>'
       }
       # The "sleep 5" below is a hack to dampen the $carp_sync when we come
       # out of promoted state. Thinking about the correct fix...
       if ! $carp_sync && $pnet && "sleep 5" every 10
               if ! $carp_sync && $pnet
                       set-state promoted
}
# if ! $pnet && $carp_up && $bnet
# set-state broute

state broute {
       init {
                run "route delete default && route add default
192.168.2.190"
               run 'echo "using bnet route" | mailx -s "which route"
<email address>'
       }
        if $carp_up && $pnet
                set-state proute
}

state proute{
        init {
                run "route delete default && route add default 192.168.3.94"
                run 'echo "using pnet route" | mailx -s "which route"
<email address>'
        }
        if $carp_sync && ! $pnet
                set-state broute
}

• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]