From: Raimundo Santos (raitechgmail.com)
Date: Tue Jun 04 2013 - 09:28:22 CDT

• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

I am guessing that the problem lies with flags S/SA.

Changing all rules to flags any, and the packets hits the rules, but things
go worse: no web navigation... this is driving me mad!

On 3 June 2013 13:09, Raimundo Santos <raitechgmail.com> wrote:

> Hi there!
>
> I asked, without an answer, something about nat-to and real IPs. Well, I
> really need an answer there, so if someone get a clue, I will be glad tho
> hear :)
>
> Now, to the new issue!
>
> Here in our WiFi ISP we are have contracted a tproxy service from FreeBSD
> Brasil. It is somehow working, but I can not figure out exactly how. Here
> is a diagram of the desired paths:
>
> http://devio.us/~raitech/Obsd53PfTproxy.png
>
> These are my rules by now:
>
> RFC1918 = "{ 172.16/12, 192.168/16, 10/8, 127/8 }"
> table <INT_NET> persist { internal nets, all valid IPs }
>
> ext_if_1 = "em0"
> ext_gw_1 = "187.72.X.X"
> ext_ip_1 = "187.72.X.X"
>
> ext_if_2 = "em1"
> ext_gw_2 = "187.72.X.X"
> ext_ip_2 = "187.72.X.X"
>
> ext_if_3 = "alc0"
> ext_gw_3 = "187.72.X.X"
> ext_ip_3 = "187.72.X.X"
>
> int_if_1 = "em2"
> int_gw_1 = "187.72.X.X"
> int_ip_1 = "187.72.X.X"
>
> squid_master_if = "em3"
> squid_master_gw = "187.72.X.X"
> squid_master_ip = "187.72.X.X"
>
> set limit states 6304000
> set limit tables 5000
> set limit src-nodes 200000
> set limit frags 3000
> set optimization aggressive
> set state-defaults pflow, no-sync
>
> set skip on lo
>
> block in log quick on { \
> $ext_if_1, \
> $ext_if_2, \
> $ext_if_3, \
> $squid_master_if, \
> $int_if_1 } from $RFC1918 label "blocking RFC1918"
>
> # trying to prioritizing ACKs...
> match set prio (3,5)
> # ... and all traffic http. https over the others
> match proto tcp to port { http, https } set prio (5,6)
> match proto tcp from port { http, https } set prio (5,6)
>
> match proto tcp to port { ssh, 9876 } set prio(5,7)
>
> pass in on $int_if_1 proto tcp from { <INT_NET>, $int_gw_1 } to port http \
> route-to ($squid_master_if $squid_master_gw)
>
> pass in on { $ext_if_1, $ext_if_2, $ext_if_3 } proto tcp from port http \
> to { <INT_NET>, $int_gw_1 } \
> route-to ($squid_master_if $squid_master_gw)
>
> pass in on $squid_master_if proto tcp from { <INT_NET>, $int_gw_1 } to \
> port http no state route-to \
> { \
> ($ext_if_1 $ext_gw_1) , \
> ($ext_if_2 $ext_gw_2) \
> } least-states label "cahce external outbound balancing"
>
> pass in on $squid_master_if proto tcp from port http \
> to { <INT_NET>, $int_gw_1 } route-to ($int_if_1 $int_gw_1) \
> label "cahce internal outbound routing"
>
> An here are a pfctl -vsr output:
>
> block drop in log quick on em0 inet from 172.16.0.0/12 to any label
> "blocking RFC1918"
> [ Evaluations: 61764339 Packets: 332 Bytes: 32854 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on em0 inet from 192.168.0.0/16 to any label
> "blocking RFC1918"
> [ Evaluations: 5883927 Packets: 114 Bytes: 28621 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on em0 inet from 10.0.0.0/8 to any label
> "blocking RFC1918"
> [ Evaluations: 5883813 Packets: 170 Bytes: 18354 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on em0 inet from 127.0.0.0/8 to any label
> "blocking RFC1918"
> [ Evaluations: 5883643 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on em1 inet from 172.16.0.0/12 to any label
> "blocking RFC1918"
> [ Evaluations: 60684174 Packets: 305 Bytes: 30912 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on em1 inet from 192.168.0.0/16 to any label
> "blocking RFC1918"
> [ Evaluations: 6862827 Packets: 93 Bytes: 9232 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on em1 inet from 10.0.0.0/8 to any label
> "blocking RFC1918"
> [ Evaluations: 6862734 Packets: 196 Bytes: 19396 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on em1 inet from 127.0.0.0/8 to any label
> "blocking RFC1918"
> [ Evaluations: 6862538 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on alc0 inet from 172.16.0.0/12 to any label
> "blocking RFC1918"
> [ Evaluations: 50726925 Packets: 304 Bytes: 30856 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on alc0 inet from 192.168.0.0/16 to any label
> "blocking RFC1918"
> [ Evaluations: 1251 Packets: 79 Bytes: 8268 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on alc0 inet from 10.0.0.0/8 to any label
> "blocking RFC1918"
> [ Evaluations: 1172 Packets: 152 Bytes: 16948 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on alc0 inet from 127.0.0.0/8 to any label
> "blocking RFC1918"
> [ Evaluations: 1020 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on em3 inet from 172.16.0.0/12 to any label
> "blocking RFC1918"
> [ Evaluations: 50726392 Packets: 304 Bytes: 30856 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on em3 inet from 192.168.0.0/16 to any label
> "blocking RFC1918"
> [ Evaluations: 13589809 Packets: 76 Bytes: 8132 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on em3 inet from 10.0.0.0/8 to any label
> "blocking RFC1918"
> [ Evaluations: 13589733 Packets: 152 Bytes: 16948 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on em3 inet from 127.0.0.0/8 to any label
> "blocking RFC1918"
> [ Evaluations: 13589581 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on em2 inet from 172.16.0.0/12 to any label
> "blocking RFC1918"
> [ Evaluations: 39571927 Packets: 10414 Bytes: 478685 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on em2 inet from 192.168.0.0/16 to any label
> "blocking RFC1918"
> [ Evaluations: 6364466 Packets: 1779 Bytes: 142401 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on em2 inet from 10.0.0.0/8 to any label
> "blocking RFC1918"
> [ Evaluations: 6362687 Packets: 32496 Bytes: 1375238 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> block drop in log quick on em2 inet from 127.0.0.0/8 to any label
> "blocking RFC1918"
> [ Evaluations: 6330191 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> match all set ( prio(3, 5) )
> [ Evaluations: 61717375 Packets: 13877464 Bytes: 10275632710 States:
> 3831 ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> match proto tcp from any to any port = 80 set ( prio(5, 6) )
> [ Evaluations: 61717375 Packets: 13877464 Bytes: 10275632710 States:
> 3831 ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> match proto tcp from any to any port = 443 set ( prio(5, 6) )
> [ Evaluations: 51200612 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> match proto tcp from any port = 80 to any set ( prio(5, 6) )
> [ Evaluations: 51200612 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> match proto tcp from any port = 443 to any set ( prio(5, 6) )
> [ Evaluations: 51200612 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> match proto tcp from any to any port = 22 set ( prio(5, 7) )
> [ Evaluations: 51200616 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> match proto tcp from any to any port = 9876 set ( prio(5, 7) )
> [ Evaluations: 51200616 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> pass all no state allow-opts
> [ Evaluations: 61717379 Packets: 61549113 Bytes: 41451833770 States:
> 0 ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> pass in on em0 inet proto tcp from any port = 80 to <INT_NET> flags S/SA
> keep state (no-sync, pflow) route-to 187.72.X.Xem3
> [ Evaluations: 61717379 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> pass in on em1 inet proto tcp from any port = 80 to <INT_NET> flags S/SA
> keep state (no-sync, pflow) route-to 187.72.X.Xem3
> [ Evaluations: 55197296 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> pass in on alc0 inet proto tcp from any port = 80 to <INT_NET> flags S/SA
> keep state (no-sync, pflow) route-to 187.72.X.Xem3
> [ Evaluations: 38378103 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> pass in on em0 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA
> keep state (no-sync, pflow) route-to 187.72.X.Xem3
> [ Evaluations: 48038032 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> pass in on em1 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA
> keep state (no-sync, pflow) route-to 187.72.X.Xem3
> [ Evaluations: 44966361 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> pass in on alc0 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA
> keep state (no-sync, pflow) route-to 187.72.X.Xem3
> [ Evaluations: 41608198 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> pass in on em2 inet proto tcp from <INT_NET> to any port = 80 flags S/SA
> keep state (no-sync, pflow) route-to 187.72.X.Xem3
> [ Evaluations: 48044445 Packets: 1439990 Bytes: 894473590 States:
> 435 ]
> [ Inserted: uid 0 pid 19584 State Creations: 40060 ]
> pass in on em2 inet proto tcp from 187.72.X.X to any port = 80 flags S/SA
> keep state (no-sync, pflow) route-to 187.72.X.Xem3
> [ Evaluations: 3694317 Packets: 12437474 Bytes: 9381159120 States:
> 3396 ]
> [ Inserted: uid 0 pid 19584 State Creations: 128206]
> pass in on em3 inet proto tcp from <INT_NET> to any port = 80 no state
> label "cahce external outbound balancing" route-to <__automatic_9ca6f8d9_0>
> least-states
> [ Evaluations: 38420511 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> pass in on em3 inet proto tcp from 187.72.X.X to any port = 80 no state
> label "cahce external outbound balancing" route-to <__automatic_9ca6f8d9_1>
> least-states
> [ Evaluations: 13586403 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> pass in on em3 inet proto tcp from any port = 80 to <INT_NET> flags S/SA
> keep state (no-sync, pflow) label "cahce internal outbound routing"
> route-to 187.72.X.Xem2
> [ Evaluations: 13731058 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
> pass in on em3 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA
> keep state (no-sync, pflow) label "cahce internal outbound routing"
> route-to 187.72.X.Xem2
> [ Evaluations: 13586403 Packets: 0 Bytes: 0 States: 0
> ]
> [ Inserted: uid 0 pid 19584 State Creations: 0 ]
>
> This is the same behavior with or without multipath routing. What
> bahavior? Well, only rules for in on em3 that are destineted to internal
> network are working, the others barelly catches a few thousands of packets.
> Very strange...
>
> But, as said before: more strange is the fact that the cache solution is
> almost working, just some delays to load a page here, youtube gasps there,
> but overall it seems to work!
>
> Tested without multipath routing, without keep state, and the behavior are
> the same.
>
> Will apreciate any kind of help on this, thank you in advance.
>
> Raimundo Santos

• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]