[nzlug] Ping/Traceroute issue.. Nagios involved.. VMWareinvolved..

[Michael Hutchinson]

Hi there,

I got another host that is doing the weird 'ping' thing from our
monitoring server, Bob. A machine sitting right next to Bob can ping the
target just fine, Bob has issues.

The tcpdump log suggests the ping packets are coming back from the
target, but for some reason ping doesn't recognise them. I used another
target, for logging reasons, where ping works fine. So here are the
logs, 192.168.51.2 is the problem target. The ping command from Bob for
target 51.2 just sits there and upon CTRL-C reports 100% packet loss. 
The other target 51.1, ping fine for Bob, and the logs reflect that the
packets are pretty much exactly the same to and from both hosts... so
what could be the issue here ?

I have played around with ttl and packetsizes but nothing changes. This
problem is intermittent, I would expect in another few hours that Bob is
able to ping the problem target again. The target, for all intents, is
fine.. Have RDP'd into it and it is all good.

Admittedly we have a bit of a 'route' but considering any other machine
on our side of the network can ping the target fine, we consider this
not to be an issue, but for those interested..

Bob -> Target map :

Bob(192.168.6.34) <-> Sonicwall(192.168.6.10) <-> Cisco
Router(192.168.6.1) <-> IPR Gateway(Telecom)(172.addressing,dynamic
routing) <-> DSLAM(telecom) <-> ADSLmodem(AR236) <-NAT->
PC(target(192.168.213..)

But.. if anything was happening there, then when we plug a box in right
next to Bob, it would not be able to ping the target properly either..
Could something be insane on Bob with the ping command ? BTW: have also
tried fping to see if my ping binary was broken, but no.. its not.


Bob:~# tcpdump -vvvv icmp | grep 192.168.51.2
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96
bytes
10:58:20.502747 IP (tos 0x0, ttl 152, id 3, offset 0, flags [DF],
length: 84) crookshanks.manux.net.nz > 192.168.51.2: icmp 64: echo
request seq 4
10:58:20.564799 IP (tos 0x0, ttl 152, id 16405, offset 0, flags [none],
length: 84) 192.168.51.2 > crookshanks.manux.net.nz: icmp 64: echo reply
seq 4
10:58:21.501970 IP (tos 0x0, ttl 152, id 4, offset 0, flags [DF],
length: 84) crookshanks.manux.net.nz > 192.168.51.2: icmp 64: echo
request seq 5
10:58:21.563406 IP (tos 0x0, ttl 152, id 16406, offset 0, flags [none],
length: 84) 192.168.51.2 > crookshanks.manux.net.nz: icmp 64: echo reply
seq 5
10:58:22.502284 IP (tos 0x0, ttl 152, id 5, offset 0, flags [DF],
length: 84) crookshanks.manux.net.nz > 192.168.51.2: icmp 64: echo
request seq 6
10:58:22.573132 IP (tos 0x0, ttl 152, id 16407, offset 0, flags [none],
length: 84) 192.168.51.2 > crookshanks.manux.net.nz: icmp 64: echo reply
seq 6
10:58:23.502803 IP (tos 0x0, ttl 152, id 6, offset 0, flags [DF],
length: 84) crookshanks.manux.net.nz > 192.168.51.2: icmp 64: echo
request seq 7
10:58:23.562834 IP (tos 0x0, ttl 152, id 16408, offset 0, flags [none],
length: 84) 192.168.51.2 > crookshanks.manux.net.nz: icmp 64: echo reply
seq 7

crookshanks:~# tcpdump -vvvv icmp | grep 192.168.51.1
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96
bytes
10:58:52.301738 IP (tos 0x0, ttl 152, id 0, offset 0, flags [DF],
length: 84) crookshanks.manux.net.nz > 192.168.51.1: icmp 64: echo
request seq 1
10:58:52.372701 IP (tos 0x0, ttl 152, id 38707, offset 0, flags [none],
length: 84) 192.168.51.1 > crookshanks.manux.net.nz: icmp 64: echo reply
seq 1
10:58:53.311979 IP (tos 0x0, ttl 152, id 1, offset 0, flags [DF],
length: 84) crookshanks.manux.net.nz > 192.168.51.1: icmp 64: echo
request seq 2
10:58:53.380485 IP (tos 0x0, ttl 152, id 38708, offset 0, flags [none],
length: 84) 192.168.51.1 > crookshanks.manux.net.nz: icmp 64: echo reply
seq 2
10:58:54.321327 IP (tos 0x0, ttl 152, id 2, offset 0, flags [DF],
length: 84) crookshanks.manux.net.nz > 192.168.51.1: icmp 64: echo
request seq 3
10:58:54.391120 IP (tos 0x0, ttl 152, id 38709, offset 0, flags [none],
length: 84) 192.168.51.1 > crookshanks.manux.net.nz: icmp 64: echo reply
seq 3
10:58:55.332160 IP (tos 0x0, ttl 152, id 3, offset 0, flags [DF],
length: 84) crookshanks.manux.net.nz > 192.168.51.1: icmp 64: echo
request seq 4
10:58:55.401460 IP (tos 0x0, ttl 152, id 38710, offset 0, flags [none],
length: 84) 192.168.51.1 > crookshanks.manux.net.nz: icmp 64: echo reply
seq 4


Cheers for any help in advance,
Michael Hutchinson
Manux Solutions Ltd
mhutchinson at manux.co.nz


-----Original Message-----
From: nzlug-bounces at linux.net.nz [mailto:nzlug-bounces at linux.net.nz] On
Behalf Of Robin Sheat
Sent: Thursday, 13 December 2007 12:37 p.m.
To: NZLUG Mailing List
Subject: Re: [nzlug] Ping/Traceroute issue.. Nagios involved..
VMWareinvolved..

On Thursday 13 December 2007 11:24:46 Michael Hutchinson wrote:
> I am not willing to install Wireshark on our server bob as it is
> considered a production server, and the pre-requisites for Wireshark
> would mean installing software that we do not want on there.
Fair enough. Wireshark can work with tcpdump files. Tcpdump on it's own
may be 
sufficient, given you're not dealing with a lot of data. 'tcpdump -s
2048 -w 
file' will produce a file you can import into wireshark for viewing
later. Or 
leave of the -w and see the data as it goes.

tcpdump will be easy to install, if it's not there already.

> I am thinking it may have something to do with netmask's, but am
unsure
> whether this could affect ping. My knowledge of appropriate netmasks
to
A netmask could affect ping, but it would be weird for it to be
intermittent.

Hey, a thought, it's not a machine with an incorrect IP address that's
only on 
sometimes is it? If another machine with the same IP address as another
came 
online, it could cause weirdness.

-- 
Robin <robin at kallisti.net.nz> JabberID: <eythian at jabber.kallisti.net.nz>

Hostes alienigeni me abduxerunt. Qui annus est?

PGP Key 0xA99CEB6D = 5957 6D23 8B16 EFAB FEF8  7175 14D3 6485 A99C EB6D