We have a EX2300 (version 18.2R3-S2.9) in a Virtual Chassis that randomly went offline today. I can SSH to it locally but it cannot ping the firewall or out to the internet. It shows the port up up to the firewall ge-0/0/24 and the firewall (MX95) also shows a connection on the port it connects to. It then came back online several hours later. Here are the logs while it was offline:
root@VC2_250> show virtual-chassis
Virtual Chassis ID: 26f8.7fac.dd96
Virtual Chassis Mode: Mixed
Mstr Mixed Route Neighbor List
Member ID Status Serial No Model prio Role Mode Mode ID Interface
0 (FPC 0) Prsnt JW3621040592 ex2300-48p 128 Master* Y VC 1 vcp-255/1/3
1 (FPC 1) Prsnt JW3621040457 ex2300-48p 128 Backup Y VC 0 vcp-255/1/2
2 vcp-255/1/3
2 (FPC 2) Prsnt JW3621040599 ex2300-48p 128 Linecard Y VC 1 vcp-255/1/2
All show log messages from today
Sep 17 02:43:24 VC2_250 xntpd: kernel time sync enabled 6001
Sep 17 03:00:27 VC2_250 xntpd: kernel time sync enabled 2001
Sep 17 04:42:52 VC2_250 xntpd: kernel time sync enabled 6001
Sep 17 04:59:56 VC2_250 xntpd: kernel time sync enabled 2001
Sep 17 05:34:08 VC2_250 xntpd: kernel time sync enabled 6001
Sep 17 05:41:40 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_UP: ifIndex 546, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-1/0/27
Sep 17 05:42:17 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_DOWN: ifIndex 546, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/27
Sep 17 05:42:21 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_UP: ifIndex 546, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-1/0/27
Sep 17 05:51:11 VC2_250 xntpd: kernel time sync enabled 2001
Sep 17 06:25:22 VC2_250 xntpd: kernel time sync enabled 6001
Sep 17 06:42:27 VC2_250 xntpd: kernel time sync enabled 2001
Sep 17 06:59:27 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_DOWN: ifIndex 546, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/27
Sep 17 07:27:00 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_UP: ifIndex 546, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-1/0/27
Sep 17 08:24:56 VC2_250 xntpd: kernel time sync enabled 6001
Sep 17 08:29:54 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_DOWN: ifIndex 546, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/27
Sep 17 08:42:02 VC2_250 xntpd: kernel time sync enabled 2001
Sep 17 09:20:58 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_UP: ifIndex 546, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-1/0/27
Sep 17 09:20:59 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_DOWN: ifIndex 546, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/27
Sep 17 09:21:03 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_UP: ifIndex 546, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-1/0/27
Sep 17 12:31:29 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_DOWN: ifIndex 546, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/27
Sep 17 12:31:35 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_UP: ifIndex 546, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-1/0/27
Sep 17 12:31:46 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_DOWN: ifIndex 546, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/27
Sep 17 12:31:50 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_UP: ifIndex 546, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-1/0/27
Sep 17 12:32:11 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_DOWN: ifIndex 546, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/27
Sep 17 12:32:31 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_UP: ifIndex 546, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-1/0/27
Sep 17 12:36:59 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_DOWN: ifIndex 546, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/27
Sep 17 12:37:18 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_UP: ifIndex 546, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-1/0/27
Sep 17 12:37:19 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_DOWN: ifIndex 546, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/27
Sep 17 12:37:23 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_UP: ifIndex 546, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-1/0/27
Sep 17 12:53:22 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_DOWN: ifIndex 552, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/33
Sep 17 12:53:24 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_UP: ifIndex 552, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-1/0/33
Sep 17 13:15:26 VC2_250 xntpd: NTP Server 132.163.96.3 is Unreachable
Sep 17 13:19:20 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_DOWN: ifIndex 546, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/27
Sep 17 13:32:28 VC2_250 xntpd: NTP Server 132.163.96.3 is Unreachable
Sep 17 13:32:42 VC2_250 last message repeated 7 times
Sep 17 13:32:44 VC2_250 xntpd: NTP Server 132.163.96.3 is Unreachable
Sep 17 13:49:46 VC2_250 xntpd: NTP Server 132.163.96.3 is Unreachable
Sep 17 13:50:02 VC2_250 last message repeated 8 times
Sep 17 14:07:03 VC2_250 xntpd: NTP Server 132.163.96.3 is Unreachable
Sep 17 14:07:13 VC2_250 last message repeated 5 times
Sep 17 14:07:19 VC2_250 last message repeated 3 times
Sep 17 14:24:23 VC2_250 xntpd: NTP Server 132.163.96.3 is Unreachable
Sep 17 14:24:39 VC2_250 last message repeated 8 times
Sep 17 14:41:43 VC2_250 xntpd: NTP Server 132.163.96.3 is Unreachable
Sep 17 14:41:59 VC2_250 last message repeated 8 times
Sep 17 14:59:01 VC2_250 xntpd: NTP Server 132.163.96.3 is Unreachable
Sep 17 14:59:13 VC2_250 last message repeated 6 times
Sep 17 14:59:17 VC2_250 last message repeated 2 times
Sep 17 15:16:20 VC2_250 xntpd: NTP Server 132.163.96.3 is Unreachable
Sep 17 15:16:36 VC2_250 last message repeated 8 times
Sep 17 15:33:24 VC2_250 sshd[28401]: error: PAM: authentication error for root from 10.21.250.253
Sep 17 15:33:24 VC2_250 sshd: SSHD_LOGIN_FAILED: Login failed for user 'root' from host '10.21.250.253'
Sep 17 15:33:24 VC2_250 sshd[28401]: Postponed keyboard-interactive for root from 10.21.250.253 port 52642 ssh2 [preauth]
Sep 17 15:33:24 VC2_250 sshd[28402]: Postponed keyboard-interactive for root from 10.21.250.253 port 52642 ssh2
Sep 17 15:33:31 VC2_250 sshd[28401]: Postponed keyboard-interactive/pam for root from 10.21.250.253 port 52642 ssh2 [preauth]
Sep 17 15:33:31 VC2_250 sshd[28401]: Accepted keyboard-interactive/pam for root from 10.21.250.253 port 52642 ssh2
Sep 17 15:33:39 VC2_250 xntpd: NTP Server 132.163.96.3 is Unreachable
Sep 17 15:33:43 VC2_250 last message repeated 2 times
Sep 17 15:33:55 VC2_250 last message repeated 6 times
Sep 17 15:35:13 VC2_250 mgd[28412]: UI_CHILD_EXITED: Child exited: PID 28424, status 1, command '/sbin/ifinfo'
Sep 17 15:50:10 VC2_250 l2cpd[4982]: Root bridge in routing-instance 'default' changed from 32768:40:8f:9d:b1:5b:66 to 32768:b4:8a:5f:11:2b:94
Sep 17 15:50:10 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_DOWN: ifIndex 594, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/24
Sep 17 15:50:14 VC2_250 mib2d[4970]: SNMP_TRAP_LINK_UP: ifIndex 594, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/24
Sep 17 15:50:16 VC2_250 l2cpd[4982]: Root bridge in routing-instance 'default' changed from 32768:b4:8a:5f:11:2b:94 to 32768:40:8f:9d:b1:5b:66
Sep 17 15:50:16 VC2_250 l2cpd[4982]: ROOT_PORT: for Instance 0 in routing-instance default Interface ge-0/0/24.0
Sep 17 15:50:16 VC2_250 l2cpd[4982]: TOPO_CH: for Instance 0 in routing-instance default generated on port ge-0/0/24.0
Sep 17 15:50:57 VC2_250 xntpd: NTP Server 132.163.96.3 is Unreachable
Sep 17 15:51:03 VC2_250 xntpd: kernel time sync enabled 6001
Sep 17 15:51:05 VC2_250 xntpd: kernel time sync enabled 2001
Show log chassd
show interface ge-0/0/24
Then it came back online and these are the messages in show log messages. Seems there was a root bridge change and the port facing the firewall bounced.
Sep 17 15:35:13 VC2_250-IND mgd[28412]: UI_CHILD_EXITED: Child exited: PID 28424, status 1, command '/sbin/ifinfo'
Sep 17 15:50:10 VC2_250-IND l2cpd[4982]: Root bridge in routing-instance 'default' changed from 32768:40:8f:9d:b1:5b:66 to 32768:b4:8a:5f:11:2b:94
Sep 17 15:50:10 VC2_250-IND mib2d[4970]: SNMP_TRAP_LINK_DOWN: ifIndex 594, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/24
Sep 17 15:50:14 VC2_250-IND mib2d[4970]: SNMP_TRAP_LINK_UP: ifIndex 594, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/24
Sep 17 15:50:16 VC2_250-IND l2cpd[4982]: Root bridge in routing-instance 'default' changed from 32768:b4:8a:5f:11:2b:94 to 32768:40:8f:9d:b1:5b:66
Sep 17 15:50:16 VC2_250-IND l2cpd[4982]: ROOT_PORT: for Instance 0 in routing-instance default Interface ge-0/0/24.0
Sep 17 15:50:16 VC2_250-IND l2cpd[4982]: TOPO_CH: for Instance 0 in routing-instance default generated on port ge-0/0/24.0
Sep 17 15:50:57 VC2_250-IND xntpd: NTP Server 132.163.96.3 is Unreachable
Sep 17 15:51:03 VC2_250-IND xntpd: kernel time sync enabled 6001
Sep 17 15:51:05 VC2_250-IND xntpd: kernel time sync enabled 2001
root@VC2_250> show spanning-tree bridge
STP bridge parameters
Routing instance name : GLOBAL
Context ID : 0
Enabled protocol : RSTP
Root ID : 32768.40:8f:9d:b1:5b:66
Root cost : 20000
Root port : ge-0/0/24
Hello time : 2 seconds
Maximum age : 20 seconds
Forward delay : 15 seconds
Message age : 1
Number of topology changes : 2
Time since last topology change : 1613 seconds
Local parameters
Bridge ID : 32768.b4:8a:5f:11:2b:94
Extended system ID : 0
show configuration protocols
protocols {
dot1x {
authenticator {
authentication-profile-name nac;
interface {
Registration {
supplicant multiple;
transmit-period 10;
mac-radius {
restrict;
}
supplicant-timeout 3;
server-timeout 5;
server-fail deny;
}
}
}
}
lldp {
interface all;
}
lldp-med {
interface all;
}
igmp-snooping {
vlan all;
}
layer2-control {
bpdu-block {
interface Registration;
}
}
rstp {
max-age 20;
hello-time 2;
forward-delay 15;
interface ge-0/0/25 {
mode point-to-point;
}
interface Management;
}
}
It's designating the firewall port as root because there is a another stack that connects to LAN 1 of the firewall (this is connected to LAN2) that it is learning through.
root@VC2_250> show spanning-tree interface
Spanning tree interface parameters for instance 0
Interface Port ID Designated Designated Port State Role
port ID bridge ID Cost
ge-0/0/24 128:490 128:505 32768.408f9db15b66 20000 FWD ROOT
ge-0/0/25 128:491 128:491 32768.b48a5f112b94 20000 FWD DESG
ge-0/0/26 128:492 128:492 32768.b48a5f112b94 200000000 BLK DIS
ge-0/0/27 128:493 128:493 32768.b48a5f112b94 200000000 BLK DIS
ge-0/0/28 128:494 128:494 32768.b48a5f112b94 200000000 BLK DIS
ge-0/0/29 128:495 128:495 32768.b48a5f112b94 200000000 BLK DIS
ge-0/0/30 128:496 128:496 32768.b48a5f112b94 20000 FWD DESG
ge-0/0/31 128:497 128:497 32768.b48a5f112b94 20000 FWD DESG
ge-0/0/32 128:498 128:498 32768.b48a5f112b94 20000 FWD DESG
ge-0/0/33 128:499 128:499 32768.b48a5f112b94 20000 FWD DESG
ge-0/0/34 128:500 128:500 32768.b48a5f112b94 20000 FWD DESG
ge-0/0/35 128:501 128:501 32768.b48a5f112b94 20000 FWD DESG
ge-0/0/36 128:502 128:502 32768.b48a5f112b94 20000 FWD DESG
ge-0/0/37 128:503 128:503 32768.b48a5f112b94 20000 FWD DESG
ge-0/0/38 128:504 128:504 32768.b48a5f112b94 200000000 BLK DIS
ge-0/0/39 128:505 128:505 32768.b48a5f112b94 200000000 BLK DIS
Root switch that is connected to LAN 1 of the firewall.
root@VC1> show spanning-tree bridge
STP bridge parameters
Routing instance name : GLOBAL
Context ID : 0
Enabled protocol : RSTP
Root ID : 32768.40:8f:9d:b1:5b:66
Hello time : 2 seconds
Maximum age : 20 seconds
Forward delay : 15 seconds
Message age : 0
Number of topology changes : 11
Time since last topology change : 1975 seconds
Local parameters
Bridge ID : 32768.40:8f:9d:b1:5b:66
Extended system ID : 0
So it looks like the root bridge changed from VC2 back to VC1 based on the error message. But why did this happen in the way it did and why did it cause loss of internet access ( is it because root bridge isn't explicitly set?).
Sep 17 15:50:16 VC2_250-IND l2cpd[4982]: Root bridge in routing-instance 'default' changed from 32768:b4:8a:5f:11:2b:94 to 32768:40:8f:9d:b1:5b:66
Still a novice at this so go easy on me thanks! :)
------------------------------
Francis Heroux
------------------------------