Junos OS

Dear all, 

I have several EX2300-T units running junos-arm-32-15.1X53-D56 that sometimes freeze and have the following messages : ( I have licenses for OSPF2 and all other services)

All my units are in Virtual-Chassis of 2 units; 

:08  CAI-EX2300-VSAT1-AB rpd[14775]: RPD_KRT_Q_RETRIES: Route Update

: No buffer space available

Oct 12 12:04:11  CAI-EX2300-VSAT1-AB kernel: rt_pfe_veto: Memory over consumed.

Op 3 err 55, rtsm_id 0:-1, msg type 10

Oct 12 12:04:16  CAI-EX2300-VSAT1-AB kernel: rt_pfe_veto: Memory over consumed.

Op 3 err 55, rtsm_id 0:-1, msg type 2

Oct 12 12:04:18  CAI-EX2300-VSAT1-AB rpd[14775]: RPD_KRT_Q_RETRIES: Route Update

: No buffer space available

Oct 12 12:04:21  CAI-EX2300-VSAT1-AB kernel: rt_pfe_veto: Memory over consumed.

Op 3 err 55, rtsm_id 5:-1, msg type 2

Oct 12 12:04:26  CAI-EX2300-VSAT1-AB kernel: rt_pfe_veto: Memory over consumed.

Op 3 err 55, rtsm_id 0:-1, msg type 10

Oct 12 12:04:28  CAI-EX2300-VSAT1-AB rpd[14775]: RPD_KRT_Q_RETRIES: Route Update

: No buffer space available

Oct 12 12:04:31  CAI-EX2300-VSAT1-AB kernel: rt_pfe_veto: Memory over consumed.

Op 2 err 55, rtsm_id 0:-1, msg type 6

Oct 12 12:04:36  CAI-EX2300-VSAT1-AB kernel: rt_pfe_veto: Memory over consumed.

Op 3 err 55, rtsm_id 5:-1, msg type 2

Oct 12 12:04:38  CAI-EX2300-VSAT1-AB rpd[14775]: RPD_KRT_Q_RETRIES: Route Update

: No buffer space available

Oct 12 12:04:41  CAI-EX2300-VSAT1-AB kernel: rt_pfe_veto: Memory over consumed.

Op 3 err 55, rtsm_id 0:-1, msg type 10

Oct 12 12:04:46  CAI-EX2300-VSAT1-AB kernel: rt_pfe_veto: Memory over consumed.

Op 2 err 55, rtsm_id 27:-1, msg type 5

Oct 12 12:04:49  CAI-EX2300-VSAT1-AB rpd[14775]: RPD_KRT_Q_RETRIES: Route Update

I could not identify what could be the cause. Any idea where it could come from? 

Failure is random and occurs once every 1-2 weeks depending the amount of units I have on the systems. 

Thank you

Best regards, 

how many routes are are you learning via OSPF and have directly connected? The error sounds like there is no more capacity in the PFE for routing entries.

The EX2300 only handles up to 512 prefixes which is seen in the datasheet on page 6: https://www.juniper.net/assets/us/en/local/pdf/datasheets/1000579-en.pdf

Hi Folks,

Just my 2 cents on this…

If the observation is repeating immediately after reboot; then it could be a scaling issue.

Else, if the objections shows up after couple of days; it could be a memory leak and a hardware BUG.

Hi Python, 

this is not just aftrer rebbot, it's usually happenning after few days which makes it quite difficult to troubleshoot. 

If it's a memory leak or hardware bug, I believe that I should open a case at JTAC?

Thank you

best regards, 

Hi Khrysthof,

A JTAC case should help in case of memory leak or hardware issue.

Memory leaks would be an issue with the version of Junos not the hardware.

I don't see any memory related PR with OSPF active in the public database.  You could try upgrading to the d57 release before opening a ticket to see if the issue is resolved.  But it is not listed as a resolved issue in the release notes.

After investigation with JTAC, I was finally advised to upgrade due to memory leak in sysctl net.routetable/sysctl_rtsock().

Hi,

Thank you for the answer. About 100 routes max. 

When I do a show route, I believe that I'm far from the limit. 

best regards,