Junos OS

 View Only
last person joined: yesterday 

Ask questions and share experiences about Junos OS.
  • 1.  MX960: High FPC CPU

    Posted 02-29-2012 02:48

    Good day,

    I have MX960 with 2 RE. ver 10.0R4.7

    Yesterday after switchover to backup RE (disk fail was the reason), I see high CPU utilization for fpc0 and fpc2  DPCE-R line cards.

    I changed the cards, sfps but it not help. Also I found out if I disable pics xe-2/0/0 and xe-2/1/0 the cpu utilization for fpc2 decrease to normal ~10-15%.

    Any ideas?

     

    evgeniy@mx960-slave> show chassis fpc

                         Temp  CPU Utilization (%)   Memory    Utilization (%)

    Slot State            (C)  Total  Interrupt      DRAM (MB) Heap     Buffer

      0  Online            29     86          2       1024       46         31

      1  Online            27     10          0       1024       46         31

      2  Online            25    100          3       1024       46         30

      3  Online            24     12          0       1024       46         31

      4  Online            25     12          0       1024       46         31

     

    Regards,

    Evgeniy


    #FPC
    #CPU
    #mx


  • 2.  RE: MX960: High FPC CPU

    Posted 02-29-2012 03:23

    Hey,

     

    You didn't say much about yout config, unfortunately. Are you using GRES+NSR? Did you see RE CPU utilization spike as well (I guess not as you didn't mention, but worth to check)? Is there anything DPC related in the logs? When you log into DPC (> start shell pfe network fpc2), can you see any errors being thrown in the console?

    The interfaces you disabled, do they have any filters applied (if so, can you remove them)? Are these part of VPLS/bridge-domain?

     

    There is number of PRs related to RPD issues during/after switchover, however based on your description there is nothing obvious matching PFE spike.

     

    Thanks,



  • 3.  RE: MX960: High FPC CPU

    Posted 02-29-2012 07:09

    Hi,
    > You didn't say much about yout config, unfortunately
    Mainly ipv4 and v6 routing. A lot of bgp sessions.

    >Are you using GRES+NSR?
    but I tried disable, without result.

    >Did you see RE CPU utilization spike as well
    no. Just everyday utilization

     PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
       11 root        1 171   52     0K    12K RUN    9293.6 81.25% idle
    88095 root        2   8  -88 38580K 16072K nanslp 180:31  6.05% chassisd
       20 root        1 -68 -187     0K    12K RUN     83.6H  1.95% irq10: em0 em1+++*
       12 root        1 -40 -159     0K    12K WAIT    34.5H  0.93% swi2: net
    87047 root        1   4    0   968M   948M kqread  58:46  0.49% rpd
     2880 root        1   4    0     0K    12K peer_s   3:01  0.24% peer proxy

     >Is there anything DPC related in the logs?
     No. Nothing strange in log about fpc

     > When you log into DPC (> start shell pfe network fpc2), can you see any errors being thrown in the console?
     No errors in console, but I see the following -

     ADPC2(mx960-slave vty)# show sched
    Total uptime 0+07:20:50, (26450900 ms), 183897628 thread dispatches
    CPU load is 100% (5 second), 91% (1 minute)
    Total network interrupt time 0 (usec)

     CPU       Name      Time(ms)
      0%       Idle      155765
     96%    Threads      25441545
      3%        ISR      853590
      3%    Level 1      818515
      0%    Level 2      31430
      0%    Level 3      5
      0%    Level 5      3640

      Threads the issues of high CPU.



  • 4.  RE: MX960: High FPC CPU

    Posted 02-29-2012 13:38

    Hey,

     

    I can see you posted some pfe logs on jnsp. What is the utilization (bps but more importantly pps) of the interfaces on those hot DPCs? What is connected from remote side? Any chance you're getting lot of ARP requests or some multicast traffic?

     

    Thanks.

     



  • 5.  RE: MX960: High FPC CPU

    Posted 03-01-2012 07:28

    Hi,

    the issue was related the bridge-domain.

    After removing this portion of configuration the CPU is ok.  But there's not any l2 loops in the topology.

    Its seems a software bug...