Junos OS

 View Only
last person joined: 7 days ago 

Ask questions and share experiences about Junos OS.

  • 1.  MX480 jtree error

    Posted 07-14-2022 14:59
    Hello All,

    We have an MX480  as our core router - which we wan to e receive full routes table from different upstream providers 
    We add 2 providers and we started receive errors in our log messages :


    Jul 14 17:19:36 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: route process failed
    Jul 14 17:19:36 TC-ISP-PE-A-RE0 fpc0 RT: Failed prefix add IPv4 - 165.220.236/23 (No memory) on FE 0
    Jul 14 17:19:36 TC-ISP-PE-A-RE0 fpc0 RT: IPv4:8 - 165.220.236/23 (add rt entry into jtree failed)
    Jul 14 17:19:36 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: rt_halp_vectors->rt_create failed
    Jul 14 17:19:36 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: proto ipv4,len 23 prefix 165.220.236/23 nh 1048589
    Jul 14 17:19:37 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: route process failed
    Jul 14 17:19:37 TC-ISP-PE-A-RE0 fpc0 RT: Failed prefix add IPv4 - 165.220.233/24 (No memory) on FE 0
    Jul 14 17:19:37 TC-ISP-PE-A-RE0 fpc0 RT: IPv4:8 - 165.220.233/24 (add rt entry into jtree failed)
    Jul 14 17:19:37 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: rt_halp_vectors->rt_create failed
    Jul 14 17:19:37 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: proto ipv4,len 24 prefix 165.220.233/24 nh 1048589
    Jul 14 17:19:37 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: route process failed
    Jul 14 17:19:37 TC-ISP-PE-A-RE0 fpc0 RT: Failed prefix add IPv4 - 165.220.232/24 (No memory) on FE 0
    Jul 14 17:19:37 TC-ISP-PE-A-RE0 fpc0 RT: IPv4:8 - 165.220.232/24 (add rt entry into jtree failed)
    Jul 14 17:19:37 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: rt_halp_vectors->rt_create failed
    Jul 14 17:19:37 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: proto ipv4,len 24 prefix 165.220.232/24 nh 1048589
    Jul 14 17:19:38 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: route process failed
    Jul 14 17:19:38 TC-ISP-PE-A-RE0 fpc0 RT: Failed prefix add IPv4 - 165.220.232/21 (No memory) on FE 0
    Jul 14 17:19:38 TC-ISP-PE-A-RE0 fpc0 RT: IPv4:8 - 165.220.232/21 (add rt entry into jtree failed)
    Jul 14 17:19:38 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: rt_halp_vectors->rt_create failed
    Jul 14 17:19:38 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: proto ipv4,len 21 prefix 165.220.232/21 nh 1048589
    Jul 14 17:19:38 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: route process failed
    Jul 14 17:19:38 TC-ISP-PE-A-RE0 fpc0 RT: Failed prefix add IPv6 - 2804:4100::/32 (No memory) on FE 0
    Jul 14 17:19:38 TC-ISP-PE-A-RE0 fpc0 RT: IPv6:8 - 2804:4100::/32 (add rt entry into jtree failed)
    Jul 14 17:19:38 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: rt_halp_vectors->rt_create failed
    Jul 14 17:19:38 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: proto ipv6,len 32 prefix 2804:4100::/32 nh 1048697
    Jul 14 17:19:39 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: route process failed
    Jul 14 17:19:39 TC-ISP-PE-A-RE0 fpc0 RT: Failed prefix add IPv6 - 2804:77c0:8000::/33 (No memory) on FE 0
    Jul 14 17:19:39 TC-ISP-PE-A-RE0 fpc0 RT: IPv6:8 - 2804:77c0:8000::/33 (add rt entry into jtree failed)
    Jul 14 17:19:39 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: rt_halp_vectors->rt_create failed
    Jul 14 17:19:39 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: proto ipv6,len 33 prefix 2804:77c0:8000::/33 nh 1048697
    Jul 14 17:19:39 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: route process failed
    Jul 14 17:19:39 TC-ISP-PE-A-RE0 fpc0 RT: Failed prefix add IPv6 - 2804:77c0::/33 (No memory) on FE 0
    Jul 14 17:19:39 TC-ISP-PE-A-RE0 fpc0 RT: IPv6:8 - 2804:77c0::/33 (add rt entry into jtree failed)
    Jul 14 17:19:39 TC-ISP-PE-A-RE0 fpc0 PFE_ERROR_FAIL_OPERATION: rt_halp_vectors->rt_create failed

    show chasis hardware 

    Hardware inventory:
    Item Version Part number Serial number Description
    Chassis JN1209B9FAFB MX480
    Midplane REV 05 710-017414 ACAW8136 MX480 Midplane
    FPM Board REV 02 710-017254 CAAN4633 Front Panel Display
    PEM 0 Rev 01 740-022697 QCS0906C0AT PS 1.2-1.7kW; 100-240V AC in
    PEM 1 Rev 01 740-022697 QCS0906C07F PS 1.2-1.7kW; 100-240V AC in
    Routing Engine 0 REV 07 740-031116 9009115480 RE-S-1800x4
    Routing Engine 1 REV 10 740-031116 9009218989 RE-S-1800x4
    CB 0 REV 15 750-031391 CAAN2940 Enhanced MX SCB
    CB 1 REV 15 750-031391 CAAN2924 Enhanced MX SCB
    FPC 0 REV 20 750-022765 YE7117 DPCE 20x 1GE + 2x 10GE R
    CPU REV 03 710-022351 YE5470 DPC PMB
    PIC 0 BUILTIN BUILTIN 10x 1GE(LAN)
    Xcvr 0 REV 02 740-013111 ZZ170120090 SFP-T
    Xcvr 1 REV 01 740-013111 T20082800359 SFP-T
    Xcvr 2 REV 01 740-013111 T20082800355 SFP-T
    PIC 1 BUILTIN BUILTIN 10x 1GE(LAN)
    PIC 2 BUILTIN BUILTIN 1x 10GE(LAN/WAN)
    PIC 3 BUILTIN BUILTIN 1x 10GE(LAN/WAN)
    FPC 5 REV 26 750-031089 CAAN8300 MPC Type 2 3D
    CPU REV 08 711-030884 CAAN8971 MPC PMB 2G
    MIC 0 REV 28 750-028387 CAAV1081 3D 4x 10GE XFP
    PIC 0 BUILTIN BUILTIN 2x 10GE XFP
    Xcvr 0 REV 01 740-031833 UW41HDD XFP-10G-LR
    Xcvr 1 AB2145 NON-JNPR T20080600582 XFP-10G-LR
    PIC 1 BUILTIN BUILTIN 2x 10GE XFP
    Xcvr 0 AB2145 NON-JNPR T12E83793 XFP-10G-LR
    Xcvr 1 NON-JNPR T12H11382 XFP-10G-LR
    MIC 1 REV 28 750-028387 CAAN2495 3D 4x 10GE XFP
    PIC 2 BUILTIN BUILTIN 2x 10GE XFP
    Xcvr 0 REV 03 740-014289 CB36BQ04W XFP-10G-SR
    Xcvr 1 NON-JNPR T20082800265 XFP-10G-SR
    PIC 3 BUILTIN BUILTIN 2x 10GE XFP
    Xcvr 0 NON-JNPR T10M61909 XFP-10G-LR
    Xcvr 1 NON-JNPR T10M61752 XFP-10G-LR
    Fan Tray Enhanced Left Fan Tray



    sh ver 

    Model: mx480
    Junos: 20.4R3.8
    JUNOS OS Kernel 64-bit [20210618.f43645e_builder_stable_11-204ab]
    JUNOS OS libs [20210618.f43645e_builder_stable_11-204ab]
    JUNOS OS runtime [20210618.f43645e_builder_stable_11-204ab]
    JUNOS OS time zone information [20210618.f43645e_builder_stable_11-204ab]
    JUNOS network stack and utilities [20210907.154329_builder_junos_204_r3]
    JUNOS libs [20210907.154329_builder_junos_204_r3]
    JUNOS OS libs compat32 [20210618.f43645e_builder_stable_11-204ab]
    JUNOS OS 32-bit compatibility [20210618.f43645e_builder_stable_11-204ab]
    JUNOS libs compat32 [20210907.154329_builder_junos_204_r3]
    JUNOS runtime [20210907.154329_builder_junos_204_r3]
    JUNOS sflow mx [20210907.154329_builder_junos_204_r3]
    JUNOS py extensions2 [20210907.154329_builder_junos_204_r3]
    JUNOS py extensions [20210907.154329_builder_junos_204_r3]
    JUNOS py base2 [20210907.154329_builder_junos_204_r3]
    JUNOS py base [20210907.154329_builder_junos_204_r3]
    JUNOS OS crypto [20210618.f43645e_builder_stable_11-204ab]
    JUNOS OS boot-ve files [20210618.f43645e_builder_stable_11-204ab]
    JUNOS na telemetry [20.4R3.8]
    JUNOS Security Intelligence [20210907.154329_builder_junos_204_r3]
    JUNOS mx libs compat32 [20210907.154329_builder_junos_204_r3]
    JUNOS mx runtime [20210907.154329_builder_junos_204_r3]
    JUNOS RPD Telemetry Application [20.4R3.8]
    Redis [20210907.154329_builder_junos_204_r3]
    JUNOS probe utility [20210907.154329_builder_junos_204_r3]
    JUNOS common platform support [20210907.154329_builder_junos_204_r3]
    JUNOS Openconfig [20.4R3.8]
    JUNOS mtx network modules [20210907.154329_builder_junos_204_r3]
    JUNOS modules [20210907.154329_builder_junos_204_r3]
    JUNOS mx modules [20210907.154329_builder_junos_204_r3]
    JUNOS mx libs [20210907.154329_builder_junos_204_r3]
    JUNOS SQL Sync Daemon [20210907.154329_builder_junos_204_r3]
    JUNOS mtx Data Plane Crypto Support [20210907.154329_builder_junos_204_r3]
    JUNOS daemons [20210907.154329_builder_junos_204_r3]
    JUNOS mx daemons [20210907.154329_builder_junos_204_r3]
    JUNOS Broadband Egde User Plane Apps [20.4R3.8]
    JUNOS appidd-mx application-identification daemon [20210907.154329_builder_junos_204_r3]
    JUNOS Services URL Filter package [20210907.154329_builder_junos_204_r3]
    JUNOS Services TLB Service PIC package [20210907.154329_builder_junos_204_r3]
    JUNOS Services Telemetry [20210907.154329_builder_junos_204_r3]
    JUNOS Services TCP-LOG [20210907.154329_builder_junos_204_r3]
    JUNOS Services SSL [20210907.154329_builder_junos_204_r3]
    JUNOS Services SOFTWIRE [20210907.154329_builder_junos_204_r3]
    JUNOS Services Stateful Firewall [20210907.154329_builder_junos_204_r3]
    JUNOS Services RTCOM [20210907.154329_builder_junos_204_r3]
    JUNOS Services RPM [20210907.154329_builder_junos_204_r3]
    JUNOS Services PCEF package [20210907.154329_builder_junos_204_r3]
    JUNOS Services NAT [20210907.154329_builder_junos_204_r3]
    JUNOS Services Mobile Subscriber Service Container package [20210907.154329_builder_junos_204_r3]
    JUNOS Services MobileNext Software package [20210907.154329_builder_junos_204_r3]
    JUNOS Services Logging Report Framework package [20210907.154329_builder_junos_204_r3]
    JUNOS Services LL-PDF Container package [20210907.154329_builder_junos_204_r3]
    JUNOS Services Jflow Container package [20210907.154329_builder_junos_204_r3]
    JUNOS Services Deep Packet Inspection package [20210907.154329_builder_junos_204_r3]
    JUNOS Services IPSec [20210907.154329_builder_junos_204_r3]
    JUNOS Services IDS [20210907.154329_builder_junos_204_r3]
    JUNOS IDP Services [20210907.154329_builder_junos_204_r3]
    JUNOS Services HTTP Content Management package [20210907.154329_builder_junos_204_r3]
    JUNOS Services Crypto [20210907.154329_builder_junos_204_r3]
    JUNOS Services Captive Portal and Content Delivery Container package [20210907.154329_builder_junos_204_r3]
    JUNOS Services COS [20210907.154329_builder_junos_204_r3]
    JUNOS AppId Services [20210907.154329_builder_junos_204_r3]
    JUNOS Services Application Level Gateways [20210907.154329_builder_junos_204_r3]
    JUNOS Services AACL Container package [20210907.154329_builder_junos_204_r3]
    JUNOS SDN Software Suite [20210907.154329_builder_junos_204_r3]
    JUNOS Extension Toolkit [20210907.154329_builder_junos_204_r3]
    JUNOS Packet Forwarding Engine Support (wrlinux9) [20210907.154329_builder_junos_204_r3]
    JUNOS Packet Forwarding Engine Support (MXSPC3) [20.4R3.8]
    JUNOS Packet Forwarding Engine Support (MX/EX92XX Common) [20210907.154329_builder_junos_204_r3]
    JUNOS Packet Forwarding Engine Support (M/T Common) [20210907.154329_builder_junos_204_r3]
    JUNOS Packet Forwarding Engine Support (aft) [20210907.154329_builder_junos_204_r3]
    JUNOS Packet Forwarding Engine Support (MX Common) [20210907.154329_builder_junos_204_r3]
    JUNOS Juniper Malware Removal Tool (JMRT) [1.0.0+20210907.154329_builder_junos_204_r3]
    JUNOS J-Insight [20210907.154329_builder_junos_204_r3]
    JUNOS jfirmware [20210907.154329_builder_junos_204_r3]
    JUNOS Online Documentation [20210907.154329_builder_junos_204_r3]
    JUNOS jail runtime [20210618.f43645e_builder_stable_11-204ab]


    request pfe execute target fpc0 command "show jtree 0 memory extensive"
    SENT: Ukern command: show jtree 0 memory extensive

    Jtree memory segment 0 (Context: 0x453798a0)
    -------------------------------------------
    Memory Statistics:
    16777216 bytes total
    16749728 bytes used
    21664 bytes available (512 bytes from free pages)
    3528 bytes wasted
    2296 bytes unusable
    32768 pages total
    32362 pages used (2571 pages used in page alloc)
    405 pages partially used
    1 pages free (max contiguous = 1)

    Partially Filled Pages (In bytes):-
    Unit Avail Overhead
    8 4872 0
    16 4304 0
    24 11208 2104
    32 384 0
    48 384 192

    Free Page Lists(Pg Size = 512 bytes):-
    Page Bucket Avail(Bytes)
    1-1 512

    Fragmentation Index = 0.976, (largest free = 512)
    Counters:
    1970524 allocs (18114 failed)
    0 releases(partial 0)
    417261 frees
    0 holds
    24043 fast-frees
    0 slow-frees
    2 pending frees(pending bytes 24)
    0 pending forced
    22 pending max hit
    17 pending max freed per call
    0 times free blocked
    0 sync writes
    Error Counters:-
    0 bad params
    0 failed frees
    0 bad cookie
    Pending list info:
    1 enabled
    256 Free size
    35 Frees per call

    Jtree memory segment 1 (Context: 0x453fbbf8)
    -------------------------------------------
    Memory Statistics:
    16777216 bytes total
    4623376 bytes used
    12150280 bytes available (12146688 bytes from free pages)
    3456 bytes wasted
    104 bytes unusable
    32768 pages total
    9031 pages used (9017 pages used in page alloc)
    13 pages partially used
    23724 pages free (max contiguous = 23724)

    Partially Filled Pages (In bytes):-
    Unit Avail Overhead
    8 2152 0
    16 432 0
    24 240 8
    48 384 64
    96 384 40

    Free Page Lists(Pg Size = 512 bytes):-
    Page Bucket Avail(Bytes)
    27-32768 12146688

    Fragmentation Index = 0.000, (largest free = 12146688)
    Counters:
    1291 allocs (0 failed)
    0 releases(partial 0)
    241 frees
    0 holds
    241 fast-frees
    0 slow-frees
    0 pending frees(pending bytes 0)
    0 pending forced
    8 pending max hit
    4 pending max freed per call
    0 times free blocked
    0 sync writes
    Error Counters:-
    0 bad params
    0 failed frees
    0 bad cookie
    Pending list info:
    1 enabled
    256 Free size
    35 Frees per call


    Context: 0x4336f830


    question  :  is there anything can be done ? or this is a limitiation of the FPC to accept less the 1milion FIB ?


    Thank you 
    Amir 






    ------------------------------
    AMIR COHEN
    ------------------------------


  • 2.  RE: MX480 jtree error

    Posted 07-15-2022 05:50
    the default DPC  is only 1M FIB, you can set memory enhance extended to 2M FIB. you can see jtree segment 0 is full, but jtree segment 1 is free. when you set memory enhance, it will load balance to jtree 1 memory.

    set chassis memory-enhanced  route

    https://www.juniper.net/documentation/us/en/software/junos/junos-overview/topics/ref/statement/memory-enhanced-edit-chassis.html
    https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/junos-software-jtree-memory-repartitioning.html

    The configuration does not become effective until you restart the FPC or DPC (on MX Series routers).

    Original Message


  • 3.  RE: MX480 jtree error

    Posted 07-15-2022 10:50

    Thank you

     

    I will try it

     

     

    Sent from Mail for Windows

     

     Link
    Intelligent Mail ™ Powered by IMP@CTIA.
    Please juniper_junosos_800e823d-246c-47ab-b66a-8207b2eb3f5a@connectedcommunity.org&dmn=gilat.net&istmp=enrich@gilat.net">do not enrich emails sent to me.



    Original Message