Routing

 View Only
last person joined: 5 days ago 

Ask questions and share experiences about ACX Series, CTP Series, MX Series, PTX Series, SSR Series, JRR Series, and all things routing, including portfolios and protocols.
  • 1.  MX80 chassis stop seeing FPC/PICs

    Posted 08-15-2023 00:22

    Hello, 

    We have an MX80 that was working fine and from nowhere we lost service and after consoling it turns out the Box cant see any physical interfaces cards even the built-in 4*10G ports. we have an extra 2 port 10G card and its also not seen. as for the routing-engine its seems ok, console to the box works fine, also we can ssh to IP on OOB Mgnt port.

    what can we do to make it see the cards again?

    root> show version 
    Model: mx80
    Junos: 18.4R3-S10
    JUNOS Base OS boot [18.4R3-S10]
    JUNOS Base OS Software Suite [18.4R3-S10]
    JUNOS Crypto Software Suite [18.4R3-S10]
    JUNOS Packet Forwarding Engine Support (MX80) [18.4R3-S10]
    JUNOS Web Management [18.4R3-S10]
    JUNOS Online Documentation [18.4R3-S10]
    JUNOS SDN Software Suite [18.4R3-S10]
    JUNOS Services Application Level Gateways [18.4R3-S10]
    JUNOS Services COS [18.4R3-S10]
    JUNOS Services Jflow Container package [18.4R3-S10]
    JUNOS Services Stateful Firewall [18.4R3-S10]
    JUNOS Services NAT [18.4R3-S10]
    JUNOS Services RPM [18.4R3-S10]
    JUNOS Services SOFTWIRE [18.4R3-S10]
    JUNOS Services Captive Portal and Content Delivery Container package [18.4R3-S10]
    JUNOS Macsec Software Suite [18.4R3-S10]
    JUNOS Services Crypto [18.4R3-S10]
    JUNOS Services IPSec [18.4R3-S10]
    JUNOS DP Crypto Software Software Suite [18.4R3-S10]
    JUNOS py-base-powerpc [18.4R3-S10]
    JUNOS py-extensions-powerpc [18.4R3-S10]
    JUNOS jsd [powerpc-18.4R3-S10-jet-1]
    JUNOS Kernel Software Suite [18.4R3-S10]
    JUNOS Routing Software Suite [18.4R3-S10]
    
    root> show chassis hardware 
    Hardware inventory:
    Item             Version  Part number  Serial number     Description
    Chassis                                T8449             MX80
    Midplane         REV 08   711-031594   ABCG4174          MX80
    PEM 0            Rev 05   740-028288   XL13218           AC Power Entry Module
    PEM 1            Rev 05   740-028288   YE01111           AC Power Entry Module
    Routing Engine            BUILTIN      BUILTIN           Routing Engine
    TFEB 0                    BUILTIN      BUILTIN           Forwarding Engine Processor
    Fan Tray                                                 Fan Tray
    
    root> 

    Regards, 

    Lish



  • 2.  RE: MX80 chassis stop seeing FPC/PICs

    Posted 08-15-2023 09:36

    Seems likely this will be a hardware failure issue since the built in fpc is missing.  Check the logs for the time period when the device went offline to see if there are any helpful messages.

    show log messages

    If this file has rolled over there are generally several back ones you can see the names of using

    show log messages?

    Have you already shutdown and started the box or a basic restart?

    You could try to force offline offline per fpc as well.

    https://www.juniper.net/documentation/us/en/software/junos/chassis/topics/ref/command/request-chassis-fpc.html



    ------------------------------
    Steve Puluka BSEET - Juniper Ambassador
    IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP - Retired)
    http://puluka.com/home
    ------------------------------



  • 3.  RE: MX80 chassis stop seeing FPC/PICs

    Posted 08-17-2023 16:32

    I shut down the box and loaded the factory default even.

    this is the chassisd logs at the moment of the crash: 

    Jul 25 11:39:46  Receive SIGHUP
    Jul 25 11:39:46  sighup_handler do new_config
    Jul 25 11:39:46  parse_configuration ddl
    Jul 25 11:39:46  fpc_is_xm_based_fpc: fru pointer is 0x267d000, fn 0x0, i2c-validity: 0
     
    Jul 25 11:39:46  fpc_is_xm_based_fpc: fru pointer is 0x26d0000, fn 0x1e4e148, i2c-validity: 0
     
    Jul 25 11:39:46  fpc_is_xm_based_fpc: fru pointer is 0x26d8000, fn 0x1e4e148, i2c-validity: 0
     
    Jul 25 11:39:46  fpc_is_xm_based_fpc: fru pointer is 0x26e0000, fn 0x1e4e148, i2c-validity: 0
     
    Jul 25 11:39:46  fpc_is_xm_based_fpc: fru pointer is 0x270a000, fn 0x0, i2c-validity: 0
     
    Jul 25 11:39:46  fpc_is_xm_based_fpc: fru pointer is 0x2765000, fn 0x1e4e148, i2c-validity: 0
     
    Jul 25 11:39:46  fpc_is_xm_based_fpc: fru pointer is 0x276d000, fn 0x1e4e148, i2c-validity: 0
     
    Jul 25 11:39:46  fpc_is_xm_based_fpc: fru pointer is 0x27a0000, fn 0x0, i2c-validity: 1
     
    Jul 25 11:39:46  fpc_is_xm_based_fpc: fru pointer is 0x27a3000, fn 0x0, i2c-validity: 1
     
    Jul 25 11:39:46  fpc_is_xm_based_fpc: fru pointer is 0x27aa000, fn 0x0, i2c-validity: 1
     
    Jul 25 11:39:46  ch_info_dynamic_power_mgmt_state_blob_get: dynamic power mgmt state not set in kernel
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Found {chassis, aggregated-devices}: Object Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  Walking Object {aggregated-devices,  }
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Found {aggregated-devices, ethernet}: Object Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  Walking Object {ethernet,  }
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Found {ethernet, device-count}: Attribute Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  Walking Object {ethernet, device-count}
    Jul 25 11:39:46  configured aggregated ethernet device count 5
    Jul 25 11:39:46  aggregated-device ethernet device-count
    Jul 25 11:39:46  configured aggregated ethernet state
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Found {chassis, tfeb}: Object Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  Walking Object {tfeb,  }
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Found {tfeb, slot}: Container Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  Walking Container {slot,  }
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Found {tfeb, slot}: Element Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  Parsing Element {slot,  }
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {slot, slotnumber}: Attribute Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Found {slot, sampling-instance}: Attribute Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  Parsing Attribute {sampling-instance,  }
    Jul 25 11:39:46  tfeb 0: sampling instance IRP-INSTANCE
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Found {chassis, alarm}: Object Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  Walking Object {alarm,  }
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Found {alarm, management-ethernet}: Object Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  Walking Object {management-ethernet, link-down}
    Jul 25 11:39:46  fxp0-link-down
    Jul 25 11:39:46  core group access set to:1
    Jul 25 11:39:46  Calling chasd_config_read_pfe_irb_options with check_only 0
    Jul 25 11:39:46  No routing-options source_routing configuration options set
    Jul 25 11:39:46  No virtual-chassis options set
    Jul 25 11:39:46  No HW FIB bulk options set
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {forwarding-options, sampling}: Object Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {forwarding-options, load-balance-label-capability}: Attribute Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  protocol-id queue-depth delete-flag
    Jul 25 11:39:46  Total Queue Allocation: 0/1024
    Jul 25 11:39:46  ch_info_pfe_hyper_mode_get: hw.chassis.pfe_hyper_mode sysctl doesn't exist in current kernel
    Jul 25 11:39:46  Entry chasd_config_read_ir_mode check 0
    Jul 25 11:39:46  Entry chasd_config_read_ir_mode_internal: init 1 ir 0 r 0 en 0
    Jul 25 11:39:46  Exit chasd_config_read_ir_mode_internal: ret 0 ir 0 r 0 en 0 chk 0
    Jul 25 11:39:46  Exit chasd_config_read_ir_mode: ret 0 ir 0 r 0
    Jul 25 11:39:46  Entry chasd_config_read_license_mode r_mode fpcs 0 ir_mode fpcs 0
    Jul 25 11:39:46  Exit chasd_config_read_license_mode r_mode fpcs 0 ir_mode fpcs 0
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {system, commit}: Object Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {system, configuration-database}: Object Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {system, login}: Object Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {system, root-authentication}: Object Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {system, services}: Object Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {system, host-name}: Attribute Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {system, time-zone}: Attribute Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {system, no-multicast-echo}: Attribute Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {system, no-redirects}: Attribute Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {system, no-ping-record-route}: Attribute Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {system, no-ping-time-stamp}: Attribute Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {system, ports}: Object Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {system, name-server}: Container Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {system, syslog}: Object Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  cfg_ddl_chasd_handle_config_option: Did not find {system, compress-configuration-files}: Attribute Config action: DAX_ITEM_UNCHANGED
    Jul 25 11:39:46  Read ifname xe-0/0/0
    Jul 25 11:39:46  Read ifname xe-0/0/1
    Jul 25 11:39:46  Read ifname xe-0/0/2
    Jul 25 11:39:46  Read ifname xe-0/0/3
    Jul 25 11:39:46  Read ifname xe-1/0/0
    Jul 25 11:39:46  Read ifname xe-1/1/0
    Jul 25 11:39:46  Read ifname ae1
    Jul 25 11:39:46  Read ifname fxp0
    Jul 25 11:39:46  Read ifname lo0
    Jul 25 11:39:46  Read ifname me0
    Jul 25 11:39:46  pic_parse_ifname: parse_interface of xe-0/0/0 found fpc 0, pic 0, port 0, channel -1
    Jul 25 11:39:46  pic_parse_ifname: parse_interface of xe-0/0/1 found fpc 0, pic 0, port 1, channel -1
    Jul 25 11:39:46  pic_parse_ifname: parse_interface of xe-0/0/2 found fpc 0, pic 0, port 2, channel -1
    Jul 25 11:39:46  pic_parse_ifname: parse_interface of xe-0/0/3 found fpc 0, pic 0, port 3, channel -1
    Jul 25 11:39:46  pic_parse_ifname: parse_interface of xe-1/0/0 found fpc 1, pic 0, port 0, channel -1
    Jul 25 11:39:46  pic_parse_ifname: parse_interface of xe-1/1/0 found fpc 1, pic 1, port 0, channel -1
    Jul 25 11:39:46  chasd_cfg_read_l2_bridge: l2_bridge_mode will be updated only in boot mode
     
    Jul 25 11:39:46  ch_create_auto_install_tmp_file AIU: Created /tmp/qfxautoconfig_tmp and enabling auto-install
    Jul 25 11:39:46  cfg_ddl_parse_mxseries_misc_conf_options AIU: Auto-install not triggered
    Jul 25 11:39:46  completed chassis state from ddl
    Jul 25 11:39:46  ch_set_non_stop_forwarding_cfg:Setting non-stop-forwarding to Disabled, source DDL
    Jul 25 11:39:46  ch_do_multichassis_overrides: Setting multichassis replication to Disabled
    Jul 25 11:39:46  fpc_post_process_config_generic1: FCOE fabric hash prev 0x1 new 0x1
     
    Jul 25 11:39:46  fpc_post_process_config_generic1: FCOE eth hash prev 0x1 new 0x1
     
    Jul 25 11:39:46  fpc_post_process_config_generic1: UFT mode prev 0x0 new 0x0
     
    Jul 25 11:39:46  fpc_post_process_config_generic1: UFT num prefix prev 0x0 new 0x0
     
    Jul 25 11:39:46  fpc_post_process_config_generic1: fpc is enhanced hashkey capable
    Jul 25 11:39:46  fpc_post_process_config_generic1: fru->fru_old_config.cfg_ucode_rebalance TRUE new TRUE
     
    Jul 25 11:39:46  fpc_post_process_config_generic, new setting : albflag(0) tol(0)
    old setting:albflag(0) tol(0)
     
    Jul 25 11:39:46  fpc_do_config...
     
    Jul 25 11:39:46  fpc_pic_post_process_config_generic:  FPC 0 PIC 0 pic type:2f9
    Jul 25 11:39:46  pic_should_bounce fpc/pic (0/0): PIC  - new speed 0 old speed 0
     
    Jul 25 11:39:46  pic_port_bounce port bounce port 0 base 0
    Jul 25 11:39:46  pic_port_bounce port bounce port 1 base 0
    Jul 25 11:39:46  pic_port_bounce port bounce port 2 base 0
    Jul 25 11:39:46  pic_port_bounce port bounce port 3 base 0
    Jul 25 11:39:46  pic_port_bounce port bounce port 0 base 0
    Jul 25 11:39:46  pic_port_bounce port bounce port 1 base 0
    Jul 25 11:39:46  pic_port_bounce port bounce port 2 base 0
    Jul 25 11:39:46  pic_port_bounce port bounce port 3 base 0
    Jul 25 11:39:46  [0,0] in pic_do_trinity_config
    Jul 25 11:39:46  fpc_pic_process_rcvrd_clk_config() fpc 0 pic 0
     
    Jul 25 11:39:46  fpc_do_config...
     
    Jul 25 11:39:46  fpc_do_config...
     
    Jul 25 11:39:47  fpc_do_config...
     
    Jul 25 11:39:47  fpc_error_post_process_config: cfg_changed =0
     
    Jul 25 11:39:47  fpc_post_process_config_generic1: FCOE fabric hash prev 0x1 new 0x1
     
    Jul 25 11:39:47  fpc_post_process_config_generic1: FCOE eth hash prev 0x1 new 0x1
     
    Jul 25 11:39:47  fpc_post_process_config_generic1: UFT mode prev 0x0 new 0x0
     
    Jul 25 11:39:47  fpc_post_process_config_generic1: UFT num prefix prev 0x0 new 0x0
     
    Jul 25 11:39:47  fpc_post_process_config_generic1: fpc is enhanced hashkey capable
    Jul 25 11:39:47  fpc_post_process_config_generic1: fru->fru_old_config.cfg_ucode_rebalance TRUE new TRUE
     
    Jul 25 11:39:47  fpc_post_process_config_generic, new setting : albflag(0) tol(0)
    old setting:albflag(0) tol(0)
     
    Jul 25 11:39:47  fpc_do_config...
     
    Jul 25 11:39:47  fpc_pic_post_process_config_generic:  FPC 1 PIC 0 pic type:2f0
    Jul 25 11:39:47  pic_should_bounce fpc/pic (1/0): PIC  - new speed 0 old speed 0
     
    Jul 25 11:39:47  pic_port_bounce port bounce port 0 base 0
    Jul 25 11:39:47  pic_port_bounce port bounce port 0 base 0
    Jul 25 11:39:47  [1,0] in pic_do_trinity_config
    Jul 25 11:39:47  fpc_pic_process_rcvrd_clk_config() fpc 1 pic 0
     
    Jul 25 11:39:47  fpc_do_config...
     
    Jul 25 11:39:47  fpc_pic_post_process_config_generic:  FPC 1 PIC 1 pic type:2f0
    Jul 25 11:39:47  pic_should_bounce fpc/pic (1/1): PIC  - new speed 0 old speed 0
     
    Jul 25 11:39:47  pic_port_bounce port bounce port 0 base 0
    Jul 25 11:39:47  pic_port_bounce port bounce port 0 base 0
    Jul 25 11:39:47  [1,1] in pic_do_trinity_config
    Jul 25 11:39:47  fpc_pic_process_rcvrd_clk_config() fpc 1 pic 1
     
    Jul 25 11:39:47  fpc_do_config...
     
    Jul 25 11:39:47  fpc_do_config...
     
    Jul 25 11:39:47  fpc_error_post_process_config: cfg_changed =0
     
    Jul 25 11:39:47  pseudo_update_devs_redundancy_group: Redundancy-group type RLT has new count 0, old count 0
     
    Jul 25 11:39:47  pseudo_update_devs_redundancy_group: Redundancy-group type RVT has new count 0, old count 0
     
    Jul 25 11:39:47 CHASSISD_PARSE_COMPLETE: Using new configuration



  • 4.  RE: MX80 chassis stop seeing FPC/PICs

    Posted 08-18-2023 06:43

    This is showing the fpc hitting an error condition but not specified.

    Recoverable errors should have been cleared by your shutdown and restart and you also factory reset.  So this is very likely a hardware failure.

    You might get some more information with show chassis fpc errors detail but I'm pretty sure you will need an official ticket and a replacement on this one.

    https://www.juniper.net/documentation/us/en/software/junos/chassis/topics/ref/command/show-chassis-fpc-errors.html



    ------------------------------
    Steve Puluka BSEET - Juniper Ambassador
    IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP - Retired)
    http://puluka.com/home
    ------------------------------