Junos OS

Expand all | Collapse all

request system snapshot recovery - how to overwrite or delete oam?

  • 1.  request system snapshot recovery - how to overwrite or delete oam?

    Posted 01-03-2019 07:13

    Happy new year 🙂

    I just upgraded an EX9214  from 17.1R1.8 to 17.2R3.4 and wanted to backup the software via snapshot recovery:

     

    root@boejuniper6> request system snapshot recovery
    Creating image ...
    Compressing image ...
    Image size is 2110MB
    ERROR: The OAM volume is too small to store a snapshot

     

    root@boejuniper6> show system snapshot

    Non-recovery snapshots:
    Snapshot snap.20190103.112901:
    Location: /packages/sets/snap.20190103.112901
    Creation date: Jan 3 11:29:01 2019
    Junos version: 17.1R1.8

    Total non-recovery snapshots: 1

    Recovery Snapshots:
    Snapshots available on the OAM volume:
    recovery.ufs
    Date created: Thu Jan 3 11:27:16 CET 2019
    Junos version: 17.1R1.8

    Total recovery snapshots: 1

     

    I didn't find any hint (neither Software Installation and Upgrade Guide nor tech pubs or knowledge base) how to delete or overwrite the existing OAM volume...

    Can anyone advice me?



  • 2.  RE: request system snapshot recovery - how to overwrite or delete oam?

     
    Posted 01-04-2019 02:57

    Try the request system storage cleanup and run the snapshot again to see if that frees up space on the right volume.

     



  • 3.  RE: request system snapshot recovery - how to overwrite or delete oam?

    Posted 01-04-2019 05:34

    Unfortunately not:

    rhellwig@boejuniper6> show system snapshot

    Non-recovery snapshots:
    Snapshot snap.20190103.112901:
    Location: /packages/sets/snap.20190103.112901
    Creation date: Jan 3 11:29:01 2019
    Junos version: 17.1R1.8

    Snapshot snap.20190104.101954:
    Location: /packages/sets/snap.20190104.101954
    Creation date: Jan 4 10:19:54 2019
    Junos version: 18.3R1.9

    Total non-recovery snapshots: 2

    Recovery Snapshots:
    Snapshots available on the OAM volume:
    recovery.ufs
    Date created: Thu Jan 3 11:27:16 CET 2019
    Junos version: 17.1R1.8

    Total recovery snapshots: 1

    {master}
    rhellwig@boejuniper6> request system storage cleanup

    List of files to delete:

    Size Date Name
    11B Jan 4 13:41 /var/jail/tmp/alarmd.ts
    ....
    0B Jan 4 10:18 /var/tmp/rtsdb/if-rtsdb
    Delete these files ? [yes,no] (no) yes
    {master}
    rhellwig@boejuniper6> request system snapshot recovery
    Creating image ...
    Compressing image ...
    Image size is 2459MB
    ERROR: The OAM volume is too small to store a snapshot

     

    In the meantime I upgraded the SW to an actual level (18.3R1.9) but this didn't help.

     

    On the second RE I also tried the request system recover oam-volume oper cmd as described in Software Installation and Upgrade Guide (p.503), but this went wrong and made the /oam unavailable...

    rhellwig@boejuniper6> show system snapshot

    Non-recovery snapshots:
    Snapshot snap.20190103.143713:
    Location: /packages/sets/snap.20190103.143713
    Creation date: Jan 3 14:37:13 2019
    Junos version: 17.2R3.4

    Snapshot snap.20190104.100147:
    Location: /packages/sets/snap.20190104.100147
    Creation date: Jan 4 10:01:47 2019
    Junos version: 18.3R1.9

    Total non-recovery snapshots: 2

    Recovery Snapshots:
    mount: /dev/oam: No such file or directory

     

    Reboot, offline/online and upgrade didn't solve this 2nd problem.



  • 4.  RE: request system snapshot recovery - how to overwrite or delete oam?

    Posted 01-04-2019 05:55

    Hello,

    Please see if this KB article https://kb.juniper.net/InfoCenter/index?page=content&id=KB32176 helps to recover OAM volume.

    HTH

    Thx

    Alex



  • 5.  RE: request system snapshot recovery - how to overwrite or delete oam?

    Posted 01-07-2019 06:27

    Hello,

    thx for that advice but it didn't help:

    root@boejuniper6:~ # fsck_ufs /dev/ada1p2
    Can't stat /dev/ada1p2: No such file or directory
    Can't stat /dev/ada1p2: No such file or directory

    or the other try:

    root@boejuniper6:~ # fsck_ufs /dev/gpt/oam

    Can't stat /dev/gpt/oam: No such file or directory
    Can't stat /dev/gpt/oam: No such file or directory

     

    It seems the RE has lost it's mind:
    root@boejuniper6> request system recover oam-volume
    NOTICE: Recovering the OAM volume ...
    (ada0:ata0:0:1:0): WRITE_DMA. ACB: ca 00 28 00 00 40 00 00 00 00 22 00
    (ada0:ata0:0:1:0): CAM status: ATA Status Error
    (ada0:ata0:0:1:0): ATA status: 41 (DRDY ERR), error: 04 (ABRT )
    (ada0:ata0:0:1:0): RES: 41 04 28 00 00 00 00 00 00 22 00
    (ada0:ata0:0:1:0): Retrying command
    ...
    (ada0:ata0:0:1:0): Error 5, Retries exhausted
    Requesting switchover due to disk failure on ada0
    Requesting reboot due to disk failure on ada0
    gpart: /dev/ada0p1: Input/output error
    gpart: autofill: No space left on device
    test: -lt: unexpected operator
    ^C[abort]

     



  • 6.  RE: request system snapshot recovery - how to overwrite or delete oam?

    Posted 04-03-2019 06:10

    Hi,

    I have almost the same issue like you have, this is in a MX960 with RE-S-1800X4 Routing Engine.

    After doing a system zeroize, i got these logs:

     

    netdaemon_disable_timer:ISSU/GRES in progress. Stopping timer for netdaemon
    (ada1:ata0:0:1:0): WRITE_DMA. ACB: ca 00 e0 91 13 40 00 00 00 00 00 00
    (ada1:ata0:0:1:0): CAM status: Command timeout
    (ada1:ata0:0:1:0): Retrying command
    (ada1:ata0:0:1:0): WRITE_DMA. ACB: ca 00 e0 91 13 40 00 00 00 00 00 00
    (ada1:ata0:0:1:0): CAM status: Command timeout
    (ada1:ata0:0:1:0): Retrying command
    (ada1:ata0:0:1:0): WRITE_DMA. ACB: ca 00 e0 91 13 40 00 00 00 00 00 00
    (ada1:ata0:0:1:0): CAM status: Command timeout
    (ada1:ata0:0:1:0): Retrying command
    (ada1:ata0:0:1:0): WRITE_DMA. ACB: ca 00 e0 91 13 40 00 00 00 00 00 00
    (ada1:ata0:0:1:0): CAM status: Command timeout
    (ada1:ata0:0:1:0): Retrying command
    (ada1:ata0:0:1:0): WRITE_DMA. ACB: ca 00 e0 91 13 40 00 00 00 00 00 00
    (ada1:ata0:0:1:0): CAM status: Command timeout
    (ada1:ata0:0:1:0): Error 5, Retries exhausted
    Requesting switchover due to disk failure on ada1
    Requesting reboot due to disk failure on ada1

     

    After reboot, the unit has booted via SATA SSD1 and disabled the CF, see below

    ada0: <TOSHIBA THNSFC256GAMJ CJLA0201> ATA8-ACS SATA 2.x device
    ada0: Serial Number Y1FS1002TLLZ
    ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
    ada0: 244198MB (500118192 512 byte sectors)
    ada0: Previously was known as ad0
    ada1 at ata0 bus 0 scbus0 target 1 lun 0
    ada1: <SMART CF 20110218> ATA-4 device
    ada1: Serial Number SPG2013050701129
    ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 512bytes)
    ada1: 3807MB (7797888 512 byte sectors)
    ada1: Previously was known as ad1
    GEOM: ada1: corrupt or invalid GPT detected.
    GEOM: ada1: GPT rejected -- may not be recoverable.

     

    Is there anyone who know how to fix this? I am facing this problem with 4 different RE-S-1800X4.

     



  • 7.  RE: request system snapshot recovery - how to overwrite or delete oam?

     
    Posted 04-03-2019 06:59

    Hi Iliass020,

     

    You seem to be facing a different issue on the MX960. Seems you had a hard disk failure on ada1, and upon restart, the GPT was unable to be found. My recommendation is to format install using a bootable USB, and check following command output to see if you have the ada1 listed there. If not, you may need to replace the RE.

     

    >> Check "show chassis hardware detail" output to check if ada1 is the hardrive, based on the RE rev number, the HD model may be different.

    Routing Engine 1 REV 07 740-031116 9009120212 RE-S-1800x4
    ada0 30533 MB UGB94BPH32H0S1-KCI 11000049731 Compact Flash
    ada1 3998 MB Virtium - TuffDrive VC P1T0200278561011 14
    usb0 (addr 1) EHCI root HUB 0 Intel uhub0
    usb1 (addr 2) product 0x0020 32 vendor 0x8087 uhub1
    DIMM 0 SGU04G72H1BD2SA-BB DIE REV-52 PCB REV-54 MFR ID-ce80
    DIMM 1 SGU04G72H1BD2SA-BB DIE REV-52 PCB REV-54 MFR ID-ce80
    DIMM 2 SGU04G72H1BD2SA-BB DIE REV-52 PCB REV-54 MFR ID-ce80
    DIMM 3 SGU04G72H1BD2SA-BB DIE REV-52 PCB REV-54 MFR ID-ce80



  • 8.  RE: request system snapshot recovery - how to overwrite or delete oam?

     
    Posted 04-03-2019 06:48

    You may try deleting the non-recovery snapshot on yoru system first first using following command- 

     

    request system snapshot delete snap.20190103.112901

     

    Sample - 

     

    root@s07-1-re0> request system snapshot
    NOTICE: Snapshot snap.20190403.064605 created successfully
     
    root@s07-1-re0> request system snapshot delete snap.20190403.064605
    NOTICE: Snapshot 'snap.20190403.064605' deleted successfully
     
    Please mark this solution as Accepted if it works.
    Kudos are appreciated as well
     

     

     



  • 9.  RE: request system snapshot recovery - how to overwrite or delete oam?

    Posted 04-04-2019 04:29

    Hi Pranavs,

    thx for your advice. But the command "request system snapshot delete snap.xx.yy" can handle only non-recovery snapshots. This works fine. My serious problem is handling the recovery snapshot. Even if I delete all non-recovery snapshots the "request system snapshot recovery" command fails:

    rhellwig@boejuniper6> request system snapshot recovery
    Creating image ...
    Compressing image ...
    Image size is 2459MB
    ERROR: The OAM volume is too small to store a snapshot



  • 10.  RE: request system snapshot recovery - how to overwrite or delete oam?

     
    Posted 04-04-2019 08:49

    Hi RTHellwig,

    Can you please check the contents of following directory -

     

    /packages/sets/active/optional

     

    There may be some unecessary packages here from previous upgrades. If you delete those, the snap size may go down and you should be able to succesfully create the recovery snap.

     

    For example -

    labroot@s07-1-re0> start shell
    % cd /packages/sets/active/optional
    % ls -ltr
    total 8
    lrwxrwxrwx 1 root wheel 36 Feb 16 09:27 jsdn -> /packages/db/jsdn-x86-32-17.3R3-S2.2
    lrwxrwxrwx 1 root wheel 76 Feb 16 09:27 junos-daemons-mx -> /packages/db/junos-daemons-mx-x86-64-20181106.231519_builder_junos_173_r3_s2
    lrwxrwxrwx 1 root wheel 87 Feb 16 09:27 junos-dp-crypto-support-mtx -> /packages/db/junos-dp-crypto-support-mtx-x86-32-20181106.231519_builder_junos_173_r3_s2
    lrwxrwxrwx 1 root wheel 82 Feb 16 09:27 junos-libs-compat32-mx -> /packages/db/junos-libs-compat32-mx-x86-64-20181106.231519_builder_junos_173_r3_s2
    lrwxrwxrwx 1 root wheel 73 Feb 16 09:27 junos-libs-mx -> /packages/db/junos-libs-mx-x86-64-20181106.231519_builder_junos_173_r3_s2
    lrwxrwxrwx 1 root wheel 76 Feb 16 09:27 junos-modules-mx -> /packages/db/junos-modules-mx-x86-64-20181106.231519_builder_junos_173_r3_s2
    drwxrwxrwx 2 root wheel 512 Feb 16 09:27 boot
    lrwxrwxrwx 1 root wheel 76 Feb 16 09:27 junos-runtime-mx -> /packages/db/junos-runtime-mx-x86-32-20181106.231519_builder_junos_173_r3_s2
    lrwxrwxrwx 1 root wheel 67 Feb 16 09:27 jweb-ex -> /packages/db/jweb-ex-x86-32-20181106.231519_builder_junos_173_r3_s2
    lrwxrwxrwx 1 root wheel 37 Feb 16 09:27 junos-install-x86-64 -> /packages/db/junos-x86-64-17.3R3-S2.2
    lrwxr-xr-x 1 root wheel 67 Feb 16 09:28 jail-runtime -> /packages/db/jail-runtime-x86-32-20180911.6c98660_builder_stable_10
    lrwxr-xr-x 1 root wheel 66 Feb 16 09:29 jpfe-X -> /packages/db/jpfe-X-x86-32-20181106.231519_builder_junos_173_r3_s2
    lrwxr-xr-x 1 root wheel 69 Feb 16 09:29 jpfe-X960 -> /packages/db/jpfe-X960-x86-32-20181106.231519_builder_junos_173_r3_s2
    lrwxr-xr-x 1 root wheel 72 Feb 16 09:29 jpfe-wrlinux -> /packages/db/jpfe-wrlinux-x86-32-20181106.231519_builder_junos_173_r3_s2
    lrwxr-xr-x 1 root wheel 76 Feb 16 09:29 jplatform-ex92xx -> /packages/db/jplatform-ex92xx-x86-32-20181106.231519_builder_junos_173_r3_s2

    I am able to succesfully create snaps on my EX9214. Can you check this output on your device and delete the packages that are present from an older code, and hence not required? May be save the previous packages mentioned above by creating a optional directory under : 

    /packages/sets/previous/

     



  • 11.  RE: request system snapshot recovery - how to overwrite or delete oam?

     
    Posted 04-04-2019 09:22

    Hey RTHellwig,

     

    I checked on your original question as well. Currently there is no manual method to delete recovery snapshot. That may get introduced in later Junos releases. In order to be able to succefully create a recovery snap, you need to decrease the size of your snap. One way to do that is to delete the optional packages that get zipped into the recovery snap by default, but they may not be needed. Please see my previous post and share the requested output so we can try to determine which packages you can delete.

     

    ========

    Kudos are appreciated.

    Mark the solution as accepted if it works.



  • 12.  RE: request system snapshot recovery - how to overwrite or delete oam?

    Posted 06-24-2019 06:33

    I'm not sure if you are still having this issue, but if not then just consider my post as documenting a possible solution for the next guy that has this issue and searches for it :  )

     

    It is actually possible to manually delete a recovery snapshot, for this you need root access to the box.

     

    Start the shell as user root:


    user@router> start shell user root
    Password:

     

    Create a temporary folder in the /var/tmp directory to use as mounting point for the OAM volume:

     

    root@router:/var/home/user # mkdir /var/tmp/oam
    root@router:/var/home/user # mount /dev/gpt/oam /var/tmp/oam

     

    CD into the mounting point just created and go into the snapshot folder:

     

    root@router:/var/home/user # cd /var/tmp/oam/snapshot/
    root@router:/var/tmp/oam/snapshot # ls
    VERSION recovery.ufs.uzip

     

    From here remove the recovery.ufs.uzip and VERSION files:

     

    root@router:/var/tmp/oam/snapshot # rm recovery.ufs.uzip

    root@router:/var/tmp/oam/snapshot # rm VERSION

     

    Unmount the OAM filesystem:

     

    root@router:/var/tmp/oam # cd /
    root@router:/ # umount /dev/gpt/oam

     

    Hope this helpes



  • 13.  RE: request system snapshot recovery - how to overwrite or delete oam?

    Posted 07-09-2019 00:59

    Hello pranavs, jminnaar,

    my last advantures...:

    I run in trouble while playing around - I don't know when and why, but sometimes I lost the /oam on RE0:

    Recovery Snapshots:
    mount: /dev/oam: No such file or directory
    ERROR: 'oam' package needs to be updated in order to use OAM functionality

    I tried to resolve this via upgrading to 19.1R1.6 but it didn't help, /oam is still missing. So I tried on RE1 the "alternative delete procedure" with good results:

    rhellwig@boejuniper6> show system snapshot

    Non-recovery snapshots:

    Snapshot snap.20190709.075643:
    Location: /packages/sets/snap.20190709.075643
    Creation date: Jul 9 07:56:43 2019
    Junos version: 19.1R1.6

    Total non-recovery snapshots: 2

    Recovery Snapshots:
    No recovery snapshots available on the OAM volume

    But nevertheless the software seems to be too big to be stored:

    rhellwig@boejuniper6> request system snapshot recovery
    Creating image ...
    Compressing image ...
    Image size is 2691MB
    ERROR: The OAM volume is too small to store a snapshot

    I' not sure if 19.1 is so much bigger than 18.3 or there is still any pointer anywhere to the oam...

    I checked the content of /packages/sets/active/optional and deleted all that didn't match with the current release. The result is:

    root@boejuniper6:/packages/sets/active/optional # ls -lkr

    total 15
    lrwxr-xr-x 1 root wheel 65 Jul 9 07:47 sflow-mx -> /packages/db/sflow-mx-x86-32-20190321.051058_builder_junos_191_r1
    lrwxr-xr-x 1 root wheel 64 Jul 9 07:47 jweb-ex -> /packages/db/jweb-ex-x86-32-20190321.051058_builder_junos_191_r1
    lrwxr-xr-x 1 root wheel 73 Jul 9 07:47 junos-runtime-mx -> /packages/db/junos-runtime-mx-x86-32-20190321.051058_builder_junos_191_r1
    lrwxr-xr-x 1 root wheel 74 Jul 9 07:47 junos-net-mtx-prd -> /packages/db/junos-net-mtx-prd-x86-64-20190321.051058_builder_junos_191_r1
    lrwxr-xr-x 1 root wheel 73 Jul 9 07:47 junos-modules-mx -> /packages/db/junos-modules-mx-x86-64-20190321.051058_builder_junos_191_r1
    lrwxr-xr-x 1 root wheel 70 Jul 9 07:47 junos-libs-mx -> /packages/db/junos-libs-mx-x86-64-20190321.051058_builder_junos_191_r1
    lrwxr-xr-x 1 root wheel 79 Jul 9 07:47 junos-libs-compat32-mx -> /packages/db/junos-libs-compat32-mx-x86-64-20190321.051058_builder_junos_191_r1
    lrwxr-xr-x 1 root wheel 41 Jul 9 07:47 junos-install-ex92xx-x86-64 -> /packages/db/junos-ex92xx-x86-64-19.1R1.6
    lrwxr-xr-x 1 root wheel 84 Jul 9 07:47 junos-dp-crypto-support-mtx -> /packages/db/junos-dp-crypto-support-mtx-x86-32-20190321.051058_builder_junos_191_r1
    lrwxr-xr-x 1 root wheel 73 Jul 9 07:47 junos-daemons-mx -> /packages/db/junos-daemons-mx-x86-64-20190321.051058_builder_junos_191_r1
    lrwxr-xr-x 1 root wheel 33 Jul 9 07:46 jsdn -> /packages/db/jsdn-x86-32-19.1R1.6
    lrwxr-xr-x 1 root wheel 73 Jul 9 07:46 jplatform-ex92xx -> /packages/db/jplatform-ex92xx-x86-32-20190321.051058_builder_junos_191_r1
    lrwxr-xr-x 1 root wheel 70 Jul 9 07:51 jpfe-wrlinux9 -> /packages/db/jpfe-wrlinux9-x86-32-20190321.051058_builder_junos_191_r1
    lrwxr-xr-x 1 root wheel 66 Jul 9 07:46 jpfe-X960 -> /packages/db/jpfe-X960-x86-32-20190321.051058_builder_junos_191_r1
    lrwxr-xr-x 1 root wheel 63 Jul 9 07:46 jpfe-X -> /packages/db/jpfe-X-x86-32-20190321.051058_builder_junos_191_r1
    lrwxr-xr-x 1 root wheel 67 Jul 9 07:46 jail-runtime -> /packages/db/jail-runtime-x86-32-20190305.df99236_builder_stable_11
    drwxr-xr-x 2 root wheel 512 Jul 9 07:47 boot



  • 14.  RE: request system snapshot recovery - how to overwrite or delete oam?

    Posted 07-09-2019 01:11

    Hi RTHellwig,

     

    If the oam partition doesn't exist or is corrupted, then a recovery snapshot will probably fail as well as it needs to be stored on this partition. Did you try to recover the OAM volume?

     

    user@router> request system recover oam-volume

     

    Regards,

    John



  • 15.  RE: request system snapshot recovery - how to overwrite or delete oam?

    Posted 10-08-2019 01:58

    Unfortunately it doesn't help:

    root@boejuniper6> request system recover oam-volume
    NOTICE: Recovering the OAM volume ...
    (ada0:ata0:0:1:0): WRITE_DMA. ACB: ca 00 28 00 00 40 00 00 00 00 20 00
    (ada0:ata0:0:1:0): CAM status: ATA Status Error
    (ada0:ata0:0:1:0): ATA status: 41 (DRDY ERR), error: 04 (ABRT )
    (ada0:ata0:0:1:0): RES: 41 04 28 00 00 00 00 00 00 20 00
    (ada0:ata0:0:1:0): Retrying command
    ...
    (ada0:ata0:0:1:0): WRITE_DMA. ACB: ca 00 28 00 00 40 00 00 00 00 20 00
    (ada0:ata0:0:1:0): CAM status: ATA Status Error
    (ada0:ata0:0:1:0): ATA status: 41 (DRDY ERR), error: 04 (ABRT )
    (ada0:ata0:0:1:0): RES: 41 04 28 00 00 00 00 00 00 20 00
    (ada0:ata0:0:1:0): Error 5, Retries exhausted
    Requesting switchover due to disk failure on ada0
    Requesting reboot due to disk failure on ada0
    gpart: /dev/ada0p1: Input/output error
    gpart: autofill: No space left on device
    test: -lt: unexpected operator

    ^C[abort]



  • 16.  RE: request system snapshot recovery - how to overwrite or delete oam?

    Posted 09-27-2019 13:26

    I had this problem and couldn't find a solution anywhere, no matter what I did, there was never enough space to do a recovery snapshot. I finally found something that works, though it's undocumented. This doesn't APPEAR to cause any problems, but who knows..

     

    From shell prompt, I ran these two commands:

     

    pkg setop rm previous
    pkg delete old

    And several hundred MB was freed up. I was able to create a recovery snapshot. YMMV



  • 17.  RE: request system snapshot recovery - how to overwrite or delete oam?

    Posted 10-08-2019 02:01

    I tried it on both REs (on backup with good OAM, on master with bad OAM) without success (nothing happened):

    {backup}
    root@boejuniper6:~ # pkg setop rm previous
    root@boejuniper6:~ # pkg delete old
    root@

    {backup}
    root@boejuniper6> request system snapshot recovery
    Creating image ...
    Compressing image ...
    Image size is 2691MB
    ERROR: The OAM volume is too small to store a snapshotboejuniper6:~ #

    -----------------------------------------------------------------------

    {master}
    root@boejuniper6:~ # pkg setop rm previous
    root@boejuniper6:~ # pkg delete old
    root@boejuniper6:~ #

    {master}
    root@boejuniper6> show system snapshot

    Non-recovery snapshots:
    Snapshot snap.20190103.143713:
    Location: /packages/sets/snap.20190103.143713
    Creation date: Jan 3 14:37:13 2019
    Junos version: 17.2R3.4

    Snapshot snap.20190104.100147:
    Location: /packages/sets/snap.20190104.100147
    Creation date: Jan 4 10:01:47 2019
    Junos version: 18.3R1.9

    Total non-recovery snapshots: 2

    Recovery Snapshots:
    mount: /dev/oam: No such file or directory
    ERROR: 'oam' package needs to be updated in order to use OAM functionality