Automation of dual-AD software upgrade
In a typical dual-aggregated Junos Fusion system software upgrade, a user needs to copy images to both aggregated devices and execute the software upgrade command. Consider a situation where each aggregated device has dual routing engines, and the complete process involves installing JUNOS software on all the four routing engines and rebooting all of them.
This can be simplified with SLAX automation. Using a SLAX script, the user can upgrade all the routing-engines by executing a single operational cli command. The user needs to copy the JUNOS software image to the aggregated device where this operational script is present.
Figure: Dual aggregated Junos-Fusion system
To upgrade the JUNOS software on all the REs, the user needs to perform the following steps:
- Copy image to aggregated devices
- Issue:
- ‘request system software add <image-location> re0’
- ‘request system software add <image-location> re1’
- ‘request system reboot both-routing-engines’
- These steps need to be repeated on the other aggregated device that is the part of Junos Fusion system
SLAX Solution:
With SLAX automation, the given script performs the following steps:
- Copies the JUNOS software image from one aggregation device to its peer aggregation device
- Collects the chassis satellite information before upgrading
- Starts installing the JUNOS software on the peer aggregation device’s routing engines and reboots those routing engines
- If parallel option is not ‘true’, the script validates the chassis satellite state once the upgrade is completed by comparing it with the previously obtained information. If the satellite(s) state is not proper then installation is aborted.
- Starts installing image on the current aggregation device’s routing engines and reboots those routing engines with a single op command
It is always recommended to upgrade one aggregated device after another in a dual aggregated system, as dual homed satellites can still be online in the aggregated device that is up, and traffic goes fine with minimal loss.
This operational script allows the user to specify if he/she wants to upgrade the entire system in parallel, or upgrade one aggregated device, wait till it comes up, then start upgrading the other.
To enable the op script, see: https://www.juniper.net/techpubs/en_US/junos15.1/topics/usage-guidelines/automation-enabling-an-op-script-and-defining-a-script-alias.html
Usage:
Syntax:
op software-add-dual-ad
Possible completions:
filename <path and package-name>
host <remote host name or IP address>
force <true to ignore warnings>
validate <true to check configuration compatibility>
parallel <true to upgrade all REs parallel>
Description:
Install a software package or bundle on routing-engines of both aggregation devices of a Junos Fusion dual AD setup
Options:
filename required argument
Value: Path to the image file
host required argument
Value: Host name or IP address in case of Junos Fusion peer ICCP address
This value will be used to copy image and install on remote side
force optional argument
Value: true | false
To force image installation to ignore warning, default value is true
validate optional argument
Value: true | false
To check compatibility with current configuration, default value is false
parallel optional argument
Value: true | false
true upgrades all REs parallel, false starts upgrading peer aggregated device and wait till it comes up with all FPCs then initiates upgrade locally
Sample output:
root@JUNOS-FUSION-01> op software-add-dual-ad filename /var/tmp/junos-install-ex92xx-x86-64-16.1-20160622.1.tgz host 10.1.1.2 force false validate false
hup
Pushing /var/tmp/junos-install-ex92xx-x86-64-16.1-20160622.1.tgz to re1:/var/tmp/junos-install-ex92xx-x86-64-16.1-20160622.1.tgz
hup
Verified junos-install-ex92xx-x86-64-16.1-20160622.1 signed by PackageDevelopmentEc_2016
Verified manifest signed by PackageDevelopmentEc_2016
Checking PIC combinations
Verified fips-mode signed by PackageDevelopmentEc_2016
Verified jail-runtime signed by PackageDevelopmentEc_2016
Verified jdocs signed by PackageDevelopmentEc_2016
Verified jpfe-X960 signed by PackageDevelopmentEc_2016
Verified jpfe-common signed by PackageDevelopmentEc_2016
Verified jpfe-wrlinux signed by PackageDevelopmentEc_2016
Verified jplatform-ex92xx signed by PackageDevelopmentEc_2016
Verified jsd signed by PackageDevelopmentEc_2016
Verified jsdn signed by PackageDevelopmentEc_2016
Verified jservices-crypto signed by PackageDevelopmentEc_2016
Adding jservices-crypto-x86-32-16.1-20160622.1 ...
Verified jservices-crypto-base signed by PackageDevelopmentEc_2016
Verified jservices-ipsec signed by PackageDevelopmentEc_2016
Verified jservices-ssl signed by PackageDevelopmentEc_2016
Verified jservices signed by PackageDevelopmentEc_2016
Adding jservices-x86-32-16.1-20160622.1 ...
Verified jservices-aacl signed by PackageDevelopmentEc_2016
Verified jservices-alg signed by PackageDevelopmentEc_2016
Verified jservices-appid signed by PackageDevelopmentEc_2016
Verified jservices-bgf signed by PackageDevelopmentEc_2016
Verified jservices-cpcd signed by PackageDevelopmentEc_2016
Verified jservices-hcm signed by PackageDevelopmentEc_2016
Verified jservices-idp signed by PackageDevelopmentEc_2016
Verified jservices-jdpi signed by PackageDevelopmentEc_2016
Verified jservices-jflow signed by PackageDevelopmentEc_2016
Verified jservices-llpdf signed by PackageDevelopmentEc_2016
Verified jservices-lrf signed by PackageDevelopmentEc_2016
Verified jservices-mobile signed by PackageDevelopmentEc_2016
Verified jservices-mss signed by PackageDevelopmentEc_2016
Verified jservices-nat signed by PackageDevelopmentEc_2016
Verified jservices-pcef signed by PackageDevelopmentEc_2016
Verified jservices-ptsp signed by PackageDevelopmentEc_2016
Verified jservices-rpm signed by PackageDevelopmentEc_2016
Verified jservices-sfw signed by PackageDevelopmentEc_2016
Verified jservices-voice signed by PackageDevelopmentEc_2016
Verified jsim-pfe signed by PackageDevelopmentEc_2016
Verified junos-daemons-mx signed by PackageDevelopmentEc_2016
Verified junos-daemons signed by PackageDevelopmentEc_2016
Verified junos-dp-crypto-support-mtx signed by PackageDevelopmentEc_2016
Verified junos-dp-crypto-support signed by PackageDevelopmentEc_2016
Verified junos-libs-compat32-mx signed by PackageDevelopmentEc_2016
Verified junos-libs-compat32 signed by PackageDevelopmentEc_2016
Verified junos-libs-mx signed by PackageDevelopmentEc_2016
Verified junos-libs signed by PackageDevelopmentEc_2016
Verified junos-modules-mx signed by PackageDevelopmentEc_2016
Verified junos-modules signed by PackageDevelopmentEc_2016
Verified junos-net-prd signed by PackageDevelopmentEc_2016
Verified junos-platform signed by PackageDevelopmentEc_2016
Verified junos-runtime-mx signed by PackageDevelopmentEc_2016
Verified junos-runtime signed by PackageDevelopmentEc_2016
Verified junos-vmguest-mtx signed by PackageDevelopmentEc_2016
Verified jweb signed by PackageDevelopmentEc_2016
Verified oam signed by PackageDevelopmentEc_2016
Verified os-compat32 signed by PackageDevelopmentEc_2016
Verified os-crypto signed by PackageDevelopmentEc_2016
Verified os-kernel-prd signed by PackageDevelopmentEc_2016
Verified os-libs-compat32 signed by PackageDevelopmentEc_2016
Verified os-libs signed by PackageDevelopmentEc_2016
Verified os-runtime signed by PackageDevelopmentEc_2016
Verified os-vmguest signed by PackageDevelopmentEc_2016
Verified os-zoneinfo signed by PackageDevelopmentEc_2016
Verified py-base signed by PackageDevelopmentEc_2016
Verified py-extensions signed by PackageDevelopmentEc_2016
Verified vrr-mx signed by PackageDevelopmentEc_2016
NOTICE: 'pending' set will be activated at next reboot...
Rebooting. Please wait ...
shutdown: [pid 26555]
Shutdown NOW!
Shutdown NOW!
System shutdown time has arrived
0
xnm:rpc results:86:(7) PCDATA invalid Char value 7
xnm:rpc results:86:(7) PCDATA invalid Char value 7
hup
Verified junos-install-ex92xx-x86-64-16.1-20160622.1 signed by PackageDevelopmentEc_2016
Verified manifest signed by PackageDevelopmentEc_2016
Checking PIC combinations
Verified fips-mode signed by PackageDevelopmentEc_2016
Verified jail-runtime signed by PackageDevelopmentEc_2016
Verified jdocs signed by PackageDevelopmentEc_2016
Verified jpfe-X960 signed by PackageDevelopmentEc_2016
Verified jpfe-common signed by PackageDevelopmentEc_2016
Verified jpfe-wrlinux signed by PackageDevelopmentEc_2016
Verified jplatform-ex92xx signed by PackageDevelopmentEc_2016
Verified jsd signed by PackageDevelopmentEc_2016
Verified jsdn signed by PackageDevelopmentEc_2016
Verified jservices-crypto signed by PackageDevelopmentEc_2016
Adding jservices-crypto-x86-32-16.1-20160622.1 ...
Verified jservices-crypto-base signed by PackageDevelopmentEc_2016
Verified jservices-ipsec signed by PackageDevelopmentEc_2016
Verified jservices-ssl signed by PackageDevelopmentEc_2016
Verified jservices signed by PackageDevelopmentEc_2016
Adding jservices-x86-32-16.1-20160622.1 ...
Verified jservices-aacl signed by PackageDevelopmentEc_2016
Verified jservices-alg signed by PackageDevelopmentEc_2016
Verified jservices-appid signed by PackageDevelopmentEc_2016
Verified jservices-bgf signed by PackageDevelopmentEc_2016
Verified jservices-cpcd signed by PackageDevelopmentEc_2016
Verified jservices-hcm signed by PackageDevelopmentEc_2016
Verified jservices-idp signed by PackageDevelopmentEc_2016
Verified jservices-jdpi signed by PackageDevelopmentEc_2016
Verified jservices-jflow signed by PackageDevelopmentEc_2016
Verified jservices-llpdf signed by PackageDevelopmentEc_2016
Verified jservices-lrf signed by PackageDevelopmentEc_2016
Verified jservices-mobile signed by PackageDevelopmentEc_2016
Verified jservices-mss signed by PackageDevelopmentEc_2016
Verified jservices-nat signed by PackageDevelopmentEc_2016
Verified jservices-pcef signed by PackageDevelopmentEc_2016
Verified jservices-ptsp signed by PackageDevelopmentEc_2016
Verified jservices-rpm signed by PackageDevelopmentEc_2016
Verified jservices-sfw signed by PackageDevelopmentEc_2016
Verified jservices-voice signed by PackageDevelopmentEc_2016
Verified jsim-pfe signed by PackageDevelopmentEc_2016
Verified junos-daemons-mx signed by PackageDevelopmentEc_2016
Verified junos-daemons signed by PackageDevelopmentEc_2016
Verified junos-dp-crypto-support-mtx signed by PackageDevelopmentEc_2016
Verified junos-dp-crypto-support signed by PackageDevelopmentEc_2016
Verified junos-libs-compat32-mx signed by PackageDevelopmentEc_2016
Verified junos-libs-compat32 signed by PackageDevelopmentEc_2016
Verified junos-libs-mx signed by PackageDevelopmentEc_2016
Verified junos-libs signed by PackageDevelopmentEc_2016
Verified junos-modules-mx signed by PackageDevelopmentEc_2016
Verified junos-modules signed by PackageDevelopmentEc_2016
Verified junos-net-prd signed by PackageDevelopmentEc_2016
Verified junos-platform signed by PackageDevelopmentEc_2016
Verified junos-runtime-mx signed by PackageDevelopmentEc_2016
Verified junos-runtime signed by PackageDevelopmentEc_2016
Verified junos-vmguest-mtx signed by PackageDevelopmentEc_2016
Verified jweb signed by PackageDevelopmentEc_2016
Verified oam signed by PackageDevelopmentEc_2016
Verified os-compat32 signed by PackageDevelopmentEc_2016
Verified os-crypto signed by PackageDevelopmentEc_2016
Verified os-kernel-prd signed by PackageDevelopmentEc_2016
Verified os-libs-compat32 signed by PackageDevelopmentEc_2016
Verified os-libs signed by PackageDevelopmentEc_2016
Verified os-runtime signed by PackageDevelopmentEc_2016
Verified os-vmguest signed by PackageDevelopmentEc_2016
Verified os-zoneinfo signed by PackageDevelopmentEc_2016
Verified py-base signed by PackageDevelopmentEc_2016
Verified py-extensions signed by PackageDevelopmentEc_2016
Verified vrr-mx signed by PackageDevelopmentEc_2016
NOTICE: 'pending' set will be activated at next reboot...
Rebooting. Please wait ...
shutdown: [pid 29823]
Shutdown NOW!
Shutdown NOW!
System shutdown time has arrived
0
]
hup
Pushing /var/tmp/junos-install-ex92xx-x86-64-16.1-20160622.1.tgz to re1:/var/tmp/junos-install-ex92xx-x86-64-16.1-20160622.1.tgz
hup
Verified junos-install-ex92xx-x86-64-16.1-20160622.1 signed by PackageDevelopmentEc_2016
Verified manifest signed by PackageDevelopmentEc_2016
Checking PIC combinations
Verified fips-mode signed by PackageDevelopmentEc_2016
Verified jail-runtime signed by PackageDevelopmentEc_2016
Verified jdocs signed by PackageDevelopmentEc_2016
Verified jpfe-X960 signed by PackageDevelopmentEc_2016
Verified jpfe-common signed by PackageDevelopmentEc_2016
Verified jpfe-wrlinux signed by PackageDevelopmentEc_2016
Verified jplatform-ex92xx signed by PackageDevelopmentEc_2016
Verified jsd signed by PackageDevelopmentEc_2016
Verified jsdn signed by PackageDevelopmentEc_2016
Verified jservices-crypto signed by PackageDevelopmentEc_2016
Adding jservices-crypto-x86-32-16.1-20160622.1 ...
Verified jservices-crypto-base signed by PackageDevelopmentEc_2016
Verified jservices-ipsec signed by PackageDevelopmentEc_2016
Verified jservices-ssl signed by PackageDevelopmentEc_2016
Verified jservices signed by PackageDevelopmentEc_2016
Adding jservices-x86-32-16.1-20160622.1 ...
Verified jservices-aacl signed by PackageDevelopmentEc_2016
Verified jservices-alg signed by PackageDevelopmentEc_2016
Verified jservices-appid signed by PackageDevelopmentEc_2016
Verified jservices-bgf signed by PackageDevelopmentEc_2016
Verified jservices-cpcd signed by PackageDevelopmentEc_2016
Verified jservices-hcm signed by PackageDevelopmentEc_2016
Verified jservices-idp signed by PackageDevelopmentEc_2016
Verified jservices-jdpi signed by PackageDevelopmentEc_2016
Verified jservices-jflow signed by PackageDevelopmentEc_2016
Verified jservices-llpdf signed by PackageDevelopmentEc_2016
Verified jservices-lrf signed by PackageDevelopmentEc_2016
Verified jservices-mobile signed by PackageDevelopmentEc_2016
Verified jservices-mss signed by PackageDevelopmentEc_2016
Verified jservices-nat signed by PackageDevelopmentEc_2016
Verified jservices-pcef signed by PackageDevelopmentEc_2016
Verified jservices-ptsp signed by PackageDevelopmentEc_2016
Verified jservices-rpm signed by PackageDevelopmentEc_2016
Verified jservices-sfw signed by PackageDevelopmentEc_2016
Verified jservices-voice signed by PackageDevelopmentEc_2016
Verified jsim-pfe signed by PackageDevelopmentEc_2016
Verified junos-daemons-mx signed by PackageDevelopmentEc_2016
Verified junos-daemons signed by PackageDevelopmentEc_2016
Verified junos-dp-crypto-support-mtx signed by PackageDevelopmentEc_2016
Verified junos-dp-crypto-support signed by PackageDevelopmentEc_2016
Verified junos-libs-compat32-mx signed by PackageDevelopmentEc_2016
Verified junos-libs-compat32 signed by PackageDevelopmentEc_2016
Verified junos-libs-mx signed by PackageDevelopmentEc_2016
Verified junos-libs signed by PackageDevelopmentEc_2016
Verified junos-modules-mx signed by PackageDevelopmentEc_2016
Verified junos-modules signed by PackageDevelopmentEc_2016
Verified junos-net-prd signed by PackageDevelopmentEc_2016
Verified junos-platform signed by PackageDevelopmentEc_2016
Verified junos-runtime-mx signed by PackageDevelopmentEc_2016
Verified junos-runtime signed by PackageDevelopmentEc_2016
Verified junos-vmguest-mtx signed by PackageDevelopmentEc_2016
Verified jweb signed by PackageDevelopmentEc_2016
Verified oam signed by PackageDevelopmentEc_2016
Verified os-compat32 signed by PackageDevelopmentEc_2016
Verified os-crypto signed by PackageDevelopmentEc_2016
Verified os-kernel-prd signed by PackageDevelopmentEc_2016
Verified os-libs-compat32 signed by PackageDevelopmentEc_2016
Verified os-libs signed by PackageDevelopmentEc_2016
Verified os-runtime signed by PackageDevelopmentEc_2016
Verified os-vmguest signed by PackageDevelopmentEc_2016
Verified os-zoneinfo signed by PackageDevelopmentEc_2016
Verified py-base signed by PackageDevelopmentEc_2016
Verified py-extensions signed by PackageDevelopmentEc_2016
Verified vrr-mx signed by PackageDevelopmentEc_2016
NOTICE: 'pending' set will be activated at next reboot...
Rebooting. Please wait ...
shutdown: [pid 27181]
Shutdown NOW!
Shutdown NOW!
System shutdown time has arrived
0
*** FINAL System shutdown message from root@JUNOS-FUSION-01 ***
System going down IMMEDIATELY
Stopping cron.
Waiting for PIDS: 17816.
.
Jul 14 02:24:00 jlaunchd: clksyncd-service (PID 18457) terminate signal 15 sent
Jul 14 02:24:01 jlaunchd: ethernet-link-fault-management (PID 18706) terminate signal 15 sent
Jul 14 02:24:01 jlaunchd: mib-process (PID 18725) terminate signal 15 sent
Jul 14 02:24:05 jlaunchd: sflow-service (PID 18715) exited with status=0 Normal Exit
Jul 14 02:24:05 jlaunchd: l2-learning (PID 18699) exited with status=0 Normal Exit
Jul 14 02:24:05 jlaunchd: app-engine-virtual-machine-management-service (PID 17828) exited with status=0 Normal Exit
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...0 0 0 done
All buffers synced.
Uptime: 1h31m12s
Khelp module "jsocket" can't unload until its refcount drops from 2 to 0.
<<<<<<<<<<<<<<<<<<<<< BOOT LOG messages >>>>>>>>>>>>>>>>>>>>>>
FreeBSD/amd64 (JUNOS-FUSION-01) (ttyu0)
login:
SLAX script details:
Arguments:
$filename - **REQUIRED** Path to the image to be installed.
$host - **REQUIRED** Host Name/IP address or ICCP IP
$force - *OPTIONAL* force addition of package (ignore warnings)
- true/false [true]
$validate - *OPTIONAL* check compatibility with current configuration
- true/false [false]
Templates:
generate-sshkey generate ssh public key on local box
copy-sshkey copy ssh public key to remote server for authentication
copy-file copy junos image to remote box
install-image install junos image on the box
build-upgrade-string build the cli to install junos image on given re
check-satellite check the status of chassis satellite
determine-local-re determine the master re
get-satellite-info get the chassis satellite information
check-ping check the reachability of the host
Implementation details:
- The script takes two necessary arguments – filename and host.
- filename should point to the JUNOS software image that is expected to be downloaded on the local device by the user before beginning the script execution.
- host is the IP address / name of the remote device to install the JUNOS software image. In the case of Junos Fusion, the ICCP address can be used as a host IP.
- For installing the software image there are two optional arguments – force and validate.
- force would ignore the warnings at the time of installation. Default force is false.
- validate would check compatibility with current configuration. Default validate is true.
- The script generates ssh key on local device using generate-sshkey
- The id_rsa.pub key generated is then copied to the list of known users on the remote device using copy-sshkey. The copy-sshkey template uses file-put rpc to copy the content of the key on the remote server. This step is required to facilitate image copy without using a password.
- The image present at path passed as filename is then copied to the /var/tmp of the remote device using copy-file template which uses file-copy rpc.
- The chassis satellite information of the peer aggregation device is obtained using the template get-satellite-info and stored in a variable locally.
- The script then determines the backup RE of the remote device using determine-local-re template and installs the image on it. After that, the image is installed on the master RE of the remote device.
- If the parallel option is not set to ‘true’,
- The script sleeps for 5 minutes, which is the expected time for the reboot of the aggregation device and waits for the interfaces to come up using the template check-ping.
- The script then checks for the satellite information and matches with the one obtained before upgrading.
- In case of mismatch the script sleeps for 10 sec and tries again to obtain the status of satellite(s). There are 10 such trials, after which the script is aborted from further upgrading.
- In the satellite(s) state(s) is proper the script goes ahead with upgrading the local device.
All this is taken care of by template check-satellite.
- The JUNOS software image is then installed on the local device REs in similar order.
- The local and remote devices are rebooted after successful installation.
Though this is written for Junos Fusion, it can be used to upgrade MC-LAG peer boxes. Also, this script can be extended to use for single RE boxes as well and to upgrade multiple devices (standalone and VCs) using a single point.
/**************************************************************************
Script - software-add-dual-ad.slax
Author - Dharmik Thakkar (dharmikt@juniper.net)
Anil Kumar A (anilak@juniper.net)
Rushi Trivedi (rtrivedi@juniper.net)
Functionality - This slax script installs JUNOS software on all the
routing engines in aggregation devices of Junos Fusion.
***************************************************************************/
version 1.0;
ns junos = "http://xml.juniper.net/junos/*/junos";
ns xnm = "http://xml.juniper.net/xnm/1.1/xnm";
ns jcs = "http://xml.juniper.net/junos/commit-scripts/1.0";
import "../import/junos.xsl";
/**************************************************************************
List of arguments for the script -
$filename - **REQUIRED** Path to the image to be installed.
$host - **REQUIRED** Host Name/IP address or ICCP IP
$force - *OPTIONAL* force addition of package (ignore warnings) - true/false [true]
$validate - *OPTIONAL* check compatibility with current configuration - true/false [false]
***************************************************************************/
var $arguments = {
<argument> {
<name> "filename";
<description> "**REQUIRED** Path to the image to be installed.";
}
<argument> {
<name> "host";
<description> "**REQUIRED** Host Name/IP address or ICCP IP";
}
<argument>{
<name> "force";
<description> "*OPTIONAL* force addition of package (ignore warnings) - true/false [true]";
}
<argument>{
<name> "validate";
<description> "*OPTIONAL* check compatibility with current configuration - true/false [false]";
}
<argument>{
<name> "parallel";
<description> "*OPTIONAL* install junos software on both ADs parellely, traffic might get dropped - true/false [false]";
}
}
param $host;
param $filename;
param $force = "true";
param $validate = "false";
param $parallel = "false";
var $username = "root";
var $password = "Embe1mpls";
/*
var $filename = "/var/tmp/junos-install-ex92xx-x86-64-16.1-20160622.1.tgz";
var $host="10.1.1.2";
*/
/*
template get-iccp-link($local){
var $iccp = <get-inter-chassis-control-protocol-information>;
var $out = jcs:execute($local,$iccp);
var $pattern = "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}";
var $peer_ip = jcs:regex($pattern,$out);
expr $peer_ip;
}
*/
/**************************************************************************
template - generate-sshkey
arguments - *$local* -> "handle to local connection"
functionality - generate ssh public key on local box
***************************************************************************/
template generate-sshkey($local){
var $remove_gen = {
<request-shell-execute> {
<command> "rm -f /root/.ssh/id_rsa";
}
}
var $add_gen = {
<request-shell-execute> {
<command> 'ssh-keygen -t rsa -f /root/.ssh/id_rsa -N ""';
}
}
var $cat = {
<request-shell-execute>{
<command> 'cat /root/.ssh/id_rsa.pub';
}
}
var $rem-key = jcs:execute($local,$remove_gen);
var $add-key = jcs:execute($local,$add_gen);
var $key = jcs:execute($local,$cat);
expr $key;
}
/**************************************************************************
template - copy-sshkey
arguments - *$local* -> "handle to local connection"
*$connection* -> "handle to remote server connection"
functionality - copy ssh public key to remote server for authentication
***************************************************************************/
template copy-sshkey($local, $connection){
var $get-key = {
<request-shell-execute> {
<command> "cat /root/.ssh/id_rsa.pub";
}
}
var $key = jcs:execute($local,$get-key);
if ($key//xnm:error) {
copy-of ($out//xnm:error);
}
var $fileput = {
<file-put> {
<filename>"/root/.ssh/authorized_keys";
<encoding>"ascii";
<permission>"0644";
<delete-if-exist>;
<file-contents> $key;
}
}
var $out = jcs:execute($connection, $fileput);
if ($out//xnm:error) {
copy-of ($out//xnm:error);
}
var $cat = {
<request-shell-execute>{
<command> 'cat /root/.ssh/authorized_keys';
}
}
var $temp = jcs:execute($connection,$cat);
if ($temp//xnm:error) {
copy-of ($out//xnm:error);
}
expr $temp;
}
/**************************************************************************
template - copy-file
arguments - *$local* -> "handle to local connection"
*$host* -> "remote server name/ip"
functionality - copy junos image to remote server
***************************************************************************/
template copy-file($local, $host){
var $dest_filename = 'root@'_ $host _ ':/var/tmp';
var $copy-rpc = <file-copy> {
<source> $filename;
<destination> $dest_filename;
};
var $out = jcs:execute($local, $copy-rpc);
expr $out;
}
/**************************************************************************
template - install-image
arguments - *$connection* -> "handle to remote server connection"
*$filename* -> "path to junos image to be installed"
*$force* -> "force addition of package"
*$validate* -> "check compatibility with configuration"
functionality - install junos image on the box
***************************************************************************/
template install-image($connection, $filename, $force, $validate) {
if( not( $filename ) ) {
<xsl:message terminate="yes"> "You must specify the JUNOS upgrade file.";
}
var $local-re = { call determine-current-re($connection); }
var $other-re = {
if( $local-re == "re0" ) {
expr "re1";
}
else {
expr "re0";
}
}
var $other-re-upgrade-string = {
call build-command-string( $re=$other-re, $filename, $force, $validate );
}
var $other-re-command = <command> $other-re-upgrade-string;
var $other-result = jcs:execute($connection, $other-re-command );
for-each( $other-result//text() ) {
expr jcs:output( . );
}
var $local-re-upgrade-string = {
call build-command-string( $re=$local-re, $filename, $force, $validate );
}
var $local-re-command = <command> $local-re-upgrade-string;
var $local-result = jcs:execute($connection, $local-re-command);
for-each( $local-result//text() ) {
expr jcs:output( . );
}
}
/**************************************************************************
template - build-upgrade-string
arguments - *$re* -> "routing engine to install image"
*$filename* -> "path to junos image to be installed"
*$force* -> "force addition of package"
*$validate* -> "check compatibility with configuration"
functionality - build the cli to install junos image on given re
***************************************************************************/
template build-command-string($re, $filename, $force, $validate) {
if( $force == "true" ) {
if( $validate == "false" ) {
expr "request system software add " _ $filename _ " reboot " _ $re _ " force " _ " no-validate";
}
else {
expr "request system software add " _ $filename _ " reboot " _ $re _ " force";
}
}
else {
if( $validate == "false" ) {
expr "request system software add " _ $filename _ " reboot " _ $re _ " no-validate";
}
else {
expr "request system software add " _ $filename _ " reboot " _ $re;
}
}
}
/**************************************************************************
template - determine-local-re
arguments - *$connection* -> "handle to server connection"
functionality - determine the master re
***************************************************************************/
template determine-current-re($connection) {
var $get-routing-engine = <get-route-engine-information>;
var $routing-engine-info = jcs:execute( $connection, $get-routing-engine );
if( not( $routing-engine-info/route-engine/slot ) ) {
expr "re0";
}
else if( $routing-engine-info/route-engine[slot == "0"]/load-average-one ) {
expr "re0";
}
else if( $routing-engine-info/route-engine[slot == "1"]/load-average-one ) {
expr "re1";
}
else {
expr "re0";
}
}
/**************************************************************************
template - check-ping
arguments - *$local* -> "handle to local device connection"
*host* -> "peer AD ip / iccp ip"
*counter* -> "counter number of retries for ping"
functionality - check the reachability of the host
***************************************************************************/
template check-ping($local, $host, $counter=0){
var $ping-rpc = {
<ping>{
<host> $host;
<count> 1;
}
}
var $ping-out = jcs:execute($local, $ping-rpc);
if($ping-out/ping-success){
expr jcs:output("!!! Ping Success. !!!");
expr "true";
}else{
if($counter < 20){
expr jcs:output("!!! Ping Failed. !!!");
expr jcs:output("!!! Sleeping for 10 sec. !!!");
expr jcs:sleep(10);
call check-ping($local, $host, $counter = $counter+1);
}else{
expr jcs:output("!!! The host is not reachable after upgradation. !!!");
expr jcs:output("!!! Aborting further upgradation of local device. !!!");
expr jcs:close($local);
}
}
}
/**************************************************************************
template - get-satellite-info
arguments - *$connection* -> "handle to peer AD connection"
functionality - get the chassis satellite information
***************************************************************************/
template get-satellite-info($connection){
var $satellite-rpc = <get-chassis-satellite-information>;
var $satellite := jcs:execute($connection, $satellite-rpc);
if(jcs:empty($satellite)){
expr "false";
}else{
var $each_satellite := $satellite/satellite;
mvar $satellite-info;
for-each($each_satellite){
append $satellite-info += <satellite>{
<slot> ./slot-id;
<state> ./operation-state;
}
}
expr jcs:output("slot-id \tsatellite state");
for-each($satellite-info/satellite){
expr jcs:output(./slot _ "\t\t" _ ./state);
}
expr $satellite-info;
}
}
/**************************************************************************
template - check-satellite
arguments - *$connection* -> "handle to peer AD connection"
*satellite-info* -> "satellite info before upgrade"
*counter* -> "counter number of retries"
functionality - check the status of chassis satellite
***************************************************************************/
template check-satellite($connection, $satellite-info, $counter=0){
var $satellite-new-info := {call get-satellite-info($connection);}
if($satellite-new-info == "false"){
expr jcs:output("!!! Satellites are not up yet. !!!");
expr jcs:output("!!! Sleeping for 10 sec. !!!");
expr jcs:sleep(10);
if($counter < 15){
call check-satellite($connection, $satellite-info, $counter = $counter+1);
}else{
expr jcs:output("!!! The satellites did not come up even after trying 15 times. !!!");
expr "false";
}
}else{
if($satellite-new-info == $satellite-info){
expr jcs:output("!!! The satellites are in proper state after peer aggregation device upgrade. !!!");
expr "true";
}else{
if($counter < 15){
expr jcs:output("!!! The satellites are not in proper state after peer aggregation device upgrade. !!!");
expr jcs:output("!!! Trying again. !!!");
expr jcs:output("!!! Sleeping for 10 sec. !!!");
expr jcs:sleep(10);
call check-satellite($connection, $satellite-info, $counter = $counter+1);
}else{
expr jcs:output("!!! The satellites are not in proper state even after trying 10 times. !!!");
expr "false";
}
}
}
}
/**************************************************************************
template - match
functionality - main template to start script execution
***************************************************************************/
match /{
<op-script-results> {
var $local = jcs:open();
var $connection = jcs:open($host, $username, $password);
if ($connection) {
expr jcs:output("!!! Generating the ssh public key on the local device. !!!");
var $cat-key := {call generate-sshkey($local);}
expr jcs:output("!!! Copying the ssh public key to the remote devices' list of known hosts. !!!");
var $copy := {call copy-sshkey($local,$connection);}
expr jcs:output("!!! Copying the JUNOS software image to the remote device. !!!");
var $copyfile := {call copy-file($local, $host);}
expr jcs:output("!!! Getting the satellite state before upgrading the aggregation device. !!!");
var $satellite-info := {call get-satellite-info($connection);}
expr jcs:output("!!! Installing JUNOS software on the remote device. !!!");
expr jcs:output("!!! Please wait while the installation is completed. !!!");
var $other-re := {call install-image($connection, $filename, $force, $validate);}
expr jcs:close($connection);
if($parallel == "false"){
expr jcs:output("!!! Sleeping for 450 sec. !!!");
expr jcs:output("!!! Waiting for interfaces to come up. !!!");
expr jcs:sleep(450);
var $ping-success = {call check-ping($local, $host);}
var $new_connection = jcs:open($host, $username, $password);
if($new_connection){
expr jcs:output("!!! Checking the satellite status before going ahead !!!");
var $satellite-status = {call check-satellite($connection = $new_connection, $satellite-info);}
if($satellite-status == "true"){
expr jcs:output("!!! Going ahead with local aggregation device upgradation. !!!");
expr jcs:close($new_connection);
}else{
expr jcs:output("!!! Aborting the upgradation for local aggregation device. !!!");
expr jcs:output("!!! There is some problem after upgradation with peer aggregation device. !!!");
expr jcs:output("!!! Manually debug the peer aggregation device. !!!");
expr jcs:close($new_connection);
expr jcs:close($local);
<xsl:message terminate="yes"> "!!! Script Aborted. !!!";
}
}else{
expr jcs:output("!!! No connection to host. !!!");
expr jcs:output("!!! Aborting further upgradation. !!!");
expr jcs:close($local);
<xsl:message terminate="yes"> "!!! Script Aborted. !!!";
}
}
expr jcs:output("!!! Installing the JUNOS software on the local device. !!!");
expr jcs:output("!!! Please wait while the installation is completed. !!!");
var $re := {call install-image($connection=$local, $filename, $force, $validate);}
}else{
expr jcs:output("!!! No connection to host. !!!");
<xsl:message terminate="yes"> "!!! Script Aborted. !!!";
}
expr jcs:close($local);
}
}