Automating Cisco IOS updates with Unimus — Part 2

Intro

  • FW image source — this time it will store not just the IOS FW images, but also two scripts. One will be a bash script which we will use to generate a list of FW images for devices to find an upgrade candidate in, and the other one will be a dedicated upgrade TCL script.
  • Cisco IOS devices. These will be downloading the TCL upgrade script, list of available images, and lastly the FW image file to update to (assuming the devices find an update candidate in the image list).
  • Unimus and our Mass Config Push feature (version 2.1.0 and newer) with a slightly different set of commands to push compared to Part 1’s preset.

Preparing the Image / FW source

  • We will serve our IOS images from this machine. Note that we are focusing only on IOS images in .bin format, not archived images in .tar format. This is done so that we can can avoid as many issues as possible with devices with insufficient free space, or ones with smaller flash capacity. These wouldn't be able to fit multiple IOS image even if their flash was empty.
  • We will generate a sorted list (supporting IOS versioning) of FW images with their respective MD5 sums and make this available for our devices to download.
  • We will serve our upgrade TCL script. Our IOS devices will download and execute this to find an update image candidate, download it, verify it, and set it as the boot image.

FW Transfer methods

SCP

HTTP

Overview of the FW image parsing bash script

#!/bin/bash

#Set a working directory in the current directory
cd "${0%/*}"

#Extract file names and MD5 sums, sort them and output into a file
md5sum *.bin | sort -V -k2 > fwlist
a63c90cc3684ad8b0a2176a6a8fe9005  c180x-advipservicesk9-mz.151-4.M12a.bin
6d0bb00954ceb7fbee436bb55a8397a9 c1900-universalk9_npe-mz.SPA.158-3.M7.bin
28518159ba5f75ef0eeb9617fd35e2ba c2800nm-advipservicesk9-mz.124-24.T4.bin
441018525208457705bf09a8ee3c1093 c3750e-ipbasek9-mz.122-55.SE5.bin
862dec5c27142824a394bc6464928f48 c3750e-universalk9-mz.122-55.SE5.bin
fd4b38e94292e00251b9f39c47ee5710 c3750e-universalk9-mz.152-4.E10.bin
1f94dacb4faf2829b0ffbb25ebd62e2e c3750-ipbasek9-mz.150-2.SE5.bin
b28cf0ed5cc0d1928ea4f6656e1c8dde c3750-ipservicesk9-mz.122-55.SE12.bin
871bdd96b159c14d15c8d97d9111e9c8 cat4500-ipbasek9-mz.150-2.SG11.bin
3287282fa1a1523a294fb018e3679872 s72033-adventerprise_wan-vz.122-33.SXI14.bin
a302a771ee0e3127b8950f0a67d17e49 s72033-ipbase-mz.151-2.SY16.bin
bbf7c6077962a7c28114dbd10be947cd s72033-ipservicesk9-mz.151-2.SY16.bin

Overview of the upgrade TCL script

tclsh

#Load and store list of available FW images
set fwrawlist [read [open fwlist]]

#Retrieve FW type of the current device for further processing
set devicefwdirty [exec show version | include System image file is]
#Get a full name of the current FW
regexp -all {:(.*?)\"} $devicefwdirty junk devicefwfull
#Get a FW type with a major release version, this prevents issues with some devices which might not be compatible (mainly storage constrains) with the next major release version
regexp -all {:(.*?-.*?-.*?\.\d\d).*?.bin} $devicefwdirty junk devicefwrelease

#Find FW update candidate
puts "Finding a viable FW update candidate..."
#Process the list, return the latest match (it will be the latest FW image)
set fwlistparsed [regexp -all -line "\\s{2}($devicefwrelease.+?.bin)$" $fwrawlist junk down_file]
#If no match is found, abort the script
if {$fwlistparsed == 0} {
puts "List of available FWs does not contain any update candidate. Aborting..."
return
}
#Run fullname string comparison, if matched, current and matched FW are identical, abort the script
if {[string compare $devicefwfull $down_file] == 0} {
puts "Current and matched FW image are identical. Aborting..."
return
}
#If the current and matched FW are not identical, start comparing them on each level
#Compare major release version
regexp -all {\.([0-9]{1,4})\-} $devicefwfull junk curfwmatch1
regexp -all {\.([0-9]{1,4})\-} $down_file junk newfwmatch1
if {$curfwmatch1 == $newfwmatch1} {
#Major release version is identical, compare minor release version
regexp -all {\-([0-9]{1,3})\.[a-zA-Z]*?[0-9]*?[a-zA-Z]*?\.bin} $devicefwfull junk curfwmatch2
regexp -all {\-([0-9]{1,3})\.[a-zA-Z]*?[0-9]*?[a-zA-Z]*?\.bin} $down_file junk newfwmatch2
if {$curfwmatch2 == $newfwmatch2} {
#Minor release version is also identical, compare revision
regexp -all {\-[0-9]{1,3}\.[a-zA-Z]*?([0-9]*?)[a-zA-Z]*?\.bin} $devicefwfull junk curfwmatch3
regexp -all {\-[0-9]{1,3}\.[a-zA-Z]*?([0-9]*?)[a-zA-Z]*?\.bin} $down_file junk newfwmatch3
if {$curfwmatch3 == $newfwmatch3} {
#Revision is also identical, in this case it suggest some other problem in versioning or FW naming
puts "Current and matched FW image and their (numeric) version seem identical, but full names are not. Aborting..."
return
} elseif {$curfwmatch3 > $newfwmatch3} {
puts "Current FW image is newer than the matched one from the list of available FWs. No update candidates were found. Aborting..."
return
} elseif {$curfwmatch3 < $newfwmatch3} {
puts "Update candidate found."
} else {
puts "Unknown error occurred during FW matching. Aborting..."
return
}
} elseif {$curfwmatch2 > $newfwmatch2} {
puts "Current and matched FW image and their (numeric) version seem identical, but full names are not. Aborting..."
return
} elseif {$curfwmatch2 < $newfwmatch2} {
puts "Update candidate found."
} else {
puts "Unknown error occurred during FW matching. Aborting..."
return
}
} elseif {$curfwmatch1 > $newfwmatch1} {
puts "Current and matched FW image and their (numeric) version seem identical, but full names are not. Aborting..."
return
} elseif {$curfwmatch1 < $newfwmatch1} {
puts "Update candidate found."
} else {
puts "Unknown error occurred during FW matching. Aborting..."
return
}

#Download FW update
#Read common arguments, abort if mandatory arguments are missing, and decide which protocol will be used
set down_prot [lindex $argv 0]
if {[string length $down_prot] == 0} {
puts "No argument was defined, please add arguments to your MCP in Unimus where you execute this command. Aborting..."
return
}
set down_addr [lindex $argv 1]
if {[string length $down_addr] == 0} {
puts "Second argument (address) is missing. Aborting..."
return
}
#Read HTTP specific arguments and build download URL
if {[string compare http $down_prot] == 0} {
set down_port [lindex $argv 2]
if {[string length $down_port] == 0} {
#Use port 80 if no custom port is defined
set down_port "80"
}
set down_url "http://$down_addr:$down_port/$down_file"
#Read SCP specific arguments and build download URL
} elseif {[string compare scp $down_prot] == 0} {
set down_user [lindex $argv 2]
if {[string length $down_user] == 0} {
puts "Third argument (user) is missing. Aborting..."
return
}
set down_pass [lindex $argv 3]
if {[string length $down_pass] == 0} {
puts "Fourth argument (password) is missing. Aborting..."
return
}
set down_url "scp://$down_user:$down_pass@$down_addr/$down_file"
} else {
puts "Unrecognized protocol. Aborting..."
return
}
puts "Downloading firmware..."
set down_result [exec copy $down_url flash:]
#Evaluate download result
if {[regexp {bytes copied} $down_result]} {
puts "Update FW image was downloaded successfully."
} elseif {[regexp {Not enough space} $down_result]} {
if {[regexp {Not enough space} $down_result]} {
puts "Error occurred during download - insufficient space left on device. Aborting..."
return
}
} elseif {[regexp {Protocol error} $down_result]} {
puts "Error occurred during download - protocol error. Aborting..."
} elseif {[regexp {busy} $down_result]} {
puts "Error occurred during download - device is busy. Aborting..."
} else {
puts "Unknown error occurred during download. Aborting..."
return
}

#Validate MD5 of the downloaded FW image
puts "Validating integrity..."
#Run validation for the downloaded FW image
set down_file_md5check [exec verify /md5 $down_file]
regexp -all -line "=\\s{1}(.+?)$" $down_file_md5check junk down_file_md5
regexp -all -line "(.+?)\\s{2}$down_file" $fwrawlist junk fwmd5tocomp
#Compare both MD5 sums
if {[string compare $down_file_md5 $fwmd5tocomp] == 1} {
puts "Update FW image validated successfully."
} else {
puts "Unknown error occurred when validating update FW image integrity, MD5 sums do not match. Aborting..."
return
}

#Set up system boot image with the downloaded update FW image
puts "Updating..."
ios_config "boot system flash:$down_file"
puts "Update is ready. Please run your reload MCP preset..."
  • Script ingests a list of available FW images from the pre-generated list we prepared earlier (with the bash script, the fwlist file).
  • Script checks the current version of IOS running on the device.
  • Script compares the current IOS version to all available FW images — note we are matching only the same major release version (if your device is running a 12.X release, you will be able to upgrade only to a newer version of the 12.X release, not to a newer major release like 15.X or 17.X).
  • If the script matches multiple upgrade candidate image files, it will always choose the last match, which will be the latest FW image (thanks to version-aware sorting in the list of FW images).
  • Script processes input arguments, evaluates them and builds a download URL for the device to download the new FW image.
  • A new FW image is downloaded. Its integrity verified and compared to the MD5 sum from our known good sums on the image server.
  • If the integrity checks out, the script sets up the FW image as the boot image and returns a final message informing the user to reload the device.

Preparing Unimus and Mass Config Push presets for IOS upgrade

Config Push preset 1 — Upgrade devices

SCP

tclsh
log_user 0
exec "copy scp://SCP_USER:SCP_PASS@FW_SRC_ADDR/fwlist flash:"
exec "copy scp://SCP_USER:SCP_PASS@FW_SRC_ADDR/ios_upgrade.tcl flash:"
tclquit
tclsh ios_upgrade.tcl PROTOCOL FW_SRC_ADDR SCP_USER SCP_PASS
SCP_USER - SCP user
SCP_PASS - SCP password
PROTOCOL - scp or http - protocol used to download an FW image
FW_SRC_ADDR - IP or hostname of FW image source device

HTTP

tclsh
log_user 0
exec "copy http://FW_SRC_ADDR/fwlist flash:"
exec "copy http://FW_SRC_ADDR/ios_upgrade.tcl flash:"
tclquit
tclsh ios_upgrade.tcl PROTOCOL FW_SRC_ADDR FW_SRC_PORT
PROTOCOL - scp or http - protocol used to download an FW image
FW_SRC_ADDR - IP or hostname of FW image source device
FW_SRC_PORT - OPTIONAL - define this argument only if your webserver is listening on a port other than 80, otherwise remove this argument altogether, script will default to port 80

Config Push preset 2 — Reload devices

tclsh
exec "reload in 3"
tclquit

Example of a successful run

Troubleshooting FAQ

Unimus returned INTERACTION_ERROR

Script returned error message Error occurred during download - device is busy. Aborting...

Script returned error message Error occurred during download - insufficient space left on device. Aborting...

Script returned error message Error occurred during download - protocol error. Aborting...

terminal monitor
debug scp all
copy scp://unimus:scppass8520@10.30.50.70/fwlist flash:
cisco#copy scp://unimus:scppass8520@10.30.50.70/fwlist flash:
Destination filename [fwlist]?
%Error opening scp://unimus:scppass8520@10.30.50.70/fwlist (Protocol error)
cisco#
*Feb 5 05:31:45.181: SSH2 CLIENT 0: kex algo not supported: client diffie-hellman-group1-sha1, server curve25519-sha256,curve25519-sha256@libssh.org
cisco#

Script returned an error message Unknown error...

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store