It is not uncommon to be performing a software upgrade on a network appliance type of device and the operation fails. Hopefully, the failure doesn’t render the hardware useless and allows for a retry of the operation. However, there are times where an upgrade will fail and the device will no longer function. This article uses a specific example to carry you through steps that can be applied to any appliance like device.

First, a few details regarding the example scenario:

A previous upgrade to an IP enabled KVM switch was causing issues with it’s normal operation. There were issues with local use using a directly attached keyboard/monitor/mouse and also when using the viewer plugin remotely. After my co-workers had complained enough, I decided it was time to downgrade the software to the previously running code, which did not have all the issues that was currently happening. Using the management software for the KVM, I downgraded 7 of 8 devices successfully. One device failed during the procedure and subsequently stopped responding on the network.

After giving sufficient time for possible self-recovery with no results, I decided it was time to investigate further. Upon inspecting the device visually, it was determined that the equipment was in recovery mode (The power light was blinking steady with no other lights on the device). This determination was made by going to the hardware manufacturers website and downloading the manual for my particular model, then looking up the device states in the troubleshooting section of the documentation.

The first thing attempted was an obvious one: Try and power cycle the hardware. After turning off and back on, the same result happened – a steadily flashing power light.

The documentation stated that when the device was in recovery mode, it would automatically attempt to download the system image via tftp from the management server. After inspecting the machine running the KVM management software, I was able to determine there was no traffic between it and the failed device. There are several ways to troubleshoot this, my particular method was to run a packet sniffer (Wireshark) from the management server to see if any requests were coming from the KVM’s IP address. If installing Wireshark (or similar program) is not an option on the machine, a portable version is available from the website that can be run out of a directory that either resides on a hard or flash drive.

At this point, a support call would have been the next course of action. However, a current maintenance contract did not exist on this equipment, so tech support was not an option. Truthfully, even if it was an option, I most likely wouldn’t be using it. I would rather be hung upside down (by my toenails), 30 feet in the air, with a pack of flesh eating Hyenas waiting underneath, for me to plummet to my death so they could consume me. Not that there is anything wrong with calling tech support, never mind – I digress…

The device is now officially ‘bricked’ (hence the title of this article). The urban dictionary defines the term as follows:

Bricked refers to ANY hardware that is unable to start up due to bad software; Usually because of a bad software flash, a modification done improperly, loss of necessary files, etc.

Thankfully, the majority of the time a device can be recovered after being in this state.

The next step in my process was to determine if a console was available. After looking at the documentation once again, I found that a serial port was available on this device for management purposes. After recording the applicable serial port settings and grabbing a null modem (serial) cable, it was off to the data center where the device was located.

My thought was to connect the serial cable between a laptop and the KVM device to see if I could get any output using a terminal program. Putty is my terminal program of choice, which has support for serial connections. I configured Putty to connect to COM1 at 9600 baud with 8 bits, No parity, and 1 stop bit (better known as eight, ‘n’, and one). The hope here was maybe the device used a bootloader which is a small piece of software that loads initially (like a BIOS) and in turn loads the full software image for the device. Many times when a bootloader can’t load the main software image, there is a very basic command line structure available to perform recovery functions such as transferring an image, re-issuing boot commands, etc.

After starting Putty and pressing the Enter key several times (which usually prompts the connected device to respond), there was no response. I’m still not sure what was going on with why the console wasn’t working, because I moved on from that very quickly. (My assumption here was the command line via serial port was only available after the firmware was correctly loaded and running on the device)

As I previously mentioned, by reading the documentation, I knew the device was supposed to request a boot image via TFTP. So, I took my laptop and connected it to an isolated switch along with the KVM device’s network interface. After starting Wireshark on the laptop and starting a capture, the KVM was powered on.

AH, progress!

^[1]

The above image displays a WireShark window running on my laptop.  When this photo was taken, there was a display filter set – so that only traffic from the KVM src mac-address was shown.  (A mac filter was used, since that was the only known information).  The mac-address is always shown, usually via a sticker on the device.  Notice it has an IP of 10.0.0.2, which obviously is hard coded in the firmware – since I didn’t have a DHCP server running on the laptop. The next thing you see is the appliance making a request via TFTP to 10.0.0.3 (again another hard coded entry in the firmware) and is requesting a file with the name DSRxx20.fl.

With this information, the laptop’s network interface can now be set statically to 10.0.0.3. The next thing I needed was a TFTP server loaded on my laptop. This is an easy task, with several available freely on the Internet, download your favorite (my recommendation is tftpd32) TFTP server and run it.

The final step is to put the firmware for the device into the TFTP server ‘home’ directory and make sure the filename matches what is being requested (in this case it was DSRxx20.fl). After the file was in place with the TFTP server running, I power cycled the appliance once again:

^[2]

As you can see from above, the transfer took place, which then the device proceeded to boot up perfectly! SUCCESS! Although this is not a universal step by step instruction on how to save any ‘bricked’ device – it should help outline the steps required to discover what is needed to bring something you are working on back to life.