The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
Upgrades and reboots are ongoing network maintenance activities. You should try to minimize the risk of disrupting the network when performing these operations in production environments and to know how to recover quickly when something does go wrong.
 Note |
This publication uses the term upgrade to refer to both Cisco NX-OS upgrades and downgrades. |
Use the following checklist to prepare for an upgrade or reboot:
| Checklist | Done |
|---|---|
| Read the Release Notes for the release to which you are upgrading or downgrading. | |
| Ensure that an FTP or TFTP server is available to download the software image. | |
| Copy the new image onto your supervisor modules in bootflash: or slot0:. | |
| Use the show install all impact command to verify that the new image is healthy and the impact that the new load will have on any hardware with regard to compatibility. Check for compatibility. | |
| Copy the startup-config file to a snapshot configuration in NVRAM. This step creates a backup copy of the startup configuration file. | |
| Save your running configuration to the startup configuration. | |
| Back up a copy of your configuration to a remote TFTP server. | |
| Schedule your upgrade during an appropriate maintenance window for your network. |
After you have completed the checklist, you are ready to upgrade or reboot the systems in your network.
Note |
It is normal for the active supervisor to become the standby supervisor during an upgrade. |
Note |
Up to 100 log messages with a severity level of critical and below (levels 0, 1, and 2) are saved in NVRAM. You can view this log at any time by entering the show logging nvram command. |
You can use the show install all status command to watch the progress of your software upgrade or to view the ongoing install all command or the log of the last installed install all command from a console, SSH, or Telnet session. This command shows the install all output on both the active and standby supervisor module even if you are not connected to the console terminal.
Troubleshooting Software Upgrades and Downgrades
| Problem | Possible Cause | Solution |
|---|---|---|
| The upgrade ends with an error | The standby supervisor module bootflash: file system does not have sufficient space to accept the updated image. | Use the delete command to remove unnecessary files from the file system. |
| The install all command is entered on the standby supervisor module. | Enter the command on the active supervisor module only. | |
| A module was inserted while the upgrade was in progress. | Restart the installation. | |
| The system experienced a power disruption while the upgrade was in progress. | Restart the installation. | |
| An incorrect software image path was specified. | Specify the entire path for the remote location accurately. | |
| Another upgrade is already in progress. | Verify the state of the system at every stage and restart the upgrade after 10 seconds. If you restart the upgrade within 10 seconds, the command is rejected. An error message displays, indicating that an upgrade is currently in progress. | |
| A module failed to upgrade. | Restart the upgrade or use the install module command to upgrade the failed module. |
You can perform an automated software upgrade on any system from the CLI.
Log into the system through the console, Telnet, or SSH port of the active supervisor.
Create a backup of your existing configuration file, if required.
| Command or Action | Purpose | |||||
|---|---|---|---|---|---|---|
|
Step 1 |
install all [nxos bootflash:filename] |
Performs the upgrade.
|
||||
|
Step 2 |
show module |
Exits the system console and opens a new terminal session to view the upgraded supervisor module. |
Troubleshooting Software System Reboots
| Problem | Possible Cause | Solution |
|---|---|---|
| A power-on or switch reboot hangs for a dual supervisor configuration | The bootflash is corrupted. | See Corrupted Bootflash Recovery. |
| The BIOS is corrupted. | Replace this module. Contact your customer support representative to return the failed module. | |
| The nx-os image is corrupted. |
Power cycle the switch if required and press Ctrl-C when the switch displays the "Loading Boot Loader" message to interrupt the boot process at the >loader prompt. |
|
| Boot parameters are incorrect. | Verify and correct the boot parameters and reboot. |
All device configurations reside in the internal bootflash. If you have a corrupted internal bootflash, you could potentially lose your configuration. Be sure to save and back up your configuration files periodically. The regular system boot goes through the following sequence:
The basic input/output system (BIOS) loads the loader.
The loader loads the nx-os image into RAM and starts the image.
The nx-os image reads the startup configuration file.
If the nx-os image on your system is corrupted and you cannot proceed (error state), you can interrupt the system boot sequence and recover the image by entering the BIOS configuration utility described in the following section. Access this utility only when needed to recover a corrupted internal disk.
 Caution |
The BIOS changes explained in this section are required only to recover a corrupted bootflash. |
| Phase | Normal Prompt (appears at the end of each phase) | Recovery Prompt (appears when the system cannot progress to the next phase) | Description |
|---|---|---|---|
| BIOS | loader> | No bootable device | The BIOS begins the power-on self test, memory test, and other operating system applications. While the test is in progress, press Ctrl-C to enter the BIOS configuration utility and use the netboot option. |
| Boot loader | Starting nx-os | loader> | The boot loader uncompresses the loaded software to boot an image using its filename as a reference. The image is made available through bootflash. When the memory test is over, press Esc to enter the boot loader prompt. |
| nx-os image | Uncompressing system | switch(boot)# | When the boot loader phase is over, press
Ctrl-] (Control
key plus right bracket key) to enter the switch(boot)# prompt. Depending on
your Telnet client, these keys might be reserved, and you might need to remap
the keystroke. See the documentation provided by your Telnet client. If the
corruption causes the console to stop at this prompt, copy the nx-os image and
reboot the system.
The nx-os image then loads the configuration file of the last saved running configuration and returns a switch login prompt. |
Use the help command at the loader> prompt to display a list of commands available at this prompt or to obtain more information about a specific command in that list.
This procedure uses the init system command, which reformats the file system of the device. Be sure that you have made a backup of the configuration files before you begin this procedure.
The loader> prompt is different from the regular switch# or switch(boot)# prompt. The CLI command completion feature does not work at the loader> prompt and might result in undesired errors. You must type the command exactly as you want the command to appear.
If you boot over TFTP from the loader> prompt, you must supply the full path to the image on the remote server.
| Command or Action | Purpose | |||
|---|---|---|---|---|
|
Step 1 |
loader> set ip ip-address Example: |
Specifies the local IP address and the subnet mask for the system. |
||
|
Step 2 |
loader> set gw gw-address Example: |
Specifies the IP address of the default gateway. |
||
|
Step 3 |
loader> cmdline recoverymode=1 Example: |
Configures the boot process to stop at the switch(boot)# prompt. |
||
|
Step 4 |
loader> boot tftp: tftp-path Example: |
Boots the nx-os image file from the required server. The switch(boot)# prompt indicates that you have a usable nx-os image. |
||
|
Step 5 |
switch(boot)# init system Example: |
Enters the nx-os system.
|
||
|
Step 6 |
switch(boot)# reload-nxos Example: |
Completes the upload of the nx-os image file. |
loader> set ip 172.21.55.213 255.255.255.224
set ip 172.21.55.213 255.255.255.224
Correct - ip addr is 172.21.55.213, mask is 255.255.255.224
Found Intel 82546GB [2:9.0] at 0xe040, ROM address 0xf980
Probing...[Intel 82546GB]
Management interface
Link UP in 1000/full mode
Ethernet addr: 00:1B:54:C1:28:60
Address: 172.21.55.213
Netmask: 255.255.255.224
Server: 0.0.0.0
Gateway: 172.21.55.193
loader> set gw 172.21.55.193
Correct gateway addr 172.21.55.193
Address: 172.21.55.213
Netmask: 255.255.255.224
Server: 0.0.0.0
Gateway: 172.21.55.193
loader> boot tftp://172.28.255.18/tftpboot/n9000-dk9.6.1.2.I1.1.bin
Address: 172.21.55.213
Netmask: 255.255.255.224
Server: 172.28.255.18
Gateway: 172.21.55.193
Filesystem type is tftp, using whole disk
Booting: /tftpboot/n9000-dk9.6.1.2.I1.1.gbin console=ttyS0,9600n8nn quiet loader
_ver="3.17.0"....
.............................................................................Im
age verification OK
Starting kernel...
INIT: version 2.85 booting
Checking all filesystems..r.r.r.. done.
Setting kernel variables: sysctlnet.ipv4.ip_forward = 0
net.ipv4.ip_default_ttl = 64
net.ipv4.ip_no_pmtu_disc = 1
.
Setting the System Clock using the Hardware Clock as reference...System Clock set. Local time: Wed Oct 1
11:20:11 PST 2013
WARNING: image sync is going to be disabled after a loader netboot
Loading system software
No system image Unexporting directories for NFS kernel daemon...done.
INIT: Sending processes the KILL signal
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2013, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
switch(boot)# When a recoverable or nonrecoverable error occurs, the system or a process on the system might reset. This table lists possible causes and solutions.
| Problem | Possible Cause | Solution |
|---|---|---|
| The system or a process on the system resets. | A recoverable error occurred on the system or on a process in the system. | The system has automatically recovered from the problem. See Recovering System Restarts. |
| A nonrecoverable error occurred on the system. | The system cannot recover automatically from the problem. See Recovering System Restarts to determine the cause. | |
| A clock module failed. | Verify that a clock module failed. Replace the failed clock module during the next maintenance window. |
Every process restart generates a syslog message and a Call Home event. Even if the event does not affect service, you should identify and resolve the condition immediately because future occurrences could cause a service interruption.
Note |
After following the steps, determine the cause and resolution for the restart condition by contacting your technical support representative and asking the representative to review your core dump. |
The copy of a specific core file to a TFTP server can be manually triggered by using the copy core://module#/pid# tftp://tftp_ip_address/file_name command.
If a supervisor failover occurs, the cores might be in the secondary logflash rather than the primary logflash.
The maximum number of times that a process can be restarted is part of the high-availability (HA) policy for any process. (This parameter is not configurable.) If the process restarts more than the maximum number of times, the older core files are overwritten.
The maximum number of core files that can be saved for any process is part of the HA policy for any process. (This parameter is not configurable, and it is set to three.)
| Command or Action | Purpose | |||
|---|---|---|---|---|
|
Step 1 |
switch# show log | include error Example: |
Displays the syslog file so you can see which process restarted and why it restarted. |
||
|
Step 2 |
switch# show processes Example: |
Displays the processes that are running and the status of each process.
|
||
|
Step 3 |
switch# show process log Example: |
Displays the processes that have had abnormal exits and if there is a stack-trace or core dump. |
||
|
Step 4 |
switch# show process log pid pid Example: |
Displays detailed information about a specific process that has restarted. |
||
|
Step 5 |
switch# show system uptime Example: |
Displays if the restart recently occurred. To determine if the restart is repetitive or a one-time occurrence, compare the length of time that the system has been up with the timestamp of each restart. |
||
|
Step 6 |
switch# show cores Example: |
Displays all cores that are presently available for upload from the active supervisor. |
||
|
Step 7 |
switch# copy core: core path Example: |
Copies the FSPF core dump to a TFTP server with an IP address. |
||
|
Step 8 |
switch# show processes log pid pid Example: |
Displays the file named zone_server_log.889 in the log directory, |
||
|
Step 9 |
switch# system cores tftp: tftp-path Example: |
Configures the system to use TFTP to send the core dump to a TFTP server. This command causes the system to enable the automatic copy of core files to a TFTP server. |
An unrecoverable system restart might occur in the following cases:
A critical process fails and is not restartable.
A process restarts more times than is allowed by the system configuration.
A process restarts more frequently than is allowed by the system configuration.
The effect of a process reset is determined by the policy configured for each process. An unrecoverable reset might cause functionality loss, the active supervisor to restart, a supervisor switchover, or the system to restart.
The last four reset-reason codes for a specific module in a given slot. If a module is absent, the reset-reason codes for that module are not displayed.
The overall history of when and why expected and unexpected reloads occur.
The time stamp of when the reset or reload occurred.
The reason for the reset or reload of a module.
The service that caused the reset or reload (not always available).
The software version that was running at the time of the reset or reload.
switch# show system reset-reason module 27
----- reset reason for Supervisor-module 27 (from Supervisor in slot 27) ---
1) At 281000 usecs after Wed Jun 26 20:16:34 2013
Reason: Reset Requested by CLI command reload
Service:
Version: 6.1(2)I1(1)
2) At 791071 usecs after Wed Jun 26 20:04:50 2013
Reason: Reset Requested by CLI command reload
Service:
Version: 6.1(2)I1(1)
3) At 70980 usecs after Wed Jun 26 19:55:52 2013
Reason: Reset Requested by CLI command reload
Service:
Version: 6.1(2)I1(1)
4) At 891463 usecs after Wed Jun 26 23:44:48 2013
Reason: Reset Requested by CLI command reload
Service:
Version: 6.1(2)I1(1)
The standby supervisor does not boot after an upgrade. You may see the following system message:
SYSMGR-2-STANDBY_BOOT_FAILED
This message is printed if the standby supervisor does not complete its boot procedure (does not reach the login prompt on the local console) 3 to 6 minutes after the loader has been loaded by the BIOS. This message is usually caused by boot variables not properly set for the standby supervisor. This message can also be caused by a user intentionally interrupting the boot procedure at the loader prompt (by pressing ESC).
Connect to the local console of the standby supervisor. If the supervisor is at the loader prompt, try to use the boot command to continue the boot procedure. Otherwise, enter the reload command for the standby supervisor from a vsh session on the active supervisor, specifying the force-dnld option. Once the standby is online, fix the problem by setting the boot variables appropriately.
| Symptom | Possible Cause | Solution |
|---|---|---|
| Standby supervisor does not boot. | Active supervisor nx-os image booted from TFTP. | Reload the active supervisor from bootflash:. |
You can recover the network administrator password using one of these methods:
From the CLI with a username that has network-admin privileges
By power cycling the device
By reloading the device
| Command or Action | Purpose | |||
|---|---|---|---|---|
|
Step 1 |
switch# show user-account Example: |
Shows that your username has network-admin privileges. |
||
|
Step 2 |
switch# config terminal Example: |
Enters global configuration mode. |
||
|
Step 3 |
switch(config)# username admin password new-password Example: |
Assigns a new network administrator password if your username has network-admin privileges.
|
||
|
Step 4 |
switch(config)# copy running-config startup-config Example: |
Copies the running configuration to the startup configuration. |
If you cannot start a session on the device that has network-admin privileges, you can recover the network administrator password by power cycling the device.
Caution |
The password recovery procedure disrupts all traffic on the device. All connections to the device will be lost for 2 to 3 minutes. |
Note |
You cannot recover the administrator password from a Telnet or Secure Shell (SSH) session to the management interface. You must have access to the local console connection. |
Note |
Password recovery updates the new administrator password only in the local user database and not on the remote AAA servers. The new password works only if local authentication is enabled; it does not work for remote authentication. When a password is recovered, local authentication is enabled for logins through a console so that the admin user can log in with a new password from a console. |
Note |
If you need to recover the password because the username was not specified in the configuration file when you performed a copy configuration-file startup-config followed by the fast-reload or reload command, you will need to perform a write erase in Step 12 below. |
On a device with two supervisor modules, you must perform the password recovery procedure on the supervisor module that will become the active module after you complete the recovery procedure. To ensure that the other supervisor module does not become active, perform one of the following tasks:
Remove the other supervisor module from the chassis.
Change the console prompt of the other supervisor module to one of the following two prompts until the recovery procedure completes:
loader >
switch(boot)#
| Command or Action | Purpose | |||
|---|---|---|---|---|
|
Step 1 |
Establish a terminal session on the console port of the active supervisor module. |
—
|
||
|
Step 2 |
If you use SSH or a terminal emulator to access the console port, go to Step 6. |
— |
||
|
Step 3 |
If you use Telnet to access the console port, press Ctrl-] (right square bracket) to verify that it does not conflict with the Telnet escape sequence. Example: |
—
|
||
|
Step 4 |
If the Telnet prompt appears, change the Telnet escape sequence to a character sequence other than Ctrl-] (right square bracket). Example: |
The example shows how to set Ctrl-\ as the escape key sequence in Microsoft Telnet.
|
||
|
Step 5 |
Press Enter one or more times to return to the Cisco NX-OS login prompt. Example: |
— |
||
|
Step 6 |
Power cycle the device. |
— |
||
|
Step 7 |
Press Ctrl-C to access the loader> prompt. Example: |
— |
||
|
Step 8 |
loader> cmdline recoverymode=1 Example: |
Enters recovery mode. |
||
|
Step 9 |
loader> boot n9000-dk9.x.x.x.bin Example: |
Restarts the device with the nx-os image to reach the switch(boot)# prompt. |
||
|
Step 10 |
Press Enter one or more times to return to the Cisco NX-OS login prompt. Example: |
— |
||
|
Step 11 |
switch(boot)# config terminal Example: |
Enters boot configuration mode. |
||
|
Step 12 |
switch(boot)(config)# admin-password new-password Example: |
Resets the network administrator password.
|
||
|
Step 13 |
switch(boot)(config)# exit Example: |
Exits boot configuration mode. |
||
|
Step 14 |
switch(boot)# load-nxos Example: |
Loads the nx-os image. You must enter the load-nxos command exactly as shown. Do not enter the image filename with this command. |
||
|
Step 15 |
Log into the device using the new administrator password. Example: |
|
||
|
Step 16 |
switch# config terminal Example: |
Enters global configuration mode. |
||
|
Step 17 |
switch(config)# username admin password new-password Example: |
Resets the new password to ensure that it is also the Simple Network Management Protocol (SNMP) password. |
||
|
Step 18 |
switch(config)# exit Example: |
Exits global configuration mode. |
||
|
Step 19 |
Insert the previously removed standby supervisor module into the chassis, if necessary. |
— |
||
|
Step 20 |
Boot the nx-os image on the standby supervisor module, if necessary. |
— |
||
|
Step 21 |
switch(config)# copy running-config startup-config Example: |
Copies the running configuration to the startup configuration. |
You can reset the network administrator password by reloading the device.
Caution |
This procedure disrupts all traffic on the device. All connections to the device will be lost for 2 to 3 minutes. |
Note |
You cannot recover the administrator password from a Telnet or Secure Shell (SSH) session to the management interface. You must have access to the local console connection. |
Note |
Password recovery updates the new administrator password only in the local user database and not on the remote AAA servers. The new password works only if local authentication is enabled; it does not work for remote authentication. When a password is recovered, local authentication is enabled for logins through a console so that the admin user can log in with a new password from a console. |
| Command or Action | Purpose | |
|---|---|---|
|
Step 1 |
Establish a terminal session on the console port of the active supervisor module. |
— |
|
Step 2 |
switch# reload Example: |
|
|
Step 3 |
loader> boot n9000-dk9.x.x.x.bin Example: |
Restarts the device with only the nx-os image to reach the switch boot prompt. |
|
Step 4 |
Reset the network administrator password by following Steps 6 through 20 in Power Cycling the Device to Recover the Administrator Password. |
— |
You must be logged in as admin to change the network administrator password.
Follow these guidelines and limitations to change an administrator password:
You must be an admin to enable or disable the CLI command, no service password-recovery.
You must be logged in as admin to change the admin password.
You cannot change the admin password from a boot prompt if the CLI was disabled by the admin on a previous boot.
Note |
If you are not logged in as admin, you see an error. |
Feedback