Cisco ONS 15454 Troubleshooting Guide, Release 4.0

General Troubleshooting

Note The terms "Unidirectional Path Switched Ring" and "UPSR" may appear in Cisco literature. These terms do not refer to using Cisco ONS 15xxx products in a unidirectional path switched ring configuration. Rather, these terms, as well as "Path Protected Mesh Network" and "PPMN," refer generally to Cisco's path protection feature, which may be used in any topological network configuration. Cisco does not recommend using its path protection feature in any particular topological network configuration.

This chapter provides procedures for troubleshooting the most common problems encountered when operating a Cisco ONS 15454. To troubleshoot specific ONS 15454 alarms, see Chapter 2, "Alarm Troubleshooting." If you cannot find what you are looking for contact the Cisco Technical Assistance Center (Cisco TAC).

This chapter includes the following sections on network problems:

•Network Troubleshooting Tests—Describes loopbacks and hairpin circuits, which you can use to test circuit paths through the network or logically isolate faults.

Note For network acceptance tests, refer to the Cisco ONS 15454 Procedure Guide.

•Identify Points of Failure on a DS-N Circuit Path—Explains how to perform the tests described in the "Network Troubleshooting Tests" section on a DS-N circuit.

•Using the DS3XM-6 Card FEAC (Loopback) Functions—Describes the Far End Alarm and Control (FEAC) functions on the DS3XM-6 card.

•Identify Points of Failure on an OC-N Circuit Path—Explains how to perform the tests described in the "Network Troubleshooting Tests" section on an OC-N circuit.

The remaining sections describe symptoms, problems, and solutions that are categorized according to the following topics:

•Restoring the Database and Default Settings—Provides procedures for restoring software data and restoring the node to the default setup.

•PC Connectivity Troubleshooting—Provides troubleshooting procedures for PC and network connectivity to the ONS 15454.

•CTC Operation Troubleshooting—Provides troubleshooting procedures for CTC login or operation problems.

•Circuits and Timing—Provides troubleshooting procedures for circuit creation and error reporting as well as timing reference errors and alarms.

•Fiber and Cabling—Provides troubleshooting procedures for fiber and cabling connectivity errors.

•Power and LED Tests—Provides troubleshooting procedures for power supply and LED indicator problems.

1.1 Network Troubleshooting Tests

Use loopbacks and hairpins to test newly created circuits before running live traffic or to logically locate the source of a network failure. All ONS 15454 line (traffic) cards, except E-Series and ML-Series Ethernet cards, allow loopbacks and hairpins.

Caution On OC-N cards, a facility loopback applies to the entire card and not an individual circuit. Exercise caution when using loopbacks on an OC-N card carrying live traffic.

Caution The facility or terminal loopback can be service-affecting. To protect traffic, apply a lockout or force switch to the target loopback port. For more information on these operations, refer to the Cisco ONS 15454 Procedure Guide.

Note Facility loopback is not available for G1000-4 cards.

A facility loopback tests the line interface unit (LIU) of a card, the EIA (electrical interface assembly), and related cabling. After applying a facility loopback on a port, use a test set to run traffic over the loopback. A successful facility loopback isolates the LIU, the EIA, or cabling plant as the potential cause of a network problem. Figure 1-1 shows a facility loopback on a DS-N card.

Figure 1-1 Facility Loopback Process on a DS-N Card

To test the LIU on an OC-N card, connect an optical test set to the OC-N port and perform a facility loopback or use a loopback or hairpin on a card that is farther along the circuit path. Figure 1-2 shows a facility loopback on an OC-N card.

Caution Before performing a facility loopback on an OC-N card, make sure the card contains at least two data communications channel (DCC) paths to the node where the card is installed. A second DCC provides a non looped path to log into the node after the loopback is applied, thus enabling you to remove the facility loopback. Ensuring a second DCC is not necessary if you are directly connected to the ONS 15454 containing the loopback OC-N card.

Figure 1-2 Facility Loopback Process on an OC-N Card

A terminal loopback tests a circuit path as it passes through the cross-connect card (XC, XCVT, or XC10G) and loops back from the card with the loopback. Figure 1-3 shows a terminal loopback on an OC-N card. The test-set traffic comes in on the DS-N card and goes through the cross-connect card to the OC-N card. The terminal loopback on the OC-N card turns the signal around before it reaches the LIU and sends it back through the cross-connect card to the DS-N card. This test verifies that the cross-connect card and terminal circuit paths are valid, but does not test the LIU on the OC-N card.

Setting a terminal loopback on the G-Series card may not stop the Tx Packets counter or the Rx Packet counters on the CTC card-level view Performance > Statistics page from increasing. The counters can increment even though the loopbacked port has temporarily disabled the transmit laser and is dropping any received packets.

The Tx Packet statistic continues to increment because the statistic is not based on the packets transmitted by the Tx laser but on the Tx signal inside the G-Series card. In normal in-service port operation, the Tx signal being recorded does result in the Tx laser transmitting packets, but in a terminal loopback this signal is being looped back within the G-Series card and does not result in the Tx laser transmitting packets.

The Rx Packet counter may also continue to increment when the G-Series card is in terminal loopback. Rx packets from any connected device are dropped and not recorded, but the internally looped back packets follow the G-Series card's normal receive path and register on the Rx Packet counter.

Figure 1-3 The Terminal Loopback Process on an OC-N Card

Figure 1-4 shows a terminal loopback on a DS-N card. The test-set traffic comes in on the OC-N card and goes through the cross-connect card to the DS-N card. The terminal loopback on the DS-N card turns the signal around before it reaches the LIU and sends it back through the cross-connect card to the OC-N card. This test verifies that the cross-connect card and terminal circuit paths are valid, but does not test the LIU on the DS-N card.

Figure 1-4 The Terminal Loopback Process on a DS-N Card

A hairpin circuit brings traffic in and out on a DS-N port rather than sending the traffic onto the OC-N card. A hairpin loops back only the specific STS or VT circuit and does not cause an entire OC-N port to loop back, thus preventing a drop of all traffic on the OC-N port. The hairpin allows you to test a specific STS or VT circuit on nodes running live traffic. Figure 1-5 shows the hairpin circuit process on a DS-N card.

Figure 1-5 The Hairpin Circuit Process on a DS-N Card

A cross-connect loopback tests a circuit path as it passes through the cross-connect card and loops back to the port being tested. Testing and verifying circuit integrity often involves taking down the whole line; however, a cross-connect loopback allows you to create a loopback on any embedded channel at supported payloads at the STS-1 granularity and higher. For example, you can loop back a single STS-1, STS-3c, STS-6c, etc. on an optical facility without interrupting the other STS circuits. Note the following restrictions to a cross-connect loopback:

You can create a cross-connect loopback on all working or protect optical ports unless the protect port is used in a 1+1 protection group and is in working mode. If a terminal or facility loopback exists on a port, you cannot use the cross-connect loopback.

Figure 1-6 shows a cross-connect loopback on an OC-N port.

Figure 1-6 The Cross-connect Loopback Process on an OC-N Card

1.2 Identify Points of Failure on a DS-N Circuit Path

Facility loopbacks, terminal loopbacks, and hairpin circuits are often used to test a circuit path through the network or to logically isolate a fault. Performing a loopback test at each point along the circuit path systematically isolates possible points of failure.

The example in this section tests a DS-N circuit on a two-node, bidirectional line switched ring (BLSR). Using a series of facility loopbacks, terminal loopbacks, and hairpins, the path of the circuit is traced and the possible points of failure are tested and eliminated. A logical progression of five network test procedures apply to this example scenario:

Note The test sequence for your circuits will differ according to the type of circuit and network topology.

1. A facility loopback on the source node DS-N

2. A hairpin on the source node DS-N

3. A terminal loopback on the destination node DS-N

4. A hairpin on the destination node DS-N

5. A facility loopback on the destination DS-N

Note All loopback tests require on-site personnel.

1.2.1 Perform a Facility Loopback on a Source DS-N Port

The facility loopback test is performed on the node source port in the network circuit, in this example, the DS-N port in the source node. Completing a successful facility loopback on this port isolates the cabling, the DS-N card, and the EIA as possible failure points. Figure 1-7 shows an example of a facility loopback on a source DS-N port.

Figure 1-7 A Facility Loopback on a Circuit Source DS-N Port

Caution Performing a loopback on an in-service circuit is service-affecting. To protect traffic, apply a lockout or force switch to the target loopback port. For more information on these operations, refer to the Cisco ONS 15454 Procedure Guide.

Procedure: Create the Facility Loopback on the Source DS-N Port

Step 1 Connect an electrical test set to the port you are testing.

Use appropriate cabling to attach the transmit (Tx) and receive (Rx) terminals of the electrical test set to the EIA connectors or DSx panel for the port you are testing. The Tx and Rx terminals connect to the same port. Adjust the test set accordingly.

Step 2 Use CTC to create the facility loopback on the port being tested:

a. In node view, double-click the card where you will perform the loopback.

b. Click the Maintenance > Loopback tabs.

c. Choose OOS_MT from the State column for the port being tested. If this is a multiport card, select the appropriate row for the port being tested.

d. Choose Facility (Line) from the Loopback Type column for the port being tested. If this is a multiport card, select the appropriate row for the port being tested.

e. Click the Apply button.

f. Click the Yes button in the Confirmation Dialog box.

Note It is normal for a LPBKFACILITY condition to appear during loopback setup. The condition clears when you remove the loopback.

Step 3 Complete the "Test the Facility Loopback Circuit" procedure.

Procedure: Test the Facility Loopback Circuit

Step 1 If the test set is not already sending traffic, send test traffic on the loopback circuit.

Step 2 Examine the traffic received by the test set. Look for errors or any other signal information that the test set is capable of indicating.

Step 3 If the test set indicates a good circuit, no further testing is necessary with the facility loopback.

a. Clear the facility loopback:

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

b. Complete the "Perform a Hairpin on a Source Node Port" procedure.

Step 4 If the test set indicates a faulty circuit, the problem might be a faulty DS-N card, faulty cabling from the DS-N card to the DSx panel or the EIA, or a faulty EIA.

Step 5 Complete the "Test the DS-N Cabling" procedure.

Procedure: Test the DS-N Cabling

Step 1 Replace the suspect cabling (the cables from the test set to the DSx panel or the EIA ports) with a known-good cable.

If a known-good cable is not available, test the suspect cable with a test set. Remove the suspect cable from the DSx panel or the EIA and connect the cable to the transmit (Tx) and receive (Rx) terminals of the test set. Run traffic to determine whether the cable is good or defective.

Step 2 Resend test traffic on the loopback circuit with a cable that is known to be good installed.

Step 3 If the test set indicates a good circuit, the problem was probably the defective cable.

a. Replace the defective cable.

b. Clear the facility loopback:

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

c. Complete the "Perform a Hairpin on a Source Node Port" procedure.

Step 4 If the test set indicates a faulty circuit, the problem might be a faulty card or a faulty EIA.

Step 5 Complete the "Test the DS-N Card" procedure.

Procedure: Test the DS-N Card

Step 1 Replace the suspect card with a known-good card. See the "Physically Replace a Card" procedure on page 2-169 for details.

Caution Removing a card that currently caries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. Consult the Cisco ONs 15454 Procedure Guide for information.

Step 2 Resend test traffic on the loopback circuit with a known-good card installed.

Step 3 If the test set indicates a good circuit, the problem was probably the defective card.

a. Return the defective card to Cisco through the returned materials authorization (RMA) process. Contact the Cisco Technical Assistance Center (Cisco TAC).

b. Replace the faulty card. See the "Physically Replace a Card" procedure on page 2-169 for details.

c. Clear the facility loopback before testing the next segment of the network circuit path.

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

d. Complete the "Perform a Hairpin on a Source Node Port" procedure.

Step 4 If the test set indicates a faulty circuit, the problem might be a faulty EIA.

Step 5 Complete the "Test the EIA" procedure.

Procedure: Test the EIA

Step 1 Remove and reinstall the EIA to ensure a proper seating:

a. Remove the lower backplane cover. Loosen the five screws that secure it to the ONS 15454 and pull it away from the shelf assembly.

b. Loosen the nine perimeter screws that hold the EIA panel in place.

c. Lift the EIA panel by the bottom to remove it from the shelf assembly.

d. Follow the installation procedure for the appropriate EIA. See the "3.6 Replace an Electrical Interface Assembly" section on page 3-17.

Step 2 Resend test traffic on the loopback circuit with known-good cabling, a known-good card, and the reinstalled EIA.

Step 3 If the test set indicates a good circuit, the problem was probably an improperly seated EIA.

a. Clear the facility loopback:

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

b. Proceed to Step 8.

Step 4 If the test set indicates a faulty circuit, the problem is probably a defective EIA.

a. Return the defective EIA to Cisco through the returned materials authorization (RMA) process. Contact the Cisco Technical Assistance Center (Cisco TAC).

b. Replace the faulty EIA. See the "3.6 Replace an Electrical Interface Assembly" section on page 3-17.

Step 5 Resend test traffic on the loopback circuit with known-good cabling, a known-good card, and the replacement EIA.

Step 6 If the test set indicates a faulty circuit, repeat all of the facility loopback procedures.

Step 7 If the test set indicates a good circuit, the problem was probably the defective EIA.

Clear the facility loopback:

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

Step 8 Complete the "Perform a Hairpin on a Source Node Port" procedure.

1.2.2 Perform a Hairpin on a Source Node Port

The hairpin test is performed on the cross-connect card in the network circuit. A hairpin circuit uses the same port for both source and destination. Completing a successful hairpin through the card isolates the possibility that the cross-connect card is the cause of the faulty circuit. Figure 1-8 shows an example of a hairpin loopback on a source node port.

Figure 1-8 Hairpin on a Source Node Port

Note The ONS 15454 does not support simplex operation on the cross-connect card. Two cross-connect cards of the same type must be installed for each node.

Procedure: Create the Hairpin on the Source Node Port

Step 1 Connect an electrical test set to the port you are testing.

a. If you just completed the "Perform a Facility Loopback on a Source DS-N Port" procedure, leave the electrical test set hooked up to the DS-N port in the source node.

b. If you are starting the current procedure without the electrical test set hooked up to the DS-N port, use appropriate cabling to attach the transmit (Tx) and receive (Rx) terminals of the electrical test set to the DSx panel or the EIA connectors for the port you are testing. The Tx and Rx terminals connect to the same port.

c. Adjust the test set accordingly.

Step 2 Use CTC to set up the hairpin on the port being tested:

a. Click the Circuits tab and click the Create button.

b. Give the circuit an easily identifiable name, such as Hairpin1.

c. Set the Circuit Type and Size to the normal preferences.

d. Uncheck the Bidirectional check box and click the Next button.

e. In the Circuit Source dialog box, select the same Node, card Slot, Port, and Type where the test set is connected and click the Next button.

f. In the Circuit Destination dialog box, use the same Node, card Slot, Port, and Type used for the Circuit Source dialog box and click the Finish button.

Step 3 Confirm that the newly created circuit appears on the Circuits tab list as a one-way circuit.

Step 4 Complete the "Test the Hairpin Circuit" procedure.

Procedure: Test the Hairpin Circuit

Step 1 If the test set is not already sending traffic, send test traffic on the loopback circuit.

Step 2 Examine the test traffic received by the test set. Look for errors or any other signal information that the test set is capable of indicating.

Step 3 If the test set indicates a good circuit, no further testing is necessary with the hairpin circuit.

a. Clear the hairpin circuit:

•Click the Circuits tab.

•Choose the hairpin circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits box.

•Confirm that the hairpin circuit is deleted form the Circuits tab list.

b. Complete the "Perform a Terminal Loopback on a Destination DS-N Port" procedure.

Step 4 If the test set indicates a faulty circuit, there might be a problem with the cross-connect card.

Step 5 Complete the "Test the Standby Cross-Connect Card" procedure.

Procedure: Test the Standby Cross-Connect Card

Step 1 Perform a reset on the standby cross-connect card:

a. Determine the standby cross-connect card. On both the physical node and the Cisco Transport Controller (CTC) window, the ACT/STBY LED of the standby cross-connect card is amber and the ACT/STBY LED of the active cross-connect card is green.

b. Position the cursor over the standby cross-connect card.

c. Right-click and choose RESET CARD.

Step 2 Initiate an external switching command (side switch) on the cross-connect cards before retesting the loopback circuit:

Caution Cross-connect side switches are service-affecting. Any live traffic on any card in the node endures a hit of up to 50 ms.

a. Determine the standby cross-connect card. The ACT/STBY LED of the standby cross-connect card is amber and the ACT/STBY LED of the active cross-connect card is green.

b. In the node view, select the Maintenance > Cross-Connect tabs.

c. In the Cross-Connect Cards menu, click the Switch button.

d. Click the Yes button in the Confirm Switch dialog box.

Note After the active cross-connect goes into standby, the original standby slot becomes active. This causes the ACT/STBY LED to become green on the former standby card.

Step 3 Resend test traffic on the loopback circuit.

The test traffic now travels through the alternate cross-connect card.

Step 4 If the test set indicates a faulty circuit, assume the cross-connect card is not causing the problem.

a. Clear the hairpin circuit:

•Click the Circuits tab.

•Choose the hairpin circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

•Confirm that the hairpin circuit is deleted form the Circuits tab list.

b. Complete the "Perform a Terminal Loopback on a Destination DS-N Port" procedure.

Step 5 If the test set indicates a good circuit, the problem might be a defective cross-connect card.

Step 6 To confirm a defective original cross-connect card, complete the "Retest the Original Cross-Connect Card" procedure.

Procedure: Retest the Original Cross-Connect Card

Step 1 Initiate an external switching command (side switch) on the cross-connect cards to make the original cross-connect card the active card.

a. Determine the standby cross-connect card. The ACT/STBY LED of the standby cross-connect card is amber and the ACT/STBY LED of the active cross-connect card is green.

b. In node view, select the Maintenance > Cross-Connect tabs.

c. From the Cross-Connect Cards menu, choose Switch.

d. Click the Yes button in the Confirm Switch dialog box.

Step 2 Resend test traffic on the loopback circuit.

Step 3 If the test set indicates a faulty circuit, the problem is probably the defective card.

a. Return the defective card to Cisco through the returned materials authorization (RMA) process. Contact the Cisco Technical Assistance Center (Cisco TAC.

b. Replace the defective cross-connect card. See Chapter 3, "Replace an In-Service Cross-Connect Card" for details.

c. Clear the hairpin circuit:

•Click the Circuits tab.

•Choose the hairpin circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

•Confirm that the hairpin circuit is deleted form the Circuits tab list.

d. Proceed to the Step 5.

Step 4 If the test set indicates a good circuit, the cross-connect card might have had a temporary problem that was cleared by the side switch.

Clear the hairpin circuit:

•Click the Circuits tab.

•Choose the hairpin circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

•Confirm that the hairpin circuit is deleted form the Circuits tab list.

Step 5 Complete the "Perform a Terminal Loopback on a Destination DS-N Port" procedure.

1.2.3 Perform a Terminal Loopback on a Destination DS-N Port

The terminal loopback test is performed on the node destination port in the circuit, in this example, the DS-N port in the destination node. First, create a bidirectional circuit that starts on the source node DS-N port and loops back on the destination node DS-N port. Then proceed with the terminal loopback test. Completing a successful terminal loopback to a destination node DS-N port verifies that the circuit is good up to the destination DS-N. Figure 1-9 shows an example of a terminal loopback on a destination DS-N port.

Figure 1-9 Terminal Loopback on a Destination DS-N Port

Caution Performing a loopback on an in-service circuit is service-affecting.

Procedure: Create the Terminal Loopback on a Destination DS-N Port

Step 1 Connect an electrical test set to the port you are testing:

a. If you just completed the "Perform a Hairpin on a Source Node Port" procedure, leave the electrical test set hooked up to the DS-N port in the source node.

b. If you are starting the current procedure without the electrical test set hooked up to the DS-N port, use appropriate cabling to attach the transmit (Tx) and receive (Rx) terminals of the electrical test set to the DSx panel or the EIA connectors for the port you are testing. Both transmit (Tx) and receive (Rx) connect to the same port.

c. Adjust the test set accordingly.

Step 2 Use CTC to set up the terminal loopback circuit on the port being tested.

a. Click the Circuits tab and click the Create button.

b. Give the circuit an easily identifiable name, such as "DSNtoDSN."

c. Set Circuit Type and Size to the normal preferences.

d. Leave the Bidirectional check box checked and click the Next button.

e. In the Circuit Source dialog box, fill in the same Node, card Slot, Port, and Type where the test set is connected and click the Next button.

f. In the Circuit Destination dialog box, fill in the destination Node, card Slot, Port, and Type (the DS-N port in the destination node) and click the Finish button.

Step 3 Confirm that the newly created circuit appears on the Circuits tab list as a 2-way circuit.

Note It is normal for a LPBKTERMINAL condition to appear during a loopback setup. The condition clears when you remove the loopback.

Step 4 Create the terminal loopback on the destination port being tested:

a. Go to the node view of the destination node:

•Choose View > Go To Other Node from the menu bar.

•Choose the node from the drop-down list in the Select Node dialog box and click the OK button.

b. In node view, double-click the card that requires the loopback, such as the DS-N card in the destination node.

c. Click the Maintenance > Loopback tabs.

d. Select OOS_MT from the State column. If this is a multiport card, select the row appropriate for the desired port.

e. Select Terminal (Inward) from the Loopback Type column. If this is a multiport card, select the row appropriate for the desired port.

f. Click the Apply button.

g. Click the Yes button in the Confirmation Dialog box.

Step 5 Complete the "Test the Terminal Loopback Circuit on the Destination DS-N Port" procedure.

Procedure: Test the Terminal Loopback Circuit on the Destination DS-N Port

Step 1 If the test set is not already sending traffic, send test traffic on the loopback circuit.

Step 2 Examine the test traffic being received by the test set. Look for errors or any other signal information that the test set is capable of indicating.

Step 3 If the test set indicates a good circuit, no further testing is necessary on the loopback circuit.

a. Clear the terminal loopback:

•Double-click the DS-N card in the destination node with the terminal loopback.

•Click the Maintenance > Loopback tabs.

•Select None from the Loopback Type column for the port being tested.

•Select the appropriate state (IS, OOS, OOS_AINS) in the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

b. Clear the terminal loopback:

•Click the Circuits tab.

•Choose the loopback circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

c. Complete the "Perform a Hairpin on a Destination Node" procedure.

Step 4 If the test set indicates a faulty circuit, the problem might be a faulty card.

Step 5 Complete the "Test the Destination DS-N Card" procedure.

Procedure: Test the Destination DS-N Card

Step 1 Replace the suspect card with a known-good card. See the "Physically Replace a Card" procedure on page 2-169 for details.

Caution Removing a card that currently caries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. Consult the Cisco ONs 15454 Procedure Guide for information.

Step 2 Resend test traffic on the loopback circuit with a known-good card.

Step 3 If the test set indicates a good circuit, the problem was probably the defective card.

a. Return the defective card to Cisco through the returned materials authorization (RMA) process. Contact the Cisco Technical Assistance Center (Cisco TAC).

b. Replace the defective DS-N card. See the "Physically Replace a Card" procedure on page 2-169 for details.

c. Clear the terminal loopback:

•Double-click the DS-N card in the destination node with the terminal loopback.

•Click the Maintenance > Loopback tabs.

•Select None from the Loopback Type column for the port being tested.

•Select the appropriate state (IS, OOS, OOS_AINS) in the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

d. Clear the terminal loopback:

•Click the Circuits tab.

•Choose the loopback circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

Step 4 Complete the "Perform a Hairpin on a Destination Node" procedure.

1.2.4 Perform a Hairpin on a Destination Node

The hairpin test is performed on the cross-connect card in the network circuit. A hairpin circuit uses the same port for both source and destination. Completing a successful hairpin through the card isolates the possibility that the cross-connect card is the cause of the faulty circuit. Figure 1-8 shows an example of a hairpin loopback on a destination node.

Figure 1-10 Hairpin on a Destination Node

Note The ONS 15454 does not support simplex operation on the cross-connect card. Two cross-connect cards of the same type must be installed for each node.

Procedure: Create the Hairpin on the Destination Node

Step 1 Connect an electrical test set to the port you are testing.

Use appropriate cabling to attach the transmit (Tx) and receive (Rx) terminals of the electrical test set to the EIA connectors or DSx panel for the port you are testing. The transmit (Tx) and receive (Rx) terminals connect to the same port. Adjust the test set accordingly.

Step 2 Use CTC to set up the hairpin on the port being tested:

a. Click the Circuits tab and click the Create button.

b. Give the circuit an easily identifiable name, such as Hairpin1.

c. Set the Circuit Type and Size to the normal preferences.

d. Uncheck the Bidirectional check box and click the Next button.

e. In the Circuit Source dialog box, select the same Node, card Slot, Port, and Type where the test set is connected and click the Next button.

f. In the Circuit Destination dialog box, use the same Node, card Slot, Port, and Type used for the Circuit Source dialog box and click the Finish button.

Step 3 Confirm that the newly created circuit appears in the Circuits tab list as a one-way circuit.

Step 4 Complete the "Test the Hairpin Circuit" procedure.

Procedure: Test the Hairpin Circuit

Step 1 If the test set is not already sending traffic, send test traffic on the loopback circuit.

Step 2 Examine the test traffic received by the test set. Look for errors or any other signal information that the test set is capable of indicating.

Step 3 If the test set indicates a good circuit, no further testing is necessary with the hairpin circuit.

a. Clear the hairpin circuit:

•Click the Circuits tab.

•Choose the hairpin circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

•Confirm that the hairpin circuit is deleted form the Circuits tab list.

b. Complete the "Perform a Facility Loopback on a Destination DS-N Port" procedure.

Step 4 If the test set indicates a faulty circuit, there might be a problem with the cross-connect card.

Step 5 Complete the "Test the Standby Cross-Connect Card" procedure.

Procedure: Test the Standby Cross-Connect Card

Step 1 Perform a reset on the standby cross-connect card:

a. Determine the standby cross-connect card. On both the physical node and the CTC window, the ACT/STBY LED of the standby cross-connect card is amber and the ACT/STBY LED of the active cross-connect card is green.

b. Position the cursor over the standby cross-connect card.

c. Right-click and choose RESET CARD.

Step 2 Initiate an external switching command (side switch) on the cross-connect cards before retesting the loopback circuit:

Caution Cross-connect side switches are service-affecting. Any live traffic on any card in the node endures a hit of up to 50 ms.

a. Determine the standby cross-connect card. The ACT/STBY LED of the standby cross-connect card is amber and the ACT/STBY LED of the active cross-connect card is green.

b. In the node view, select the Maintenance > Cross-Connect tabs.

c. In the Cross-Connect Cards menu, click the Switch button.

d. Click the Yes button in the Confirm Switch box.

Note After the active cross-connect goes into standby, the original standby slot becomes active. This causes the ACT/STBY LED to become green on the former standby card.

Step 3 Resend test traffic on the loopback circuit.

The test traffic now travels through the alternate cross-connect card.

Step 4 If the test set indicates a faulty circuit, assume the cross-connect card is not causing the problem.

a. Clear the hairpin circuit:

•Click the Circuits tab.

•Choose the hairpin circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

•Confirm that the hairpin circuit is deleted form the Circuits tab list.

b. Complete the "Perform a Facility Loopback on a Destination DS-N Port" procedure.

Step 5 If the test set indicates a good circuit, the problem might be a defective cross-connect card.

Step 6 To confirm a defective original cross-connect card, complete the "Retest the Original Cross-Connect Card" procedure.

Procedure: Retest the Original Cross-Connect Card

Step 1 Initiate an external switching command (side switch) on the cross-connect cards to make the original cross-connect card the active card.

a. Determine the standby cross-connect card. The ACT/STBY LED of the standby cross-connect card is amber and the ACT/STBY LED of the active cross-connect card is green.

b. In node view, select the Maintenance > Cross-Connect tabs.

c. In the Cross-Connect Cards menu, click the Switch button.

d. Click the Yes button in the Confirm Switch dialog box.

Step 2 Resend test traffic on the loopback circuit.

Step 3 If the test set indicates a faulty circuit, the problem is probably the defective card.

a. Return the defective card to Cisco through the returned materials authorization (RMA) process. Contact the Cisco Technical Assistance Center (Cisco TAC).

b. Replace the defective cross-connect card. See Chapter 3, "Replace an In-Service Cross-Connect Card".

c. Clear the hairpin circuit before testing the next segment of the network circuit path.

•Click the Circuits tab.

•Choose the hairpin circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

d. Proceed to Step 5.

Step 4 If the test set indicates a good circuit, the cross-connect card might have had a temporary problem that was cleared by the side switch.

Clear the hairpin circuit:

•Click the Circuits tab.

•Choose the hairpin circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

Step 5 Complete the "Perform a Facility Loopback on a Destination DS-N Port" procedure.

1.2.5 Perform a Facility Loopback on a Destination DS-N Port

The facility loopback test is performed on the node source port in the circuit, in this example, the destination DS-N port in the destination node. Completing a successful facility loopback on this port isolates the possibility that the destination node cabling, DS-N card, LIU, or EIA is responsible for a faulty circuit. Figure 1-11 shows an example of a facility loopback on a destination DS-N port.

Figure 1-11 Facility Loopback on a Destination DS-N Port

Caution Performing a loopback on an in-service circuit is service-affecting.

Procedure: Create a Facility Loopback Circuit on a Destination DS-N Port

Step 1 Connect an electrical test set to the port you are testing:

a. If you just completed the "Perform a Hairpin on a Destination Node" procedure, leave the electrical test set hooked up to the DS-N port in the destination node.

b. If you are starting the current procedure without the electrical test set hooked up to the DS-N port, use appropriate cabling to attach the transmit (Tx) and receive (Rx) terminals of the electrical test set to the DSx panel or the EIA connectors for the port you are testing. Both transmit (Tx) and receive (Rx) connect to the same port.

c. Adjust the test set accordingly.

Step 2 Use CTC to create the facility loopback on the port being tested:

a. In node view, double-click the card where the loopback will be performed.

b. Click the Maintenance > Loopback tabs.

c. Select Facility (Line) from the Loopback Type column for the port being tested. If this is a multiport card, select the row appropriate for the desired port.

d. Click the Apply button.

e. Click the Yes button in the Confirmation Dialog box.

Note It is normal for a LPBKFACILITY condition to appear during loopback setup. The condition clears when you remove the loopback.

Step 3 Complete the "Test the Facility Loopback Circuit" procedure.

Procedure: Test the Facility Loopback Circuit

Step 1 If the test set is not already sending traffic, send test traffic on the loopback circuit.

Step 2 Examine the test traffic received by the test set. Look for errors or any other signal information that the test set is capable of indicating.

Step 3 If the test set indicates a good circuit, no further testing is necessary with the loopback circuit.

Clear the facility loopback:

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

The entire DS-N circuit path has now passed its comprehensive series of loopback tests. This circuit qualifies to carry live traffic.

Step 4 If the test set indicates a faulty circuit, the problem might be a faulty DS-N card, faulty cabling from the DS-N card to the DSx panel or the EIA, or a faulty EIA.

Step 5 Complete the "Test the DS-N Cabling" procedure.

Procedure: Test the DS-N Cabling

Step 1 Replace the suspect cabling (the cables from the test set to the DSx panel or the EIA ports) with a cable that is known to be good.

•If a cable that is known to be good is not available, test the suspect cable with a test set. Remove the suspect cable from the DSx panel or the EIA and connect the cable to the transmit (Tx) and receive (Rx) terminals of the test set. Run traffic to determine whether the cable is good or defective.

Step 2 Resend test traffic on the loopback circuit with a cable that is known to be good installed.

Step 3 If the test set indicates a good circuit, the problem was probably the defective cable.

a. Replace the defective cable.

b. Clear the facility loopback:

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

The entire DS-N circuit path has now passed its comprehensive series of loopback tests. This circuit qualifies to carry live traffic.

Step 4 If the test set indicates a faulty circuit, the problem might be a faulty card or a faulty EIA.

Step 5 Complete the "Test the DS-N Card" procedure.

Procedure: Test the DS-N Card

Step 1 Replace the suspect card with a known-good card.

Caution Removing a card that currently caries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. Consult the Cisco ONs 15454 Procedure Guide for information.

Step 2 Resend test traffic on the loopback circuit with a known-good card installed.

Step 3 If the test set indicates a good circuit, the problem was probably the defective card.

a. Return the defective card to Cisco through the returned materials authorization (RMA) process. Contact the Cisco Technical Assistance Center (Cisco TAC).

b. Replace the faulty card. See the "Physically Replace a Card" procedure on page 2-169 for details.

c. Clear the facility loopback:

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

The entire DS-N circuit path has now passed its comprehensive series of loopback tests. This circuit qualifies to carry live traffic.

Step 4 If the test set indicates a faulty circuit, the problem might be a faulty EIA.

Step 5 Complete the "Test the EIA" procedure.

Procedure: Test the EIA

Step 1 Remove and reinstall the EIA to ensure a proper seating.

a. Remove the lower backplane cover, loosen the five screws that secure it to the ONS 15454, and pull it away from the shelf assembly.

b. Loosen the nine perimeter screws that hold the EIA panel in place.

c. Lift the EIA panel by the bottom to remove it from the shelf assembly.

d. Follow the installation procedure for the appropriate EIA. See the "3.6 Replace an Electrical Interface Assembly" section on page 3-17 for details.

Step 2 Resend test traffic on the loopback circuit with known-good cabling, a known-good card, and the reinstalled EIA.

Step 3 If the test set indicates a good circuit, the problem was probably an improperly seated EIA.

Clear the facility loopback:

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

The entire DS-N circuit path has now passed its comprehensive series of loopback tests. This circuit qualifies to carry live traffic.

Step 4 If the test set indicates a faulty circuit, the problem is probably the defective EIA.

a. Return the defective EIA to Cisco through the returned materials authorization (RMA) process. Contact the Cisco Technical Assistance Center (Cisco TAC) at 1-800-553-2447 or obtain a directory of toll-free Cisco TAC telephone numbers at the following URL: http://www.cisco.com/warp/public/687/Directory/DirTAC.shtml

b. Replace the faulty EIA. See Chapter 3, "Replace an Electrical Interface Assembly" for details.

Step 5 Resend test traffic on the loopback circuit with known-good cabling, a known-good card, and the replacement EIA.

Step 6 If the test set indicates a faulty circuit, repeat all of the facility loopback procedures.

If the faulty circuit persists, contact the Cisco Technical Assistance Center (Cisco TAC).

Step 7 If the test set indicates a good circuit, the problem was probably the defective EIA.

Clear the facility loopback:

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

The entire DS-N circuit path has now passed its comprehensive series of loopback tests. This circuit qualifies to carry live traffic.

1.3 Using the DS3XM-6 Card FEAC (Loopback) Functions

The DS3XM-6 card supports Far End Alarm and Control (FEAC) functions that are not available on basic DS-3 cards. Click the Maintenance tab at the DS3XM-6 card view to reveal the two additional function columns. Figure 1-12 shows the DS3 subtab and the additional Send Code and Inhibit FE Lbk function columns.

Figure 1-12 Accessing FEAC Functions on the DS3XM-6 Card

The far end in FEAC refers to the piece of equipment that is connected to the DS3XM-6 card and not the far end of a circuit. In Figure 1-13, if a DS3XM-6 (near-end) port is configured to send a Line Loop Code, the code will be sent to the connected test set, not the DS3XM-6 (far-end) port.

Figure 1-13 Diagram of FEAC

1.3.1 FEAC Send Code

The Send Code column on the maintenance tab of a DS3XM-6 port only applies to out-of-service (OOS_MT, OOS_AINS) ports configured for CBIT framing. The column lets a user select No Code (the default) or Line Loop Code. Selecting Line Loop Code inserts a line loop activate FEAC (Far End Alarm and Control) in the CBIT overhead transmitting to the connected facility. This code initiates a loopback from the facility to the ONS 15454. Selecting No Code sends a line-loop-deactivate FEAC code to the connected equipment, which will remove the loopback. You can also insert a FEAC for the 28 individual DS-1 circuits transmuxed into a DS-3 circuit.

1.3.2 FEAC Inhibit Loopback

The DS3XM-6 ports and transmuxed DS-1s initiate loopbacks when they receive FEAC Line Loop codes. If the Inhibit Loopback check box is checked for a DS-3 port, then that port will ignore any received FEAC Line Loop codes and will not loop back. The port can still be put into loopback manually using the Loopback Type column even if the Inhibit Loopback check box is selected. Only DS-3 ports can be configured to inhibit responses to FEAC loopback commands, individual DS-1 ports cannot inhibit their responses.

1.3.3 FEAC Alarms

The node raises a LPBKDS1FEAC-CMD or LPBKDS3FEAC-CMD condition for a DS-1 or DS-3 port if a FEAC loopback code is sent to the far end.

If the ONS 15454 port is in loopback from having received a loopback activate FEAC code, a LPBKDS1FEAC or LPBKDS3FEAC condition occurs. The condition will clear when a loopback deactivate FEAC command is received on that port.

A DS3E card will respond to, and can inhibit, received FEAC DS3 level loopback codes. A DS3E card cannot be configured to send FEAC codes.

1.4 Identify Points of Failure on an OC-N Circuit Path

Facility loopbacks, terminal loopbacks, and cross-connect loopback circuits are often used together to test the circuit path through the network or to logically isolate a fault. Performing a loopback test at each point along the circuit path systematically isolates possible points of failure.

The example in this section tests an OC-N circuit on a three-node, bidirectional line switched ring (BLSR). Using a series of facility loopbacks and terminal loopbacks, the path of the circuit is traced and the possible points of failure are tested and eliminated. A logical progression of seven network test procedures apply to this example scenario:

Note The test sequence for your circuits will differ according to the type of circuit and network topology.

1. A facility loopback on the source node OC-N port

2. A terminal loopback on the source node OC-N port

3. A cross-connect loopback on the source OC-N port

4. A facility loopback on the intermediate node OC-N port

5. A terminal loopback on the intermediate node OC-N port

6. A facility loopback on the destination node OC-N port

7. A terminal loopback on the destination node OC-N port

Note All loopback tests require on-site personnel.

1.4.1 Perform a Facility Loopback on a Source-Node OC-N Port

The facility loopback test is performed on the node source port in the network circuit, in this example, the source OC-N port in the source node. Completing a successful facility loopback on this port isolates the OC-N port as a possible failure point. Figure 1-14 shows an example of a facility loopback on a circuit source OC-N port.

Figure 1-14 Facility Loopback on a Circuit Source OC-N Port

Caution Performing a loopback on an in-service circuit is service-affecting.

Procedure: Create the Facility Loopback on the Source OC-N Port

Step 1 Connect an optical test set to the port you are testing.

Use appropriate cabling to attach the transmit (Tx) and receive (Rx) terminals of the optical test set to the port you are testing. The transmit (Tx) and receive (Rx) terminals connect to the same port. Adjust the test set accordingly.

Step 2 Use CTC to create the facility loopback circuit on the port being tested:

a. In node view, double-click the card where you will perform the loopback.

b. Click the Maintenance > Loopback tabs.

c. Choose OOS_MT from the State column for the port being tested. If this is a multiport card, select the appropriate row for the desired port.

d. Choose Facility (Line) from the Loopback Type column for the port being tested. If this is a multiport card, select the appropriate row for the desired port.

e. Click the Apply button.

f. Click the Yes button in the Confirmation Dialog box.

Note It is normal for a LPBKFACILITY condition to appear during loopback setup. The condition clears when you remove the loopback.

Step 3 Complete the "Test the Facility Loopback Circuit" procedure.

Procedure: Test the Facility Loopback Circuit

Step 1 If the test set is not already sending traffic, send test traffic on the loopback circuit.

Step 2 Examine the traffic received by the test set. Look for errors or any other signal information that the test set is capable of indicating.

Step 3 If the test set indicates a good circuit, no further testing is necessary with the facility loopback.

a. Clear the facility loopback:

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

b. Complete the "Perform a Terminal Loopback on a Source-Node OC-N Port" procedure.

Step 4 If the test set indicates a faulty circuit, the problem might be a faulty OC-N card.

Step 5 Complete the "Test the OC-N Card" procedure.

Procedure: Test the OC-N Card

Step 1 Replace the suspect card with a known-good card. See the "Physically Replace a Card" procedure on page 2-169 for details.

Caution Removing a card that currently caries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. Consult the Cisco ONs 15454 Procedure Guide for information.

Step 2 Resend test traffic on the loopback circuit with a known-good card installed.

Step 3 If the test set indicates a good circuit, the problem was probably the defective card.

a. Return the defective card to Cisco through the returned materials authorization (RMA) process. Contact the Cisco Technical Assistance Center (Cisco TAC).

b. Replace the faulty card. See the "Physically Replace a Card" procedure on page 2-169 for details.

c. Clear the facility loopback:

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

Step 4 Complete the "Perform a Terminal Loopback on a Source-Node OC-N Port" procedure.

1.4.2 Perform a Terminal Loopback on a Source-Node OC-N Port

The terminal loopback test is performed on the node destination port in the circuit, in this example, the destination OC-N port in the source node. First, create a bidirectional circuit that starts on the node source OC-N port and loops back on the node destination OC-N port. Then proceed with the terminal loopback test. Completing a successful terminal loopback to anode destination OC-N port verifies that the circuit is good up to the destination OC-N. Figure 1-15 shows an example of a terminal loopback on a destination OC-N port.

Figure 1-15 Terminal Loopback on a Source-Node OC-N Port

Caution Performing a loopback on an in-service circuit is service-affecting.

Procedure: Create the Terminal Loopback on a Source Node OC-N Port

Step 1 Connect an optical test set to the port you are testing:

a. If you just completed the "Perform a Facility Loopback on a Source-Node OC-N Port" section, leave the optical test set hooked up to the OC-N port in the source node.

b. If you are starting the current procedure without the optical test set hooked up to the OC-N port, use appropriate cabling to attach the transmit (Tx) and receive (Rx) terminals of the optical test set to the port you are testing. Both transmit (Tx) and receive (Rx) connect to the same port.

c. Adjust the test set accordingly.

Step 2 Use CTC to set up the terminal loopback circuit on the port being tested.

a. Click the Circuits tab and click the Create button.

b. Give the circuit an easily identifiable name, such as "OCN1toOCN2."

c. Set Circuit Type and Size to the normal preferences.

d. Leave the Bidirectional check box checked and click the Next button.

e. In the Circuit Source dialog box, fill in the same Node, card Slot, Port, and Type where the test set is connected and click the Next button.

f. In the Circuit Destination dialog box, fill in the destination Node, card Slot, Port, and Type (the OC-N port in the source node) and click the Finish button.

Step 3 Confirm that the newly created circuit appears on the Circuits tab list as a 2-way circuit.

Note It is normal for a LPBKTERMINAL condition to appear during a loopback setup. The condition clears when you remove the loopback.

Step 4 Create the terminal loopback on the destination port being tested:

a. In node view, double-click the card that requires the loopback, such as the destination OC-N card in the source node.

b. Click the Maintenance > Loopback tabs.

c. Select OOS_MT from the State column. If this is a multiport card, select the row appropriate for the desired port.

d. Select Terminal (Inward) from the Loopback Type column. If this is a multiport card, select the row appropriate for the desired port.

e. Click the Apply button.

f. Click the Yes button in the Confirmation Dialog box.

Step 5 Complete the "Test the Terminal Loopback Circuit" procedure.

Procedure: Test the Terminal Loopback Circuit

Step 1 If the test set is not already sending traffic, send test traffic on the loopback circuit.

Step 2 Examine the test traffic being received by the test set. Look for errors or any other signal information that the test set is capable of indicating.

Step 3 If the test set indicates a good circuit, no further testing is necessary on the loopback circuit.

a. Clear the terminal loopback:

•Double-click the OC-N card in the source node with the terminal loopback.

•Click the Maintenance > Loopback tabs.

•Select None from the Loopback Type column for the port being tested.

•Select the appropriate state (IS, OOS, OOS_AINS) in the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

b. Clear the terminal loopback circuit:

•Click the Circuits tab.

•Choose the loopback circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

c. Complete the "Perform a Facility Loopback on an Intermediate-Node OC-N Port" procedure.

Step 4 If the test set indicates a faulty circuit, the problem might be a faulty card.

Step 5 Complete the "Test the OC-N card" procedure.

Procedure: Test the OC-N card

Step 1 Replace the suspect card with a known-good card. See the "Physically Replace a Card" procedure on page 2-169 for details.

Caution Removing a card that currently caries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. Consult the Cisco ONs 15454 Procedure Guide for information.

Step 2 Resend test traffic on the loopback circuit with a known-good card.

Step 3 If the test set indicates a good circuit, the problem was probably the defective card.

a. Return the defective card to Cisco through the returned materials authorization (RMA) process. Contact the Cisco Technical Assistance Center (Cisco TAC).

b. Replace the defective OC-N card. See the "Physically Replace a Card" procedure on page 2-169 for details.

c. Clear the terminal loopback before testing the next segment of the network circuit path.

•Double-click the OC-N card in the source node with the terminal loopback.

•Click the Maintenance > Loopback tabs.

•Select None from the Loopback Type column for the port being tested.

•Select the appropriate state (IS, OOS, OOS_AINS) in the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

d. Clear the terminal loopback circuit before testing the next segment of the network circuit path.

•Click the Circuits tab.

•Choose the loopback circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

Step 4 Complete the "Create the XC Loopback on the Source OC-N Port" procedure.

1.4.3 Create the XC Loopback on the Source OC-N Port

The XC loopback test occurs on the cross-connect card in a network circuit. An XC loopback circuit uses the same port for both source and destination. Completing a successful XC loopback through the cross-connect card isolates the possibility that the cross-connect card is the cause of the faulty circuit. Figure 1-16 shows an example of an XC loopback on a source OC-N port.

Figure 1-16 XC Loopback on a Source OC-N Port

Note An XC loopback breaks down an existing circuit path and creates a new cross-connect circuit (a hairpin), while the source of the original path is set to inject a line-side AIS-P condition. For instance, if you create an XC loopback from node A to node C through an intermediate node (B), the connection between A and B will be broken. Node C will be left with a one-way hairpin circuit back to A, and node A will inject an AIS-P signal. For more information about this condition, see the "2.6.3 AIS-P" section on page 2-22.

Step 1 Connect an optical test set to the port you are testing.

Note Refer to the manufacturer's instructions for detailed information on connection and setup of the optical test set.

a. If you just completed the "Perform a Terminal Loopback on a Source-Node OC-N Port" section, leave the optical test set hooked up to the OC-N port in the source node.

b. If you are starting the current procedure without the optical test set hooked up to the OC-N port, use appropriate cabling to attach the transmit (Tx) and receive (Rx) terminals of the optical test set to the port you are testing. The transmit (Tx) and receive (Rx) terminals connect to the same port.

c. Adjust the test set accordingly.

Step 2 Use CTC to put the circuit being tested out of service:

a. In node view, double-click the card where the test set is connected. The card view appears.

b. In card view, click the Provisioning > Line tabs.

c. Choose OOS (Out of Service) or OOS_MT (Out of Service Maintenance) from the Status column for the port being tested.

d. Click Apply.

e. Click Yes in the confirmation dialog.

Step 3 Use CTC to set up the XC loopback on the circuit being tested:

a. In card view, click the Provisioning > SONET STS tabs.

b. Click the check box in the XC Loopback column for the port being tested.

c. Click Apply.

d. Click Yes in the confirmation dialog.

Step 4 Complete the "Test the XC Loopback Circuit" procedure.

Procedure: Test the XC Loopback Circuit

Step 1 If the test set is not already sending traffic, send test traffic on the loopback circuit.

Step 2 Examine the test traffic received by the test set. Look for errors or any other signal information that the test set is capable of indicating.

Step 3 If the test set indicates a good circuit, no further testing is necessary with the cross-connect.

a. Clear the XC loopback:

•In card view, click the Provisioning > SONET STS tabs.

•Uncheck the check box in the XC Loopback column for the circuit being tested.

•Click Apply.

•Click Yes in the confirmation dialog.

b. Complete the "Perform a Facility Loopback on an Intermediate-Node OC-N Port" procedure.

Step 4 If the test set indicates a faulty circuit, there might be a problem with the cross-connect card.

Step 5 Complete the "Test the Standby Cross-Connect Card" procedure.

Procedure: Test the Standby Cross-Connect Card

Step 1 Perform a reset on the standby cross-connect card:

a. Determine the standby cross-connect card. On both the physical node and the CTC window, verify that the active cross-connect card displays a green ACT LED and the standby cross-connect card displays an amber SBY LED.

b. Position the cursor over the standby cross-connect card.

c. Right-click and choose RESET CARD.

Step 2 Initiate an external switching command (side switch) of the cross-connect cards before retesting the loopback circuit:

Caution Cross-connect side switches are service-affecting. Any live traffic on any card in the node endures a hit of up to 50 ms.

a. Determine the standby cross-connect card. The active cross-connect card displays a green ACT LED and the standby cross-connect card displays an amber SBY LED.

b. In the node view, select the Maintenance > Cross-Connect tabs.

c. In the Cross-Connect Cards menu, click the Switch button.

d. Click the Yes button in the Confirm Switch dialog box.

Note After the active cross-connect goes into standby, the original standby slot becomes active. This causes the ACT LED to become green on the former standby card.

Step 3 Resend test traffic on the loopback circuit.

The test traffic now travels through the alternate cross-connect card.

Step 4 If the test set indicates a faulty circuit, assume the cross-connect card is not causing the problem.

Clear the XC loopback circuit:

•Click the Circuits tab.

•Choose the XC loopback circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

•Confirm that the XC loopback circuit is deleted form the Circuits tab list.

Step 5 If the test set indicates a good circuit, the problem might be a defective cross-connect card.

Step 6 To confirm a defective original cross-connect card, complete the "Retest the Original Cross-Connect Card" procedure.

Procedure: Retest the Original Cross-Connect Card

Step 1 Initiate an external switching command (side switch) on the cross-connect cards to make the original cross-connect card the active card.

a. Determine the standby cross-connect card. The ACT/STBY LED of the standby cross-connect card is amber and the ACT/STBY LED of the active cross-connect card is green.

b. In node view, select the Maintenance > Cross-Connect tabs.

c. In the Cross-Connect Cards menu, click the Switch button.

d. Click the Yes button in the Confirm Switch dialog box.

Step 2 Resend test traffic on the loopback circuit.

Step 3 If the test set indicates a faulty circuit, the problem is probably the defective card.

a. Return the defective card to Cisco through the returned materials authorization (RMA) process. Contact the Cisco Technical Assistance Center (Cisco TAC.

b. Replace the defective cross-connect card. See the "Replace an In-Service Cross-Connect Card" procedure on page 3-1 for details.

c. Clear the XC loopback circuit:

•Click the Circuits tab.

•Choose the XC loopback circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

Step 4 If the test set indicates a good circuit, the cross-connect card might have had a temporary problem that was cleared by the side switch.

Clear the XC loopback circuit:

•Click the Circuits tab.

•Choose the XC loopback circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

1.4.4 Perform a Facility Loopback on an Intermediate-Node OC-N Port

The facility loopback test is performed on the node source port in the network circuit, in this example, the source OC-N port in the intermediate node. Completing a successful facility loopback on this port isolates the OC-N port as a possible failure point. Figure 1-17 shows an example of a facility loopback on a intermediate node circuit source OC-N port.

Figure 1-17 Facility Loopback on an Intermediate-Node OC-N Port

Caution Performing a loopback on an in-service circuit is service-affecting.

Procedure: Create the Facility Loopback on an Intermediate-Node OC-N Port

Step 1 Connect an optical test set to the port you are testing:

a. If you just completed the "Perform a Terminal Loopback on a Source-Node OC-N Port" section, leave the optical test set hooked up to the OC-N port in the source node.

b. If you are starting the current procedure without the optical test set hooked up to the OC-N port, use appropriate cabling to attach the transmit (Tx) and receive (Rx) terminals of the optical test set to the port you are testing. Both transmit (Tx) and receive (Rx) connect to the same port.

c. Adjust the test set accordingly.

Step 2 Use CTC to set up the facility loopback circuit on the port being tested.

a. Click the Circuits tab and click the Create button.

b. Give the circuit an easily identifiable name, such as "OCN1toOCN3."

c. Set Circuit Type and Size to the normal preferences.

d. Leave the Bidirectional check box checked and click the Next button.

e. In the Circuit Source dialog box, fill in the same Node, card Slot, Port, and Type where the test set is connected and click the Next button.

f. In the Circuit Destination dialog box, fill in the destination Node, card Slot, Port, and Type (the OC-N port in the intermediate node) and click the Finish button.

Step 3 Confirm that the newly created circuit appears on the Circuits tab list as a 2-way circuit.

Note It is normal for a LPBKFACILITY condition to appear during a loopback setup. The condition clears when you remove the loopback.

Step 4 Create the facility loopback on the destination port being tested:

a. Go to the node view of the intermediate node:

•Choose View > Go To Other Node from the menu bar.

•Choose the node from the drop-down list in the Select Node dialog box and click the OK button.

b. In node view, double-click the card that requires the loopback, such as the destination OC-N card in the intermediate node.

c. Click the Maintenance > Loopback tabs.

d. Select OOS_MT from the State column. If this is a multiport card, select the row appropriate for the desired port.

e. Select Terminal (Inward) from the Loopback Type column. If this is a multiport card, select the row appropriate for the desired port.

f. Click the Apply button.

g. Click the Yes button in the Confirmation Dialog dialog box.

Note It is normal for a LPBKFACILITY condition to appear during loopback setup. The condition clears when you remove the loopback.

Step 5 Complete the "Test the Facility Loopback Circuit" procedure.

Procedure: Test the Facility Loopback Circuit

Step 1 If the test set is not already sending traffic, send test traffic on the loopback circuit.

Step 2 Examine the traffic received by the test set. Look for errors or any other signal information that the test set is capable of indicating.

Step 3 If the test set indicates a good circuit, no further testing is necessary with the facility loopback.

a. Clear the facility loopback:

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the confirmation dialog box.

b. Clear the facility loopback circuit:

•Click the Circuits tab.

•Choose the loopback circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

c. Complete the "Perform a Terminal Loopback on an Intermediate-Node OC-N Port" procedure.

Step 4 If the test set indicates a faulty circuit, the problem might be a faulty OC-N card.

Step 5 Complete the "Test the OC-N Card" procedure.

Procedure: Test the OC-N Card

Step 1 Replace the suspect card with a known-good card. See the "Physically Replace a Card" procedure on page 2-169 for details.

Caution Removing a card that currently caries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. Consult the Cisco ONs 15454 Procedure Guide for information.

Step 2 Resend test traffic on the loopback circuit with a known-good card installed.

Step 3 If the test set indicates a good circuit, the problem was probably the defective card.

a. Return the defective card to Cisco through the returned materials authorization (RMA) process. Contact the Cisco Technical Assistance Center (Cisco TAC).

b. Replace the faulty card. See the "Physically Replace a Card" procedure on page 2-169 for details.

c. Clear the facility loopback:

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

d. Clear the facility loopback circuit:

•Click the Circuits tab.

•Choose the loopback circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

Step 4 Complete the "Perform a Terminal Loopback on an Intermediate-Node OC-N Port" procedure.

1.4.5 Perform a Terminal Loopback on an Intermediate-Node OC-N Port

The terminal loopback test is performed on the node destination port in the circuit, in this example, the destination OC-N port in the intermediate node. First, create a bidirectional circuit that starts on the node source OC-N port and loops back on the node destination OC-N port. Then proceed with the terminal loopback test. Completing a successful terminal loopback to anode destination OC-N port verifies that the circuit is good up to the destination OC-N. Figure 1-18 shows an example of a terminal loopback on an intermediate node destination OC-N port.

Figure 1-18 Terminal Loopback on an Intermediate-Node OC-N Port

Caution Performing a loopback on an in-service circuit is service-affecting.

Procedure: Create the Terminal Loopback on an Intermediate-Node OC-N Port

Step 1 Connect an optical test set to the port you are testing:

a. If you just completed the "Perform a Facility Loopback on an Intermediate-Node OC-N Port" section, leave the optical test set hooked up to the OC-N port in the source node.

b. If you are starting the current procedure without the optical test set hooked up to the OC-N port, use appropriate cabling to attach the transmit (Tx) and receive (Rx) terminals of the optical test set to the port you are testing. Both transmit (Tx) and receive (Rx) connect to the same port.

c. Adjust the test set accordingly.

Step 2 Use CTC to set up the terminal loopback circuit on the port being tested.

a. Click the Circuits tab and click the Create button.

b. Give the circuit an easily identifiable name, such as "OCN1toOCN4."

c. Set Circuit Type and Size to the normal preferences.

d. Leave the Bidirectional check box checked and click the Next button.

e. In the Circuit Source dialog box, fill in the same Node, card Slot, Port, and Type where the test set is connected and click the Next button.

f. In the Circuit Destination dialog box, fill in the destination Node, card Slot, Port, and Type (the OC-N port in the intermediate node) and click the Finish button.

Step 3 Confirm that the newly created circuit appears on the Circuits tab list as a 2-way circuit.

Note It is normal for a LPBKTERMINAL condition to appear during a loopback setup. The condition clears when you remove the loopback.

Step 4 Create the terminal loopback on the destination port being tested:

a. Go to the node view of the intermediate node:

•Choose View > Go To Other Node from the menu bar.

•Choose the node from the drop-down list in the Select Node dialog box and click the OK button.

b. In node view, double-click the card that requires the loopback, such as the destination OC-N card in the intermediate node.

c. Click the Maintenance > Loopback tabs.

d. Select OOS_MT from the State column. If this is a multiport card, select the row appropriate for the desired port.

e. Select Terminal (Inward) from the Loopback Type column. If this is a multiport card, select the row appropriate for the desired port.

f. Click the Apply button.

g. Click the Yes button in the Confirmation Dialog dialog box.

Step 5 Complete the "Test the Terminal Loopback Circuit" procedure.

Procedure: Test the Terminal Loopback Circuit

Step 1 If the test set is not already sending traffic, send test traffic on the loopback circuit.

Step 2 Examine the test traffic being received by the test set. Look for errors or any other signal information that the test set is capable of indicating.

Step 3 If the test set indicates a good circuit, no further testing is necessary on the loopback circuit.

a. Clear the terminal loopback:

•Double-click the OC-N card in the intermediate node with the terminal loopback.

•Click the Maintenance > Loopback tabs.

•Select None from the Loopback Type column for the port being tested.

•Select the appropriate state (IS, OOS, OOS_AINS) in the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

b. Clear the terminal loopback circuit:

•Click the Circuits tab.

•Choose the loopback circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

c. Complete the "Perform a Facility Loopback on a Destination-Node OC-N Port" procedure.

Step 4 If the test set indicates a faulty circuit, the problem might be a faulty card.

Step 5 Complete the "Test the OC-N card" procedure.

Procedure: Test the OC-N card

Step 1 Replace the suspect card with a known-good card. See the "Physically Replace a Card" procedure on page 2-169 for details.

Caution Removing a card that currently caries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. Consult the Cisco ONs 15454 Procedure Guide for information.

Step 2 Resend test traffic on the loopback circuit with a known-good card.

Step 3 If the test set indicates a good circuit, the problem was probably the defective card.

a. Return the defective card to Cisco through the returned materials authorization (RMA) process. Contact the Cisco Technical Assistance Center (Cisco TAC).

b. Replace the defective OC-N card. See the "Physically Replace a Card" procedure on page 2-169 for details.

c. Clear the terminal loopback:

•Double-click the OC-N card in the source node with the terminal loopback.

•Click the Maintenance > Loopback tabs.

•Select None from the Loopback Type column for the port being tested.

•Select the appropriate state (IS, OOS, OOS_AINS) in the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

d. Clear the terminal loopback circuit:

•Click the Circuits tab.

•Choose the loopback circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

Step 4 Complete the "Perform a Facility Loopback on a Destination-Node OC-N Port" procedure.

1.4.6 Perform a Facility Loopback on a Destination-Node OC-N Port

The facility loopback test is performed on the node source port in the network circuit, in this example, the source OC-N port in the destination node. Completing a successful facility loopback on this port isolates the OC-N port as a possible failure point. Figure 1-19 shows an example of a facility loopback on a destination node circuit source OC-N port.

Figure 1-19 Facility Loopback on a Destination Node OC-N Port

Caution Performing a loopback on an in-service circuit is service-affecting.

Procedure: Create the Facility Loopback on a Destination Node OC-N Port

Step 1 Connect an optical test set to the port you are testing:

a. If you just completed the "Perform a Terminal Loopback on an Intermediate-Node OC-N Port" procedure, leave the optical test set hooked up to the OC-N port in the source node.

b. If you are starting the current procedure without the optical test set hooked up to the OC-N port, use appropriate cabling to attach the transmit (Tx) and receive (Rx) terminals of the optical test set to the port you are testing. Both transmit (Tx) and receive (Rx) connect to the same port.

c. Adjust the test set accordingly.

Step 2 Use CTC to set up the facility loopback circuit on the port being tested.

a. Click the Circuits tab and click the Create button.

b. Give the circuit an easily identifiable name, such as "OCN1toOCN5."

c. Set Circuit Type and Size to the normal preferences.

d. Leave the Bidirectional check box checked and click the Next button.

e. In the Circuit Source dialog box, fill in the same Node, card Slot, Port, and Type where the test set is connected and click the Next button.

f. In the Circuit Destination dialog box, fill in the destination Node, card Slot, Port, and Type (the OC-N port in the destination node) and click the Finish button.

Step 3 Confirm that the newly created circuit appears on the Circuits tab list as a 2-way circuit.

Note It is normal for a LPBKFACILITY condition to appear during a loopback setup. The condition clears when you remove the loopback.

Step 4 Create the facility loopback on the destination port being tested:

a. Go to the node view of the destination node:

•Choose View > Go To Other Node from the menu bar.

•Choose the node from the drop-down list in the Select Node dialog box and click the OK button.

b. In node view, double-click the card that requires the loopback, such as the destination OC-N card in the destination node.

c. Click the Maintenance > Loopback tabs.

d. Select OOS_MT from the State column. If this is a multiport card, select the row appropriate for the desired port.

e. Select Terminal (Inward) from the Loopback Type column. If this is a multiport card, select the row appropriate for the desired port.

f. Click the Apply button.

g. Click the Yes button in the Confirmation Dialog box.

Note It is normal for a LPBKFACILITY condition to appear during loopback setup. The condition clears when you remove the loopback.

Step 5 Complete the "Test the Facility Loopback Circuit" procedure.

Procedure: Test the Facility Loopback Circuit

Step 1 If the test set is not already sending traffic, send test traffic on the loopback circuit.

Step 2 Examine the traffic received by the test set. Look for errors or any other signal information that the test set is capable of indicating.

Step 3 If the test set indicates a good circuit, no further testing is necessary with the facility loopback.

a. Clear the facility loopback:

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the confirmation dialog box.

b. Clear the facility loopback circuit:

•Click the Circuits tab.

•Choose the loopback circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

c. Complete the "Perform a Terminal Loopback on a Destination Node OC-N Port" procedure.

Step 4 If the test set indicates a faulty circuit, the problem might be a faulty OC-N card.

Step 5 Complete the "Test the OC-N Card" procedure.

Procedure: Test the OC-N Card

Step 1 Replace the suspect card with a known-good card. See the "Physically Replace a Card" procedure on page 2-169 for details.

Caution Removing a card that currently caries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. Consult the Cisco ONs 15454 Procedure Guide for information.

Step 2 Resend test traffic on the loopback circuit with a known-good card installed.

Step 3 If the test set indicates a good circuit, the problem was probably the defective card.

a. Return the defective card to Cisco through the returned materials authorization (RMA) process. Contact the Cisco Technical Assistance Center (Cisco TAC).

b. Replace the faulty card. See the "Physically Replace a Card" procedure on page 2-169 for details.

c. Clear the facility loopback:

•Click the Maintenance > Loopback tabs.

•Choose None from the Loopback Type column for the port being tested.

•Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

d. Clear the facility loopback circuit:

•Click the Circuits tab.

•Choose the loopback circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

Step 4 Complete the "Perform a Terminal Loopback on a Destination Node OC-N Port" procedure.

1.4.7 Perform a Terminal Loopback on a Destination Node OC-N Port

The terminal loopback test is performed on the node destination port in the circuit, in this example, the destination OC-N port in the destination node. First, create a bidirectional circuit that starts on the node source OC-N port and loops back on the node destination OC-N port. Then proceed with the terminal loopback test. Completing a successful terminal loopback to a node destination OC-N port verifies that the circuit is good up to the destination OC-N. Figure 1-20 shows an example of a terminal loopback on an intermediate node destination OC-N port.

Figure 1-20 Terminal Loopback on a Destination Node OC-N Port

Caution Performing a loopback on an in-service circuit is service-affecting.

Procedure: Create the Terminal Loopback on a Destination Node OC-N Port

Step 1 Connect an optical test set to the port you are testing:

a. If you just completed the "Perform a Facility Loopback on a Destination-Node OC-N Port" procedure, leave the optical test set hooked up to the OC-N port in the source node.

b. If you are starting the current procedure without the optical test set hooked up to the OC-N port, use appropriate cabling to attach the transmit (Tx) and receive (Rx) terminals of the optical test set to the port you are testing. Both transmit (Tx) and receive (Rx) connect to the same port.

c. Adjust the test set accordingly.

Step 2 Use CTC to set up the terminal loopback circuit on the port being tested.

a. Click the Circuits tab and click the Create button.

b. Give the circuit an easily identifiable name, such as "OCN1toOCN6."

c. Set Circuit Type and Size to the normal preferences.

d. Leave the Bidirectional check box checked and click the Next button.

e. In the Circuit Source dialog box, fill in the same Node, card Slot, Port, and Type where the test set is connected and click the Next button.

f. In the Circuit Destination dialog box, fill in the destination Node, card Slot, Port, and Type (the OC-N port in the destination node) and click the Finish button.

Step 3 Confirm that the newly created circuit appears on the Circuits tab list as a 2-way circuit.

Note It is normal for a LPBKTERMINAL condition to appear during a loopback setup. The condition clears when you remove the loopback.

Step 4 Create the terminal loopback on the destination port being tested:

a. Go to the node view of the destination node:

•Choose View > Go To Other Node from the menu bar.

•Choose the node from the drop-down list in the Select Node dialog box and click the OK button.

b. In node view, double-click the card that requires the loopback, such as the destination OC-N card in the destination node.

c. Click the Maintenance > Loopback tabs.

d. Select OOS_MT from the State column. If this is a multiport card, select the row appropriate for the desired port.

e. Select Terminal (Inward) from the Loopback Type column. If this is a multiport card, select the row appropriate for the desired port.

f. Click the Apply button.

g. Click the Yes button in the Confirmation Dialog dialog box.

Step 5 Complete the "Test the Terminal Loopback Circuit" procedure.

Procedure: Test the Terminal Loopback Circuit

Step 1 If the test set is not already sending traffic, send test traffic on the loopback circuit.

Step 2 Examine the test traffic being received by the test set. Look for errors or any other signal information that the test set is capable of indicating.

Step 3 If the test set indicates a good circuit, no further testing is necessary on the loopback circuit.

a. Clear the terminal loopback:

•Double-click the OC-N card in the intermediate node with the terminal loopback.

•Click the Maintenance > Loopback tabs.

•Select None from the Loopback Type column for the port being tested.

•Select the appropriate state (IS, OOS, OOS_AINS) in the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

b. Clear the terminal loopback circuit:

•Click the Circuits tab.

•Choose the loopback circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

c. The entire OC-N circuit path has now passed its comprehensive series of loopback tests. This circuit qualifies to carry live traffic.

Step 4 If the test set indicates a faulty circuit, the problem might be a faulty card.

Step 5 Complete the "Test the OC-N Card" procedure.

Procedure: Test the OC-N Card

Step 1 Replace the suspect card with a known-good card. See the "Physically Replace a Card" procedure on page 2-169 for details.

Caution Removing a card that currently caries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. Consult the Cisco ONs 15454 Procedure Guide for information.

Step 2 Resend test traffic on the loopback circuit with a known-good card.

Step 3 If the test set indicates a good circuit, the problem was probably the defective card.

a. Return the defective card to Cisco through the returned materials authorization (RMA) process. Contact the Cisco Technical Assistance Center (Cisco TAC).

b. Replace the defective OC-N card. See the "Physically Replace a Card" procedure on page 2-169 for details.

c. Clear the terminal loopback:

•Double-click the OC-N card in the source node with the terminal loopback.

•Click the Maintenance > Loopback tabs.

•Select None from the Loopback Type column for the port being tested.

•Select the appropriate state (IS, OOS, OOS_AINS) in the State column for the port being tested.

•Click the Apply button.

•Click the Yes button in the Confirmation Dialog box.

d. Clear the terminal loopback circuit:

•Click the Circuits tab.

•Choose the loopback circuit being tested.

•Click the Delete button.

•Click the Yes button in the Delete Circuits dialog box.

The entire OC-N circuit path has now passed its comprehensive series of loopback tests. This circuit qualifies to carry live traffic.

1.5 Restoring the Database and Default Settings

This section contains troubleshooting for node operation errors that require restoration of software data or the default node setup.

1.5.1 Restore the Node Database

Symptom: One or more node(s) are not functioning properly or have incorrect data.

Table 1-1 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Restore the Database

Note The following parameters are not backed up and restored: node name, IP address, mask and gateway, and IIOP port. If you change the node name and then restore a backed up database with a different node name, the circuits will map to the new renamed node. Cisco recommends keeping a record of the old and new node names.

Caution E1000-2 cards lose traffic for approximately 90 seconds when an ONS 15454 database is restored. Traffic is lost during the period of spanning-tree reconvergence. The CARLOSS alarm appears and clears during this period.

Caution If you are restoring the database on multiple nodes, wait until the TCC+/TCC2 reboot has completed on each node before proceeding to the next node.

Step 1 Log into the node where you will restore the database.

a. On the PC connected to the ONS 15454, start Netscape or Internet Explorer.

b. In the Netscape or Internet Explorer Web address (URL) field, enter the ONS 15454 IP address.

A Java Console window displays the CTC file download status. The web browser displays information about your Java and system environments. If this is the first login, CTC caching messages appear while CTC files are downloaded to your computer. The first time you connect to an ONS 15454, this process can take several minutes. After the download, the CTC Login dialog box appears.

c. In the Login dialog box, type a user name and password (both are case sensitive) and click the Login button. The CTC node view window will appear.

Step 2 Ensure that there are no ring or span (four-fiber only) switch events; for example, ring-switch east or west, and span-switch east or west. In network view, click the Conditions tab and click Retrieve Conditions to view a list of conditions.

Step 3 If there are switch events that need to be cleared, in node (default) view, click the Maintenance > BLSR tabs and view the West Switch and East Switch columns.

a. If there is a switch event (not caused by a line failure), clear the switch by choosing CLEAR from the drop-down menu and click Apply.

b. If there is a switch event caused by the Wait to Restore (WTR) condition, choose LOCKOUT SPAN from the drop-down menu and click Apply. When the LOCKOUT SPAN is applied, choose CLEAR from the drop-down menu and click Apply.

Step 4 In node view, click the Maintenance > Database tabs.

Step 5 Click Restore.

Step 6 Locate the database file stored on the workstation's hard drive or on network storage.

Step 7 Click the database file to highlight it.

Step 8 Click Open. The DB Restore dialog box appears.

Caution Opening a restore file from another node or from an earlier backup might affect traffic on the login node.

Step 9 Click Yes.

The Restore Database dialog box monitors the file transfer.

Step 10 Wait for the file to complete the transfer to the TCC+/TCC2 card.

Step 11 Click OK when the "Lost connection to node, changing to Network View" dialog box appears. Wait for the node to reconnect.

Step 12 If you cleared a switch in Step 3, reapply the switch as needed.

1.5.2 Restore the Node to Factory Configuration

Symptom    A node has both TCC+/TCC2 cards in standby state, and you are unable reset the TCC+/TCC2 cards to make the node functional.

Table 1-2 describes the potential cause(s) of the symptom and the solution(s).

Caution If you are restoring the database on multiple nodes, wait until the TCC+/TCC2 cards have rebooted on each node before proceeding to the next node.

Caution Cisco strongly recommends that you keep different node databases in separate folders. This is because the reinitialization tool will choose the first product-specific software package in the specified directory if you only use the Search Path field. You might accidentally copy an incorrect database if multiple databases are kept in the specified directory.

Note If the software package files and database backup files are located in different directories, complete the Package and Database fields (Figure 1-21).

Note The following parameters are not backed up and restored: node name, IP address, mask and gateway, and IIOP port. If you change the node name and then restore a backed up database with a different node name, the circuits map to the new renamed node. Cisco recommends keeping a record of the old and new node names.

Procedure: Use the Reinitialization Tool to Clear the Database and Upload Software (Windows)

Note The TCC+/TCC2 cards reboot several times during this procedure. Wait until they are completely rebooted before continuing.

Step 1 Insert the system software CD containing the reinit tool (Figure 1-21) into the local craft interface PC drive. If the CTC Installation Wizard opens, click Cancel.

Step 2 To find the recovery tool file, go to Start > Run > Browse and select the CD drive.

Step 3 On the CD drive, go to the CISCO15454 folder and set the Files of Type drop-down menu to All Files.

Step 4 Select the RE-INIT.jar file and click Open to open the reinit tool (Figure 1-21).

Figure 1-21 Reinitialization Tool in Windows

Step 5 If the node you are reinitializing is an end network element (ENE) in a proxy server network, enter the IP address of the gateway network element (GNE) in the GNE IP field. If not, leave it blank.

Step 6 Enter the node name or IP address of the node you are reinitializing in the Node IP field (Figure 1-21).

Step 7 Verify that the Re-Init Database, Upload Package, and Confirm check boxes are checked. If one is not checked, click the check box.

Step 8 In the Search Path field, verify that the path to the CISCO15454 folder on the CD drive is listed.

Caution Cisco strongly recommends that you keep different node databases in separate folders. This is because the reinit tool chooses the first product-specific software package in the specified directory if you use the Search Path field instead of the Package and Database fields. You might accidentally copy an incorrect database if multiple databases are kept in the specified directory.

Caution Before you perform the next step, be sure you are uploading the correct database. You cannot reverse the upload process after you click Yes.

Step 9 Click Go.

Step 10 A confirmation dialog box opens (Figure 1-22). Click Yes.

Step 11 The status bar at the bottom of the window displays Complete when the node has activated the software and uploaded the database.

Note The Complete message only indicates that the TCC+/TCC2 successfully uploaded the database, not that the database restore was successful. The TCC+/TCC2 then tries to restore the database after it reboots.

Step 12 If you are logged into CTC, close the browser window and disconnect the straight-through LAN cable from the RJ-45 (LAN) port on the TCC+/TCC2 or on the hub or switch to which the ONS 15454 is physically connected. Reconnect your straight-through LAN cable to the LAN port and log back into CTC. Refer to the Cisco ONS 15454 Procedure Guide.

Step 13 Manually set the node name and network configuration to site-specific values. Refer to the Cisco ONS 15454 Procedure Guide for information on setting the node name, IP address, mask and gateway, and IIOP port.

Figure 1-22 Confirm NE Restoration

Procedure: Use the Reinitialization Tool to Clear the Database and Upload Software (UNIX)

Note JRE 1.03_02 must also be installed on the computer you use to perform this procedure.

Note The TCC+/TCC2 cards will reboot several times during this procedure. Wait until they are completely rebooted before continuing.

Step 1 Insert the system software CD containing the reinit tool, software, and defaults database into the local craft interface PC drive. If the CTC Installation Wizard opens, click Cancel.

Step 2 To find the recovery tool file, go to the CISCO15454 directory on the CD (usually /cdrom/cdrom0/CISCO15454).

Step 3 If you are using a file explorer, double click the RE-INIT.jar file to open the reinit tool (Figure 1-23). If you are working with a command line interface, run java -jar RE-INIT.jar.

Figure 1-23 Reinitialization Tool in UNIX

Step 4 If the node you are reinitializing is an end network element (ENE) in a proxy server network, enter the IP address of the gateway network element (GNE) in the GNE IP field. If not, leave it blank.

Step 5 Enter the node name or IP address of the node you are reinitializing in the Node IP field (Figure 1-23).

Step 6 Verify that the Re-Init Database, Upload Package, and Confirm check boxes are checked. If any are not checked, click that check box.

Step 7 In the Search Path field, verify that the path to the CISCO15454 folder on the CD drive is listed.

Caution Cisco strongly recommends that you keep different node databases in separate folders. This is because the reinit tool chooses the first product-specific software package in the specified directory if you use the Search Path field instead of the Package and Database fields. You might accidentally copy an incorrect database if multiple databases are kept in the specified directory.

Caution Before you perform the next step, be sure you are uploading the correct database. You cannot reverse the upload process after you click Yes.

Step 8 Click Go.

Step 9 A confirmation dialog box opens (Figure 1-22). Click Yes.

Step 10 The status bar at the bottom of the window displays Complete when the node has activated the software and uploaded the database.

Note The Complete message only indicates that the TCC+/TCC2 successfully uploaded the database, not that the database restore was successful. The TCC+/TCC2 then tries to restore the database after it reboots.

Step 11 If you are logged into CTC, close the browser window and disconnect the straight-through LAN cable from the RJ-45 (LAN) port on the TCC+/TCC2 or on the hub or switch to which the ONS 15454 is physically connected. Reconnect your straight-through LAN cable to the LAN port and log back into CTC. Refer to the Cisco ONS 15454 Procedure Guide.

Step 12 Manually set the node name and network configuration to site-specific values. Refer to the Cisco ONS 15454 Procedure Guide for information on setting the node name, IP address, mask and gateway, and IIOP port.

1.6 PC Connectivity Troubleshooting

This section contains troubleshooting procedures for PC and network connectivity to the ONS 15454.

1.6.1 Unable to Verify the IP Configuration of your PC

Symptom    When connecting your PC to the ONS 15454, you are unable to successfully ping the IP address of your PC to verify the IP configuration.

Table 1-3 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Verify the IP Configuration of Your PC

Step 1 Open a DOS command window by selecting Start > Run from the Start menu.

Step 2 In the Open field, type command and then click the OK button. The DOS command window will appear.

Step 3 At the prompt in the DOS window, type one of the following appropriate commands:

•For Windows 98, NT, and 2000, type ipconfig and press the Enter key.

•For Windows 95, type winipcfg and press the Enter key.

The Windows IP configuration information appears, including the IP address, subnet mask, and the default gateway.

Step 4 At the prompt in the DOS window, type ping followed by the IP address shown in the Windows IP configuration information previously displayed.

Step 5 Press the Enter key to execute the command.

If the DOS window appears multiple (usually four) replies, the IP configuration is working properly.

If you do not receive a reply, your IP configuration might not be properly set. Contact your Network Administrator for instructions to correct the IP configuration of your PC.

1.6.2 Browser Login Does Not Launch Java

Symptom    The message "Loading Java Applet" does not appear and the JRE does not launch during the initial login.

Table 1-4 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Reconfigure the PC Operating System Java Plug-in Control Panel

Step 1 From the Windows start menu, click Settings > Control Panel.

Step 2 If Java Plug-in Control Panel does not appear, the JRE might not be installed on your PC.

a. Run the Cisco ONS 15454 software CD.

b. Open the [CD drive]:\Windows\JRE folder.

c. Double-click the j2re-1_3_1_02-win icon to run the JRE installation wizard.

d. Follow the JRE installation wizard steps.

Step 3 From the Windows start menu, click Settings > Control Panel.

Step 4 In the Java Plug-in Control Panel window, double-click the Java Plug-in 1.3.1_02 icon.

Step 5 Click the Advanced tab on the Java Plug-in Control Panel.

Step 6 From the Java Run Time Environment menu, select JRE 1.3 in C:\ProgramFiles\JavaSoft\JRE\1.3.1_02.

Step 7 Click the Apply button.

Step 8 Close the Java Plug-in Control Panel window.

Procedure: Reconfigure the Browser

Step 1 From the Start Menu, launch your browser application.

Step 2 If you are using Netscape Navigator:

a. On the Netscape Navigator menu bar, click the Edit > Preferences menus.

b. In the Preferences window, click the Advanced > Proxies categories.

c. In the Proxies window, click the Direct connection to the Internet check box and click the OK button.

d. On the Netscape Navigator menu bar, click the Edit > Preferences menus.

e. In the Preferences window, click the Advanced > Cache categories.

f. Confirm that the Disk Cache Folder field shows one of the following paths:

•For Windows 95/98/ME, C:\ProgramFiles\Netscape\Communicator\cache

•For Windows NT/2000, C:\ProgramFiles\Netscape\<username>\Communicator\cache.

g. If the Disk Cache Folder field is not correct, click the Choose Folder button.

h. Navigate to the file listed in Step f, and click the OK button.

i. Click the OK button on the Preferences window and exit the browser.

Step 3 If you are using Internet Explorer:

a. On the Internet Explorer menu bar, click the Tools > Internet Options menus.

b. In the Internet Options window, click the Advanced tab.

c. In the Settings menu, scroll down to Java (Sun) and click the Use Java 2 v1.3.1_02 for <applet> (requires restart) check box.

d. Click the OK button in the Internet Options window and exit the browser.

Step 4 Temporarily disable any virus-scanning software on the computer. See the "Browser Stalls When Downloading CTC JAR Files From TCC+/TCC2" section.

Step 5 Verify that the computer does not have two network interface cards (NICs) installed. If the computer does have two NICs, remove one.

Step 6 Restart the browser and log on to the ONS 15454.

1.6.3 Unable to Verify the NIC Connection on Your PC

Symptom    When connecting your PC to the ONS 15454, you are unable to verify the NIC connection is working properly because the link LED is not illuminated or flashing.

Table 1-5 describes the potential cause(s) of the symptom and the solution(s).

1.6.4 Verify PC Connection to the ONS 15454 (ping)

Symptom    The TCP/IP connection was established and then lost.

Table 1-6 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Ping the ONS 15454

Step 1 Display the command prompt:

a. If you are using a Microsoft Windows operating system, from the Start Menu choose Run, type command prompt in the Open field of the Run dialog box, and click OK.

b. If you are using a Sun Solaris operating system, from the Common Desktop Environment (CDE) click the Personal Application tab and click Terminal.

Step 2 For both the Sun and Microsoft operating systems, at the prompt type:

ping ONS-15454-IP-address

For example:

ping 192.1.0.2.

Step 3 If the workstation has connectivity to the ONS 15454, the ping is successful and displays a reply from the IP address. If the workstation does not have connectivity, a "Request timed out" message appears.

Step 4 If the ping is successful, an active TCP/IP connection exists. Restart CTC.

Step 5 If the ping is not successful, and the workstation connects to the ONS 15454 through a LAN, check that the workstation's IP address is on the same subnet as the ONS node.

Step 6 If the ping is not successful and the workstation connects directly to the ONS 15454, check that the link light on the workstation's NIC is illuminated.

1.6.5 The IP Address of the Node is Unknown

Symptom    The IP address of the node is unknown and you are enable login.

Table 1-7 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Retrieve Unknown Node IP Address

Step 1 Connect your PC directly to the active TTC+/TCC2 card Ethernet port on the faceplate.

Step 2 Start the Sniffer application on your PC.

Step 3 Perform a hardware reset by pulling and reseating the active TCC+/TCC2 card.

Step 4 After the TCC+/TCC2 card completes resetting, it will broadcast its IP address. The Sniffer software on your PC will capture the IP address being broadcast.

1.7 CTC Operation Troubleshooting

This section contains troubleshooting procedures for CTC login or operation problems.

1.7.1 Unable to Launch CTC Help After Removing Netscape

Symptom    After removing Netscape and running CTC using Internet Explorer, the user is unable to launch the CTC Help and receives an "MSIE is not the default browser" error message.

Table 1-8 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Reset Internet Explorer as the Default Browser for CTC

Step 1 Open the Internet Explorer browser.

Step 2 From the menu bar, click Tools > Internet Options. The Internet Options window appears.

Step 3 In the Internet Options window, click the Programs tab.

Step 4 Click the Internet Explorer should check to see whether it is the default browser check box.

Step 5 Click the OK button.

Step 6 Exit any and all open and running CTC and Internet Explorer applications.

Step 7 Launch Internet Explorer and open a new CTC session. You should now be able to access the CTC Help.

1.7.2 Unable to Change Node View to Network View

Symptom    When activating a large, multi node BLSR from Software Release 3.2 to Software Release 3.3, some of the nodes appear grayed out. Logging into the new CTC, the user is unable to change node view to network view on any and all nodes, from any workstation. This is accompanied by an "Exception occurred during event dispatching: java.lang.OutOfMemoryError" in the java window.

Table 1-9 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Reset the CTC_HEAP Environment Variable for Windows

Step 1 Exit any and all open and running CTC and Netscape applications.

Step 2 From the Windows Desktop, right-click on My Computer and choose Properties in the popup menu.

Step 3 In the System Properties window, click the Advanced tab.

Step 4 Click the Environment Variables button to open the Environment Variables window.

Step 5 Click the New button under the User variables field or the System variables field.

Step 6 Type CTC_HEAP in the Variable Name field.

Step 7 Type 256 in the Variable Value field, and then click OK to create the variable.

Step 8 Click OK in the Environment Variables window to accept the changes.

Step 9 Click OK in the System Properties window to accept the changes.

Restart the browser and CTC software.

Procedure: Reset the CTC_HEAP Environment Variable for Solaris

Step 1 From the user shell window, kill any CTC applications.

Step 2 Kill any Netscape applications.

Step 3 In the user shell window, set the environment variable to increase the heap size:

% setenv CTC_HEAP 256 

Restart the browser and CTC software in the same user shell window.

1.7.3 Browser Stalls When Downloading CTC JAR Files From TCC+/TCC2

Symptom    The browser stalls or hangs when downloading a CTC JAR file from the TCC+/TCC2 card.

Table 1-10 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Disable the VirusScan Download Scan

Step 1 From the Windows start menu, choose Programs > Network Associates > VirusScan Console.

Step 2 Double-click the VShield icon listed in the VirusScan Console dialog box.

Step 3 Click the Configure button on the lower part of the Task Properties window.

Step 4 Click the Download Scan icon on the left of the System Scan Properties dialog box.

Step 5 Uncheck the Enable Internet download scanning check box.

Step 6 Click Yes when the warning message appears.

Step 7 Click OK on the System Scan Properties dialog box.

Step 8 Click OK on the Task Properties window.

Step 9 Close the McAfee VirusScan window.

1.7.4 CTC Does Not Launch

Symptom    CTC does not launch, usually an error message appears before the login window appears.

Table 1-11 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Redirect the Netscape Cache to a Valid Directory

Step 1 Launch Netscape.

Step 2 Display the Edit menu.

Step 3 Choose Preferences.

Step 4 Under the Category column on the left side, expand the Advanced category and choose the Cache tab.

Step 5 Change your disk cache folder to point to the cache file location.

The cache file location is usually C:\ProgramFiles\Netscape\Users\yourname\cache. The yourname segment of the file location is often the same as the user name.

1.7.5 Sluggish CTC Operation or Login Problems

Symptom    You experience sluggish CTC operation or have problems logging into CTC.

Table 1-12 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Delete the CTC Cache File Automatically

Caution All running sessions of CTC must be halted before deleting the CTC cache. Deleting CTC cache might cause any CTC running on this system to behave in an unexpected manner.

Step 1 Enter an ONS 15454 IP address into the browser URL field. The initial browser window shows a Delete CTC Cache button.

Step 2 Close all open CTC sessions and browser windows. The PC operating system does not allow you to delete files that are in use.

Step 3 Click the Delete CTC Cache button on the initial browser window to clear the CTC cache. Figure 1-24 shows the Delete CTC Cache window.

Note For CTC releases prior to 3.0, automatic deletion is unavailable. For CTC Cache file Manual deletion, see the "Delete the CTC Cache File Manually" procedure.

Figure 1-24 Deleting the CTC Cache

Procedure: Delete the CTC Cache File Manually

Caution All running sessions of CTC must be halted before deleting the CTC cache. Deleting CTC cache might cause any CTC running on this system to behave in an unexpected manner.

Step 1 To delete the jar files manually, from the Windows Start menu choose Search > For Files or Folders.

Step 2 Enter *.jar in the Search for files or folders named field on the Search Results dialog box and click Search Now.

Step 3 Click the Modified column on the Search Results dialog box to find the jar files that match the date when you downloaded the files from the TCC+/TCC2. These files might include CTC*.jar, CMS*.jar, and jar_cache*.tmp.

Step 4 Highlight the files and press the keyboard Delete key.

Step 5 Click Yes at the Confirm dialog box.

1.7.6 Node Icon is Grey on CTC Network View

Symptom    The CTC network view shows one or more node icons as grey in color and without a node name.

Table 1-13 describes the potential cause(s) of the symptom and the solution(s).

1.7.7 CTC Cannot Launch Due to Applet Security Restrictions

Symptom    The error message "Unable to launch CTC due to applet security restrictions" appears after you enter the IP address in the browser window.

Table 1-14 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Manually Edit the java.policy File

Step 1 Search your computer for this file and open it with a text editor (Notepad or Wordpad).

Step 2 Verify that the end of this file has the following lines:

		// Insert this into the system-wide or a per-user java.policy file.

	// DO NOT OVERWRITE THE SYSTEM-WIDE POLICY FILE--ADD THESE LINES!

	grant codeBase "http://*/fs/LAUNCHER.jar" {

permission java.security.AllPermission;

	};

Step 3 If these five lines are not in the file, enter them manually.

Step 4 Save the file and restart Netscape.

CTC should now start correctly.

Step 5 If the error message is still reported, save the java.policy file as (.java.policy). On Win95/98/2000 PCs, save the file to the C:\Windows folder. On WinNT4.0 PCs, save the file to all of the user folders on that PC, for example, C:\Winnt\profiles\joeuser.

1.7.8 Java Runtime Environment Incompatible

Symptom    The CTC application does not run properly.

Table 1-15 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Launch CTC to Correct the Core Version Build

Step 1 Exit the current CTC session and completely close the browser.

Step 2 Start the browser.

Step 3 Type the ONS 15454 IP address of the node that reported the alarm. This can be the original IP address you logged on with or an IP address other than the original.

Step 4 Log into CTC. The browser downloads the jar file from CTC.

Note After Release 2.2.2, the single CMS.jar file evolved into core and element files. Core files are common to the ONS 15454, ONS 15454 SDH, and ONS 15327, while the element files are unique to the particular product. For example, the ONS 15327 Release 1.0 uses a 2.3 core build and a 1.0 element build. To display the CTC Core Version number, from the CTC menu bar click Help > About CTC. This lists the Core and Element builds discovered on the network.

1.7.9 Different CTC Releases Do Not Recognize Each Other

Symptom    This situation is often accompanied by the INCOMPATIBLE-SW alarm.

Table 1-17 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Launch CTC to Correct the Core Version Build

Step 1 Exit the current CTC session and completely close the browser.

Step 2 Start the browser.

Step 3 Type the ONS 15454 IP address of the node that reported the alarm. This can be the original IP address you logged on with or an IP address other than the original.

Step 4 Log into CTC. The browser will download the jar file from CTC.

Note After Release 2.2.2, the single CMS.jar file evolved into core and element files. Core files are common to the ONS 15454, ONS 15454 SDH, and ONS 15327, while the element files are unique to the particular product. For example, the ONS 15327 Release 1.0 uses a 2.3 core build and a 1.0 element build. To display the CTC Core Version number, from the CTC menu bar click Help > About CTC. This lists the Core and Element builds discovered on the network.

1.7.10 Username or Password Do Not Match

Symptom    A mismatch often occurs concurrently with a NOT-AUTHENTICATED alarm.

Table 1-18 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Verify Correct Username and Password

Step 1 Ensure that your keyboard Caps Lock key is not turned on and affecting the case-sensitive entry of the username and password.

Step 2 Contact your system administrator to verify the username and password.

Step 3 Call Cisco TAC to have them enter your system and create a new user name and password.

1.7.11 No IP Connectivity Exists Between Nodes

Symptom    The nodes have a grey icon and is usually accompanied by alarms.

Table 1-19 describes the potential cause(s) of the symptom and the solution(s).

1.7.12 DCC Connection Lost

Symptom    The node is usually accompanied by alarms and the nodes in the network view have a grey icon. This symptom is usually accompanied by an EOC alarm.

Table 1-20 describes the potential cause(s) of the symptom and the solution(s).

1.7.13 "Path in Use" Error When Creating a Circuit

Symptom    While creating a circuit, you get a "Path in Use" error that prevents you from completing the circuit creation.

Table 1-21 describes the potential cause(s) of the symptom and the solution(s).

1.7.14 Calculate and Design IP Subnets

Symptom    You cannot calculate or design IP subnets on the ONS 15454.

Table 1-22 describes the potential cause(s) of the symptom and the solution(s).

1.7.15 Ethernet Connections

Symptom    Ethernet connections appear to be broken or are not working properly.

Table 1-23 describes the potential cause(s) of the symptom and the solution(s).

Figure 1-25 Ethernet Connectivity Reference

Procedure: Verify Ethernet Connections

Step 1 Verify that the alarm filter is turned OFF.

Step 2 Check for SONET alarms on the STS-N that carries the VLAN #1 Ethernet circuit. Clear any alarms by looking them up in Chapter 2, "Alarm Troubleshooting."

Step 3 Check for Ethernet-specific alarms. Clear any raised alarms by looking up that alarm in Chapter 2, "Alarm Troubleshooting."

Step 4 Verify that the ACT LED on the Ethernet card is green.

Step 5 Verify that Ports 1 and 3 on ONS 15454 #1 and Ports 1 and 2 on ONS 15454 #2 have green link-integrity LEDs illuminated.

Step 6 If no green link-integrity LED is illuminated for any of these ports:

a. Verify physical connectivity between the ONS 15454s and the attached device.

b. Verify that the ports are enabled on the Ethernet cards.

c. Verify that you are using the proper Ethernet cable and that it is wired correctly, or replace the cable with a known-good Ethernet cable.

d. Check the status LED on the Ethernet card faceplate to ensure the card booted up properly. This LED should be steady green. If necessary, remove and reinsert the card and allow it to reboot.

e. It is possible that the Ethernet port is functioning properly but the link LED itself is broken. Run the procedure in the "Lamp Test for Card LEDs" section.

Step 7 Verify connectivity between device A and device C by pinging between these locally attached devices (see the "Verify PC Connection to the ONS 15454 (ping)" section). If the ping is unsuccessful:

a. Verify that device A and device C are on the same IP subnet.

b. Display the Ethernet card in CTC card view and click the Provisioning > VLAN tabs to verify that both Port 1 and Port 3 on the card are assigned to the same VLAN.

c. If a port is not assigned to the correct VLAN, click that port column in the VLAN row and set the port to Tagged or Untag. Click Apply.

Step 8 Repeat Step 7 for devices B and D.

Step 9 Verify that the Ethernet circuit that carries VLAN #1 is provisioned and that ONS 15454 #1 and ONS 15454 #2 ports also use VLAN #1.

1.7.16 VLAN Cannot Connect to Network Device from Untag Port

Symptom    Networks that have a VLAN with one ONS 15454 Ethernet card port set to Tagged and one ONS 15454 Ethernet card set to Untag might have difficulty implementing Address Resolution Protocol (ARP) for a network device attached to the Untag port (Figure 1-26). They might also see a higher than normal runt packets count at the network device attached to the Untag port. This symptom/limitation also exists when ports within the same card or ports within the same chassis are put on the same VLAN, with a mix of tagged and untagged.

Figure 1-26 A VLAN with Ethernet ports at Tagged and Untag

Table 1-24 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Change VLAN Port Tag and Untagged Settings

Step 1 Display the CTC card view for the Ethernet card involved in the problem VLAN.

Step 2 Click the Provisioning > VLAN tabs (Figure 1-27).

Figure 1-27 Configuring VLAN Membership for Individual Ethernet Ports

Step 3 If the port is set to Tagged, continue to look at other cards and their ports in the VLAN until you find the port that is set to Untag.

Step 4 At the VLAN port set to Untag, click the port and choose Tagged.

Note The attached external devices must recognize IEEE 802.1Q VLANs.

Step 5 After each port is in the appropriate VLAN, click Apply.

1.7.17 Cross-Connect Card Oscillator Fails

Symptom: The XC, XCVT, or XC10G card can be affected by this problem. It is indicated by a CTNEQPT-PBPROT or CTNEQPT-PBWORK condition raised against all I/O cards in the node. The following conditions might also be raised on the node:

•SWMTXMOD against one or both cross-connect cards

•SD-L against near-end or far-end line cards

•AIS-L against far-end line cards

•RFI-L against near-end line cards

Table 1-25 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Resolve the XC Oscillator Failure When Slot 8 XC Card is Active

Step 1 If the CTNEQPT-PBPROT condition is reported against all I/O cards in the node and the Slot 8 cross-connect card is active, right-click the Slot 10 cross-connect card.

Step 2 Choose Reset Card, then click OK. (Slot 8 remains active and Slot 10 remains standby.)

Step 3 If the alarm remains, reseat the Slot 10 card.

Step 4 If CTNEQPT-PBPROT does not clear, replace the Slot 10 cross-connect card with a spare card.

Step 5 If CTNEQPT-PBPROT does not clear, replace the spare card placed in Slot 10 with the original cross-connect card.

Step 6 Right-click the Slot 8 card and choose Reset Card.

Step 7 Click OK to activate the Slot 10 card and place the Slot 8 card in standby.

Step 8 If you then see the CTNEQPT-PBWORK condition raised against all I/O cards in the node, verify that CTNEQPT-PBPROT has cleared on all I/O cards. Seeing CTNEQPT-PBWORK on the cards indicates that Slot 8 card has a bad oscillator. If this is indicated, complete the following substeps. Otherwise, go toStep 9.

a. Replace the Slot 8 cross-connect card with a spare card. (Slot 8 remains standby.)

b. Reseat the Slot 10 cross-connect card to activate the Slot 8 card and make Slot 10 standby.

c. Verify that the CTNEQPT-PBWORK condition has cleared on all I/O cards.

Step 9 If you see CTNEQPT-PBPROT reported against all I/O cards in the node, this indicates that the Slot 10 card has a bad oscillator. If so, complete the following steps:

a. Replace the Slot 10 cross-connect card with a spare card. (The Slot 8 card is now active.)

b. Reseat the Slot 8 cross-connect card to make Slot 10 active.

c. Verify that the CTNEQPT-PBPROT condition has cleared on all I/O cards.

Procedure: Resolve the XC Oscillator Failure When Slot 10 XC Card is Active

Step 1 If the CTNEQPT-PBWORK condition is reported against all I/O cards in the node and the Slot 10 card is active, right-click the Slot 8 cross-connect card.

Step 2 Choose Reset Card and click OK. (Slot 10 remains active and Slot 8 remains standby.)

Step 3 If the CTNEQPT-PBWORK condition does not clear, reseat the Slot 8 cross-connect card.

Step 4 If the condition does not clear, replace the Slot 8 cross-connect card with an identical, spare card.

Step 5 If the condition does not clear, replace the spare card placed in Slot 8 with the original cross-connect card.

Step 6 Right-click the Slot 10 cross-connect card.

Step 7 Choose Reset Card and click OK. The Slot 8 cross-connect card becomes active and Slot 10 becomes standby.

Step 8 If you have switched the Slot 8 card to active and continue to see CTNEQPT-PBWORK reported against all I/O cards in the node, this indicates the Slot 8 card has a bad oscillator. If this is indicated, complete the following substeps. If not, go to Step 9.

a. Replace the Slot 8 cross-connect card with a spare card. (The Slot 10 card is made active.)

b. Reseat the Slot 10 cross-connect card to make Slot 8 active.

c. Verify that the CTNEQPT-PBWORK condition has cleared on all I/O cards.

Step 9 If you then see the CTNEQPT-PBPROT condition raised against all I/O cards, verify that CTNEQPT-PBWORK has cleared on the I/O cards. This indicates that Slot 10 has a bad oscillator. If so, complete the following substeps:

a. Replace the Slot 10 cross-connect card with a spare card. (Slot 10 remains standby.)

b. Reseat the Slot 8 cross-connect card to activate the Slot 10 card and make Slot 8 standby.

c. Verify that the CTNEQPT-PBPROT condition has cleared on all I/O cards.

1.8 Circuits and Timing

This section provides solutions to circuit creation and reporting errors, as well as common timing reference errors and alarms.

1.8.1 Circuit Transitions to Partial State

Symptom    An automatic or manual transition of a circuit from one state to another state results in one of the following partial state conditions:

•OOS_PARTIAL  At least one of the connections in the circuit is in OOS state and at least one other connection in the circuit is in IS, OOS_MT, or OOS_AINS state.

•OOS_MT_PARTIAL  At least one connection in the circuit is in OOS_MT state and at least one other connection in the circuit is in IS, OOS_MT, or OOS_AINS state.

•OOS_AINS_PARTIAL  At least one connection in the circuit is in the OOS_AINS state and at least one other connection in the circuit is in IS or OOS_AINS state.

Table 1-26 describes the potential cause(s) of the symptom and the solution(s).

Procedure: View the State of Circuit Nodes

Step 1 Click the Circuits tab.

Step 2 From the Circuits tab list, select the circuit with the *_PARTIAL state condition.

Step 3 Click the Edit button. The Edit Circuit window appears.

Step 4 In the Edit Circuit window, click the State tab.

The State tab window lists the Node, CRS End A, CRS End B, and CRS State for each of the nodes in the circuit.

1.8.2 AIS-V on DS3XM-6 Unused VT Circuits

Symptom    An incomplete circuit path causes an alarm indications signal (AIS).

Table 1-27 describes the potential cause(s) of the symptom and the solution(s).

Procedure: Clear AIS-V on DS3XM-6 Unused VT Circuits

Step 1 Determine the affected port.

Step 2 Record the node ID, slot number, port number, or VT number.

Step 3 Create a unidirectional VT circuit from the affected port back to itself, such as Source node/Slot 2/Port 2/VT 13 cross connected to Source node/Slot 2/Port 2/VT 13.

Step 4 Uncheck the bidirectional check box in the circuit creation window.

Step 5 Give the unidirectional VT circuit an easily recognizable name, such as "delete me."

Step 6 Display the DS3XM-6 card in CTC card view. Click the Maintenance > DS1 tabs.

Step 7 Locate the VT that is reporting the alarm (for example, DS3 #2, DS1 #13).

Step 8 From the Loopback Type list, choose Facility (line) and click Apply.

Step 9 Click Circuits.

Step 10 Find the one-way circuit you created in Step 3. Select the circuit and click Delete.

Step 11 Click Yes in the Delete Confirmation dialog box.

Step 12 Display the DS3XM-6 card in CTC card view. Click Maintenance > DS1.

Step 13 Locate the VT in Facility (line) Loopback.

Step 14 From the Loopback Type list, choose None and then click Apply.

Step 15 Click the Alarm tab and verify that the AIS-V alarms have cleared.

Step 16 Repeat this procedure for all the AIS-V alarms on the DS3XM-6 cards.

1.8.3 Circuit Creation Error with VT1.5 Circuit

Symptom    You might receive an "Error while finishing circuit creation. Unable to provision circuit. Unable to create connection object at <node name>" message when trying to create a VT1.5 circuit in CTC.

Table 1-28 describes the potential cause(s) of the symptom and the solution(s).

1.8.4 Unable to Create Circuit From DS-3 Card to DS3XM-6 Card

Symptom    You cannot create a circuit from a DS-3 card to a DS3XM-6 card.

Table 1-29 describes the potential cause(s) of the symptom and the solution(s).

1.8.5 DS3 Card Does Not Report AIS-P From External Equipment

Symptom    A DS3-12/DS3N-12/DS3-12E/DS3N-12E card does not report STS AIS-P from the external equipment/line side.

Table 1-30 describes the potential cause(s) of the symptom and the solution(s).

1.8.6 OC-3 and DCC Limitations

Symptom    Limitations to OC-3 and DCC usage.

Table 1-31 describes the potential cause(s) of the symptom and the solution(s).

1.8.7 ONS 15454 Switches Timing Reference

Symptom    Timing references switch when one or more problems occur.

Table 1-32 describes the potential cause(s) of the symptom and the solution(s).

1.8.8 Holdover Synchronization Alarm

Symptom    The clock is running at a different frequency than normal and the HLDOVRSYNC alarm appears.

Table 1-33 describes the potential cause(s) of the symptom and the solution(s).

1.8.9 Free-Running Synchronization Mode

Symptom    The clock is running at a different frequency than normal and the FRNGSYNC alarm appears.

Table 1-34 describes the potential cause(s) of the symptom and the solution(s).

1.8.10 Daisy-Chained BITS Not Functioning

Symptom    You are unable to daisy-chain the BITS.

Table 1-35 describes the potential cause(s) of the symptom and the solution(s).

1.8.11 Blinking STAT LED after Installing a Card

Symptom    After installing a card, the STAT LED blinks continuously for more than 60 seconds.

Table 1-36 describes the potential cause(s) of the symptom and the solution(s).

Table 1-36 Blinking STAT LED on installed card 

Possible Problem

Solution

The card cannot boot because it failed the Power On Shelf Test (POST) diagnostics.

The blinking STAT LED indicates that POST diagnostics are being performed. If the LED continues to blink more than 60 seconds, the card has failed the POST diagnostics test and has failed to boot. If the card has truly failed, an EQPT alarm is raised against the slot number with an "Equipment Failure" description. Check the alarm tab for this alarm to appear for the slot where the card was installed. To attempt recovery, remove and reinstall the card and observe the card boot process. If the card fails to boot, replace the card. Caution Removing a card that currently caries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. Consult the Cisco ONs 15454 Procedure Guide for information.

1.9 Fiber and Cabling

This section explains problems typically caused by cabling connectivity errors. It also includes instructions for crimping CAT-5 cable and lists the optical fiber connectivity levels.

1.9.1 Bit Errors Appear for a Traffic Card

Symptom    A traffic card has multiple bit errors.

Table 1-37 describes the potential cause(s) of the symptom and the solution(s).

1.9.2 Faulty Fiber-Optic Connections

Symptom    A line card has multiple SONET alarms and/or signal errors.

Table 1-38 describes the potential cause(s) of the symptom and the solution(s).