You get a call from a user stating that they have experienced poor voice quality issues with their Cisco IP phone.
Your mind goes numb, your palms get sweaty, and your eyes glaze over. What do you do? Where do you start? It’s so much easier to deal with problems where something stops working completely, rather than trying to resolve an intermittent fault such as bad voice quality.
Why are poor voice quality issues so difficult to tackle?
It’s usually because there is a myriad of different causes, and the methodology to investigate those causes is not always readily or intuitively clear. For voice troubleshooters, it is important to have a battle plan in place to quickly zero in on causes of voice degradation and to diagnose, identify, and resolve them as quickly as possible.
This article will focus on this battle plan so that you can identify the problem quickly, and determine the first troubleshooting steps, which are often the most difficult to decide upon.
Build pro IOS configs. FAST.
Identify the voice quality issue
When you initially hear about the problem, it’s usually described to you verbally by a user who has experienced it. This description can be subjective and may require clarification. Get as much information as you can, and if possible, experience the problem for yourself. If you can reasonably easily recreate the issue and hear the poor voice quality, this can go a long way in identifying the problem. If it’s not possible, get as good a description as you can.
Next, ask questions that will give you an idea of the scope of the problem. Is it something that only a single user is experiencing, a small group of users, or most users? Are the problems observed for all calls, internal calls, inter-site calls, or PSTN calls? The answers to these questions along with experiencing the problem for yourself will give you a good foundational understanding of the problem.
Classify the voice quality issue
Cisco IP Phone Voice quality issues usually fall into one of three categories, and problems for each of the categories may manifest as follows:
Includes problems occurring on voice gateways, CUBE, routers, and switches, involving things like jitter, packet loss, and latency.
Network related voice quality symptoms include:
- Metallic sounding voice
- Intermittent gaps in the voice
- One way voice
- Calls being connected successfully but without any sound in either direction
- Occur only on certain types of calls such as calls to the PSTN, or inter-site calls
- Usually not isolated to a single user, but to a group of users having common network characteristics
Isolated to specific device models or firmware.
Model or firmware related symptoms include:
- Quality degradation manifests itself in the same way only on devices that have the same firmware or are of the same model
Which involves handsets, headsets, and spiral phone cords.
Equipment related voice quality symptoms include:
- White noise or static similar to that heard on AM radio, usually (but not always) taking place when the handset or headset cable is moved or pulled
- Intermittent voice heard on the far end only when using the speakerphone
- Problem is always limited to a single user
By answering most of the questions mentioned before, you may already be able to identify which of the three categories the problem belongs to.
This document can also help you with that task.
Out of the three categories, firmware and equipment related voice quality issues are among the easiest to identify, and their resolution is for the most part quite straightforward. Even so, such voice quality problems are generally rarer. The most common category, and also the most difficult to confront, is the network related voice issues.
If you find that the problem is isolated to a single user, or to a single model of phone, then you should proceed with the troubleshooting process to first determine if it is a firmware or equipment related issue. Here are a few steps to start you off that will help you identify firmware and equipment related voice quality problems.
- Compare the problematic phone with other phones of the same model, running the same firmware version. If the problem is replicated only on phones of the same model and the same firmware, then the next thing you should do is update the firmware. There are several ways to find the firmware version installed on a Cisco IP phone:
- If you have access to the physical phone, press Settings, scroll down and select Status. Scroll down and select Firmware Versions, and there you will see the firmware load version. This process may differ somewhat depending on the phone model.
- Using the Call Manager administration web interface, go to the Phone Configuration page of the device in question. There you will see the Active Load ID as shown below, which is the firmware that is currently running on the device.
- If you have CLI access to the switch to which the phone is connected, you can use CDP to determine the firmware. By issuing the show cdp neighbor <interface> detail command, you can see the model number and the firmware version of the phone, as shown below:
4506_Switch#show cdp neighbor gigabitEthernet 2/17 detail ------------------------- Device ID: SEPA4563041DEFE Entry address(es): IP address: 172.16.5.100 Platform: Cisco IP Phone 7965, Capabilities: Host Phone Two-port Mac Relay Interface: GigabitEthernet2/17, Port ID (outgoing port): Port 1 Holdtime : 138 sec Second Port Status: Up Version : SCCP45.9-4-2SR1-1S advertisement version: 2 Duplex: full Power drawn: 12.000 Watts Power request id: 57086, Power management id: 3 Power request levels are:12000 0 0 0 0 Management address(es):
- Another way to view the firmware load is to go to the IP phone’s web page by navigating to the IP address of the phone itself. You’ll be able to see the firmware version under the Device Information section, similar to the following image. Keep in mind that your display may differ based on the phone model:
- If you’ve ruled out a firmware or model related problem, the next step is to determine if it is an equipment related issue. To do this, determine the following:
- Is the problem the same when using the handset, the speakerphone, or the headset (if there is one)?
- If the problem is isolated to the handset, check the spiral cable connections, and/or swap out the handset and the spiral cable with one from a working phone and test it.
- If the problem is isolated to the headset, check the headset connections, and/or swap out the headset with one from a working phone and test it.
- If the problem is only with the speakerphone, the phone hardware itself may need to be repaired or replaced.
- In some cases, poor voice quality may also be due to a bad patch cable connecting the phone to the wall jack, a bad wall jack or patch panel, or a bad patch cable from the patch panel to the switch. Bad connections can cause packet loss on the physical connection, something that will be examined when looking at the network related issues as well. To determine if this is the problem, follow these steps:
- Take the phone and plug it in directly to the port of the switch with a new patch cable and test it. If the issue disappears, the problem exists on a portion of the cabling from the switch to the workstation location. Test all passive cabling connections.
- If it does not solve the problem, attempt to plug in another working IP phone directly into the port of the switch. If the new phone operates normally, then the problem is most likely a hardware issue with the IP phone.
- If the new phone experiences the same voice degradation, then the problem may be with the hardware of the particular port on the switch.
The above steps do not necessarily have to be taken in this order. The purpose of laying them out in this fashion is to allow you to understand the thought process involved with the troubleshooting procedure. If the problem has not been solved, or if you find that it manifests itself to a whole group of users, then you should proceed to troubleshoot it as a network related voice quality issue.
At this point, you have either ruled out firmware or equipment related issues either because you’ve gone through the above troubleshooting procedure, or you’ve initially identified the issue as network related Cisco IP phone voice quality issue.
Firmware and equipment issues are troubleshot for the most part at the end device itself. Network related voice quality issues may have to be diagnosed at any part of the network where voice packets are switched or routed. This means that you may need to employ the use of Wireshark at particular locations, as well as look at interface statistics of specific network devices. If your network has additional tools, such as Netflow, these may also be very helpful.
These next steps will help you to zero in even more on the problem, and on the particular culprit on the network causing the voice degradation.
Determine the type of problem
Problems with voice quality are usually due to either jitter, latency, or packet loss. All of these can be measured at the phone itself, or at any location on the network.
- On the phone menu, go to Admin Settings > Status > Call Statistics. The menu may differ slightly depending on the phone model. Here you can see information including:
- Avg Jitter – This is the estimated average jitter observed on the RTP stream of packets. Here two values are displayed, the average jitter in milliseconds, and the current audio frame buffer depth in milliseconds. Jitter should typically be less than 30 ms to achieve good voice quality.
- Max Jitter – This is the maximum jitter experienced during this particular call. Jitter may peak at values exceeding 100 ms within the duration of a call, but this may take place for a very small period of time. This value is not as significant unless the Avg Jitter is constantly high.
- Rcvr Discarded – This displays the number of packets that have been discarded due to packets arriving too late (extensive latency), or bad packets. If this value is continually increasing, it is an indication of a problem of latency or packet/frame corruption. If this value is non zero but constant, then there’s not much to worry about.
- Rcvr Lost Packets – This is the number of packets that have been lost in transit. Again, if the value is continually increasing, there is a problem on the network, but a constant non-zero value is not worrisome.
- You will also see several MOS scores that evaluate the quality of the voice. Typical MOS scores for common codecs include:
- G.711 – 4.2
- G.722 – 4.5
- G.729 – 3.91
- To examine these phenomena at other locations along the network, you have several options:
- You can use Wireshark with a SPAN port on the switch where the end device is located or using the SPAN to PC feature on the phone itself. You can then use Wireshark’s telephony tools to isolate particular SIP flows, and thus the specific phone conversations that are experiencing problems. Wireshark will measure the values of jitter, latency, and lost packets.
- You can examine the port statistics on particular ports of a switch or router.
- You can use a network monitoring tool to view collected data on these phenomena.
Determine where the problem takes place
Possibly the most difficult thing to determine is which port or which location of the network should be examined. This depends on the extent of voice degradation and the type of calls that experience it.
- If it is experienced by IP phones in a particular subnet only, check the statistics at the default gateway of that subnet. Once again, this can be done using Wireshark or using established monitoring tools such as Netflow.
- If it is experienced for particular types of calls (i.e. PSTN, or over the WAN), examine the ports over which those calls are routed, such as the voice gateway or the WAN router.
What do I do once the location of the problem is determined?
At this point, you should have detected the latency and/or packet loss and/or extensive jitter that exists on the network, and you should have located where this is taking place. How do you resolve it? Well, it’s important to note that these problems are almost always due to network congestion. The next step will involve taking a look at QoS mechanisms at those points of the network, and examining if they need to be checked, revamped, or, if the problems are severe enough, even if the network design needs to change.
By taking specific steps that will rule out or identify particular causes of poor voice quality issues, you can slowly but surely zero in on the specific problem. As you do this more often, and encounter and successfully deal with such problems, you will get better and faster at it, and much of it will become second nature.
Build pro IOS configs. FAST.