How to punch SIP One Way Audio in the face

And why should you punch it in the face?
To be honest, mostly revenge…
Because one-way audio and no audio problems hurt.
Really bad.
Especially when they hit you by surprise, as they usually do.

---

The thing is, I used to think that if we have signaling and the call is up,
the audio part should be obvious. Even more so, once you have audio, it should work in both ways,
I mean, why the hell wouldn’t it, right?
Wrong.

Even more so, once you have audio, it should work in both ways,
I mean, why the hell wouldn’t it, right?
Wrong.

ahhh how young and naive I was,
and how painful was the blow.

let’s stuff some cotton wool into this metaphorically bleeding nose
and understand how audio streams behave.

How RTP streams are born

First of all, you probably know that RTP (Real-Time Transport Protocol) is the protocol responsible to send our voice over the IP network.
But who sends it?
And where to?

I’ll start by mentioning that RTP streams (audio/media streams) take different paths from the signaling sessions.
This is how it (very roughly) works in a Cisco-based UC environment:

You enter the digits of your bestie’s number on your SIP phone and press dial.
The phone sends a SIP request to the CUCM, it’s an all-knowing god.
CUCM then looks for a match in its dial plan database for your bestie’s number.
Once a match is found, CUCM tries to set up a SIP session with the destination phone.

In these SIP signaling sessions, between each phone and the CUCM,
audio parameters are being exchanged.
These parameters include codecs, DTMF types, fax properties etc.
This is done using SIP SDP.
One of these parameters include the destination IP for the media stream
(marked with ‘c=’ in the SDP).

So that in the message your phone sends to the CUCM,
the phone would publish it’s own IP address in the ‘c=’ attribute in the SDP.
Stating, this is where RTP should be sent to, makes sense?
CUCM in its turn, when engaging with your bestie’s phone, will publish your phone’s IP address in the ‘c=’ attribute, because it would want it to send the media directly to your phone.

It works the same for the other direction.
Your bestie’s phone publishes it’s an IP address (in the SDP of the SIP message)
and CUCM sends it over to your phone.
This instructs your phone to send RTP voice packets directly to your bestie’s phone.

Now, sometimes, CUCM can decide not to publish the IP addresses of the phones to each other, but instead, it instructs them to send their RTP streams to itself.
Why?
Good question.
In cases, the call requires MTP or a conference bridge for example.

CUCM can also instruct the phones to send their RTP to a transcoder on a router or an MCU.
So that when the initial SIP session is done,
the phone starts to send RTP with the negotiated codec to the destination IP address and port.

If the process interests you and you want to dig deeper, we go over the process in great detail in the SIP troubleshooting course.

One more thing we have to understand is the nature of RTP.
RTP is an application layer protocol that uses UDP to send audio and video.
When a SIP session (a call) is established, we will be looking at two,
unidirectional, non-related RTP streams.

That is why it’s quite possible that only one of these streams will encounter a problem.
The problem can be one-way audio or no audio at all when both RTP streams can’t find their way.
Each phone will listen and send RTP on different port numbers, as published in the SDP.

Now that you understand the basics of how RTP works,
we can go ahead and start punching one way audio in the face.
Or in more civilized words, troubleshoot one-way audio.

Troubleshooting one-way audio and no audio

The process includes three steps:
1. Find where RTP streams are sent to.
2. Verify whether RTP packets are received.
3. Work through possible issues.

The logic is simple,
we first have to understand where each entity sends it’s RTP stream, if at all.
If it does, we have to verify if these voice packets are received at the other end.
We will then go over the possible issues and understand how to solve them.

Another important thing to notice before we begin.
Although each RTP stream is unidirectional and so-called independent,
in a VoIP call, the direction of the RTP stream is mirrored.
So that if Phone A sends RTP to Phone B, Phone B will send it’s RTP to Phone A.

If we take a conference call, for example, Phone A will send RTP to the Conference Bridge (CFB) and the CFB will send RTP to Phone A.
In a conference call, there will be several parties, each will exchange it’s RTP with the CFB.

Finding RTP stream destination

With SIP one-way voice issues, we are usually looking for a specific audio path that can’t reach its destination, thus,
we first have to realize which stream we are looking for.

If phone A can’t hear phone B, the logic will be to take a look at the source
(phone B) and find where it sends its RTP to.
Goes the same for CUBE or a Voice Gateway.
Next, we should check if the destination of our audio stream receives the voice packets.

Let’s review the methods we can use to find and troubleshoot the RTP stream path.

Phone web interface

If we can’t hear audio that should be coming from a Cisco IP phone, we can easily check where it sends the RTP traffic using its web interface.
Find out your phone’s IP address in the CUCM and open it in a web page.
Once you are there, reproduce the issue and depending on the model,
find the relevant stream.

But wait, that’s not all.
This is also the place where you can actually see if packets are being sent (which they usually will) and much more importantly – whether voice packets are being received.

The application is simple,
If you make a call and you can’t hear anything, check the streaming statistics on your phone to actually verify if you have incoming voice packets.
This is the most accurate way to pinpoint one-way audio and no audio problems,
use it as often as you can.

CDR

Unfortunately, we don’t live in a perfect world.
Sometimes you won’t be able to reproduce the one-way audio issue.
In that case, you can use the built-in CDR in CUCM to find the direction of the RTP stream.

Go to Cisco Unified Serviceability–>Tools–>CDR Analysis and Reporting.
Then, CDR–>Search–>By user/phone number/SIP URL
find your call and click view on the CDR-CMR dump.
This is where you can see the media (RTP) ip address and port number of both the originator of the call and of the destination.

SIP Voice Gateway and CUBE

The commands in this section are full of great info,
with or without reference to one way audio issues,
so feel free to play with them.

Not a very popular command, but very useful nonetheless is the:

show voip rtp connections

Whenever you need help to find the relevant stream by the called and the calling number, use:

show voip rtp connections detail

More then that, in scenarios where the IOS router is the CFB, transcoder or MTP, you’ll be able to see all of the connected parties and their RTP streams.
Pretty cool.

Cool as it is, it might not be enough.
Think of a scenario where an internal caller complains that the PSTN party can’t hear them.

Tricky?
A bit, because you can’t monitor the PSTN network…
What you can do, is to check if voice packets arrive in the Voice GW or CUBE.

To do that, we need to find our CallId on the Voice GW.
You can easily extract it from the ‘show voip rtp connection detail’
or ‘show call active voice brief’ (the first number after the colon).

Now run the command:

show call active voice callid (callid number)

Just remember that this method is used to troubleshoot VOIP call legs,
and not ISDN (PRI) call legs.
Meaning that if CUBE is in use(like in the example above),
you can see the ITSP (Internet Telephony Service Provider)
VoIP call leg statistics as well,
including the number of packets coming in, out and where to and from.

Exactly what the doctor ordered for one way audio problems.

Empty Voice Packets

Stop.
It is so common to blame the network or the FW side for one way audio that we often come predetermined with wrong assumptions.

It is necessary to verify both the direction of the media stream and
whether voice packet actually arrives at their destination.

If the incoming media packets count is zero, it’s a good idea to explore
IP Routing and FW issues as detailed below,
but,
if you can see packets coming in and there is still no audio,
you should be ready to explore other options.

When RTP packets do arrive but contain no audio,
it’s very likely that the call goes through an IOS device.
It might be a conference call terminated on the IOS, a transcoded call or just a call coming through a PRI interface.
In all of these scenarios, a DSP is involved.
The DSP is responsible to decode and encode the voice packets,
so when the DSP isn’t functioning properly you can expect plenty of awkward silence
and late nights rebooting routers and replacing faulty DSPs.

Early-Media issues

But wait,
Before you replace the DSP, in some cases, when you call to the PSTN,
you will hear silence and the call would disconnect after a few seconds.

This can be caused by Early-Media compatibility issues.
To verify that your SIP Trunk has a SIP-Profile with SIP Rel1xx set to
‘Send PRACK if 1xx Contains SDP’.

There were also reports the command

voice rtp send-recv

helped in some cases of silence and one way audio to the PSTN, when there was no ISDN Connect message, so don’t hesitate to try it.

Troubleshooting network related One Way Audio and No Audio

Time to work our magic.
Now that you have verified that packets are not coming in
and you have the source and the destination of the RTP streams,
it’s time to solve the mystery.

There are several reasons for one way audio and no way audio
that are related to the IP network.

IP Routing

Let’s start with the easy part.
Simply put, the problem may very well be that our source
(Phone A at 10.0.0.1) doesn’t have a route to the destination (Phone B at 192.168.1.1).

Now just like RTP streams, routes are independent.
So the fact that A can reach B doesn’t mean that B can reach A.
The easiest way to verify that would be to use Trace Route and Ping to verify connectivity.

How?
simply.
Get to the source’s Default Gateway and ping the destination using the source interface of our source (10.0.0.254 for example).

If you are the networking guy, check the routing table on both the Default Gateways
(of A and B).
If networking isn’t your gig, ask someone to work with you.
As long as you properly verified the RTP stream flow,
this step should be relatively straight forward.

Firewall

Warning,
This is where things might get somewhat complex.
That’s why we are going to simplify them as much as possible.

Firewalls use two methods to allow voice (RTP) traffic.

One method is using an Access List rule to allow RTP.
CUCM by default will negotiate UDP ports 16384 – 32767 for audio.

Do check that these ports are open in each direction, as RTP streams are independent of each other and unidirectional.

Now, since the security guys would rarely be happy to open ~32k ports,
there is another method of dynamically opening specific UDP ports per direction per call.
This is done using SIP Inspection, a.k.a SIP ALG.

The idea is simple.
The FW inspects SIP traffic that’s going through it
and looks in the SDP for the negotiated media IP and RTP port.
The FW then allows the specific RTP streams to go through.

That’s all very smart but what happens if the FW is in the RTP path, but not in the SIP path?
We’ve already established that the signaling (SIP) path is different than the media (RTP) path.
In that case, there is no way the FW would dynamically open the UDP ports for RTP,
as it can’t see the SIP SDP negotiations.

The solution here is to talk to the security guys,
verify the method they are using,
verify that RTP ports are open in each direction,
or when using SIP ALG, that the signaling is actually going through the FW.

NAT

In case you’ve missed the memo,
NAT and VoIP are arch enemies.

The reason for that is the way that RTP is negotiated.

With SIP, IP addresses for the RTP streams are sent in the SDP header, inside the message body.

The way NAT works is that it changes the Network Layer Source IP address (internal IP address) with the NAT device’s (public) IP address.
You can see how useless it is for SIP, because our internal phone’s IP address, or even CUBE’s internal IP address is being published in the SDP header going to the internet.
Needless to say that these internal IP addresses are not routable on the internet.
So while SIP signaling might work, there is no way that we are ever going to receive the audio packets that the ITSP sends to our phone at 10.0.0.1.

VoIP and NAT deserve their own post, but in brief, you have two options:
1. Enable SIP ALG on the FW that connects you to the public network.
What SIP ALG will do is find the internal IP addresses in the SIP headers and replace them with its own IP address, and then proxy the RTP stream in.
2. Configure CUBE to use the public IP address in the SDP with Internet-facing SIP sessions.
This is done using SIP-Profiles which is a broad and advanced topic.

To conclude, VoIP is a unique application that runs over the IP network,
and it is our profession and thus our responsibility to understand how it works and help the networking and security teams to do their job, by providing them specific instructions for configuration and troubleshooting.

I hope this post will help you punch One Way Audio and No Audio in the face.
Let me know when you do in the comments below, it will personally make me feel better.
Thanks in advance.

---