Edit: This post is full of SIP debugs, and i honestly do my best to explain every step, but i still think it should come as an extension to essential understanding of SIP protocol. If you are new to SIP, and even more so if you work with SIP daily, i really recommend you to take the Learn SIP course. SIP is the VOIP standard and it’s here to stay, your new SIP skills will be the butter to your career.
SIP doesn’t usually stress me out anymore and after a dozen of expressway implementation neither does NATed VOIP. But apparently Murphy always keeps another trick up his sleeve.
The topology was pretty straight forward. Cisco CUBE was in a DMZ, behind an Internet facing FW which was doing the NAT for us.
Cisco CUBE: An unknown identity
It started off with a loud squeak, a sign of what’s about to come..
I set up the SIP Trunk from CUCM towards Cisco CUBE and from Cisco CUBE towards ITSP (Internet Telephony Service Provider) and tried to call.
First try, no luck.
Went over my configuration again. All checked out fine.
The first setback came early in the game forcing me to look under the hood.
*I didn’t want to use the customer logs so I replicated the issues in my own environment.
Debug ccsip messages
The outcome was similar to this:
Sent: INVITE sip:firstname.lastname@example.org:5060 SIP/2.0 Via: SIP/2.0/UDP 192.168.1.20:5060;branch=z9hG4bK2C21C6 Remote-Party-ID: "ucpros" <sip:email@example.com>;party=calling;screen=yes;privacy=off From: "ucpros" <sip:firstname.lastname@example.org>;tag=410D9A-182 To: <sip:email@example.com> Date: Wed, 31 Aug 2016 11:33:26 GMT Call-ID: 90F1E4EF-6EA511E6-8015F4F4-11BF0349@192.168.1.20 Supported: timer,resource-priority,replaces,sdp-anat Min-SE: 1800 Cisco-Guid: 3121089664-0000065536-0000000003-0167880896 User-Agent: Cisco-SIPGateway/IOS-15.5.3.S2 Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY, INFO, REGISTER CSeq: 101 INVITE Timestamp: 1472643206 Contact: <sip:firstname.lastname@example.org:5060> Call-Info: <sip:192.168.1.20:5060>;method="NOTIFY;Event=telephone-event;Duration=2000" Expires: 180 Allow-Events: telephone-event Max-Forwards: 68 Session-Expires: 1800 Content-Length: 0 . . . Received: SIP/2.0 604 Unknown Identity Via: SIP/2.0/UDP 192.168.1.20:5060;branch=z9hG4bK2C21C6;received=220.127.116.11 From: "ucpros" <sip:email@example.com>;tag=410D9A-182 To: <sip:firstname.lastname@example.org>;tag=5A36FA5D-57C6C0860003EE14-43C32700 Call-ID: 90F1E4EF-6EA511E6-8015F4F4-11BF0349@192.168.1.20 CSeq: 101 INVITE Server: ser (3.3.0-pre1 (i386/linux)) Warning: 392 192.168.0.85:5060 "Noisy feedback tells: pid=1391 req_src_ip=192.168.0.86 req_src_port=5060 in_uri=sip:email@example.com:5060 out_uri=sip:firstname.lastname@example.org:5060 via_cnt==1" Content-Length: 0
Unknown identity, that’s a new one. I was expecting a digest challenge with a
407 Proxy Authentication Required but it wasn’t my day now was it..
It was about to be a long evening and i had to make it shorter.
I added the Alias command so that writing sip would invoke the command debug ccsip messages.
CUBE(config)#alias exec sip debug ccsip messages
After some brainstorming, googling and testing i was onto it. Apparently, ITSP expects to see your allocated DID range / username in the From field whenever you make a call.
My was showing the internal directory which I had to change to my allocated username.
From: "ucpros" <sip:email@example.com>;tag=410D9A-182 Shows internal extension: wrong From: "ucpros" <sip:firstname.lastname@example.org>;tag=410D9A-182 Shows allocated username: right From: "ucpros" <sip:email@example.com>;tag=410D9A-182 Shows DID number from allocated range: right
Usually you would just have to update the External Phone Number Mask on the CUCM so it would match ITSP’s expectations, but if your case is a bit more complicated you would have to use SIP profiles to get the job done.
You can use the following set of commands to modify the From header
voice class sip-profiles 1 request INVITE sip-header From modify "(<.*:)(.*@)" "\1ucpros@"
This command replaces anything that is before the @ with my ITSP username.
You would then have to attach it the proper outgoing dial-peer.
Dial-peer voice 1 voip Description outgoing to ITSP voice-class sip profiles 1
After a rough start I was hoping for some improvement but success wasn’t giving itself up that easily. Unlike the setbacks, which continued pouring in, in buckets.
I could call now, but that was not enough.
Audio? One way, can’t hear a thing. Not a big surprise when you are behind NAT.
A quick debug ccsip messages revealed that voice packets are sent but none are received.
SIP BYE message indicated that PR=0 (Packets Received).
Received: BYE sip:135601EA-57C6DA3C000A0C9F-43F35700@192.168.1.20:5060;transport=tcp SIP/2.0 Via: SIP/2.0/TCP 192.168.1.10:5060;branch=z9hG4bK2b4048cc72 From: "ucpros" <sip:firstname.lastname@example.org>;tag=193~caf26ba8-507b-44f8-ac22-461d3d689977-18522847 To: <sip:email@example.com>;tag=A5803C-1389 Date: Wed, 31 Aug 2016 13:23:08 GMT Call-ID: firstname.lastname@example.org User-Agent: Cisco-CUCM11.0 Max-Forwards: 70 P-Asserted-Identity: "ucpros" <sip:email@example.com> CSeq: 102 BYE Reason: Q.850;cause=16 Session-ID: 25a321302cb9fe0e3578b2978beaa192;remote=a1fd93dd8b22cbd2ca21f6fe533ab193 P-RTP-Stat: PS=970,OS=155200,PR=0,OR=0,PL=0,JI=0,LA=0,DU=19 Content-Length: 0
Red lights and yellow light bulbs were flooding my brain.
“You did configure the FW to allow SIP ALG, right?” I asked the customer.
“Sure!” he replied.
It was agreed that the customer’s FW would provide NAT support via SIP ALG (Application Layer Gateway).
The point of ALG is to check the Application layer of the packet (the SIP header in our case) and replace any IP addresses (usually LAN addresses) with it’s own address.
Something was missing, the FW wasn’t doing it’s job.
After a few sighs and clicks, the customer asked me to try again.
The second hurdle was behind me, but it was far from being the end of it.
No incoming calls
I could make outbound calls just fine but incoming calls were intermittently failing. I was baffled, I had to see what was going on the Cisco CUBE side.
Debug ccsip messages was there for the rescue once again.
My goal was to first verify that I’m getting the proper invite from ITSP.
I was. And then I wasn’t. When calls were going through I could see the proper invite reach the Cisco CUBE. When they didn’t, Cisco CUBE was desolated. There were no SIP packets coming in.
“How could it be?” I was thinking.
I had to narrow it down, find some consistencies and patterns.
I found that calls from some sources were always reaching Cisco CUBE, while others were never coming through. Who is to blame??
I had to check again. I registered my softphone (i like the free MicroSIP) to the ITSP and tried to call it. Calls were reaching it no matter what was the source.
YES!! It was my fault! A sense of hope flooded my body.
If I am to blame, surely I can fix it.
Quickly, what is the difference between Cisco CUBE and my softphone? Err.. Registration! I have to check my registration status.
CUBE#show sip-ua register status
Flat-line sound was filling my ears. I was so sure about my configuration that I missed the obvious. After all, outgoing calls and even some of the incoming ones were working well..
CUBE#show sip-ua register status Registrar is not configured
I was missing the registrar command under SIP-UA. Outgoing calls were working because I had the authentication command in place which would pass the auth’ challenge from ITSP.
CUBE(config)sip-ua CUBE(config-sip-ua)#credentials username ucpros password 7 0442A4334 realm iptel.org CUBE(config-sip-ua)#registrar dns:iptel.org expires 3600
CUBE#show sip-ua register status Line peer expires(sec) reg survival P-Associ-URI ================================ ========== ============ === ======== ============ ucpros -1 23 yes normal
The end was close, or so I thought.
CUCM: 404 Not Found
I could see the incoming INVITE messages from the ITSP but something had changed and I couldn’t quite point my finger on it. My Jabber wasn’t ringing. Now what??
Poker night was getting further away while a failed cut-off was getting closer by the minute.
I had to figure out what was wrong.
Debug ccsip messages.
Incoming INVITE. Check.
Outgoing INVITE to CUCM. Check.
200 OK from CUCM. Missing. How come?
Went through all of the CUCM incoming call flow. Flawless.
Back to Cisco CUBE. Checking out the SIP messages I’ve noticed something was wrong.
First look, nothing, looks great. The To field in the invite message to CUCM contained the right number.
Sent: INVITE sip:UCPROS-TRUNK@192.168.1.10:5060 SIP/2.0 Via: SIP/2.0/UDP 192.168.1.20:5060;branch=z9hG4bK12C7B7 Remote-Party-ID: <sip:firstname.lastname@example.org>;party=calling;screen=no;privacy=off From: <sip:email@example.com>;tag=BED29A-150B To: <sip:firstname.lastname@example.org> Date: Wed, 31 Aug 2016 18:47:10 GMT Call-ID: 28CC4870-6EE211E6-802DCA05-6CA790D5@192.168.1.20 Supported: 100rel,timer,resource-priority,replaces,sdp-anat Min-SE: 1800 Cisco-Guid: 0684376435-1860309478-2150091269-1822920917 User-Agent: Cisco-SIPGateway/IOS-15.5.3.S2 Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY, INFO, REGISTER CSeq: 101 INVITE Timestamp: 1472669230 Contact: <sip:email@example.com:5060> Call-Info: <sip:192.168.1.20:5060>;method="NOTIFY;Event=telephone-event;Duration=2000" Expires: 180 Allow-Events: telephone-event Max-Forwards: 2 Session-Expires: 1800 Content-Type: application/sdp Content-Disposition: session;handling=required Content-Length: 270 v=0 o=CiscoSystemsSIP-GW-UserAgent 3195 4421 IN IP4 192.168.1.20 s=SIP Call c=IN IP4 192.168.1.20 t=0 0 m=audio 8020 RTP/AVP 0 8 101 c=IN IP4 192.168.1.20 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-15 a=ptime:20
Yet CUCM couldn’t find it.
Received: SIP/2.0 404 Not Found Via: SIP/2.0/UDP 192.168.1.20:5060;branch=z9hG4bK12C7B7 From: <sip:firstname.lastname@example.org>;tag=BED29A-150B To: <sip:email@example.com>;tag=380~caf26ba8-507b-44f8-ac22-461d3d689977-18522858 Date: Wed, 31 Aug 2016 18:47:10 GMT Call-ID: 28CC4870-6EE211E6-802DCA05-6CA790D5@192.168.1.20 CSeq: 101 INVITE Allow-Events: presence Reason: Q.850;cause=1 Server: Cisco-CUCM11.0 Session-ID: 00000000000000000000000000000000;remote=a1fd93dd8b22cbd2ca21f6fe533ab380 Content-Length: 0
I had to take another look, something was amiss.
Right! The SIP request URI, it made no sense.
SIP-REQ-URI is the first row of the SIP header and it was showing a variation of my ITSP username while the To field had the called number. Problem is, CUCM uses the SIP-REQ-URI to route calls!
Timing couldn’t be worse but I knew what I had to do.
SIP profiles is the way to customize SIP headers in Cisco CUBE.
Not the easiest job on the planet but definitely one of the most powerful features Cisco CUBE has to offer.
My first try was to replace the SIP INVITE header.
voice class sip-profiles 2 request INVITE sip-header To copy "sip:(.*)@" u10 request INVITE sip-header SIP-Req-URI modify "UCPROS-TRUNK@(.*)" "\u10@\1"
we would then have to associate the SIP Profile to the incoming dial peer.
voice service voip sip sip-profiles inbound Dial-peer voice 2 voip Description Incoming from ITSP voice-class sip profiles 2 inbound
Basically what I did here was to copy the called number from the To header to the SIP-REQ-URI header instead of my ITSP username.
It was working! I was exuberant. The happiest person within a hundred mile radius!
Until I wasn’t.
Test calls had strange behaviors.
They wouldn’t disconnect, reconnect or properly placed on hold. Basically any mid call feature was misbehaving in one way or another.
My patience was running on fumes now.
I was already late for the first round of bets and I really wanted the cut-off to end smoothly. No loose ends on this one.
I had to see what was happening SIP wise on the Cisco CUBE.
Debug ccsip messages revealed the problem.
Even though I changed the INVITE SIP header to reflect the proper called number to CUCM, all of the other messages that had abnormal SIP-REQ-URI were just blanks for CUCM.
It couldn’t understand BYE or any other SIP message that were coming from the ITSP.
Consequently, this one was pretty straight forward.
All i had to do was to change the INVITE to ANY on my SIP-profile to apply the changes on all of the incoming SIP messages from ITSP.
It would look something like that:
Voice class sip-profiles 2 request ANY sip-header To copy "sip:(.*)@" u10 request ANY sip-header SIP-Req-URI modify "UCPROS-TRUNK@(.*)" "\u10@\1"
Success!! At last! What’s more, I managed to make it to the poker game before the blinds went up. Not to bad for a Tuesday night.
Let’s try to wrap things up here. There are few things that you would have to pay attention to when things go south on your CUBE implementation.
- First of all, debug ccsip messages is your best friend, use it. do notice that sometimes this debug won’t show you the register messages to your Registrar and you would have to use debug ccsip all. Only use the all command in a non-production router or during a maintenance window as it might crash your router.
- Check your From field in outgoing calls. If the number/user is not associated to you by the ITSP, your call will probably be rejected.
- Don’t forget to verify your registration status with show sip-ua register status.
Do not get confused by outgoing calls that are working.
Remember that you will need both the Authentication and the Credentials commands under sip-ua.
- One-way or no-way audio is pretty common. It usually caused by NAT issues.
If your Cisco CUBE has only LAN interfaces, remember that someone has to replace the IP Addresses in the application layer (SIP headers), so either use SIP-Profiles or SIP-ALG on the FW.
- CUCM routes calls using the SIP-REQ-URI header, not the To header.
Do verify what you receive from ITSP and change accordingly using SIP-Profiles.
In addition, there are some very good docs and tools out there, don’t be afraid to use it.
- I have written a multi-part blog post on how to configure CUBE step-by-step, you can find very useful information there. Part 1 Part 2
Let me know if you have any questions or had any issues with your ITSP deployment in the comments!
Best of luck!