atcomsystems.ca/forum Forums TDM Business Telephone Systems Toshiba Telephones & Systems DID problem

Hi all.

I'm having a weird issue with DID numbers that point to IP phone extensions on a remote IPU.

For example I have 2 buildings with CIX40 systems. The PRI resides at a 3rd building with the phone system being a CIX670. All of these systems are networked, but when I try to assign a DID in emanager to one of the extensions on the CIX40's, I run into a weird problem.

When making the call, the caller cannot be heard. However, the caller CAN hear the person who answers the call.

I have no issues assigning DID's to the extensions at the building with the CIX670. They work fine. Just not at the remote sites with the CIX40's.


Anyone heard of this before?

Hi AndyK, welcome to the board!

Just to clarify...

1. Are the remote buildings (CIX40's) networked via Q-Sig over point-to-point T1's?

2. Do the CIX40's use the PRI on the CIX670 for in/outbound calls?

First suggestion/question - Have you tried changing the CoDec's or the Voice Packet Transmission Interval?

Thanks for responding!

Well, the buildings are networked with Proxim Tsunami wireless bridges. We aren't using point-to-point T1's for that.

The CIX 40's do use the PRI on the CIX670 for outgoing calls most definitely, but right now most incoming calls to those buildings are transferred through vociemail.

For example, someone dials the main number to our company and the voicemail auto attendant answers. They dial the extension for that person at the remote building and their phone rings. They answer and all is good.

When I assign DID's to the same remote building extension, that's where the caller can't be heard, but the caller can hear the person who answered their call.

I'm not really at expert level yet with eManager so it's hard to say about Q-Sig. I can tell you what I've got though for setups in ILG.

On the 670, it shows a group number 1 for ISDN. This is the DID CO service type, and the line type is CO. Private service type is Standard.

There is also a group number 20 for ISDN. In there the service type is DIT and the line type is TIE. Private service type is Q-SIG.

That same group 20 for ISDN is on both CIX 40's under the ILG setup.

As for as CoDec's or Voice Packet Transmission I don't really know where to look.

The software was given to me by the vendor when we got our system. They gave me a crash course but nothing too in depth, so I'm learning a lot by the seat of my pants.

Hopefully, that info gives you a better idea of what may be going on. I can't help but think it's something to do with either the networking portion or something with the incoming line over the network since it's the incoming caller that can't be heard.

Anyway, thanks for any advice you can offer.

Andy

A bit of an update here. I was tinkering today and found out something even more strange.

I was trying different things trying to get it to work and decided to call the DID number, then do a transfer to my phone which is at the building with the 670 and PRI.

I transferred the call to my phone and it worked with no problem. Not that surprising I guess.

BUT

Afterward, I tried the DID again. This time it worked!!

It only worked a few times before it stopped working. If it stops working I repeat the above steps. Call the DID answer the call...transfer it to my phone, answer the call at my phone wait a few seconds, and hang up. When I call the DID immediately afterward, it works again.

Not sure if this helps at all, but I found it interesting.

I think you get the award for the best described, most puzzling question of the year so far.

You're telling me!
I've been trying to resolve this one for about a week now! It just seems so weird and hard to pin point. I've looked through a bunch of settings in eManager, and I finally figured out what MacOSX was talking about above and played with those settings a bit, and it wasn't that. Only thing I can think is something isn't shaking hands properly 100% of the time with the CIX 40 and the 670 with the PRI at the remote location.

Nobody seems to have any ideas here, so I'm just going to keep throwing out what I'm doing to try and resolve this so it might spike some ideas.

I decided to take a phone programmed for the main building with the CIX640 over to one of the remote buildings with a CIX40 to try and see if a call through a DID would work, or it would not work.

I did this basically to eliminate the wireless link(Proxim Tsunami) as the problem.

The wireless has been eliminated. A call to the phone through DID worked fine, even at that building.

Now my next step will be looking at how those remote systems use the PRI. The problem must be with that part of the configuration, right?

Anyone here know what I should typically see in my eManager if I have two buildings using CIX40's and the main building with the PRI using a CIX640? I'm looking for things I can tweak, maybe find a mistake, etc.

Any help would be great...I know this is a difficult one.

All your phones are IP phones? Intersting how a phone registeredto the CIX670 IPU works correctly accross the network.

Typically in an VoIP environment, the 2 endpoint devices will communicate directly after a call is established. I believe that for the IP phones registered to the local CIX system will talk directly to the IPU on the main CIX 670 system. The local IPU on the CIX40 will establish the call and will no pass audio traffic.

If that is the case, the the problem is between the Main IPU and the remote IP phone, where the RTP traffic is't passing correctly. Since the caller can not be hear I would guess it is a problem from the main IPU to the remote IP phone.
The best way to verfy this is to use wireshark at one of the endpoints.

I would guess that all the IP addresses are local network IP since you are not routing through the WAN.

Just to throw stuff out there, you could try setting the NAT, no peer to peer setting in the IPT to enabled. This is usually not set in a LAN enviroment, but it causes all audio traffic to be sent to the local IPU. I'm not 100% sure if it effects IPT to IPU traffic. For that matter I'm not 100% sure the local IPU is bybassed across a Stratanet network either.

Also, sometimes Diffserv or 802.11Q and P can block audio if the LAN devices don't understand the signaling. Try disabling them to see.

We do have a few Digital telephones. They run off of the CIX670 because there's some digital station cards in it. However, most are IP phones. There's no digital phones at the remote sites I'm talking about. All of the IP's for the phones themselves are a VLAN. VLAN 10 to be exact, then our data network is on VLAN 1.

Thanks for the ideas, I'll try them on Monday and give an update as to what happens. I've already glanced in eManager and found the NAT peer to peer setting, and I do have wireshark. Anything in particular I should be looking for on Wireshark when I run it?

One dumb question though, where would I change those Diffserv or 802.11Q settings you're talking about? Would that be on the Tsunami?

Thanks again!

I have not used Vlans on Toshiba systems as of yet. Like I said, preferably the NAT setting should be left disable a LAN setup, but I was throwing stuff out to try.

As far as Diffserv and 802.11Q, they are both QOS with ideally should help with the quality of the audio. Diffserv is enabled in Prg 150, System IP Data, FB 03. If it is enabled, then FB 04 defines it ad TOS or DSCP, and for DSCP you assign it a number in FB 06.

802.11Q is enabled in PRg 161, IPU configuration. You wouldn't want to change VLAN, but you can try to disable packet prioritization.

Normally for either QOS setting, any device that does not recognize the header info will just ignore it and pass it through, but I have seen cases where the audio gets blocked, usually across a WAN.

As far as Wireshark, if I were doing it I would look for the RTP traffic to/from the IP phone to see which device IP it is talking to during a call, just to varify my thought that the IP phone is probably talking directly to the IPU on the remote side. Then I would try various point in the network, were possible, to see if any device on the network is not passing the audio, such as either side of your router.

Obviously you could mirror a port on some switches. If you had a data hub (a 10BT in an old pile of junk somewhere) you could move the snoop point around a bit, but maybe just mirroring the data switch will give you the info you need.

You would see some Megaco traffic to that phone also. I'd be more concerned about the Realtime Traffic.

I looked for the Diffserv, 802.11Q and P settings, and they were already disabled.

I also tried enabling NAT no peer to peer and it made no difference.

Looks like it's back to the drawing board.
Maybe I'll give wireshark a go.

Andy,
your problem might be a firmware issue on the MIPU cards. there was some knwo issues to cause intermittent audio on Stratanet calls.

I never did figure out this problem, however I do know that the firmware was just upgraded on those MIPU's in December or so for an audio related problem. The audio isn't intermittent, there's just a lack of it completely on the caller's end meaning the caller can't hear the person they called.

This only happens on our remote IPU's at other buildings that are connected via Tsunami wireless links.

I am pretty sure this came out in november, but you want the MIPU firmware to be MIPU01_19

"Audio will no longer be lost over StrataNet IP when the MIPU firmware sends an ARP request and waits for a reply to recover the ARP table. If the packet send request generated before a receive ARP reply, the MIPU firmware reads the MAC address from ARP table with all zeroes for the destination ethernet address. (Note: In Wireshark search for the following command, “eth.dst == 00:00:00:00:00:00”) (CTXR520-0064)."

Also in the DID tables try setting the data destination and the data digits to match the audio settings, I have seen that correct some strange issues in the past.

Well the data digits thing in the DID tables did not work.

My MIPU version is MIPU01_18DA200 on all IPU cards.

Looks a bit behind what you say it should be.
Does a dealer have to update it?

Interesting update on this...

I found that there is also a one way audio problem when I try to do an "announce" to a phone on one of the remote IPU's as well. A regular call where I actually pick it up on the remote end instead of pressing TALK locally and listening for the tone and then background noise works fine.

I'm beginning to think this problem and the problem plaguing incoming DID calls to these extensions on the remote IPU's are one in the same.

I'm also trying to set up Wireshark to watch the network for this: eth.dst == 00:00:00:00:00:00
to see if it is in fact that firmware problem.

I have it running on a computer that is connected to the voice vlan only, but I don't see the packets I'm looking for. I'm kind of a novice with it, so I'm not really sure how to set it up properly to watch for what I need to see.

Anyway, if anyone has an input that would be great...

wireshark will not capture the packets unless you and the IPU are plugged in thru a true HUB, not a Switch or you have a managed switch and you set a port to "mirror" the port that the IPU card is plugged into.

Ah Ha- of course...makes sense. Thanks.

Ok I went ahead and did some tests with Wireshark and I saw something interesting in there when I tried the announcing calls, and through the DID.
I was running Wireshark from a PC connected to a mirrored switch port at the remote building. I mirrored the port connected to the IPU in the CIX 40.

I was seeing ICMP Destination unreachable (port unreachable) or Type 3 Code 3. When I initiated the call, first a large ammount of UDP packets were passed until the call was answered, and from then on I would see mixed UDP and RTP which seemed to be fine, then here and there these ICMP errors popped up. Sometimes after I ended the call, but they occurred 17 times in the few moments I was capturing packets.

Basically from what I was able to gather in the details there was some sort of communication attempt between 192.168.14.8 which is the IPU at that remote building, and then 192.168.14.7 which is one of the IPU cards at the main building with the CIX 670.

In further detail it says the frame contained the following protocols eth:ip:icmp:ip:udp:data

I noticed the following in the info about UDP:
The source port was 20994 and the destination was 21016

Under internet protocol(there are two of them) on the first it says the source was 192.168.14.7 and destination was 192.168.14.8

On the second the source was 192.168.14.8(remote building IPU) destination was 192.168.14.7(main bulding )

Whenever I saw RTP packets they were always between the IPT at the remote site and the IPU (192.168.14.8) at the remote site.

UDP packets were always between the main building IPU and the remote building IPU. 192.168.14.7 was always the source, 192.168.14.8 was always the destination. Source ports and destination seemed to change but on this capture they were 21016 for source and 20994 for destination.

It did not seem to make a difference what way I called.

A call where I pressed talk and announced to another IPT did not work while a call where I let it ring and then picked it up worked just fine. On the calls where I announced I noticed there was NO audio passing either way. On DID calls to the remote IPT there was audio passing one way from the IPT to the caller's cell phone, but words spoken into the cell phone could not be heard on the IPT.

I did NOT see any ethernet destinations for an invalid mac or 00:00:00:00:00:00

Not once in any of the captures I did.


I'm not sure if any of this means anything to anyone, but hey, I'm bouncing ideas around....

I saved my capture so I can refer to it again if need be.

Hi Andy,
I'm new here so treat me nice! ;>)
Would I be correct in assuming that you are using just one MIPU card in each site to service both IP phones and IP network trunk (Stratanet) calls?
Every time we experience one way audio, it ALWAYS turns out to be a router or Port blocking problem. I'be willing to bet that if the CIX40 and the CIX670 were in the same room, and on the same subnet, you would not be having this problem. I'll bet there's nothing wrong with the telephone equipment. Do you have routers along with the wireless gear?

I'm relatively new as well so no worries about getting any crap from me...I'm always grateful to receive responses!

Anyway, what we've got at our "main building" if you will is a CIX 670 with two IPU cards in it. One is addressed 192.168.14.6 and the other 192.168.14.7 respectively. As far as the Stratanet goes, node 11, which is the 670 is only listed with the ip route of 192.168.14.7 which I thought was a little weird since there are two IPU's in that, but eh...maybe it doesn't matter?

At the remote sites (there are two of them) we have CIX 40's and the remote sites are networked to the main site with one Tsunami Wireless bridge each. No routers. I do have an inter vlan router I recently set up, but this problem was persisting long before. The data and voice networks are somewhat seperate. There are two vlans one for data, one for voice. All the phones are obviously all on the same subnet.

I've not seen the Tsunami's ever block any kind of traffic, but I'm not an expert with those either. I also confirmed that by setting up a phone from the main building(connecting to an IPU from the main building at the remote site) over at one of the remote sites. It worked fine. No audio loss in any case. I would have to think if it was the Tsunami blocking traffic, I would have experienced problems with audio loss.

All of the routers we have with the exception of the inter vlan one are on the data subnet, and have nothing to do with the phone systems.

It seems as though regardless Wireshark is telling me that something's not able to get through someplace. I'm left to believe it has to be the CIX 40 IPU talking to the IPU on the 670. The ICMP errors I was seeing in Wireshark and mentioned above looked like it had a lot to do with those two IPU cards. Those being 192.168.14.7 at the main site with the 670 and 192.168.14.8 at the remote site with the 40.

Andy, I had a similar problem at a multi site customer They had a cix670 and three cix40's

they would get one way audio, the problem was the wrong gateway was set on the mipu cards at the remote ends. I could take an ip phone from the remote and plug in at the main and everything worked fine.

i could ping everything and i could call the remote but would loose the audio

check all locations and make sure the gateways on the mipu/lipus are correct. just a shot

Hey, glad to see you gave wireshark a try. I'm busy packing so I can't take a lot of time to try to figure out all the information you posted, but I hope it at least points you to the cards that are having trouble talking to each other. This should give you an idea of how the communications travels from point to point.

Like Canukvoip said it is almost always something blocking the data stream. The 1st hard part is finding out what is blocking it. The harder part is fixing it.

Are your Tsunami's configurable as to whether they pass broadcast traffic? If they're blocking ARP you could get this.

It might be productive to set up a *pair* of wiresharks on each side of the bridge, and make sure their clocks are both NTP synced to something, and then compare the traffic you see.

Got all the IPU's on the same gateway. Previously they had no gateways assigned but I did assign them each the same gateway because I started testing a remote IP phone over the internet awhile back.

This problem goes back to before then, so I'm not sure it has much to do with that. I could be wrong though!

Something's not right somewhere however with the audio stream. It's fine from IPT to IPT on the calls(when both people on the call are using the receiver and not voice first announcing) over the stratanet from like the remote building to the main building and the opposite. It's just when you try to announce to an IPT extension at the remote building that you get no audio at all. BUT you CAN announce from the remote building to an IPT at the main building...which is weird.

Externally you get one way audio on DID calls to the extensions at the remote buildings. The only person who can hear anything is the outside caller.

So I'm going to sound a bit like Dr. House here but..."What causes the following symptoms?" lol

Here's a quick summary

1.Audio working fine on calls where the caller picks up the receiver on the IPT at the remote building and dials, and the other person answers. It also works the same if you dial from the main to the remote the same exact way.

2.Audio does not work at all when you dial an IPT's extension at the main building from the remote building and press TALK to "announce" to that extension. No warning tone is heard, and no audio is heard through the speaker from the other end, and no audio is heard through the IPT that was dialed/called. So basically, no audio at all. Even if you then pick up the receiver at either end you still have nothing.

3.When a DID number is assigned to an extension at a remote building and it is dialed from the outside, the phone rings and displays the caller ID with no problem, however when you answer it you can't hear the caller. However, they can hear you just fine.

4.Wireshark captures revealed that when a call placed through said DID from #3, there were ICMP errors logged and somewhere..somehow...we've got 17 instances of Type 3 Code 3 Destination unreachable (port unreachable) for these port numbers that are UDP: The source port was 20994 and the destination was 21016. This was logged from a communication between the IPU at the remote side and the IPU at the main side.

Refering to 4.
For sure, something is blocking those ports. The ICMP errors thing bugs me. I thought ICMP referred to ping...
I maintain that if you took the CIX 40 and plugged it into the same L2 switch that the CIX670 was in, or if you used a cross over cable and plugged the two MIPU cards together, it would work first time/every time.
There's something on the network blocking ports me thinks lads.

I have a stupid question.
What is the DHCP server for the phones?
And, are there any other "rogue" DHCP server devices on the site that you may or may not know about?
If the phone got an address from an incorrect device, the gateway might be wrong.
If you went to an IP phone on the CIX670 site and checked the IP address/Subnet/Gateway you could compare that to a phone on the remote site.
Might shed some light on the subject!

Canuckvoip,

The DHCP server for the phones is running off of the "backbone" switch. There are about 7 of these Adtran NetVanta managed switches total between all three buildings. Only one gives out DHCP though. It gives out IP's for 192.168.14.0/24 and then a default gateway of 192.168.14.4 which is my inter vlan router. On the other side of that router is the data network or Vlan 1, which is 192.168.4.0/24- The DHCP server for that network runs off of a Windows DHCP server.

DHCP server is off on the Tsunami wireless bridges and each has a static address on the data vlan. That seems to have no bearing because if it did my main building phone wouldn't work at the remote building.

The phones at the remote buildings would all have the same gateway because they receive it from DHCP. I programmed a phone from the remote building and I have it over here at the main building. It's aimed at the IPU from that remote building. I verified that it's gateway, etc were correct. I have tested a phone from the main building over at a remote building as well. Aiming that one at the main building's IPU. There's no audio problems meaning the problem seems to be directly with the IPU at the remote building talking to the one at the main building. Using the phone from the main building at the remote building verifies that when the IPU on the CIX 40 is out of the mix, no trouble. As soon as we need to initiate communication between the IPU's at the main building and then the ones at the remote buildings...then there's problems.

Just another thought on this....

I remember when we first put this phone system in there was somewhat of a complaint from the folks who work in these remote buildings that they didn't want anyone to be able to "yell" at them through the phone. They would have been talking about the feature to press TALK and do a voice announce. I'm not sure what became of it, but I've noticed we can do a setting on our system that will allow the voice first announce. These settings work fine here at the main building. I set the parameter under 204-05 on a phone at the remote building and its like it completely ignores it. When I dial the remote extension from the main building it still does tone first! Is there any way that this could have been completely disabled? In a sense that anyone doing a voice announce from the main building to a remote one just wouldn't be able to make it happen? It seems if you dial from one extension to another within the remote building voice announce works fine, but from the main to the remote you get silence and nothing happens, though the phone does display "CALLING 407" then "ANNOUNCE TO 1200 CONFERENCE" once the connection has been established.

Re: Just another thought on this....

Andy, I have to admit that I haven't had experience with this personally. That being said, the way Toshiba (and others) network systems together via ethernet is called Qsig.
Qsig was and still is available on PRI (TDM) and now on ethernet/Wan/extended Lan connections.
It is "kind of like Tapi" in that there are a defined/limited number of features available. Things like M/W lights, caller ID etc passed between systems.
It was meant as a way for multiple systems to "feel" as if they were on one system.
Not all features are going to be passed between systems. Period!
Each site has its own processor, so if the voice first/tone first info is not passed via Qsig it won't have any effect on the destination processor (CIX670 to CIX40).
So, (unless I'm reading this wrong), the feature enabled on the CIX40 will affect those CIX40 callers calling other CIX40 users ON THE SAME CPU/SYSTEM.
Same like same on the main CIX670 site.
This I believe has no bearing on the one way voice situation on the network overall.
I think...
Dave

Re: one way audio...
Did you go to a CIX670 IP phone and look at/verify the IP/Subnet/Gateway addresses?
Did you physically go to a CIX40 IP phone and compare that to the other?
What are the addresses seen? This MUST be done at the set.
I know we assume that the DHCP works, but we all also know what assuming does...
;>)
Dave

I've tested this with a phone that does have the correct gateway on it, and it still does not work. I have a test phone for each site set up at this main site. Both have the right gateway. I also went to both sites and saw that each phone is getting the proper default gateway.

The one way audio problem persists.

As far as the QSIG thing, the CIX's are definitely networked through QSIG so I guess I see what you mean there. I know that not all of the features are available because the voicemail works differently here on the main site that it does at the remote sites. They can still access it fine, but at the main site it gives you options to use the softkeys for the different menu options where as the remote sites it does not. I just thought maybe since that was an audio problem, then it may be related to the other audio problem with the DID's.

What you said makes sense though about the gateways because the same applies if you have remote sites with computers..they need a static route if they don't use the router that connects the networks as the gateway. It just doesn't seem to be the problem here.

Well, an update on this after a month or so...

I had my dealer come out to take a look at the problem. Apparently this portion of things was still covered under our warranty.

Anyway, in a nut shell...we've rebooted the phone systems, updated the MIPU software, changed the protocol on the PRI from Nat'L ISDN to Nat'L ISDN Nortel...,none of these worked.

The dealer is stumped, and so is Toshiba. Toshiba has looked at the programming for the phone system and found nothing out of order, nor did the dealer.

We're going to try updating the software for all of the phone systems on Tuesday. We'll see what happens.

Well, call this one resolved!

The problem was resolved by upgrading the phone system software to the latest version.

Interestingly enough this also resolved the issue with not being able to announce to stations at the remote locations as well. Awesome!

Good to know.

To Andy and all the people who replied...
I should be dead cuz if you stuck a gun to my head I would have bet my life on a router/port blocking problem.
Holy guacamole!
I suck ;>)
Thanks!

Dave

No worries Dave! I appreciated your suggestions and that you wanted to help. I don't believe it's a bad thing to throw stuff out there especially when you've had a similar experience.

Thanks to everyone who contributed.

In more detail for those who want to know, Toshiba told the tech from our dealer working on this problem to change the ChMethod all to Channel Number B and click the checkbox for 08,09, and 11 on program 302 for PRI and IP QSIG. Strangely enough before we did the upgrade we could not do this. Each time we submitted the changes, eManager put them right back to what they were before. The software upgrade however, automatically made the changes. The only other thing we needed to do immediately following the upgrade was reset the PRI.

Toshiba also said they wanted to delete the channel group(this was after we couldn't get the changes to stick), but were unable to do so since it was during business hours. I'm not sure if this may have resolved the issue alone without the upgrade. It would be interesting if someone with the same problem tries this method and resolves the problem as well.

I guess there are a few possible solutions to this, but if you're with a dealer and have the means to get the software upgrades, I would just go for that.

Thanks again everyone!