atcomsystems.ca/forum Forums TDM Business Telephone Systems Avaya - Lucent Telephones & Systems Software initiated cold start

Any ideas on what causes this? This is from the event log. Have replaced the 509 processor already with no improvement.

Simple Name Count Last Reset Event 16 06/02/2009 11:11:02 AM software initiated - cold start

I've never found the admin software error logs of much use Bill.
What problems are you having with the system/systems? Out of the blue restarts?

Also is it a R.7 that you are working with? If so what version?

Yes they are seeing out of the blue restarts. The version is pacs7.91.0
The customer has 6 18d sets and 7 analog sets.

Have you ruled out the posibilty of having A/C power problems?
And is the system on a battery backup?

System is not on battery backup. In the past I believe power fails cause a different message from software restart.

I would put it on a good battery backup and see what happens.
Like I said before, I've never found the admin software error message log on a Partner worth much.

Have replaced or isolated all packs and carrier. Have tested or removed all extension wiring. Have installed an APC? battery backup which indicates no issues through it's software. The analog sets are polycom units with headsets. They have lights when in use, but I've removed half of them in case there was too great a current drain when in use. It seems the only other area might be power surge on one or more c/o lines or a problem with one of the 5 18D sets. Any more ideas?

Any more ideas?

None of what you attempted or are thinking except for the UPS would have any effect on your problem. Next step is to replace the ACS.

-Hal

Do you have PC VM in it? I had ones ACS was resetting itself for split of a second cause by PC VM card in slot 2. If I changed a slot to 1 all was OK.

Have already replaced that with the same result. I'm down to removing c/o lines one at a time and swapping out the 18D sets one at a time. Open to any suggestions.

Ahh, Ok, I didn't remember that. Is this a standalone processor? No carrier or expansion mods? Is there a PC card VM as vad asks?

-Hal

The system has a 509 processor, a 308 card in the first slot, a 400 card in the second slot, and a second 400 card in the 4th slot. There is a small VM card in the processor. There are 6 18d sets, 3 of which are key sets and 3 are pool sets. There are 10 analog sets and a credit card machine. The Auto Attendant answers all 15 c/o lines and uses selector codes for direct transfer. Lines 2 thru 15 are in the main pool.
I've called the main number, which hunts, from 5 different phones at the same time (well as fast as I could dial). Two answer and the rest ring and the system does not restart.

Since you replaced the processor I think you are going to have to either start replacing the VM card and the expansion mods as well as the 5 slot carrier (one at a time!) or remove everything, operate with just the processor and start adding things back one at a time. I suspect it will operate just fine with just the standalone ACS.

That is where I would start looking, I seriously doubt a phone would cause this. This is also why, if you are going to service and install these systems you are going to need to have spare equipment on hand or at least be willing to purchase same. That's the only way you are going to diagnose weird problems like this.

-Hal

Hi,

I work with Bill. Here is a review of what has been done so far.

New 509 Processor, 308EC, (2) 400 and large pc mail card installed. Installed 6 18-Button Display phones and the customer used 9 plantronics T10 analog headsets for their telemarketers.

After installation, system would crash. Log shows "software initiated cold start".

We started by adding a UPS to the system. No change.

Swapped out 509 Processor. No change.
Swapped out 5-slot carrier. No change.
Swapped out 308 expansion. No change.
Swapped out 400 mods. No change.

Found 2 pairs reverse on cable for Ext. 10. Corrected and system stayed up for 2 days without crashing. Then crashed.

Have checked all other cabling. Have removed groups of the analog headsets. System continues to crash.

We are taking each CO line out of the system one at a time to see if that could be the cause.

Have not swapped out VM card yet.

We are currently trying a new 18-button phone at ext. 10 and will rotate it through the other locations as the system crashes.

Any other suggestions?

Pull or swap out the VM card. What exactly happens when the system "crashes"? Does this happen during the day, after hours? Any particular pattern?

-Hal

How about loop current? High loop current can cause systems to do funny things. Seems to me you've pretty well replace all equipment.

Nahh, something's not right with this story. I want to know exactly what the system does. Does it act like you pulled the plug or what?

-Hal

I agree with Hal. Something is missing here.

Swapped out the VM card yesterday, system crashed this morning.

Right before it goes down, all of the buttons on the 18-Ds become inoperable. Then the system shut down, drops all calls and then power cycles back to life.

Moved the replacement 18-D to ext. 11 and removed the existing 18-d from the system.

So now, other than the 5 remaining ext. with 18-Ds, we've replaced every component of the system except the OHP 6000 music on hold.

Jim

I see you installed a UPS have you tried pulling the UPS power plug from the wall to be sure the system doesn't restart if the UPS loses power ?

Now that you mention it I had a UPS (APC) that caused something similar with an R6. Dumped the UPS and plugged it in directly and no problem for years.

-Hal

It's got to be ac power related...I would start
looking at what else could be plugged into the same circuit as your Partner...you have replaced
everything else EXCEPT your AC power source.
It just could be dirty power coming from the
electrical panel or a circuit braker gone bad.
Just a thought!!!!!

Originally the system was not on a UPS. So when it started crashing, we installed the APC UPS and have been monitoring it with the APC supplied software. It indicates no spikes or outages. We also ran an extension cord from another part of the office to the UPS to try to rule out the outlet it was on.

Jim

Been following this thread and it looks like you have replaced everything but the building. The only time I have ever had this problem was when the system was in the vicinity of a aluminum welder. Buy chance is that the case?

That is my next thought also. What kind of facility is this and what is around it?

Unless you got a bad string of ACSs it ain't the system.

-Hal

A warm start of a system is initiated with a warm start manager disabling incoming signals to the system and initiating at least a two-phase warm start procedure. In the first phase, being an intra-process phase, each process is checked for integrity of its own data structures. When data structures fail this check and cannot be recovered, a cold start is initiated. In the second phase, being an inter-process phase, entities that each process owns are checked to ensure that all other processes have a consistent image of the entities. Those entities that do not have a consistent image across the processes are removed. In an optional third phase of the warm start procedure, a determination is made as to which of the removed entities can be recreated immediately, and those entities are recreated.

Sharmagaurav1, what are you talking about???

Hal?? comments?

Think he means the translisilator and the muffler bearing are crossed in the cable............at least that's what I got out of that?

...bob...

Makes perfect sense to me. :shrug:

-Hal

System continues to crash, although we've had periods of 5-7 days when it does not crash. Have not been able to identify a pattern.

This customer sells vacuum cleaners and they take old vacuum cleaners in as trade-ins. The old vacuum cleaners and their new demo units are stored in the same room as system. This may be a dumb question, but is it possible that having that many electric motors that I assume have magnets in them create some kind of interference? They have someone who comes by from time to time to take away the old vacuums, which could explain why there are periods when the system doesn't crash.

Just a thought.

Clutching at straws aren't ya. You might be onto something though. Any sources of high powered RF around? Truckers with linears on their CBs for instance?

I assume that when you take one of these ACS processors back to your shop it will run without problem right?

-Hal

Nah, I have been down the trucker high powered linear thing several times.
All you will get there is bleed over. With the most server cases you will even get bleed over across the system phones at idle. But never a power down of a system.

Sounds like you are just going to have to live with this system and do some very basic troubleshooting to get to the bottom of this.

One other condition is: low power output...is the
power outlet isolated from other plugs....I would
look into that....Or change out the Partner to a
Legend...

Sounds like you are just going to have to live with this system and do some very basic troubleshooting to get to the bottom of this.

I agree. There has to be something you are not seeing.

-Hal

The customer just switched dialtone providers on Thursday. We will see what happens next.

I have also placed another 509 processor in the same phone closet connected to nothing but the power supply. It has cold started once vs. multiple times for the entire system.

Other ideas that have been floated:

The system has the Avaya Large PC Mail card. This company runs a lot of help wanted ads and gets flooded with calls. Is it likely that the system gets overloaded with all of the calls trying to get through the 2 ports on the PC card? I would have thought it would just be "ring no answer" until a port frees up.

Another theory is that the telemarkets using the analog headsets get their line access via a pool. Extensions 14 & 15 (18-d's) have access to these lines on their phones and select lines by pushing a button. Is it possible that there is some sort of collission confusing the system if someone in telemarketing is trying to get the same line at the same time. Seems unlikely to me, but throwing it out there.

Jim

I have also placed another 509 processor in the same phone closet connected to nothing but the power supply. It has cold started once vs. multiple times for the entire system.

Good idea! And since it also restarted with nothing connected to it that kind of blows your other theories out of the water, doesn't it.

It's either a power problem or RF.

-Hal

From everything I've read...you should have an electrician out the check the power panel...to
rule a voltage spike or drop.
there is either a huge electrical motor attached
to the panel, maybe a bad ground...it could be
a old power panel....etc, etc.
I'm out of possible solutions to your problem..
but, I would never let this issue carry on this
long...

I agree. Just because you have a UPS on the line doesn't mean that you have clean power. They will take over when there is no power but most are straight through and no better than a power strip when normal power is present.

-Hal

There is a gadget called a Dranetz meter. They can be rented as well as purchased. A million years ago when electronic systems first came out they were even more sensitive to spikes then the current ones.

We had a Siemens SD-192 switch in the Galleria Building (53rd & Lexington) here in NYC that would restart twice a day. We did everything you did, including replacing the switch and reprogramming all the software. Nothing helped.

We put a Deltec line conditioner on the AC and the problem went away. But why? The Dranetz meter showed a spike of 400+ volts on all the 120 Volt circuits in the building (a large skyscraper).

It turns out that when the (humongous) motors that ran the HVAC kicked on or off (usually twice a day), it put a spike on the building AC for less then 5 milliseconds. It wasn't enough to bother anything else, but it fooled the Power Failure Transfer card on the switch into thinking that the power had gone out and come back on. Hence the reload.

See if you can rent one of these babies. You could also try running the system off a portable generator.

Sam

I think your on the right track Sam.
Another way that we have checked these type issues is to take a good true RMS voltmeter that has a PC link.
Hook a ole cheap laptop that you have laying around to it and let it run a data logger for a week or so and then compare notes.
We had one a few years back that had a "Wild Leg" on their 3-phase that was causing about the same problem.
I still think this is a power issue, instead of phone equip.

I still think this is a power issue, instead of phone equip.

It CAN'T be the phone equipment (like we have been saying all along) since he now has another ACS sitting there with nothing connected to it and it does the same damn thing!

Sorry, but somebody needs to have better troubleshooting skills.

-Hal

I give that a big :thumb: !

I have never heard of or witnessed standard loop
start lines ever bring a switch down....now I've
seen a DS1 card cause issues, but not what you
have been reporting....hopefully you will find
your bad power problem and soon.

An electriciation is needed. Stop waisting time and have the customer call one in and see what he finds.

Landlord's electrician is scheduled to come out on Monday. The stand alone processor that we placed in the phone closet has only gone down once, whereas the system has gone down many times during the same period.

Once is enough to clinch it. If it did the same thing there has to be a common denominator. The power is the only thing.

And don't expect sparkie to come up with anything.

-Hal

Ditto.
More than likely sparkie is going to say "Call the people you bought your phone system from".

That is why you always need hard proof to back your argument before you call any other vendor and/or service provider. If it was me in this situation I would run a data logger on the A/C power for a few days or week and see what you get.

Good luck, hope it works out for ya.

The APC UPS is monitoring the power and their reports show no spikes or outages.

Throw that friggin thing away and get one that is always on line. It's also a poor excuse for power quality monitoring. Rent a real logger and learn how to use it.

-Hal

I still would pull the UPS power plug from the wall a dozen or so times and see if you can duplicate it

does the ACS plug directly into the the UPS or is there a outlet strip or surge protector inline ?

I ran into a similar problem with a panamax tower max surge suppressor and a APC UPS

sandman used to sell a inexpensive power monitor you might want to give him a call

Nah Skip, that's no better than what's included with the UPS. He needs to rent this for a week, hook it up and let it run. That's the only way to prove or disprove a power problem.

-Hal

all his APC software is telling him is power in

pulling the plug and watching tells us if sufficient power is getting to the ACS

quick and easy test that I always do on any install before I leave

4 months later and its still not resolved. You
are spending way to much time on this...hopefully
the electrician will find the problem and fix it.

No, I wasn't arguing with that Skip. That's a good thing to do.

4 months later and its still not resolved.

I'm surprised the customer is still in business.

-Hal

maybe the AA greeting says something like

"if this call is disconnected for any reason please call back ...We value your business"

"quick whats your number in case the system re sets"

Electrician was not much help. He is going to install a dedicated circuit for the system on Wednesday.

The system had just gone from last Tuesday until this Monday without crashing. Then, after our tech left the site yesterday after meeting with the electrician, it crashed twice.

My tech thought last week that perhaps the Auto Attendant on the large mail card was getting confused when too many calls were coming in, so he set up a delayed ring to help relieve any congestion. Then the system stayed up for almost a week, so we thought we were on to something. But again, two crashes yesterday.

My tech thought last week that perhaps the Auto Attendant on the large mail card was getting confused when too many calls were coming in, so he set up a delayed ring to help relieve any congestion.

I don't know how much clearer we can be. You have an ACS with nothing connected to it and not handling calls and it crashes too. Do you really think too many calls are causing this?

I don't want to be rude but if I were your customer I would have fired you and called another company in long ago. It doesn't look like you have a clue.

-Hal

AMEN to that,Hal. This has been painful to
read over the months...You have tried the patience of the guru's in this forum...and have
been given very clear answers to the problem and
yet refuse to move forward...time to bring another company who can get this resoloved...I
suggest the Mod's close this topic.

Thanks for the feedback guys.

The electrician installed the new dedicated circuit this morning. We have brought up the idea of using the monitoring equipment to the customer and they are reluctant to spend the money. I should point out that the reason the customer has not fired us is because we have not charged them for any of 30+ visits to try to resolve this issue. The customer wants us to prove that it is the phone system causing the problem and make it go away, which we've been unable to do. They don't want to spend any money on their own to research any other issues, i.e. AC power.

We checked the log on the two processors (the system and the stand alone) today. The system had reset twice this week. The stand alone has still only reset 1 time in total in over 3 weeks. An interesting twist is the last reset on the system had a 2008 date. All of the other resets have had the correct date. We verified that the system date was correct.

The customer wants us to prove that it is the phone system causing the problem

So, you are not the company that originally installed it and you were brought in by the customer to show that the phone system is causing the problem?

-Hal

No. We installed the system in late May and yes, we charged them for that. After the system crashed a few times, we convinced the customer to let us install the APC ups/surge protector.

Prior to that the customer was using a few generic 4-line phones for their admin people and had all of their telemarketers on single line phones. But they had never had a "phone system" before.

I'm anxious to see if the new dedicated circuit helps. Since May we've had two 1-week periods where the system did not crash at all, so we'll need at least a few weeks of stability to see if the problem is actually resolved.

I trust when it is proved the AC was the problem you will bill him for all your time.

Update. After the new dedicated circuit was installed we went 2 weeks without a crash, which was a new record. We thought "Problem solved". Customer called today and said they system started crashing again on Wednesday. Now it is doing it more frequently and it seems to take a little longer (a few seconds more) to come back up.

You still have power problems, period. Even a dedicated circuit can cause it if the leg is bad.Is there another business close that has a switch that isn't having problems or is there a server room, if so run an extension cord and use that power for a while to test, at this point and as others have said I'm sure it's the building power and not the system.

Thanks Russ. We did run an extension cord from another part of the office months ago with the same result. As Hal has pointed out, a power analyzer seems to be the only way to prove it is the power.

Power analyzer is the answer.
Let cust understand this is all billable when you prove it is the power.

This is undoubtedly a power issue that must be resolved. If their business is not important enough to spend the money to rent an analyzer - I'm not sure I'd want them as a customer. This business has to work both ways.

That being said, I had a similar problem for 2 days with a V6 or 7 Partner at a body shop after an install. Checked power 10 ways from Sunday. Replaced the processer, backplane, 308's - everthing. I decided to spend the 3rd morning there myself and get it figured out. I had a new APC and all.

The system was in the shop area and in view of the office window to the shop. I was getting ready to call the electrician in. Got a cup of coffee and went to update the client.

As I am standing there, I see one of the guys plug a vacuum cleaner into the APC and turn it on. Damned if the system didn't crash at the same time! Within 10 minutes, the APC was mounted 10 feet up in the air. No problems since.

What kind of business did you say this is?

Ed

Vacuums. But here is the kicker. They do a lot of outbound telemarketing. The service department is closed on Sundays, so there are no vacuums running on Sundays. But the system will still crash when there are only 3-5 people in the office on Sunday doing outbound telemarketing. It is possible that they are also getting inbound calls to the Auto Attendant from the multiple help wanted ads they run each week.

Do you use a power analizer yet?

OK, I've got a customer who had these two reset events in the error log:



The first one is a power cycle, the 2nd is the same thing you are chasing. (Software Initiated)

You've replaced the processor - did you just swap the PC card and restore? Or did you reprogram from scratch?

I once had a system that somehow got 2 - Line 3 buttons on a set! Where line 4 should have been, was a 2nd Line 3 button. When the user tried to press "line 4", the system restarted.

If you did a restore, I would back it up, default it, and reprogram from scratch.

Still doesn't explain why a second stand alone processor with nothing connected to it sitting at the same location does the same thing.

Tommy, I can understand your point but you have a power cycle not a power disturbance. What caused that second event? Were you able to determine it?

I think there's more going on here than what we are being told and Jim @ DTS is able to determine.

-Hal

I worked in a city that had its own power company
and they created major problems for computers and
phone systems....hi & low power cycles, outages,
spikes, etc. We were replacing power suppies and
ckt cards all the time...the only relief the customer got was getting power filters installed
at the power panel. This stopped the damages.

Lets look at this problem from another perspective.
You might have a jack without a phone that has damage either by corrosion or phisical damage. This jack might be still connected to Partner. Is there same amount of cables plugged in into ACS as the amount of phones?
Also are you an Avaya BP? if you buying Avaya business phone system from anybody other then Jenne or Catalist you can very easy have bad processors in your hands.

Maybe one, but odds of three bad processors is small. Even from a non-authorized source.

If it were a bad jack, how would that affect the 2nd processor that has nothing attached, in the same room, exactly the same problem?

Power Analyzer? Do it and get this off your hands.

ED

Exactly. Do a power analysis to confirm or rule out the obvious before you jump to any more illogical ideas.

-Hal

The cust is responsible for power. You are helping them resolve the issue. Get the power anyalizer and let the customer pay for it. This should resolve the problem and move on.

It is not the power.

Customer continued to refuse to pay for a power analyzer. Eventually, we found a company willing to lend us one, hoping they would be able to prove it was the power and sell the customer a high-end UPS. Power came back just fine. System has been shutting down at least once a day, but with more than 10 days of power monitoring we have seen no relation between any power fluctuations and the system crashing.

I was leaning towards and really hoping we could prove it was the power. The fact they put in a new circuit and it went 2 weeks without going down certainly pointed in that direction. But unless the APC UPS monitoring software is wrong and the power analyzer reporting is wrong, it isn't the power causing (at least directly) the crashes.

Thoughts? Suggestions?

Thanks.

Only other possibility is RF.

-Hal

I realize this is a few years old, but I thought I should go back and update this story. After denying it was a software issue for over 18 months, Avaya finally admitted it was a software probelm and issued a fix for it.

Refresh my memory after all these years. Were these ACSs all R7s?

-Hal

Thank's for the update.