Obsession memory issues?

    Hi All

I have an Obsession II SPS with what I believe might be memory issues that are showing themselves in two areas.

A little background on the system: the processor lives in the dimmer room, not by the faceplate, and is not accessible (or not without a bit of "effort") by the students here.  The console is rarely shut down.  This has not been a problem until recently, although ever since the system was installed in 2002, the console would crash if it wasn't used for a few weeks.  It didn't used to be a problem.  Reboot of the system took about 3-4 minutes and away we'd go. 

The first problem: 

Recently, however, boot times are all over the map, sometimes taking as long as 20 minutes to reboot.  I called Tech Support about this issue.  I had assumed that due to a couple recent power outages that the system had been corrupted by the sudden shutdown.  Then it happened without a power loss.  The message logging is enabled, and at one point got to be quite high, topping 500k in size.  The tech (don't remember his name) said to clear that out as the system doesn't like large chunks of memory used like that.  I cleared the log as directed.  The boot was somewhat faster, but not markedly so.  Fast forward a few months.  The console had crashed a couple of times, again after not being used for some time.  Thankfully, up to this point the thing hasn't crashed during a show (knocking wood).  When I rebooted the system, the message log was a smallish 48-50k, but the last line was a stream of random characters and then looped back to the beginning.  I cleared out the log and went on with my life as usual.  The last time I booted the machine, a student had shutdown the system.  I was expecting a long reboot, but noooo....  it booted in a chipper 3-4 minutes.  I'm starting to think it's possessed at this point.

The second problem:

 Lately saves to disk are taking an extremely long time.  This afternoon I saved a show with a patch, 2 moving lights patched, and 4 dummy cues and it took 4 minutes to write to the hard drive.  Now I *know* this isn't a full drive even though there is the final version of every show file since the system was installed. The 40GB disk that's in the machine should be able to hold hundreds of thousands of show files based on the size of the average file.

 I'm concerned that since this is an DOS based platform and that the hard drive hasn't been formatted in about 7 years that it could stand a defrag, maybe?  That might answer both of my problems.  Then again, I might be way off base. Our road house has a Dual Processor system that they rarely shut down, and it works fine.  It's only about a year younger than my system.

 I bow to the collective wisdom of the ETC Forum.  Perhaps someone can shed some light on this and/or offer suggestions.  In the mean time, I am going to start regularly shutting the system down and rebooting.  It seems to like that. 

 

Thanks!

....Ron






 

  • Hi Ron-

    Sorry to hear that you're having these problems. I would suggest you get back in touch with either your local support (ETC LA office) or ETC technical support and discuss options for deep-clearing the desk. A defrag may also be in order. If you have a dark period, it may be worth having the RPU looked at either in LA or in the Wisconsin HQ.

    This could also be a hardware problem, but it's best to start with software.

    -luke- 

  • Ron, it's hard to say anything specific about your system from so far away.  But I have to comment that it's unusual to leave a light board on for months at a time.  And when you add in that it's in an inaccessible room (do you have the key, or do you mean you only reboot it from the facepanel?), then I have ask, how does the humidity and temperature in that room vary when the building is unoccupied?  How often have the rack filters and the machine interiors been cleaned?  How long can anyone expect a constantly running hard drive to last?

    The comparison isn't very relevant, but I'll comment that after many years of use, my Obsession I had very little fragmentation of the hard drive.  The test program I ran (without ETC's permission) suggested that defragmentation was not necessary.  On the other hand, in modern hard drives, it's very hard for the software to see what the actual geometry of the hard drive is-it's concealed by the drive's interface.

    You aren't specific, but much of your post sounds like LightNetwork posts from high school students who are not allowed to change the lamps in their lekos, and they don't have any spare lamps anyway.  How can they expect the Obsession to work forever without maintenance?  Do the school busses work forever without oil changes?  (Not directed at you, sorry!)

    Tim

  • Thanks to those of you who have replied to my post.  You all had good suggestions.

    Let me clear up some items from replies to my original post.

    It may be "unusual" to leave a lighting console on for months at a time, but it's not unusual to leave an enterprise grade computer on for that long.  The brain unit of the Obsession II is nothing more than an OEM PC.  Mine happens to be made by Dell.  The machine could just as easily have been a server in some co-location offering forth this very website.  It just happens to have the ETC software on it, which doesn't come anywhere near using all the capabilities of the computer. 

    The room that the processor unit is in happens to be the dimmer room for the theatre. This room is only inaccessible to the students.  I have a key, and use it a lot.  It also houses the ethernet distribution for the ETCNet and the NCE computer.  The room is very clean.  I keep a sharp eye on the gear in the room (it's just down the hall from my office), and needs cleaning only about every other year.  The room is not humid (nor is it overly dry) and the dimmers keep it a comfortable 74 degrees regardless of what the temp is elsewhere in the building.

    As I understand it, the current expected life span of a hard drive is about 5 years, which is the age of my drive.

    Which brings me to what has happened since my original post.  I called ETC Tech Support and explained my problem.  Turns out they read my post so they were kinda familiar with what I needed (Hi Terry!).  Following the manufacturers suggestions, I booted the console into DOS and ran Scandisk.  That was the straw that broke the hard drive.  The scanner lit up like a Vegas casino and then ground to a halt.  Can you say hard drive failure?  I knew I was in trouble when the first cluster showed a surface error. The machine no longer recognizes that it has a hard drive. The machine is getting packed up and shipped to refreshingly brisk Wisconsin this afternoon.

    Tim, even though you said the last part of your quote wasn't directed at me, please allow me to respond anyway.  The lighting design students here are all grad students.  They spend a large amount of time with hands on the gear.  They do everything that needs to be done with the exception of configure the DMX nodes on the system, and the only reason they don't is that they feel they have enough to worry about and leave it to me to do.  I don't expect any of my gear to last forever, and with seven venues under my jurisdiction, I have a lot of gear go south.  For budget reasons I try to do as much repair in-house as I can.  And yes, I am specifically vague at times in my posts.  Mostly it's because I don't want to mire my posts with tedious detail.  I wasn't expecting to find "the answer" via my post, merely to see if others have had similar experiences.


    ....Ron

     

  • Ron-

    I just read your post and "the bells went off"--you had the same symptoms I had and it looked like you were about "due" for a new HD. (our symptoms only showed for about 3 days before the final crash though)  Our second HD is on about year 4 of service...

    (we have the same system and went through the same symptoms...) our original HD lasted just under 4 years...now just keep an eye on that internal master power supply, ours went at year 6.5....

      

  • Scott, it's not a very useful comparison, but the original factory-installed "Quantum Lightning [sic] ProDrive" in our Obsession I is well over 12 years old.  It certainly runs 250 to 300 shows a year, and about as many rehearsals.  But it's probably turned on (daily) for only an average of 50-60 hours a week, 10 months a year.  (I do think it's starting to make bearing noises.)  They don't make them like they used to.

    On the other hand, the Western Digital (...name brand) in my Dell P3 desktop failed almost the day after the Dell warranty of one year.  Luckily, Western Digital had a longer warranty, and I got a new one without cost.

    The point I was trying to make about being left on all the time was not limited to heat and wear and tear.  I didn't say so, but there is no doubt that Obsession I has memory leaks and perhaps other software flaws.  As good as Obsession II is, no one would be willing to say that there are no undetected flaws in the software.  Because there is dynamic memory allocation, it is reasonable to guess that these will get worse with longer run times.  By comparison, it's universally recommended to hard-Reset a Hog II at least weekly.  And I can tell you that the first thing I do when my Obsession I gets touchy is to do a 7-8-9 hard-reset, even though it's booted daily.

    I'm not "proud" to be Windows® user (!) But there's no doubt that Windows 98® had internal time counters that crashed the system after (I forget the precise number...) 48 days or so!  My wife is in the software quality business, so I think I can say without fear of refutation that no academic or professional computer scientist believes that software defects can be reduced to zero.

  • Quantum was actually a highly respected hard drive manufacturer back in the day.  They were bought up by Maxtor about 10 years ago.  As I recall, high-end and server-grade hard drives were their specialty.  Apple used a lot of Quantum Fireball hard drives in their early Power Macs and probably the late-model Quadras.  "Lightning" was, in fact, the model name of the off-the-shelf drive that ETC used at the time.  :-)

    You're right though: They don't make them like they used to.
     



    [edited by: mgarrison at 10:00 AM (GMT -6) on Fri, Mar 14 2008]
  • Hi All,

     

    Just thought I'd send an update slash resolution to my Obsession conundrum.

    I send the brain back to sunny Wisconsin.  They replaced the hard drive, floppy drive, the motherboard battery, and the CPU fan.

    They even managed to save some of the show files that were on the old HD, although I really was only interested in the two rep plot files (which were readable, but I had to dig to an earlier saved version to find one of them).  The system crashed while saving a showfile to the faceplace floppy drive, but a reboot fixed the problem.  I should mention that I haven't seen the processor boot up this fast since the console was new. Yay!

    I wasn't aware that there was a problem with the floppy on the processor, but I guess that explains why I couldn't back anything up using that drive just before the thing went south.  I'm also not sure why they put a new CPU fan in as the fan that was in there was only a couple years old.  IThe fan was replaced when it made more noise than the nine Sensor 48 racks it shares the room with and you could hear it screaming in the hall outside the dimmer room.

     Anyway, thanks to the tech folks at ETC for a fast turnaround and a job well done.  See you again in five more years?  :)

     

    ....Ron


     

  • tbuchman said:
    And I can tell you that the first thing I do when my Obsession I gets touchy is to do a 7-8-9 hard-reset, even though it's booted daily.

    What's a 7-8-9 hard reset?

    Will King

    Asst. Electrician, Phantom of the Opera, Music Box Company

    2 Obesession Processor Units running software version 2.4.3 in Full Tracking Backup mode

    DMX Output Selector

    Playback Console

    Submaster Console

    All connected via ETCnet through a 3com ethernet hub

  • a little more aggressive than the strand 2-4-6  :)

  • In a 7-8-9 reset, you place three fingers on those number keys and depress them, and then turn the power key switch on. You continue to hold down the three number keys until the board boots up. It will then look similar to a board that's had a System Reset done from the Setup menu, but Victor Seastone told me that it is a more thorough clear process.

Related