i960InFifoEmptiedTimeout

OBS made an unexpected yet rather graceful exit, while idle; appears to be functioning normally after cycling power.

i appreciate that not every buffer flush can be logged, so there's no way to conclusively analyze this instance. i wonder if there are any known causes, common or otherwise?

i will have the unit opened to replace the battery, and will be restoring the cmos file; is there any maintenance specific to the 960 board that should be performed while i'm in there?

 

12/16 09:45:04 112 I/O device online: type=23 addr=00:c0:16:00:21:b5
12/16 09:45:04 003 Phantom Toggle is Off
12/16 09:47:22 112 I/O device online: type=11 addr=01:00:00:00:00:00
12/16 09:47:22 003 Phantom Toggle is Off
12/16 10:12:01 299 p83isa: i960InFifoEmptiedTimeout
12/16 10:12:01 299 p83isa: Isa960Failure()!
01/01 00:00:47 100 Temporary network disable call failed
01/01 00:00:47 000 Initializing system
01/01 00:00:47 000 Software Version: 5.1.0.9.0.50 Build: 1057789153 Size: 1536
01/01 00:00:47 001 Shadow buffer 0x011628F0 Shadow pointer 0x01163000
01/01 00:00:47 120 CONFIG.DAT - Message Logging enabled
01/01 00:00:47 120 HRDSET.DAT - Hours from GMT -8:00
01/01 00:00:47 120 HRDSET.DAT - Daylight savings option selected
01/01 00:00:47 120 CONFIG.DAT - IP Address 10.101.30.101
01/01 00:00:47 120 CONFIG.DAT - TCP/IP Subnet Mask 255.255.0.0
01/01 00:00:48 120 CONFIG.DAT - Gateway IP 10.101.30.101
01/01 00:00:48 120 CONFIG.DAT - DMX Data Distribution IP 236.1.0.39
01/01 00:00:48 110 Slave is iPentium
01/01 00:00:48 120 Setting time zone TIMEZONE=LOC::480:040102:102802
01/01 00:00:48 120 HRDSET.DAT - ETCNet2
01/01 00:00:48 120 HRDSET.DAT - fileserver disabled
01/01 00:00:48 120 HRDSET.DAT - Redundant Tracking disabled
01/01 00:00:48 400 Exit FTB Tracking Indeterminate
01/01 00:00:48 400 Enter FTB Primary Power Up
01/01 00:00:48 120 HRDSET.DAT - Tracking Unit A
01/01 00:00:48 120 HRDSET.DAT - DMX from backup unit on
01/01 00:00:48 120 HRDSET.DAT - Offline Editing Disabled
01/01 00:00:48 120 HRDSET.DAT - Default Printer Letter
01/01 00:00:48 001 Initializing Network Interface
01/01 00:00:50 004 BAD CMOS FILE EXISTS!
01/01 00:00:50 004 AND CMOS IS STILL BAD!
01/01 00:00:51 299 Successfully downloaded isa960io.hex to ISA i960
01/01 00:00:52 299 Getting channel count from ISA i960
01/01 00:00:52 299 Sent channel count request
01/01 00:00:52 299 System Channel count is: 1536
01/01 00:00:52 299 Reset ops
01/01 00:00:52 100 Tracking system starting up
01/01 00:01:13 221 New Device at addr=df:71:7f is of unknown type.
01/01 00:01:13 400 Exit FTB Primary Power Up
01/01 00:01:13 400 Enter FTB Acting Primary
01/01 00:01:13 221 Device at addr=df:71:7f is of unknown type.
01/01 00:01:13 299 Rediscovered device, addr=00:04:76:df:71:7f
01/01 00:01:13 001 File  not found on disk for network file write
01/01 00:01:13 111 Multiuser enabled
01/01 00:01:13 299 etcLinkInitFailed = false
01/01 00:01:17 288 CDisplayEtherNet::OpenVideoSystem
01/01 00:01:17 299 Sending SMPTE_RECV_DISABLE message to ISA i960
01/01 00:01:17 299 Sending SMPTE_RECV_DISABLE message to ISA i960
01/01 00:01:17 299 Sending SMPTE_RECV_DISABLE message to ISA i960
01/01 00:01:17 299 Sending SMPTE_RECV_DISABLE message to ISA i960
01/01 00:01:18 095 Clearing all
01/01 00:01:18 299 Sending DMX_PARAM_SET message to ISA i960
01/01 00:01:19 299 Sending SMPTE_RECV_DISABLE message to ISA i960
01/01 00:01:19 299 Sending SMPTE_RECV_DISABLE message to ISA i960
01/01 00:01:20 299 Sending DMX_PARAM_SET message to ISA i960
01/01 00:01:20 299 Sending DMX_PARAM_SET message to ISA i960
01/01 00:01:20 299 Sending SMPTE_RECV_DISABLE message to ISA i960
01/01 00:01:20 299 Sending SMPTE_RECV_DISABLE message to ISA i960
01/01 00:01:20 299 Sending DMX_PARAM_SET message to ISA i960
01/01 00:01:20 299 Sending DMX_PARAM_SET message to ISA i960
01/01 00:01:20 221 Device at addr=df:71:7f is of unknown type.
01/01 00:01:20 299 Sending DMX_PARAM_SET message to ISA i960
01/01 00:01:20 299 Sending DMX_PARAM_SET message to ISA i960
01/01 00:01:21 299 Sending ISA_IO_ENABLE message to ISA i960
01/01 00:01:21 300 seeking Net1/Net2 beacon: 1
01/01 00:01:21 Init task messaging
01/01 00:01:21 221 Device at addr=df:71:7f is of unknown type.
01/01 00:01:21 001 Doing Reconstruct()
01/01 00:01:21 095 Clearing all
01/01 00:01:22 299 Sending DMX_PARAM_SET message to ISA i960
01/01 00:01:22 299 Reset ops
01/01 00:01:22 299 Sending SMPTE_RECV_DISABLE message to ISA i960
01/01 00:01:22 299 Sending SMPTE_RECV_DISABLE message to ISA i960
01/01 00:01:23 299 Sending DMX_PARAM_SET message to ISA i960
01/01 00:01:23 299 Sending DMX_PARAM_SET message to ISA i960
01/01 00:01:23 299 Sending SMPTE_RECV_DISABLE message to ISA i960
01/01 00:01:23 299 Sending SMPTE_RECV_DISABLE message to ISA i960
01/01 00:01:24 001 Restoring groups
01/01 00:01:24 001 Restoring Effects
01/01 00:01:24 001 Restoring cues
01/01 00:01:28 001 Restoring subs
01/01 00:01:35 299 Sending DMX_PARAM_SET message to ISA i960
01/01 00:01:35 299 Sending DMX_PARAM_SET message to ISA i960
01/01 00:01:36 299 Sending SMPTE_RECV_DISABLE message to ISA i960
01/01 00:01:36 299 Sending SMPTE_RECV_DISABLE message to ISA i960
01/01 00:01:37 299 Sending DMX_PARAM_SET message to ISA i960
01/01 00:01:37 299 Sending DMX_PARAM_SET message to ISA i960
01/01 00:01:37 001 In Reconstruct() SRAM checksum = aef2
01/01 00:01:37 001 After Reconstruct()
01/01 00:01:37 299 Initializing Net2 Video
01/01 00:01:37 299 Net2 Video initialization complete
01/01 00:01:39 299 GLOBL_Initialization complete = true
01/01 00:01:39 300 seeking Net1/Net2 beacon: 2
01/01 00:01:40 000 Initialization completed
01/01 00:01:44 300 seeking Net1/Net2 beacon: 3
01/01 00:01:49 300 seeking Net1/Net2 beacon: 4
01/01 00:05:10 112 I/O device online: type=23 addr=00:c0:16:00:21:b5
01/01 00:05:10 003 Phantom Toggle is Off
01/01 00:05:10 112 I/O device online: type=23 addr=00:c0:16:00:21:b5
01/01 00:05:10 003 Phantom Toggle is Off
01/01 00:05:18 299 Sending DMX_PARAM_SET message to ISA i960
01/01 00:05:18 299 EtcLinkAddress->IsValid() = false
01/01 00:05:18 001 I/O Selector Output: primary



[edited by: quinn at 1:26 AM (GMT -6) on Tue, Dec 21 2010]
  • quinn, if you can't wait for a staff reply, consider this link, after making sure it matches your console:

    http://community.etcconnect.com/wikis/products/knowledgebase-obsession-1-cmos-setup.aspx

    Have you gently vacuumed lately? Although you have to be very careful of the vanes, I cleaned the cooling fans once a year with damp Q-tips. The dust deposits have to decrease air volume.

     

  • Ooh yeah, finally a good ol' obsession question I can sink my teeth into, heh heh (I work on these if you can't tell).  First off, if you don't already know, you can reset your cmos easily if you have Vsn5.1.1 or 5.1.0 by going into DOS (hold left shift on keyboard as it boots) and typing OBSNCMOS at the DOS prompt (or you can use a boot disk).  This program will reset your CMOS for you.  This program can be easily copied to a floppy and used on ANY version of software by the way.  But you will still have to reset the time manually of course or you will get a CMOS TIME AND DATE NOT SET error.  Anyway, it may very well be your CMOS battery that is the cause, but the following error lines are not actually errors;

    01/01 00:00:50 004 BAD CMOS FILE EXISTS!
    01/01 00:00:50 004 AND CMOS IS STILL BAD!

    People often see this and think it is an error.  It is not.  These lines are part of the obsession program checking to see that the cmos is set correctly.  Why it was written to say that the CMOS is bad I do not know (something to do with Y2K), but these lines come up in every boot in any version.  In Version 5.1.x the OBSNCMOS.EXE program was included which will reset the CMOS automatically, which may be why it fails the first boot but not the next.

    The second most common cause of ISA960 failures is low DC voltages.  Open the case and find the main motherboard power connector.  Put a meter on the red and black wires and if it is any less than 5 volts, there is a very good chance that voltage is dropping over a DC connection.  With the meter still on the main DC 5V line, wiggle the power connector coming out of the power supply on the power distro board.  Then wiggle the hard drive connections.  If you see the voltage suddenly pop back up to proper voltage, then you have some corrosion on the connections.  The biggest killer of almost all electronics is corroded low voltage DC connections and the 5 volt line is the first to be hit, which makes the ISA960 card the first thing to go.  I replace the main DC power harness on older models all the time, but I also eliminate the CPU fansink's hard drive power splice connector, which also drops voltage very commonly.  I put the power for the CPU fan now on the motherboard's fan connection beside the RAM.  If you want to make your own on your old fan, the pinout of the mobo's fan connector is GND/12V/GND, so just make sure the red wire is in the middle.  Otherwise, if your processor hasn't seen repair in a while, it would be a good idea to send it in, and I will go over it with a fine tooth comb, resolder a few things and make sure that all the inherent flaws and wear are dealt with.  It is usually a fairly inexpensive repair unless I end up needing to replace boards, which is rare.  If I find errors on your hard drive, it could be a little expensive, but not too bad.  Most of the repairs I do on Obsession 2s are free.

    For your final question, no, there are no user serviceable items on the ISA960 card to check other than makings sure connections are tight.  If the card edge is lifted near the edge of the case, look underneath to make sure than the bottom of the card's metal flange is not stuck on the case, although I've never seen this be an issue.

  • Oh yeah, one more thing, and I cannot say this enough. 

    Make sure that you have less than 50 shows saved on the hard drive and that your message log is purged periodically!!!  The obsession may say that it has 96% of space left, but the ISA card only has so much RAM which it uses to keep track of show names and message logs in addition to all the show info.  Remember this is a very old computer now and it has limitations.  When it reaches these limitations, it starts exhibiting some pretty weird software glitches, sometimes ghosting channels or subs, flickering lights, adding channels to cues, or just plain locking up.  This is usually a major problem with tracking backup systems, as it checks both hard drives and compares show data on boot which loads up the RAM on the ISA960 and locks up.  But, I have seen obsessions with over 1000 shows run fine and then some that have just over 100 shows that lock up.  Usually the problem occurs because of the lack of separate SAVE and SAVE AS functions, and a new copy of the show is saved each time, making a list of show revisions (by design).  So just keep an eye on your show number and periodically delete unused show revisions, and don't forget to purge the message log once in a while.



    [edited by: pflaniga at 4:07 PM (GMT -6) on Tue, Dec 21 2010]
  • tbuchman said:

    quinn, if you can't wait for a staff reply, consider this link, after making sure it matches your console:

    http://community.etcconnect.com/wikis/products/knowledgebase-obsession-1-cmos-setup.aspx

    Have you gently vacuumed lately? Although you have to be very careful of the vanes, I cleaned the cooling fans once a year with damp Q-tips. The dust deposits have to decrease air volume.

     

    Just FYI, I know it wasn't clear in the original post, and I only know it because I'm familiar with the message log, but the link you referenced is for an Obsession 1.  Quinn has an Obsession 2.  Just wanted to deter any confusion.  As a matter of fact, now that I look at that page, it needs some serious revision, which I will attend to now. . .

  • i'll set triggers on the +5V and GND lines for a few days to attempt to confirm a distrubance prior to rerouting as suggested.

    i don't doubt that this system will continue to operate correctly after applying the excellent information that's been provided; thank you kindly.

     

    though minimizing the working set of showfiles and logs absolutely is sound, good advice, i would expect that an out-of-memory exception would be thrown and handled not when the FIFO in question is being "emptied", but rather when it is first allocated, which in this instance would have been just under 4.5 billion msec earlier.

    so, if you're still in the mood for feasting, i've prepared a second course for you. feel free to bring a friend from dev/test:


    Isa960Failure() is called by the i960InFifoEmptiedTimeout handler, which is invoked when its associated timer times out.

    is that timer created with a fixed timeout value?

    if so, is that timeout value of a duration that suggests that the timer is in place to trap:

    component-level failures, such as a sag->reset on, say, 485 transcievers powered by the ISA bus?
    single function performance, such as a for loop iterating over, in this case, 80686 (!) lines of message log?
    long-running logic, such as an Acting Primary's attempt to transfer while in an SPS configuration?



    [edited by: quinn at 6:38 PM (GMT -6) on Wed, Dec 22 2010]
  • Ok, I realize that you really want to catch the source of this little bugger, but honestly, I have never had this level of understanding be helpful to fixing problems.  This is partly because, even if it was, it would not be helpful to fixing it when I cannot even get to the card to do the physical troubleshooting required, and I don't have access to a very complicated logic analyzer (I have one, and have used it to find problems occasionally, but it is limited).  Even if I did have the tools and the ability to get the card in a position to troubleshoot (it will not work on a riser or slot bender due to timing issues), doing so would require a lot more time and frustration than simply going after more obvious problems. 

    If the suggestions previously mentioned turn out to not be the source of the problem, then either you have cold solder joints on one or more chips, or you have a bad chip (or both).  Although it doesn't happen a lot, ISA960 cards are notorious for intermittent issues.  Most of these I have found to be attributed to the aforementioned voltage or show related issues, but occasionally I come across a card that simply doesn't want to work consistently.  If I do manage to get the problem to occur consistently, it is usually solved with a simple reflow of the solder on the card (mostly the 2 microprocessor chips that have tiny pins).  I also look for chips that are getting warmer than usual (or sometimes have a tiny dot of physical damage where a transistor blew) and replace them. 

    In the other 1% of most instances, I just end up replacing the card.  I cannot send a repair back to a customer with questionable, intermittent hardware.  There has been only a couple of instances where I had thought that I had fixed the card when a reflow caused the issues to go away, only to have it come back in again later, and if I still cannot pin down the problem, the card gets replaced for free under repair warranty.  We don't mess around here, and we really hate it when things come back in for repair again, so we make darn sure it's fixed.

    Basically, you are way over my head on the matter, and, although I understand what you said, I have no idea about the specifics of the ISA error itself because I have never needed it.  Programmers in R&D are the only ones who would have known the answer to your questions over 10 years ago, but now, not so much.  Sorry, but that feast is just a little too rich for me and a bit cold for R&D ;).

    Wow, I really blather on and on in these posts, sorry. . .

     

     



    [edited by: pflaniga at 11:48 AM (GMT -6) on Tue, Dec 28 2010]
Related