• Date
    Casey
  • Date
  • Replies 15 replies
  • Subscribers 109 subscribers
  • Views 15798 views
  • Users 0 members are here

Net3 ACN Gateway Crashing - N34G-4OUT

Hey All,

I'm a dealer as well as a freelancer here in Chicago.

Today I'm out at the set of Empire Season 2 helping them shoot a couple of shots in studio.  They are having an issue that I need help troubleshooting.  I sold them the hardware (new Ion, new gateway, and a bunch of other things) around June of this year, so I have a number of vested interests in getting this fixed for them.

They have this gateway setup on a very simple network:  Netgear Prosafe Switch FS108P, Ion 2k, Airport WAP (for Ipad Mini), and this NET3 gateway.  They have an identical setup for some other stages here and are experiencing the same crashing issues.  These rigs are outputting between 600 and 1200 addresses depending on which stage they are on on a given day.  They are on for an average of 15 hours a day, and 5 or 6 days a week, and are power cycled at least once a day, usually twice a day (during moves).

4 times since July 1 they have had a gateway just completely drop connection to the consoles.  The gateway doesn't loose power, it shows as not having connectivity to the console (all four ports flashing).  They have cycled power on the entire control rig (including switches and consoles) and had it come back up.  This isn't during fades, large changes or anything.  Just sitting idle for hours, and then bang, nothing.  I'm not clear if the dimmers maintain state or turn off, but I can find out if that would inform the situation.

Today being a shooting day, there isn't much I can do to troubleshoot, or really even poke around in the GCE.  However, what should I have them look for?

The first things I thought of to check is:

Turn off RDM

Check ethernet cables

Ensure static IPs

Cycle power routinely (but they say they do already)

What else should I recommend.  It is a pretty new node, and in fact, if I had to guess, it likely isn't the node that is the issue.  To me it sounds like a networking issue.  The FS108P is unmanaged, but either way if it was working great, I wouldn't think that switch would dump connectivity for no reason.

Also, this happens to both rigs, which are both almost completely new, so it sounds like a config issue more than an anomaly.  Is there a firmware update for these gateways?  What else am I missing.

Tech support, if you want to reach out, I can send SNs, software versions, or whatever else you might want to know.

Thanks all,

Casey

  • Hi Casey,

    can you list following Information for each product:

    1a.) Ion IP:

    Subnet Mask:

    Default Gateway:

    1b.) Gateway IP:

    Subnet Mask:

    Default Gateway

    1c.) Airport WAP IP:

    Subnet Mask

    Default Gateway

    1d.) Ipad Mini IP:

    Subnet Mask:

    Default Gateway:

    2.) What protocols are enabled in Ion?

    3.) What Eos Software is installed on Ion ? Eos v2.3.1?

    4.) What Firmware is installed on Net3 Gateway? v6.0.1 is current - best to Update Gateways to this (with UpdaterAtor)

    5.) Do all Show files do this? Could you mail me a copy of one of the Showfiles? cc (a-t) etcconnect (d-ot) com

    6.) Gateway is getting power from PSU or only PoE from Switch?

    7.) Could you send the Ion logs and date/time when such a drop out happened?

    Good Luck!

  • in reply to corey cascio

    Hey Corey,
    I'll send you what I can now. The rest might have to come later when we aren't shooting (edit: I'll edit this with more info as it becomes available)

    1a.) Ion IP: 10.101.100.101 (static)

    Subnet Mask: can't check now

    Default Gateway: can't check now

    1b.) Gateway IP: 10.101.50.101  (it is set to get an IP from DHCP (from the console, I presume).  Also has set to maintain IP on reboot.)

    Subnet Mask:  255.255.0.0

    Default Gateway  0.0.0.0

    ---It also has set the update IP to be 10.101.100.101 so that they can update the software from the console, I presume---


    1c.) Airport WAP IP: 10.101.50.104

    Subnet Mask 255.255.0.0

    Default Gateway can't check now

    1d.) Ipad Mini IP: DHCP from Airport WAP

    Subnet Mask: 255.255.0.0

    Default Gateway:

    2.) What protocols are enabled in Ion? Only sACN and DMX

    3.) What Eos Software is installed on Ion ? v2.2.1 I don't know why they haven't updated yet. I'll ask.

    4.) What Firmware is installed on Net3 Gateway? v6.0.1 is current - best to Update Gateways to this (with UpdaterAtor) v6.0.1.9.0.4

    5.) Do all Show files do this? Could you mail me a copy of one of the Showfiles? cc (a-t) etcconnect (d-ot) com I'll ask for permission to do so.

    6.) Gateway is getting power from PSU or only PoE from Switch? PoE from switch. That switch has a budget of 53W and the gateway is the only device it is powering via PoE.

    7.) Could you send the Ion logs and date/time when such a drop out happened? I cannot, but have asked that if it happens again that they do so.

    EDIT: Updated 10:13am with GCE details.  Also, they had RDM enabled on all ports.  I disabled it.  And I got permission to send over the show file so I will do so now.

  • in reply to Casey
    Corey and I are working through this via email. It sounds like a number of small setup issues working together to occasionally create IP address problems. I think the largest solution will be to change everything to static IPs, but there are a few other small suggestions Corey gave us to put the network more in line with best practices.

    I just wanted to post back here that we are getting the support that we needed.
  • If you continue to have problems... Try connecting the console to gateway directly, with a network crossover cable, and powering gateway off of a wallwart. (Thus eliminating the switch and poe, for testing...)
  • in reply to murrro

    Just for your reference and convenience:

    A cross-over cable is not required with any of our current generation consoles or gateways due to Auto MDI-X (https://en.wikipedia.org/wiki/Medium-dependent_interface#Auto_MDI-X

  • in reply to marcusbirkin

    Is the more info on this thread? I have a brand new system which is crashing, network error sounds very similar.  I have Ion, 4-port and 2-port Net3 gateways, net gear power safe 8-port network switch with 4-port PoE.  All cat5e has been checked for standard B and length.  IP addresses are in the correct 10.101.range.  i appreciate any insight!  I have not even be able to build a show yet because I can't get the system stable.  Thanks. 

  • in reply to KarenG

    Hi Karen,

    What is crashing? Is the Eos Software on Ion crashing? 

    Could you list all three parts of the Ion, and Gateways here? 

    Network Address Settings: 

    Ion IP:

    Subnet Mask: 255.x.x.x

    Default GatewaY: x.x.x.x

    Thanks, 

  • in reply to KarenG
    Hello Karen,

    What is the exact model number of the switch?

    Are the symptoms by any chance: works for awhile (about 5mins) then stop?

    Regards,

    Marcus
  • in reply to corey cascio
    Corey,
    This is Philip Wheeler, a tech working on this KarenG. This is Gamut Theatre install (#300439700, Job #15037024)
    Ion is not crashing - the 'crash' is all lights out, then the Ion still functions but nothing happens
    Two Net3 2-port Gateways, one Net3 4-port, all in PoE side of the switch, Ion and dRD12 unpowered
    Still haven't confirmed IP address of the NetGear switch - all others (as KG stated) are in correct 10.101.xx.xx range
    Subnet Masks are all 255.255.0.0
    IP Gateways were all 0.0.0.0 (or 1.1.1.1, can't recall @ the moment)
    BUT...all are Dynamic. A very clear next step to me is try with Static IP in the Gateways.
    RDM Disabled on all Gateway outputs
    If I can cause the light dump with any change to the control pathway - which I can - then (I'm guessing) anytime Dynamic IP considers a change it could be a cause.
    All Gateways are PoE using the NetGear switch. Another next step could be wallwarts.

    Once set, what must be altered in the Ion to ensure good continual control comm?
    What can we check/confirm on the Ion to establish/verify this?
  • in reply to marcusbirkin
    Netgear Prosafe GS108PE
    Main symptom: works most of the time, but some change is attempted and all the lights go out - see also my reply to corey
  • in reply to PhilipW

    Hi Philipp,

    Could be an issue with settings on Managed Switch. I recommend unmanaged Switches. 

    Could be an issue with a same IP Address on multiple devices. 

    Could be an issue with the EOs Software or Ion Hardware.

    How often does this issue occur? How long does it last? How does problem go away? 

    Would be good to get sACNview Software running on a PC in network when problem occurs to see if console is sending sACN. 

    - Set all IP Addresses to Static in 10.101. Subnet with Subnet Mask 255.255.0.0 and Default Gateway/Router IP of 10.101.1.1. 

    - Create a Network Plan of all devices and how they are connected and their network address info. 

    - Disable DHCP Servers in Network

    - Look at Settings in Switch. 

    - If problem continues, send us results of sACNview when problem is occurring plus Eos Logfiles, with time/date of occurrence and answers to above questions. 

    - PoE wouldn't be an issue here. 

    Thanks and Good Luck!

  • in reply to corey cascio
    Corey - thanks so much!
    The problem occurs seemingly randomly, but we've seen it happen with fixture manipulations (e.g. getting a mover up & running) or something that causes a solid level of signal through the system. Once the lights go out, Ion can no longer perform any functions with the fixtures until the P-ACP is reset or the whole system is cold-booted (power cycle of Ion only does not fix)
    - Managed vs. Unmanaged switches?
    I'm working on getting an sACN view, working on getting info on the switch (software''s not cooperating)
    Gonna call to confirm tinier questions on configuring the gateways, then do that.
    Still wondering what confirming I can be doing on the Ion setup/configuration regarding this issue
  • in reply to PhilipW

    Paradigm is also involved?

    Maybe the Paradigm is causing issues? Who programmed the Paradigm? They need to be involved in this troubleshooting process. 

    Alone that you say that the problem stops when ypu reset Paradigm, points to an issue involving Paradigm. 

    You could set the sACN Priority of Ion to 200 and see if problem stops. This should only be done with caution though - not that this interferes with the Paradigm's control of certain address like relays or such. 

    And then afterwards you need to determine where the issue is. 

    As suggested - you should involve the person who programmed the paradigm here. 

  • in reply to PhilipW

    Hello Philip,

    The switch will be at fault in this situation.

    This range of Netgear switches are designed as edge switches in small business, not as standalone products in a lighting network. As such the default configuration will cause problems.

    You will need to download the "NetGear Prosafe Plus Utility" can configure the swich. The option you want to disable is "IGMP Snooping"

    The manual for "NetGear Prosafe Plus Utility" can be found here: http://www.downloads.netgear.com/files/ProSafe_Plus_Utility_3_0%20_UM_23Sept10.pdf

    Technical explanation on why these switchers don't work out of the box:

    As stated above, these switchers are designed as edge switchers, intended to extend a network in a office when you need an extra port or two. It is there intension that a central core switcher is used.

    The switch therefor doesn't want to do anything that the core switcher should be in charge of.

    IGMP Management is the function of routing multicast (in the case of a lighting network - the ACN and sACN data) to the correct location. It requires two functions to operate: IGMP Querier (Only 1 per network - On the core switch) and IGMP Snooping (enabled on every device). With out the IGMP Querier, things will work for a short time and then fail.

    These Netgear switchers don't support being an IGMP Querier as they expect this to be done by the core switch, however they do, by default, enable IGMP snooping (So that they are easy to install).

    Regards,

    Marcus

  • in reply to marcusbirkin
    Corey & Marcus:

    We've disabled RDM and set Static IP addresses for all the Net3 Gateways in the system.
    (Most were already disabled, but all of them had been dynamic IP)

    We've also disabled DHCP on the Ion console.
    This appears to have stabilized the system, as we've had no dropoffs since.

    We were able to make these fixes and stabilize the system largely through your collective help here in this forum, and with extensive assistance from the Tech Support folks (special thanks to Ezra who worked patiently with me on the phone last Friday). Thanks you , all, for your expertise and assistance!
Related

www.etcconnect.com