Two Spaces, Two Ions, Two Unisons, and WiFi

I agree with Steve on the recommendation to split the two spaces and isolate them for each other, and especially from the building network.  Having the theatrical network for each space as self-contained as possible will make troubleshooting easier.  Problems on a mixed network usually manifest themselves as you have experienced: "it was all working fine and then..."

This problem is best handled by calling ETC's tech service and getting on the phone with a support person.  They can pull up the original configs and drawings, work through any changes that have been made over the years and get everything working.

Without seeing the config files and control layout for your system, people in this forum are only guessing at how to help.

-Todd

I agree with Steve on the recommendation to split the two spaces and isolate them for each other, and especially from the building network.  Having the theatrical network for each space as self-contained as possible will make troubleshooting easier.  Problems on a mixed network usually manifest themselves as you have experienced: "it was all working fine and then..."

This problem is best handled by calling ETC's tech service and getting on the phone with a support person.  They can pull up the original configs and drawings, work through any changes that have been made over the years and get everything working.

Without seeing the config files and control layout for your system, people in this forum are only guessing at how to help.

-Todd

I truly thank you guys for taking the time to answer.  Todd,I know people are only guessing, which is precisely why I am giving every bit of information that I have. I am trying to avoid a call to ETC since I am currently in off hours.  Ultimately that was my next step.  I usually as a IT administrator turn to forums for these things because getting a person on the phone who has a clue can be difficult at best ... then again that is with Cisco, MS, and Dell... ETC can be a little different which is something I do appreciate about them.

Firstly, the WiFi works happily with all other aspects of the building networks.  the AP's are all working in harmony and due to the roaming I have setup users can wander around the building and never see a drop off even when switching AP's.  Well technically there is a 5ms delay when switching AP's, but that is invisible to most.  Cisco's backhaul algorhythms are AWESOME!.  So there is no worry of MAC assignments, or WifI channeling issues.  Like I said this WiFi system is an enterprise level system and is function as intended in regards to IP for talking to three other domains in the building.  I really am quite proud of it.

However, By ETC's documentation, this SHOULD work.  And in fact it DOES work flawlessly, but then one console drops off.  WHY? That is the real problem... mostly because it doesn't make sense.  There are no ethernet loops, all the IP's are unique to the device, and there is not excess traffic on this segment of the network.

There is no good explanation for it.  It seems to me there is an IP conflict somewhere, but all the IP setting are correct based on Cicso AND etc netowrking protocols, except I have made a small adjustment in the IP ranges to fit our setup a little better(which etc states is okay if the site requires adjustment).  When I go in Monday, I will set everything into ETC defined IP ranges to see if that clears it up.  If there is a flaw in the ETC software or hardware that doesn't allow this to work as designed, then I am sure we all would want to know about it so it can be fixed... I don't think that is the case here, I really am convinced I am just missing a setting somewhere.  By what I read in ETC documentation I should be able to drop 5 Ions, 10 RVI's, a couple of Net3 gateways configured on different universes, 30 dimmer racks, and 20 WiFi devices running iRFR or aRFR all onto the same network and be able to setup segemented control all over the place...  This works on other protocols and networked consoles I have used, why not on ETC?  I can ping ALL devices except the one Unison control that is physically isolated from the other network (which is expected)...  But when the console drops off I also stop being able to ping it... change physical ports on the switch all works again.

As a building, We have ALWAYS wanted to be unified across the spaces for a variety of reason specific to this building.  Please don't be insulting by saying there is no good reason to do this, when a.) there is and b.) it should work whether or not it is a typical setup or if you think another might be better. 

One reason we want this unification being the NLT space also gets converted into a banquet hall several times a year and at such times we have large full building events that span the entire building and we only have budget for one lighting tech to come in for and we are trying to simplify things for that person and eliminate them having to run all the way across the building to make adjustments.  NLT's Ion talking back  to the CE is not nessecary.  Really we only want one direction, although by default it should work both ways.

Ideally we should be able to drop a Net3 gateway down out in the lobby control some movers and a set of LED's out there, talk to the dimmers and LED's in NLT and run the majority of the show in CE space from one console during these times and be able to have building wide a/iRFR access to the system for focus out in the lobby, NLT, CE, etc.   This SHOULD work just fine.  The WiFi is an added bonus and again... it works beautifully.... then one of the consoles drops off the network and stops transmitting on the network... with no good reason, then when you change ports because the switches send a layer 2 broadcast packet, all works again begin the switch MAC tables update.  That says there is something wrong in IP land.  If I can talk to all the things I want to and it all works as intended, then just stops working with no real rhyme or reason for it, that is a problem and should be able to be addressed.

I don't mind the headache getting this all working... in fact I invite it because I KNOW when I get the proper setting it will work as designed unless the software is flawed.   

Honeslty, we don't mind there are two unison processors. . The problem is this issue manifested itself at install and instead of the installers taking the time to figure it our and make it work as intended they gave up, and severed the network connection and walked away. I understand why they are in place and was just thinking of options to eliminate the unwanted cross talk between spaces.  The unison issue really is of very littel concern... ALTHOUGH, the TD did mention that he wants to be able to administer and manage the dimmer racks, unison, etc from his office which has a single port connection to the dimmer room... and there isn't any reason he should not be able to.  BUt like I said the Unison cross talk is sort of back burner. 

Has ANYONE out there done a full building setup like this?  We do have reasons we want this to work and it SHOULD work... I need to find out why it is not, so I can make it work.

dlderry said:

 ALTHOUGH, the TD did mention that he wants to be able to administer and manage the dimmer racks, unison, etc from his office which has a single port connection to the dimmer room... and there isn't any reason he should not be able to.  BUt the Unison cross talk is sort of back burner.

Again, Why ?.  I manage my system, which is essentially half what yours is (one space currently).  I pretty much never have to deal with my CEM+ racks. They work so I leave them alone.  There is nothing I need to do except a once a year check of software updates.  Ditto Unison.  Any changes I do to Unison are done off-line in the Light Manager software and that's very rare.  The system as configured works, so I don't muck with it and I would never want the CMEi accessible from the network in any event, there's too much possibility for a problem.  

In my opinion, you are asking for trouble keeping the system configured as such.  It is WAY too complicated and that makes it nearly impossible or difficult at best for phone tech. support to help you.  

As to the Ion as one desk for both spaces for key events ?, I'd do a DMX to DMX send of data, I.E, setup back to back nodes, one per network, take DMX out and put it back to the 2nd network.

"In theory I should be able to drop 5 ions onto the system with a couple of RVI's and a bunch of WiFi devices and make them all point the directions I want them to. This SHOULD work just fine. The WiFi is an added bonus and again... it works beautifully."

And am I reading this correctly that you are connecting the consoles to the network via WiFi, as opposed to hard wired ?. 

Here's a quote from Steve Terry on ControlBooth http://www.controlbooth.com/forums/lighting-electrics/29920-irfr-lost-connection.html

"The situation described is one where the device stops working when the 2.4 gHz band gets crowded. In this situation, the user of the mission-critical device (you) has no control over other users in contention for bandwidth. My observation is that people tend to get seduced by the convenience of wireless devices, but when they stop working, those people tend to get amnesia and forget that they cannot own the bandwidth exclusively--it's a limited, finite resource!

That is why I always recommend a wired solution when that is possible, especially is a mission-critical, real time application such as a professional lighting system. This is especially true for DMX512 links or network links carrying DMX512 data.

If you lose the RFU, it's an inconvenience. If you lose control of the dimmers and the movers, it's a disaster."

I truth, I am impressed at your apparent knowledge of your system and the steps you have taken to get it all working.  It is not something I would attempt, as at my level of knowledge I still consider it all voodoo to a degree.  The multiple console setup, as example is something discussed in great detail in the 4 day Ion class I took in NYC (highly recommended), and I would as result be approaching your setup with a good deal more desire to simplify everything.  One over-riding philosophy would be to keep it as simple as possible, if for no other reason that if YOU get in a car accident on the way to work tomorrow, who's going to keep it all running ?.  

You ARe reading incorrectly.  The consoles are hardwired. EVERYTHING is hardwired except the WiFi setup... which is purely to pass aRFR and iRFR app data through and to let designers drop thier laptop on the desk with Eos Client open in event the building's RVI is tied up in the other space.  Consoles, racks, etc in this case will always be hardwired.  Trying to pass a full 32 universes of EDMX or ACN through WIFi is ridiculous, yes that will jam the bandwidth.  I have the LIghtign Wifi on a dedicated channel of the wifi spectrum and our access points also are setup is isolate rogue WiFi data and push it out.  Also they dynamically adjust all other SSID's channels when a channel gets over loaded.   It's kind of funny to watch NLT's ME setup his linksys router and wonder why the signal drops off when he gets more than 10' away from it.  Cisco's Aironet Enterprise WiFi is a BEAUTIFUL thing.  Each AP can broadcast 25 SSID's simulataneously and I got the lower end one.  The big daddies make me want to cream my khakis since they can also sense when a microwave gets turned on and adjust the signal to compensate for it.

The problem though is not with the WiFi at all... The WifI will stay connected to a single console all day long and I can ping all the racks and goods all day long except the one console keeps dropping off the hardwired network.  It is in the hardwired network which in this building has always been a problem.  The installer said he would do a service call and try setting till it worked, but honestly... after talking to him, I have ZERO faith in his abilities as a network engineer especially seeing as his eyes glazed at several points in talking about what I am doing here.  The service call to 'try settings' tells me he has NO CLUE as to how to fix the problems.

And you hit on half the issue... both CMEi's are NOT visible from the CE side of the network, you physically have to go to the dimmer racks and connect to the switch we have isolated... which for us and the size of the spaces is a kind of a huge pain in the ass especially on the full building events.  We have ETCnet dedicate ethernet all over the building, I want to make it all work.

Thank you for the compliment.  Part of this is coming from my knowledge as a CCNA and serving as a Server/IT admin for IBM for a number of years before I returned to theatre.  Simple is usually the best... the KISS principle is almost always in effect, but we want to enable some of the more advanced features in the building because we are starting to have requests for them as a road house and enabling them require us to complicate things a tad. 

I just want to make it all happy sicne my bulding network is happy, I think ETCnet should be too.

Sidenote, we don't want to do DMX hard link because of the cable paths, cable lengths, etc. AND gigabit ethernet is capable of pushing a ridiculous amount of data with error correction, DMX technically has none, they are all broadcast packets (unless I don't understand DMX packets as well as I think I do, which is entirely possible.)  The ethernet has very nice shorter runs hardwired into the building so it is FAR more convienient and once we get working the way it should, will allow us a TON of maneuverability within the building.

That article is correct, losing RFR is inconvient, and I personally would never try to pass DMX in voume we do for show of JUST WiFi.  I will end up calling ETC on tuesday and seeing what thier network folks can tell me.  They were really good about explaining why ETC doesn't support VLAN'ing.

Entertainment networking != office networking.
The goals are different and the usage patterns are different:

• Office networking is generally high-bandwidth-high-latency, everything buffered somewhere
• Entertainment networking is (usually) low-bandwidth, always requires low-latency, nothing is buffered and thus out-of-order packets are discarded. (Can't have a fade go 0% - 100% - 50% -100% instead of 0% - 50% - 100% - 100%)

Thus they should be kept physically separate whenever possible.

A switch port shutting down is often caused by it detecting looped traffic or packet storm (often a symptom of the former).
Obviously STP can do that (although we don't recommend STP due to the increased latency and out-of-order packetry)

I'd suggest grabbing a bit of paper and drawing out both of lighting networks - from your description it's quite a lot more complex than is usual for such a venue.

That will often make it obvious if there's a loop or other mistake somewhere.

However your first step is to define the goal - namely, why do you want these networks joined?

The Why can be answered in this statement by the OP:

"One reason we want this unification being the NLT space also gets converted into a banquet hall several times a year and at such times we have large full building events that span the entire building and we only have budget for one lighting tech to come in for and we are trying to simplify things for that person and eliminate them having to run all the way across the building to make adjustments. NLT's Ion talking back to the CE is not nessecary. Really we only want one direction, although by default it should work both ways. "

Thus they can save labor by not needing an add'l console operator when they only need one desk for the event, just running two spaces.  Very much like a Unison/Paradigm setup for a conference facility when a temporary dividing wall opens to make it all one room.

If it was once a year I'd do it as a DMX out to DMX in, with nodes at the switch rack.  Because it's several times per year I can understand the desire for an easier and more elegant solution.  I would still keep the systems off the general building network as well as isolated from each for day-to-day operations, as well as for ease of troubleshooting and redundancy, but would then set it up as a simple as needed e-net patch of the CE console now controlling both networks, with the NLT Ion off line and powered down,.  That avoids the need to configure Master/Slave for the 2 Ions.  Attention would then be needed for all the IP addresses.  

I think everyone is getting caught up on the office network and wifi, and I can see why... The way enterprise controller based WiFi works is not what you seem to think it is, at least from the responses I have been seeing.  Right now think of it as I have added a single genereic linksys WiFi router to the system to get aRFR and iRFR data into the mix.  The only place the office and ETC network touch is at the single port/interface on the WIFI controller which is configured to talk to ONLY the lighting network.  That traffic does not cross out of the controller.  It never intersects with the business network at switches, routers, etc.  It is easily removable from the situation, and I have removed it from the mix to get the other parts working. From my experimentation it will work just fine once the rest of the network is working as designed.... so lets make it a non-issue.  I think the fact that I am fighting multiple problems is hanging us up in sussing this out... Sooooo lets not even think about Unison and WiFi at this moment.  I forget I am OCD and a little ADD and tend to try and put every detail ut there even when some of them don't pertain to a particular problem.  Right now, I think the first step is figuring out why I have two consoles and one gets booted through the hardline Ethernet dedicated to Lighting.

I am 98% positive there are no PHYSICAL Ethernet loops.  Every device only has one connection and the switches all talk to each other at a single point.  I am going to diagram this out later today after meetings and post, maybe people can see something I am not.

I dug into the switches yestarday and I THINK I might have found something that might be cause a network loopback at the switch level according to ETC's documentation, I need to verify this though.  They say to set all net 3 gear is to have a DG of 10.101.1.1 but they say to set net 2 devices to a DG of the same as the IP address because this forces the device to broadcast to all IP's on that network.  As far as I can tell we should be talking all net 3 anyhow.  Two Ion's, RVI, Censor+.   The switches when installed were set with a DG of the switches IP address.  ie. the booth switch is 10.101.2.1.  The way I understand hardware switches is when they recieve a broadcast packet they forward that to all MAC addresses.  Is it is possible what is happening is

10.101.3.1 is receiving a broadcast packet from 10.101.2.1 then re-broadcasting to all network.  in turn,

10.101.2.1 is receiving a broadcast packet from 10.101.3.1 then re-broadcasting to all network. IN turn,

10.101.3.1 is receiving a broadcast packet from 10.101.2.1 then re-broadcasting to all network.... you see where I am going here.  By ETC's documentation shouldn't the switches have a DG of 10.101.1.1?  I THINK that might be an ethernet loop at the switch level and would fit the criteria set down by Richard to cause port shutdown.

I might not fully understand switching protocols, but without a physical hardware loop (I have been up and down EVERY connection with a fine tooth comb.

We have number of reason for this needing to work.  Full building events are one, the second is lately we have been doing double billed shows and each company wants to bring in thier own console.  All consoles involved speak ACN and Artnet.(Hog 3, Ion, and Chamsys MQ)  We want to be able to drop all the console on the desk connect through ethernet/ACN and have them talk to the dimmers without having to swap DMX hardlines 8 times during a show.  If this only had happened once I wouldn't sweat it, now it is being requested three times or more a year.... So You can see why my frustration level builds because all of this SHOULD work, but has not worked since install.

Again thank you very much for taking the time to read this mess and make responses.  Every idea that pops up at least leads me to being able to try something else.

'D'

Your broadcast/default gateway question:

A switcher will transmit a broadcast packet on all interfaces other than the one it was recived on, if you have no loops then it'll not cause a storm.

Unless you have a routed network (i.e. router/Layer 3 switcher) leave the Default gateways tobe the self IP.

Your unison question:

It sounds like the mimic protocol is causeing problems. If the unison configs where designed with being two sepperate networks, then preset/macro names could be the same on both configs/processors.

The correct way to fix this is to have the configs checked out for duplicate preset names, etc. Ask who ever installed this unison system to check the configs.

The short term/bodge way (only if the processors never need to comunitcate) is to block the multicast address 236.1.0.5 in the switchers - I stress that this is a very drastic and un approved "fix"

Your drop out question:

This could be related to ARP tables in the switcher, does rebooting the switch bring the port back up? Does it have a web config you can view to see the port status?

EDMX and sACN are multicast fire and forget UDP packets, RFR in unicast TCP. With out a valid ARP table the desk could in theory still control dimmers, but appear offline to the remotes.

Your WiFi AP could also have multicast storm control, and in turn temporarily blocking unicast. If you can, filter the multicast packets from the AP at the switcher (be careful to only block the AP port). - This would disable laptop client connections, but I belive that is technicaly unsupported by ETC anyway. Mulicast transmition over 802.11 is very bandwidth hungry.

Or it could be related to STP, which can create funky results if miss-configed.

What model switcher, what settings?

Marcus

There we go Marcus, thank you for answering.  I have actually never powered the switches down.  Let me double check with the questions you asked and I will post opefully tommorow.  Looking at the setup in your signature, I am guessing you too think we can get this all working how it should be.

Will report when I am in the building tommorrow.

'D'

Finally I was able to get in the space and check this out again.

I reset the booth switch and the aRFR and iRFR  started working again, so I suspect we did indeed have a bad ARP table in at least one of the switches.  So one problem down.  We are waiting on some down time to mess with the unisons.  We have been in non-stop techs in the two spaces.

Our switches are all LinSys/Cisco - SRW224G4P

10.101.2.1 - Booth

10.101.3.1- NSC DImmer Racks

10.101.4.1 - NLT Dimmer Racks.

We are still having trouble getting a clietn PC to connect to the console.  At this point... one step at a time.

***UPDATE***

We finally had some down time for me to suss this puppy out. 

Marcus, you da man!  On one of the three switches STP was check.  I disabled is and suddenly everything is playing nicely... well has been for about 5 hours... I will keep monitoring over the next few days to make sure it sticks, but all is looking well.

We can:

Connect via Ion Client through ethernet.

Connect iRFR and aRFR though the enterprise wifi

Connect to all the dimmer CEM+'s via Ethernet or Wi-Fi

All nodes are playing nice.

Still working on :

CMEi cross talk issues

WishList -

Client Connectivity via WiFi.  I know it is not supported but damn it would be awesome if it was.  I know several other consoles are able to do this, wonder why ETC chose to not support it. 

Thank you very much for the suggestions along the way.

dlderry said:

WishList -

Client Connectivity via WiFi.  I know it is not supported but damn it would be awesome if it was.  I know several other consoles are able to do this, wonder why ETC chose to not support it. 

 

Thank you very much for the suggestions along the way.

Not sure if you've tried it, but I connect as a client wirelessly. I know it's not recommended, but it has been working for me.

i remember a post, where the reaction time of the client was terribly slow, but as long as you have a sufficient and stable wifi connection and as long as it isn't critical to the show, why not use wifi for a client...

In looking at your config, your have 4 racks set to be 10.101.101.101-104, and the next space has 3 racks set to 10.101.101.111 - 103.

Can I assume you mean 113 as the final octet for the second set of racks, not 103?

Are these CEM+ racks? One thing to remember with CEM+ racks is the third octet specifies a group number. This has no bearing on DMX , EDMX or sACN addressing or who can speak to the racks form where. It isolates certain aspects of a multivenue form other venues for the purpose of configuration for one.  Groups are also isolated as they can be quite chatty online, depending on what synchronization and communications racks are trying to do with each other.  When we do larger CEM+ systems we try to seperate them by venues into rack groups.  For instance in your case,  10.101.101.101-104 and 10.101.102.101-103.  The way your system is set up apears that  you could be pressing buttons in the first venues racks, and accidentally change settings in the second venue racks. Since racks synchronize by group, and share some information, keeping them seperate can prevent problems from spreading across rack groups. I'm sur eit will work just fine in the same group/ IP range, but there is always the consideration of reducing potetnial  problems.  How are these racks venues addressed for control by Venue/Rack?

Your system sounds eloquently done and the possibilities and flexibilities can be very nice to have.  A risk of having a level of complexity such as yours, is if you leave your position, for instance, a lottery win, and the people who stay behind or replace you (in case you were in a pool with your entire staff)  do not have your knowledge of the system for troubleshooting, it can complicate future trouble shooting. Just something to think about. This can be offset by thoroughly documenting your layout in the new configurations, in Network structure, physical wiring layouts, backed up configs, Circuit maps and such, and thoroughly training of staff, shortly before retiring to your private island.

 The Installer probably didn't have much say by the time he received the paperwork and arrived onsite. Typically 12-18 months after the process started. He was probably following the standards we teach our installers. Most venues do not have your level of  knowledge or comfort with networking systems, and we try to keep troubleshooting simplified by these standards. It helps in those instances where there is a problem shortly before a performance, when time is of the essence.

Systems are specified by consultants to be laid out in certain ways, using these standards to create stable independant systems.  While it can be useful in a more complex layout as you are implementing,  in most systems we do not recommend  DHCP, or cross room control even though they can be made to work. 

Another consideration is the business end of how a job gets completed.  Without going way off topic,  there are many reasons a job gets installed as documented, submitted and approved by multiple groups, and to make major layout changes during installation can cause political issues for that job, and long term issues for the people involved on future jobs.  

bosox242 said:

Not sure if you've tried it, but I connect as a client wirelessly. I know it's not recommended, but it has been working for me.

For the past 4-5 years I've only used the client wirelessly. Best part about it is that the client software isn't restricted to 2 screens. I've worked on 3 screens from 1 laptop, and 4 screens with two laptops, all connected wirelessly. (I don't like using the page button).

Only down side is wireless client can't keep up with fast effects.