Network slows down after enabling ACN with CEM+ modules

So. I have nearly the exact same problem. 5 racks SP48 2.4k, SP6 (6k), 3 SP12 (12k) All CEM+ running 3.1.2

Ion is running the latest Beta.1.9.11 Build 23

Rack networking, but each rack is in its own group. I would have preferred to set up a group of all 5 racks, but they simply wouldn't recognize each other, and I could at least get the Ion to recognize all 5 and make adjustments from the console.

Wireless is an Apple Airpot Extreme (AAE). iPhone and iPad. Ethernet is connected from console to Ethernet LAN port (WAN is empty). Additional LAN ports feed a switch at Racks, and laptop to configure the router. 10.101.124.101 and router address 10.101.1.1. Bridge mode.

I have been through nearly every combination of trying to get the console to be the DHCP server with the AAE in bridge mode, or have the AAE as the DHCP server. Router addresses as the device's own IP and as 10.101.1.1 Subnet mask is always 255.255.0.0.

Currently iPad & iPhone each work (and only work) with static IPs 10.101.125.101 & 102 Router 10.101.1.1 Console is 10.101.100.101 AEE is 10.101.124.101, with 10.101.1.1 in

At the console networking is off, console has a manual address, with gateway as 10.101.1.1

As soon as I connect the cable running to the racks (through unmanaged ethernet switches) and ACN & DMX on, the i devices start to have a multiple second lag, and I bet the signal strength of the console would drop from 5 bars to 1 though all the CEM+s show up on the ION About screen.

This shoot lasts 4 days (and this is day 3) but I set up all of my shows with networked Sensor racks, an Ion and a wireless router. This all worked as expected until the 5.0+ phone updates. Devices are on 5.1 currently.

So I guess I am asking what am I doing wrong?

Console: 

Should network services be on? DHCP? Routered network? Gateway addresses? I have set them to the IP settings here (http://community.etcconnect.com/wikis/products/knowledgebase-etc-network-ip-addresses.aspx)

Base Station

I couldn't get it to work either receiving or sending DHCP. What should the router address be? Always 10.101.1.1 for all devices?

Finally, I suppose the cable to the racks could be the problem, but the similarity between my case and the case above makes me think not. I cannot turn off DMX since I don't have a gateway, we are shooting today, and I am controlling our racks, a bunch of Kinoflo Image 85s, and 5 installed Sensor racks of the house system on old CEMs. I am using the lower part of one universe and the upper part of another for our racks and theirs.

If this relates to why the CEM+s wouldn't talk to each other I am all for it. Last time it happened on a different job it was because of a bad CEM+ that I kept kicking up the chain as I had only one spare instead of a spare for each rack, but, I was satisfied with putting each rack in its own group. 10.101.10x.101 with gateways the same.

Any assistance would be welcome.

James McNeal

ETCP Certified Entertainment Electrician

IATSE Local #52, ACT

Programmer, HBO Films Project

As of the end of my last show I was really only able to trace it back to the the fact that the Ion is not powerful enough (cpu-wise) to handle that much network traffic. I did not get a chance to try upgrading the switches from 10/100 to 1000 based switches.  This is a "network bogging down" effect and I wanted to try that next, but I was using CBS in-house stuff. 

I'd like to point out that this is only an RFR/iRFR problem when the Sensor racks are on the network with the Ion.  I had DimStat software running on my laptop so I could monitor the racks from there (keeping the racks off of the Ion's network) and everything worked great.

I think it'll get sorted out over time from two improvements:

1.  faster cpu power

2. better software/protocol implementation in both the CEM modules and EOS.

#2 being the place for more immediate improvements.  When they updated to 1.9.8 it got a little better, but the iRFR lag was still way too much.  It took ETC over a year to fix the USB bug with changing the channel on the RFR from the EOS Shell via the USB connection, requiring an ethernet connection through the GCE as a workaround....  So there is plenty of room for improvement in the implementation of network protocols into EOS.

@James - the thing that confuses me in what you wrote was the "networking at the console is off".   The setup should start with the Ion as the DHCP Server for your own, independent "network".  So yes, turn that on and start with configuring the Ion's start address, subnet & gateway.  I've got other threads in the ION forum that show the settings.. 

The only device that will automatically get it's settings from the ION's DHCP is your WAP (for your iPod/iPad app), if your router doesn't have WAP mode, then just get a WAP, not a router (they're different).  If you're using DHCP in your router, that's the first place to start making changes.  All you need is wireless for the remotes, so it only needs to be an access point.

As far as each device (rack, iPad, etc) You manually assign each device it's own IP (even though your using dynamic addressing - don't do that), set the IP yourself for each rack.  (i.e. 10.101.1.x   X=unique IP) each rack has to have it's own IP in a group.  If you set them all the same, that's why it didn't work.

You should be able to do this with your laptop.  Plug it into the switch at your racks and login to the IP address that brings up ETC Connect (CEM access)  and you can set it all from there..  I'm away from my notes right now but can go dig them out if you can't find the other threads on this in the forum.

Cheers,

Jason

As of the end of my last show I was really only able to trace it back to the the fact that the Ion is not powerful enough (cpu-wise) to handle that much network traffic. I did not get a chance to try upgrading the switches from 10/100 to 1000 based switches.  This is a "network bogging down" effect and I wanted to try that next, but I was using CBS in-house stuff. 

I'd like to point out that this is only an RFR/iRFR problem when the Sensor racks are on the network with the Ion.  I had DimStat software running on my laptop so I could monitor the racks from there (keeping the racks off of the Ion's network) and everything worked great.

I think it'll get sorted out over time from two improvements:

1.  faster cpu power

2. better software/protocol implementation in both the CEM modules and EOS.

#2 being the place for more immediate improvements.  When they updated to 1.9.8 it got a little better, but the iRFR lag was still way too much.  It took ETC over a year to fix the USB bug with changing the channel on the RFR from the EOS Shell via the USB connection, requiring an ethernet connection through the GCE as a workaround....  So there is plenty of room for improvement in the implementation of network protocols into EOS.

@James - the thing that confuses me in what you wrote was the "networking at the console is off".   The setup should start with the Ion as the DHCP Server for your own, independent "network".  So yes, turn that on and start with configuring the Ion's start address, subnet & gateway.  I've got other threads in the ION forum that show the settings.. 

The only device that will automatically get it's settings from the ION's DHCP is your WAP (for your iPod/iPad app), if your router doesn't have WAP mode, then just get a WAP, not a router (they're different).  If you're using DHCP in your router, that's the first place to start making changes.  All you need is wireless for the remotes, so it only needs to be an access point.

As far as each device (rack, iPad, etc) You manually assign each device it's own IP (even though your using dynamic addressing - don't do that), set the IP yourself for each rack.  (i.e. 10.101.1.x   X=unique IP) each rack has to have it's own IP in a group.  If you set them all the same, that's why it didn't work.

You should be able to do this with your laptop.  Plug it into the switch at your racks and login to the IP address that brings up ETC Connect (CEM access)  and you can set it all from there..  I'm away from my notes right now but can go dig them out if you can't find the other threads on this in the forum.

Cheers,

Jason

The hard part about this is that something changed. I have setup systems with as many as 13 racks all in the same group, accessed through either iPad or iPhone, and had the Ion seeing each CEM+ with no lag on the iPad.. I realize there are lots of variables at play, but as for console, router, iPad, I've. been plug and play since the iRFR came out. All I would have to change was the console password as I would get a different console for each job.

I turned off networking as it was the only way I could get the i's to join the wireless network and see the Ion (10.101.100.101 255.255.0.0 10.101.1.1)  WAP in bridge mode. Using only Ethernet ports. 

The port speed isn't the issue. All of my switches used 10/100/1000.

As for the racks. That was a different problem. I updated software so they were all 3.1.2 I started with just one rack on Ethernet. Deleted it's config. Assigned it an IP .101.101 and turned on networking.  Then I assigned the next rack an IP 101.102, and added that rack from the first CEM. Transfer would always fail no matter which rack I started with or which I was adding. After wrap on Friday, I reinstalled the software update, and tried again using just two of the SP12s 100AFs. Failed again. l also tried adding the racks from Web Connect. Could add the rack, but could not send the config to Rack 2. Had to give up. 

James

what happens when you ping each rack?

Sounds like the rack is loosing the network. Maybe once you assign it's static IP the other  'stuff' don't see it anymore.

Never got that far. After several attempts at trying to put them in the same group, once I got them working as separate groups (of just one rack) and ACN I had to move on.

James

Ah, now I see. We did change the Multicast address used for inter-rack communication in CEM+ v3.0, so I think you were going from the old group mutlicast IPs to the new ones.

This Multicast address survives an Abort Boot (otherwise it wouldn't be able to send configs between racks), so you'd need to manually upload the new config into each rack the very first time after this update.

After this update, the first time you Generate Defaults on the first rack in the group, just keep adding the racks to get the final config you want.
Then set Group/Rack/Net on all the other CEM+, but leave them with no config.

Then log into the Rack 1 CEM+ to download the complete system config file.

Finally, log into each of the other CEM+ and upload this file.

Once this first 'new version' config file has loaded the first time, the inter-rack multicast will have been updated to the new one so you shouldn't need to do this again.

Wow. Never would have gotten that on my own. Will try it out on the next one. Thanks.

Jim

Can you sticky this note somewhere?  That's a big one!!

Well, the change of default Multicast is in the release notes...

I live in hope that somebody out there reads them, though I know it's a very long shot!