Latency Issues

O.K So brief run-down of the system.

1 x ETC EOS Classic 2.4.0 (only console on the system)

On Stage RACK.

1 x Unmanaged 10/100 switch

2 x NET3 4 Port Node

1 x Show Control Gate way

---- from this rack on stage, we run a 90m Ethercon snake to FOH Control to another smaller rack which includes 

 

1 x Unmanaged 10/100 Switch

1 x NET3 4 Port Node

The EOS connects to this switch.

 

During technical rehearsals, we ran another cable from this switch to the EOS which was in the stalls. We were getting massive lag issues sometimes up to 7-8 seconds worth. We ran a separate Cat5 from the stage rack to the desk as it was closer and it seemed to solve the problem and thought it was length of cable. 

Move the desk into control box for first preview and plug back directly into the switch on the FOH rack so only 90m Snake and a 2m CAT5 from the switch to the EOS. Again, massive lag back during the preview.

Again we think the snake is longer than it says it is and luckily video are using a 75m snake so were used a spare connection on that snake and took ours out of the system. The lag disappeared and all seems to be fine. Show reports start coming in after I leave and lag is coming back every now and then and now out of ideas!! 

 

Any help of thoughts would be appreciated.

 

John

Parents
  • Hello John,

    I believe I spoke to you on the phone when you where in tech.

    -Could you confirm the make and model of the switches used?

    -Could you also, while the EOS running, in a cue, and lag is apparent: connect a laptop (set to an appropriate IP and mask) to the EOS switch (the built in one) and then ping the on-stage gateways in turn.

    -When the lag is apparent, could you take a picture (or note down) of the network switches link speed and duplex LEDs.

    -Inside the EOS there is a network switch, could you confirm the DIP switch settings on that board?

    -Could you confirm with a network cable tester, that all cores of every network cables is of a suitably low resistance.

    -With a laptop connected (and set to an appropriate IP and mask) to the onstage switch, please run this: www.dropbox.com/.../ETC_IPERF_SERVER.exe and then on the EOS (via software update button) please run this: www.dropbox.com/.../ETC_IPERF_CLIENT.exe The IP address of the laptop should be entered in the prompt. If we could then see the resultant output (a phone picture is fine).

    Feel free to email us direct at techservltd@etcconnect.com and/or marcus.birkin@etcconnect.com

    Regards,

    Marcus
Reply
  • Hello John,

    I believe I spoke to you on the phone when you where in tech.

    -Could you confirm the make and model of the switches used?

    -Could you also, while the EOS running, in a cue, and lag is apparent: connect a laptop (set to an appropriate IP and mask) to the EOS switch (the built in one) and then ping the on-stage gateways in turn.

    -When the lag is apparent, could you take a picture (or note down) of the network switches link speed and duplex LEDs.

    -Inside the EOS there is a network switch, could you confirm the DIP switch settings on that board?

    -Could you confirm with a network cable tester, that all cores of every network cables is of a suitably low resistance.

    -With a laptop connected (and set to an appropriate IP and mask) to the onstage switch, please run this: www.dropbox.com/.../ETC_IPERF_SERVER.exe and then on the EOS (via software update button) please run this: www.dropbox.com/.../ETC_IPERF_CLIENT.exe The IP address of the laptop should be entered in the prompt. If we could then see the resultant output (a phone picture is fine).

    Feel free to email us direct at techservltd@etcconnect.com and/or marcus.birkin@etcconnect.com

    Regards,

    Marcus
Children
  • Hi. Pitching in on this one, as I'm presently at the venue in question ...

    Couple of minor amendments to John's description of the system. The switch at the control end isn't being used at the moment. One ethernet o/p on Eos connects to a 4-port gateway driving the venue dimmers, another one directly feeds the snake running to the onstage racks. Snake length is 75m (it says on the end, and I believe it!).

    Investigation revealed the existence of an unnecessary 20m extension between Eos and control snake - that's now removed, so let's see if it makes a difference.

    Switch in onstage rack is a Cisco SR216.

    Difficult to achieve any fault-finding "while the lag is apparent" ... it only ever manifests itself during a performance, and then only for a handful of cues before going away again. I can't be pinging gateways or photographing switches while op'ing the show, and there generally isn't anyone else around to do so. The first opportunity we have to do this will be next week when John rejoins us and there are two of us around who can look into this simultaneously.

    Haven't had a window of opportunity to open the console and look at DIP switch settings, nor to ring out the network cables in the snake with a tester.

    Photo of the output from ETC_IPERF_CLIENT.exe is attached.

     

    I'll see what happens during the performance tonight - after that, I'm not back with it until next week, when John also returns.

    Gareth.

  • Hello Gareth,

    These figures would suggest that your network is running at 10Mbit and at half duplex. This is far from ideal.
    Unfortunately the Cisco SR216 doesn't have a speed duplex indicator, so we can not instantly establish where the speed drop is located.
    I would maybe try the iperf test again, but with the laptop directly connect to the EOS - this would establish the maximum speed obtainable.

    A 95m (75m + 20m) is very close to absolute limit of 100m port-to-port for Ethernet and it is generally accepted that 90m should be the maximum working limit. At long distances Ethernet is unpredictable. I would wager that this would be the cause.

    Regards,

    Marcus

  • Hi Marcus,

    Update from tonights show report.

    Lag is still apparent, about 5-6 cues went with Lag issues, note the lag is with the output to the rig, the desk is changing as it should, sometimes its steppy, sometimes it go and delays/ snaps up.

    Im slightly concerned as this is the second snake which is now defiantly under 90m's in length.

    Ill give a call tomorrow to the office and have a chat/update.

    John

  • Hello John,

    Ethernet length is a very important aspect and is often over looked. It worth noting that due to way ethernet cable is constructed, that the outside jack length is not a true indication of the length of the twisted cores within.

    I'd also note that the Cisco SD216 is a rebadged clone of a cheap netgear switch.


    I would say that you have a couple of issue on this setup.

    So now we need to establish where the speed drop is occurring, as I would expect the iPerf results to be in the region of 80-90Mbits/sec and at the very least 10MBits/sec.
    With that in mind, could you iPerf test between the eos and the laptop in the following setups:
    1)Laptop plugged directly into the EOS switch
    2)Laptop plugged directly into the EOS motherboard (you will need to open the console up for this)
    3)Laptop plugged directly into the far end of the long snake while connected to the EOS switch (i.e. Not with the on-stage switch)
    4)Moving the on-stage switch to the control room and via short cables, laptop via the switch.

    Regards,

    Marcus

  • Hi.

    A quick update for anyone reading this thread.

    It was diagnosed finally as a fault Mother Board and swapped out and all happy.

    Thanks for all the help!
Related