Hey Everyone,
I have 2 Gios that have been running as master and backup for like 10 years. We had to move to a temporary space and I switched their roles since we had to run one console temporarily and we wanted to use the one we always operate on (the original backup) for the dumb reason that it has all the tape marks.
In the temporary space, everything worked perfectly fine for over a month. The role swap and the physical move are the only thing that changed. No settings were touched as everything plugged in and worked right away. No software updates at this time.
Now we had a very disastrous power outage to the building one night. These consoles weren't even powered on when it happened. It messed up a bunch of things across all departments.
The next morning I go in and suddenly the "new" backup will just not find the master no matter how many reboots I tried. The procedure for the last 10+ years was always boot up the master and then boot up the backup. Just this time it didn't work. I tried a backup cable run and still nothing. The network green light was blinking both times.
Oddly enough, I noticed our Paradigm panel wasn't connecting either. So I tried that same backup cable and it synched right up. So it wasn't the cable.
My coworker found a workaround where if you start the backup first and then the master it would sync every time - like power on the backup and wait until the beeping stops and then power on the master and worked like a charm.
Now, that was a couple months ago. Since then we have a new control room with an XE RPU as master, I upgraded everything to 3.2.7 at the time. I updated the backup Gio too but we didn't really need it yet. But it will become a client and operate as our floor board whereas now we have an Ion as a floor client. I updated that Ion today with the same USB stick I used on everything else. It synched up right away after I changed master to the RPU, no issues, no delay.
I rolled the old backup Gio out and used the same cable in the same port and tried to run it as client and it would just not see the master still. For fun I switched the IP addresses on the 2 ports and switched the cable and it still wouldn't find the master. Did a deep clean. Still nothing. Didn't try the power on the client first thing since that's way too much of a hassle to do that now. And shouldn't be unnecessary.
The one thing I didn't try was seeing if Concert on that Gio could see all the gateways so I'm kicking myself for not thinking of that.
Now here's the annoying part - I set up a little temporary network completely separate with that Gio as client and yet another, separate Ion as Master and it synched up right away. On both ports. I even shut down the Gio and unplugged it for 30sec to simulate the "start the client first" and couldn't re-create the problem.
So the only thing that changed when this first started was the consoles roles which worked for over a month even with a physical location change. Otherwise, it's coincidental that the problem started the morning after the power outage.
I'm pretty much dumbfounded. Unlike many other people's connectivity issues there is no PC involved or wifi and all these consoles were talking perfectly together for many years. I will double check the fixture library versions but I don't see how that could have caused it because I never updated anything when the problem started. The time delay was just because it wasn't a priority.
Any thoughts?