Have You Tried Turning It Off And On Again?

Those of you familiar with British sitcoms might be aware of the show The IT Crowd, about an IT support office for a huge but mysterious company. Their catchphrase is the title of this blog. The reason I’m bringing this up is that C&IT is going to do just that this coming Sunday. Everything you know and love will go away from midnight Saturday night till 10 AM Sunday morning, and this blog is intended to provide a sense of why this is being done and what effects it will have.

As you might imagine, C&IT has hundreds of servers, running Pipeline, Blackboard, Banner and even each other. The last bit is because much of the C&IT infrastructure runs on virtual machines rather than having one operating system per machine, and there is also complex load balancing going on. When there are thousands of people visiting Blackboard at the same time a ‘traffic cop’ assigns them to different routes to the basic Blackboard files.
Consequently, the electrical power demands of these hundreds of units are very large, and require  a very elaborate system to assure continuous power. The system includes an enormous battery back-up system, and beyond that, a natural gas-powered generator to power the entire building independently when power problems occur. All this is necessary to deal with the vagaries of electrical supply in the city of Detroit, especially during the peak-demand summer months.

The electricity comes into the primary room to the un-interruptable power supply (UPS) system and is then routed to power distribution units (PDU’s) where the power is transformed from 480 volts to 208 volts before being distributed through panels that are similar to the circuit breaker panels in your basement. Over the years the number of servers has increased, and it’s time to rewire the PDU’s  in order to make sure that servers are connected redundantly to the PDU’s and subsequently the breakers. But, as you know if you’ve ever thought about doing this at home, you need to shut off the entire power supply before you touch anything. So, early on Sunday morning (specifically 12:01 AM) we’ll start shutting down all the computers. Because they are all interconnected, this is a complex and slow process. Then the electrical guys will do the rewiring, and finally we’ll turn it all back on again, which is again, a very slow and careful process. This is why we’re allocating ten hours for the complete change. It’s possible it will take less time, but just to be sure, we’re being very cautious.

So, everything you normally use (Blackboard, Pipeline, Banner, Wayne Connect email…) will be turned off between midnight and 10 AM Sunday morning. We’re hoping, because the university is closed Monday in observance of Martin Luther King Day, that this will not be too disruptive.