Averting a Disaster

I have been wrestling with an issue with our Internet firewall recently, and the culmination of troubleshooting efforts boiled down to a simple fact: a module inside the firewall would have to be rebooted.  This ends up being a big deal because while our firewall is rebooting, network traffic cannot pass thru it, effectively isolating WSU until the firewall properly initializes again.

While this is a minor annoyance when we have to do this at home with our Cable/DSL Modems, it has the potential of being something very nasty at an institution as large as Wayne State University.  That is all time that off-site students cannot access their Blackboard sessions, faculty cannot collaborate with other Universities, and prospective students cannot browse our webpages looking for that perfect program to enroll in.

Thankfully, in working with the Network Engineering group, the Information Security Office has multiple redundant systems setup for exactly this purpose.  With a few keystrokes, the Internet traffic was instantly rerouted thru our secondary Internet firewall, picking up the 161,000 network connections with ease.  Now that our troublesome firewall was “out of the loop”, we were able to run the diagnostic commands to restart certain modules without causing a moment of downtime.  This, in turn, helped resolve several production issues that have been growing over the past few weeks.

Exercises like this should be a reminder on how important it is to build redundancy in the systems that we create.  While the above was a controlled event, it as just as important to be ready in the case of an unexpected failure, such as a power supply failing or a backhoe digging up your fiber connection.  When dealing with large enterprise systems (including our Internet backbone), effective redundancy, Disaster Recovery, and Business Continuity Planning must be built into your methods and practices.  Without these things, it will be impossible to deliver the quality of services that our consumers live to expect!


How Vital is Network Security?

The Information Security Office(ISO) is responsible for the implementation, maintenance and troubleshooting of numerous firewalls located throughout campus.  These firewalls are dedicated hardware devices that sit on the network between key areas acting as “gatekeepers” – they can be programmed to a very fine detail as to which kinds of traffic or activity should be permitted or denied.

If a new server or network service is being implemented here at Wayne State, we may have to alter the configuration on one or more of our firewalls in order for it to work properly.  In order to do this, the Firewall Rule Change Request web-based form can be submitted.  Using this form allows us to have a single, standardized, and audit-able way to perform configuration changes on our firewalls.

We understand that the open nature of education clashes with the closed nature of security.  While to some it may seem inconvenient to involve an additional party (the Information Security Office) whenever you would like to make a change to a server or service, maintaining a proper level of IT security is paramount to ensuring that we have the minimum amount of risk or chance of a data breach or compromise.  The last thing that I ever want to see is a headline in the Detroit Free Press that a hacker compromised any of our student or financial data.

For a 24 hour period spanning between April 5th and 6th, our security devices provided the following protection:

  • 61.6 million blocked connections from over 1 million Internet hosts headed to WSU non-server networks;
  • 5 million successful connections to WSU servers from the Internet;
  • Over 2000 prevented specific attacks from the Internet to WSU non-server networks;
  • Over 800 prevented specific attacks from the Internet to WSU servers.

It is always enlightening to see the actual, proven benefits of the diligence and hard work when everybody comes together!