William Dougherty – Systems Support
Richard Hach – Network Admin
Carl Harris – Network Engineering
Mary Beth Nash – Legal Counsel
Jeff Crowder – Program Director
They did a review of the communications infrastructure and IT supporting response and recovery.
Themes/Conclusions
– Systems owned and controlled by the university generally adapted to crisis conditions. Provider-owned systems could not.
– Current IT infrastructure supports response and recovery, but does little for mitigation and preparedness.
Richard Hach –
Stress factors – web site had as much in the single day as they had previously seen in their busiest month; 300% increase in calls in the campus phone system; Blacksburg central office had capacity issues. The PSTN is configured for average peak loads – not peak usage in a crisis. Cell networks became congested and blocked calls.
Cellular providers dispatched technicians to add capacity to their networks. By 4/17 Sprint, US Cellular, and Verizon each had “cell on light truck” systems operating on campus and had provided emergency-use phones and accessories.
Carl Harris –
Technical staff know what the institutional assets are and how to adapt them to a crisis. They had to: Install phone and data comm for 9 geographically dispersed command centers and media workrooms and counseling centers.
Brought on an additional gigabit of Internet connectivity in 10 minutes thanks to Nat’l Lambda Rail connection
The web hosting infrastructure was stressed – database calls, rss feeds, etc. The web design folks aren’t always aware of what’s efficient. Extra servers came from surplus pile of stuff that had recently come out of service.
Radio communicatons – responders from several jurisdictions all with different radios. Radio transmission was less than ideal in some locations. Deficiencies in interoperability and coverage of police, fire, and rescue radios are decades-old problems in the US.
Notifications systems: used: Broadcast email to all @vt.edu addresses (with listserv); broadcast voicemail to campus phones; recorded message on the weather hotline; vt.edu web site and news web site; university switchboard; public media (tv, radio, newspapers); siren systems. A short list of vendors for cell phone alerts was identified prior to event, but they expedited after. selected 3n – does text messages, phone calls, email. Elected to hide vendor’s implementation and limit amount of selection choices. What vendors say about ability of systems to scale isn’t necessarily true – run stress tests. Students have phones off when in class, or don’t get signals in buildings, etc. Not a panacea. How to notify visitors on campus?
William Dougherty –
Data collection and preservation
April 16 meeting with central IT support staff (SS, Web hosting, DBMS) – knew they were going to be asked to preserve information. Started holding backup media.
Over next week spent time with law enforcement – fed, state, and campus. Brought warrants and subpoenas. Tried to draw relationships between victims and shooter. Reviewed email and web content.
April 23 – first preservation memo issued by legal counsel asking people to preserve any information that might pertain to event.
May 9 met with consultant hired by counsel (Servient)
May 10, meeting with departmental IT representatives
Took images of individual hard drives and other media of “persons of interest”. Finished last image Jan 8. Names on list changed regularly. Now determining how to restore data and make it searchable for discovery requests.
Stats: 27 departments interviewed, 149 individual data custodians (over 200 total images); 5 Tb of storage for these images; 10,000+ tapes stored from backup systems – over 900 TB stored. 5 TB of log files, including email, courseware, student systems, etc. Estimate 1400 person-hours spent on imaging process alone (and counting) – always had two people on site. All machines implemented were owned by the University. They requested that people who did work during the crisis on personal machines get them relevant information. The preservation memo put people on notice to preserve any information that they have.
GPG encryption was used for storage, with keys passed to legal counsel in sealed envelopes.
Out of 27 departments, only 4 had their own fileshare or backup systems – the rest use the central systems.
Information and Communications Infrastructure Group report (pdf)
Oren Sreebny 