[CSG Winter 08] Technology issues and implications around the VT shootings – Panel: IT Infrastructure relating to th

William Dougherty – Systems Support Richard Hach – Network Admin Carl Harris – Network Engineering Mary Beth Nash – Legal Counsel Jeff Crowder – Program Director They did a review of the communications infrastructure and IT supporting response and recovery. Themes/Conclusions – Systems owned and controlled by the university generally adapted to crisis conditions. Provider-owned … Continue reading “[CSG Winter 08] Technology issues and implications around the VT shootings – Panel: IT Infrastructure relating to th”

Advertisement

William Dougherty – Systems Support

Richard Hach – Network Admin

Carl Harris – Network Engineering

Mary Beth Nash – Legal Counsel

Jeff Crowder – Program Director

They did a review of the communications infrastructure and IT supporting response and recovery.

Themes/Conclusions

– Systems owned and controlled by the university generally adapted to crisis conditions. Provider-owned systems could not.

– Current IT infrastructure supports response and recovery, but does little for mitigation and preparedness.

Richard Hach –

Stress factors – web site had as much in the single day as they had previously seen in their busiest month; 300% increase in calls in the campus phone system; Blacksburg central office had capacity issues. The PSTN is configured for average peak loads – not peak usage in a crisis. Cell networks became congested and blocked calls.

Cellular providers dispatched technicians to add capacity to their networks. By 4/17 Sprint, US Cellular, and Verizon each had “cell on light truck” systems operating on campus and had provided emergency-use phones and accessories.

Carl Harris –

Technical staff know what the institutional assets are and how to adapt them to a crisis. They had to: Install phone and data comm for 9 geographically dispersed command centers and media workrooms and counseling centers.

Brought on an additional gigabit of Internet connectivity in 10 minutes thanks to Nat’l Lambda Rail connection

The web hosting infrastructure was stressed – database calls, rss feeds, etc. The web design folks aren’t always aware of what’s efficient. Extra servers came from surplus pile of stuff that had recently come out of service.

Radio communicatons – responders from several jurisdictions all with different radios. Radio transmission was less than ideal in some locations. Deficiencies in interoperability and coverage of police, fire, and rescue radios are decades-old problems in the US.

Notifications systems: used: Broadcast email to all @vt.edu addresses (with listserv); broadcast voicemail to campus phones; recorded message on the weather hotline; vt.edu web site and news web site; university switchboard; public media (tv, radio, newspapers); siren systems. A short list of vendors for cell phone alerts was identified prior to event, but they expedited after. selected 3n – does text messages, phone calls, email. Elected to hide vendor’s implementation and limit amount of selection choices. What vendors say about ability of systems to scale isn’t necessarily true – run stress tests. Students have phones off when in class, or don’t get signals in buildings, etc. Not a panacea. How to notify visitors on campus?

William Dougherty –

Data collection and preservation

April 16 meeting with central IT support staff (SS, Web hosting, DBMS) – knew they were going to be asked to preserve information. Started holding backup media.

Over next week spent time with law enforcement – fed, state, and campus. Brought warrants and subpoenas. Tried to draw relationships between victims and shooter. Reviewed email and web content.

April 23 – first preservation memo issued by legal counsel asking people to preserve any information that might pertain to event.

May 9 met with consultant hired by counsel (Servient)

May 10, meeting with departmental IT representatives

Took images of individual hard drives and other media of “persons of interest”. Finished last image Jan 8. Names on list changed regularly. Now determining how to restore data and make it searchable for discovery requests.

Stats: 27 departments interviewed, 149 individual data custodians (over 200 total images); 5 Tb of storage for these images; 10,000+ tapes stored from backup systems – over 900 TB stored. 5 TB of log files, including email, courseware, student systems, etc. Estimate 1400 person-hours spent on imaging process alone (and counting) – always had two people on site. All machines implemented were owned by the University. They requested that people who did work during the crisis on personal machines get them relevant information. The preservation memo put people on notice to preserve any information that they have.

GPG encryption was used for storage, with keys passed to legal counsel in sealed envelopes.

Out of 27 departments, only 4 had their own fileshare or backup systems – the rest use the central systems.

Information and Communications Infrastructure Group report (pdf)

[CSG Winter 08] Technology issues and implications around the VT shootings

Larry Hincker – Crisis communication VA Tech was an all-military school until 1964. 49 people shot, 32 dead. About 1500 first responders the first day, 27 ambulances from 14 agencies and give hospitals. Everybody who was take to the hospitals alive survived. Lots of police on campus – it wasn’t until about 24 hours into … Continue reading “[CSG Winter 08] Technology issues and implications around the VT shootings”

Larry Hincker – Crisis communication

VA Tech was an all-military school until 1964.

49 people shot, 32 dead.

About 1500 first responders the first day, 27 ambulances from 14 agencies and give hospitals. Everybody who was take to the hospitals alive survived.

Lots of police on campus – it wasn’t until about 24 hours into the event until the police we convinced it was a single shooter.

36,000 people from around the world posted condolences to a web site, 28,382 unique stories logged in first two weeks.

Dealing with the White House was not easy.

The media city – about 1,000 journalists and crew – 125 satellite trucks, 60 international agencies.

Crisis Communications –

– Have a plan – he was struck by how closely they hewed to the plan, even though they didn’t pull it off the shelf during the event.

– Have your CEO visible.

– Have a designated spokesperson

– Have a communications command center.

– Communicate as much as possible … as often as possible. Stay on message. Use other experts where necessary. ID target audiences and flood them with information.

Shooting over at 10 am. Posted statements to the web simultaneously. First news conference at noon, two more on first day. 11 news conferences across 8 days.

Running the press conference … walking the tightrope. Issuing a statement alone won’t work. Manage info needs day to day – topics moved from the murder, to gun control, to handling troubled students. Let the air out of the balloon – let people talk.

CEO out front – President at all press conferences on Monday, and on national news shows, etc.

Staying on message – what is the message when you feel like this? The message – we will not be defined by this tragedy, focus on the families, focus on helping the campus heal.

Setting up the media city – how to manage the media arriving in huge numbers? Where to put them, how to manage the phone calls (350 interview requests on Monday). Immediately added telecom capacity and prioirity, added mobile towers. Shfited calls to Joint Information Center by Tuesday afternoon.

Joint Information Center – it was extremely helpful that the state declared an emergency and helped support. Location afforded access to bathrooms, place to file stories, space to set up program sets. IT lifted the network security and enabled open access. They provided meals from local restaurants.

The role of the Web – a powerful tool in a crisis. Mike Dame – Director of Web Communications

Benefited from work on redesigning web site which built relationships between IT and Media Relations. The Web is the nexus of the University’s communication plan.

The had a plan for what to do on the web when a crisis arose. Learned a lot from a previous event (when an escaped convict was seen near campus).

First blast email sent at 9:26 with subject line “Shooting on campus”

9:31 – alert posted to home page

9:44 web staff overhears chatter on police scanner about gunshots – alerted IT staff to expect a huge spike – they started adding servers.

Went to a light version of the home page – just text and a the logo.

Goals for day of tragedy – communicate essential info, home page about brevity and clarity. Navigation reduced to pertinent sections – About VT, Administration, Campus Maps, Campus Buildings. Expand server load balancing. Establish communications workflows with the administration.

Launched lite version of home page at 10:17.

at 10:33 shut down news database to reduce load and go to an alert “details page”

new servers came on line in early afternoon.

Details page was presented in a very blog-like fashion – anything they knew they put on there. All multimedia files were kept on a separate server.

Next day talked and agreed on coming up with a new design for the home page – simple and austere that reflects a grieving university. Decided to think web 2.0 – engaging people in various ways and allow them to participate. Put students and families first in all decisions. Choose words carefully to aid healing. Created the condolences web site. Set up live video streaming for Convocation, partnering with Athletics.

Went to the new In Memoriam page at 1:57 right before the convocation started.

Used blog software they already had for the condolences site. 35,000 entries in 72 hours.

University began releasing victim names on 4/19. Those were displayed prominently on the home page. Worked with families to get photos and biographies for the victims.

Used In Memoriam page to target messages to specific communities, including students and the media. When classes resumed on 4/23 they made classroom buildings off limits to the media.

Went from an average of 10k hits per hour to over 150k hits per hour. 432 GB of data transferred on April 16 – a normal day is about 15 GB.

Transition to “normalcy” – tragedy still an open wound on campus. Transitioned to a blend of normal look with the remembering page. Reintroduced school colors, enlarged logo to make the “Invent the future” message stand out more. Still devoted the main content area to information and news about the events. Back to normal home page on May 15 – day after semester ended, but left the black ribbon on the top right which linked to the In Memoriam page.

(back to Larry) –

Surrogate Spokespersons

Coping with the media circus – on Wednesday got the reporters out and onto campus. The students presented themselves very well. Got a petition from student government asking whether they could ban the media from campus. Sent a letter to the entire campus community saying they couldn’t ban the media, but that they didn’t have to talk to the media if they didn’t want to. Put 152 signs on campus asking the media to respect and asking them not to go into buildings. Did media training for selected students.

When is the crisis over? When you say it is. Ran out of information.

Stabilizing a wounded community – continuing the web story. Had 400 reporters back on campus for commencement, which was hard.

Hard to convey the frantic nature of things.

After the healing concert they shut down and started concentrating on returning to normal.

Long Term Challenges –

Not to be defined; wearing the mantle without owning the issues; when is it right to move on?; Focus on core brand messages – “Invent the Future”; Focus on core mission – learning, discovery, engagement.

It may take a generation to get the image of VT back to what it was before – think U Texas with the tower shootings, or Kent State.