A lesson in real time…

One of my jobs is to instruct first year college students about how to succeed in academics, partly because it is very different from what is expected of a student before they are in college. If for nothing else, the anty is upped once they’ve entered higher education. Yes, many of the things a student may learn in the beginning of their university career might be similar to the advanced work a student has done in their last final years before college, but no matter what it is usually a shock.

This year, especially this month, because of the 2012 U.S. Presidential Election, the world is and has been watching the United States of America, and the now re-elected President Obama.
One reason people have been watching or paying attention is the ability of the Obama re-election campaign to utilize social media and technology, with an amazing dream team, to help give the incumbent the edge necessary to win re-election. Well, that and the utter failure of the other guy, Romney’s social media and technology team. Ultimately, regarding the technological success of campaign tech during this election, it was campaign ORCA versus campaign NARWAL.

However, what might this have to do with first-year student success? Well, it never seems to be enough that I can show my students how to plan ahead, how to create a backwards plan so that they know what type of timeline to create and maintain once they have many a projects on their plate that generally all happen to be due around the same time. It isn’t enough that I make it an assignment during their second semesters, no – so many still fail. However, that is the point, to try and fail and then try again. It is better, I tell them, to do this now, then to wait until they are out on their own beyond these safe “Ivory” walls when they are presumed experts at one thing or another, only to prove to their bosses and managers that they in fact have failed – miserably.

One of the jobs of college is to prepare to for life after college, hopefully for lifelong learning. Higher education is supposed to teach you how to think for yourself, plan ahead, avert disaster and succeed. If you do not try and fail, and prepare, prepare, prepare – you will have to do all of these things after your journey through higher education and the process will be much more painful (see 2012 Romney Presidential Campaign). This is the precise reason why professionals (and volunteers) working in disaster relief and preparedness run drills and role-plays. Heck, it is the reason why I run language role-plays with my English language learning students – so when they get in the middle of the disaster or stressful situation that will only prove their competence, they are prepared!

Here, Alexis C. Madrigal writing for The Atlantic online, explains the secret to the Obama campaigns’ technological win in “When the Nerds Go Marching In.”

The Obama campaign’s technologists were tense and tired. It was game day and everything was going wrong.

Josh Thayer, the lead engineer of Narwhal, had just been informed that they’d lost another one of the services powering their software. That was bad: Narwhal was the code name for the data platform that underpinned the campaign and let it track voters and volunteers. If it broke, so would everything else.

They were talking with people at Amazon Web Services, but all they knew was that they had packet loss. Earlier that day, they lost their databases, their East Coast servers, and their memcache clusters. Thayer was ready to kill Nick Hatch, a DevOps engineer who was the official bearer of bad news. Another of their vendors, PalominoDB, was fixing databases, but needed to rebuild the replicas. It was going to take time, Hatch said. They didn’t have time.

They’d been working 14-hour days, six or seven days a week, trying to reelect the president, and now everything had been broken at just the wrong time. It was like someone had written a Murphy’s Law algorithm and deployed it at scale.

And that was the point. “Game day” was October 21. The election was still 17 days away, and this was a live action role playing (LARPing!) exercise that the campaign’s chief technology officer, Harper Reed, was inflicting on his team. “We worked through every possible disaster situation,” Reed said. “We did three actual all-day sessions of destroying everything we had built.”

Hatch was playing the role of dungeon master, calling out devilishly complex scenarios that were designed to test each and every piece of their system as they entered the exponential traffic-growth phase of the election. Mark Trammell, an engineer who Reed hired after he left Twitter, saw a couple game days. He said they reminded him of his time in the Navy. “You ran firefighting drills over and over and over, to make sure that you not just know what you’re doing,” he said, “but you’re calm because you know you can handle your shit.”

The team had elite and, for tech, senior talent — by which I mean that most of them were in their 30s — from Twitter, Google, Facebook, Craigslist, Quora, and some of Chicago’s own software companies such as Orbitz and Threadless, where Reed had been CTO. But even these people, maybe *especially* these people, knew enough about technology not to trust it. “I think the Republicans fucked up in the hubris department,” Reed told me. “I know we had the best technology team I’ve ever worked with, but we didn’t know if it would work. I was incredibly confident it would work. I was betting a lot on it. We had time. We had resources. We had done what we thought would work, and it still could have broken. Something could have happened.”

In fact, the day after the October 21 game day, Amazon services — on which the whole campaign’s tech presence was built — went down. “We didn’t have any downtime because we had done that scenario already,” Reed said. Hurricane Sandy hit on another game day, October 29, threatening the campaign’s whole East Coast infrastructure. “We created a hot backup of all our applications to US-west in preparation for US-east to go down hard,” Reed said.

“We knew what to do,” Reed maintained, no matter what the scenario was. “We had a runbook that said if this happens, you do this, this, and this. They did not do that with Orca.”

And that is an excellent example of how to plan for failure, in order to be as successful as you can possibly be and the real-time example of what hard work and its reward look like that will now and for the foreseeable future share with my students.

 

For more information about these two technological campaigns:
Gallagher, Sean. “Inside Team Romney’s whale of an IT meltdown.” arstechnica.com. Condé Nast. 09.11.2012 Web. 17.11.2012. <http://arstechnica.com/information-technology/2012/11/inside-team-romneys-whale-of-an-it-meltdown/>.

hotdamn. “Romney had ORCA, Obama Had Narwhal. How Superior Technology Ruled the Day.” DailyKos.com. Kos Media, LLC. Blog. 16.11.2012. Web. 17.11.2012. <http://www.dailykos.com/story/2012/11/16/1162469/-Romney-had-ORCA-Obama-had-Narwal-How-Superior-Technology-Ruled-the-Day>.
Jennings, Natalie. “Romney digital director: Orca wasn’t a loss.” WashingtonPost.com. The Washington Post. Blog. 09.11.2012. Web. 17.11.2012. <http://www.washingtonpost.com/blogs/election-2012/wp/2012/11/09/romney-campaign-orca-wasnt-a-loss/>.

Madrigal, Alexis C. “When the Nerds Go Marching In.” The Atlantic.com. The Atlantic Monthly Group. 16.11.2012. Web. 17.11.2012. <http://www.theatlantic.com/technology/archive/2012/11/when-the-nerds-go-marching-in/265325/>.

Pruim, Andrew. “ORCA vs. Narwhal offers lessons in campaign technology.” Calvin College Chimes. Calvin College Chimes. 14.11.2012. Web. 17.11.2012.
<http://www.calvin.edu/chimes/2012/11/14/orca-vs-narwhal-offers-lessons-in-campaign-technology/>.

Rousseau. “Lessons IT can take away from Romney campaign’s Orca project?”  IT World Answers. The IDG Network. 15.11.2012. Web. 17.11.2012. <http://www.itworld.com/answers/topic/big-data/question/lessons-it-can-take-away-romney-campaigns-orca-project>.

UC Strategy Staff. “Project Orca: The Mismanaged Mobile and IT Solution That Failed Mitt Romney.” Unified Strategy Solutions. UCStrategies.com. 12.11.2012. Web. 17.11.2012. <http://www.ucstrategies.com/unified-communications-newsroom/project-orca-the-mismanaged-mobile-and-it-solution-that-failed-mitt-romney.aspx>.