This November will mark 12 years since the launch of Xbox Live, Microsoft’s online gaming service that currently boasts more than 48 million global users. Ensuring all of those Live members are constantly connected with each other and enjoying all of the service’s many features is paramount to delivering the Xbox experience gamers have come to expect. To pull it off, Microsoft has staffers working around the clock at its Xbox Live Operations Center (XOC) in Redmond, Washington, reports Total Xbox.
As part of their job function, XOC staffers respond to clear-cut problems. The majority of their time, however, is dedicated to ferreting out issues that no one knows about. Team members search for irregularities and investigate them to discover whether or not they are problems that need solving. They can’t guess at whether something is a threat to uninterrupted Xbox Live service or not; they have to know. They have to find out.
Microsoft Gaming Ninja (yes, that’s his real job title) Eric Neustadter informed Total Xbox that his team has to be ready for anything at all times. “It’s a bit like being in the fire department or police, where you have to think through what would I do in [a worst case] scenario. My boss was in the fire department for years, and he went from doing that to running services like this. In a lot of ways, it is a similar mindset.”
Redundancy is Neustadter’s go-to tool for preventing an Xbox Live doomsday scenario. “We have all the things you need to run 24/7/365, like multiple power grids connected into the campus,” he explained. “When that fails, we have diesel generators with huge tanks out there so that we can run for roughly 72 hours without power, and then we can bring in tankers with more diesel and refill the tanks while they’re running. We also have a disaster recovery site… somewhere else on a different power grid with almost all of this replicated so we can go over there and work instead.”