#Facebook critical ops software

The Facebook operations team carefully tracks equipment failure rates, and the data is reviewed when the company makes supply chain decisions, Eberly said. The server is designed so drives and components can be replaced without tools. The result: Facebook has reduced the time needed to repair servers by 54 percent. An example: all servers are designed to be serviced from the front, so data center staffers have no need to enter the hot aisle. Design Supports ServiceabilityĪt Facebook, the operations team's time and workloads are considered during Facebook's hardware design. This matters to us.'Įberly is a veteran of the data center industry, beginning her career at McKesson in 1998, followed by stints at colocation pioneer Exodus Communications and Critical Path. 'The way you do that is to give them the opportunity to work on high-value tasks. 'We want to hang onto our talent,' she said. It's because Facebook values its workers, according to Eberly. The emphasis on automation is not because Facebook is interested in unmanned data centers, or having robots operate facilities. 'Our goal is not to deploy a technician to the data center floor unless they actually have to physically handle a server,' said Eberly. If CYBORG exhausts automated repair options, it will send an alert to the ticketing system to dispatch a data center staffer to investigate the issue.

#Facebook critical ops software

That includes software known as CYBORG, which detects problems with servers and attempts to fix the problems.

To manage all that activity, Facebook has developed software to automate many aspects of data center operations. The company now stores more than 240 billion photos, and adds 7 petabytes of photo storage each month. Each day, Facebook users share 4.75 billion content items and 'like' 4.5 billion items. The company's success affirms the potential of using an integrated approach in which the operations team works closely with other teams in IT and facilities.ĭata center operations is a critical skill at Facebook, which now has 1.15 billion users, including 720 million who log in daily.

Eberly was the keynote speaker Tuesday morning at the 7x24 Exchange 2013 Fall Conference, speaking on 'Operations at Scale.'įacebook's performance appears to break new ground in the server-to-admin ratio, which has rarely exceeded 10,000 to 1 (see High Scalability for more). That includes the scalability of the people that work in the company's data centers.Įach Facebook data center operations staffer can manage at least 20,000 servers, and for some admins the number can be as high as 26,000 systems, according to Delfina Eberly, Director of Data Center Operations at Facebook.

SAN ANTONIO - Facebook has been an industry leader in building its Internet infrastructure for scalability. In terms of hardware, Facebook, because it runs such an enormous volume of servers, focuses on serviceability, including starting from the ground up by influencing server design to ensure easiest and least time consuming methods to repair equipment in the data hall. Delfina Eberly, Director of Data Center Operations, Facebook, presented the Tuesday morning keynote about optimizing data center operations.