Real-Time Energy Optimization and Best Practices In a Mission-Critical Data Center

As one of the world’s largest Internet commerce platforms,  eBay demands extreme data center reliability; any downtime would impact transactions worth more than $2,000 USD a second. Thus, its global data center team must excel simultaneously in delivering uptime and flexibility while keeping costs down — a truly complex challenge, requiring constant innovation to succeed.

Since its inception, eBay amassed a portfolio of 12 data center sites to keep up with its constantly increasing compute load. Unsurprisingly, eBay’s data center operating costs are also increasing every year as business growth continues unabated. 

In response, eBay created and began executing a four-year data center plan with an ambitious goal: to cut power costs in half, while simultaneously doubling compute performance. At the same time, this new infrastructure design would enable greater operational agility and increased reliability.

To achieve these goals, eBay would consolidate its assets into a smaller number of data centers, each designed from the ground up to be extremely energy-efficient without compromising uptime.

“A successful data center strategy assumes that computing demand will grow every year yet enables this growth without proportionate increases in operational expense,” explains eBay’s Senior Director of Data Center Strategy and Operations. “We have to break the linear relationship between compute load and operational cost. This requires regular,  disruptive leaps in our energy efficiency, computational power and server utilization profiles.”

In May 2010, eBay opened the flagship facility and cornerstone of its new data center strategy, Project Topaz. Located in South Jordan, Utah, Topaz is the single most significant capital investment in eBay’s history and houses more than a third of its worldwide server infrastructure. Despite deploying all the necessary redundancies required for a Tier IV data center, the Topaz facility can still boast a Power Usage Effectiveness (PUE) metric of only 1.4.

Maximum Rack Efficiency Enabled Through Best Practices

By employing innovative energy-efficient mechanisms in its infrastructures — such as an optimized building design, water- side economizers, rainwater collection and dynamic controls throughout the cooling and electrical distribution subsystems – the Topaz facility provides power and cooling to the raised floor with remarkably minimal overhead. Complementing these facility measures, Topaz also implements many best practices on its raised floor space, working in tandem with the cooling plant to contribute to the site’s LEED Gold design. Five key design features of the Topaz raised floor space are particularly compelling for their value and applicability to all modern data centers:

1. Hot Aisle Containment

To achieve maximum cooling efficiencies, eBay employed a hot aisle containment design that had been carefully designed, prototyped and measured before construction.

eBay’s data center architecture team outfitted full-scale mock-ups of multiple containment designs at their San Jose headquarters. They experimented with different strategies (hot aisle vs.  cold aisle containment), material selection (containment doors vs. plastic sheathing) and dimensions. This design process determined an optimal balance of cost, flexibility and reliability before mass deployment. eBay found that hot aisle containment provided more generous buffer times than cold aisle containment in the event of a CRAC failure. The team deployed containment aisles comprising rows of 45U cabinets and partnered with their cabinet and containment vendor to ensure a modular design while still minimizing leakage.

2. 400V Distribution To Each Cabinet

With multiple blade servers deployed in certain cabinets, eBay wanted any given rack to be capable of provisioning loads up to 17kW. While this is achievable with traditional 208V distribution,  Topaz enjoys significant cost efficiencies by extending higher-voltage distribution to each cabinet. By supplying 400V power to each rack, eBay minimizes step-down transformation and line transmission loss while simultaneously reducing copper utilization. Specifically, eBay provides 400V 3-phase power to two Raritan intelligent rack power distribution units (also known as “plug strips‘’ or “rack PDUs”) in each cabinet.

The rack PDU, with internal line-to-neutral (3-phase WYE) wiring, supplies each server with 240V, single-phase power — well within the operating range of the power supplies of virtually any IT equipment.

By eliminating unnecessary voltage transformations, 400V power reduces energy costs by approximately 2-3% vs. 208V distribution; and approximately 4-5% vs. 120V distribution.

As a global vendor of rack power distribution units, Raritan provides an extensive range of product configurations that accept 400V input. While 400V distribution is relatively new in North America, Raritan has deployed 400V systems, which is the nominal standard in Australia and in many European countries.

3. Detailed Server Power Instrumentation

Arguably the most important economic metric for eBay’s data center operations team is its aggregate cost per search. And because server CPU capabilities grow so quickly with each product generation, eBay replaces its servers in two-year refresh cycles, as performance-per-watt gains outweigh the capital costs of new assets.

With each server procured, however, eBay can only truly minimize its cost per search by factoring in all costs of running the server, not only capital expenses. Notably, energy costs are the single largest operational expense for any given asset.

To enable maximum savings, eBay deployed rack PDUs from Raritan that provide precise energy consumption data for every single power supply, of every single server. Accurate to IEC/ANSI “billing grade” standards, Raritan’s PX family of rack PDUs provides continuous sub +/- 1% accurate kilowatt-hour information for each individual asset in the data center. Built with both IT and facility standards in mind,  the Raritan PX can pass this information upstream to eBay’s building management and asset tracking systems in real-time, achieving what The Green Grid terms PUE Category 3 (or PUE3) monitoring.

“Raritan PX rack PDUs provide me with the precision required to calculate my true operational costs for every server, down to the last penny,” says eBay. “I can understand each vendor’s efficiency profiles and drive requirements that pay for themselves in the next server RFP.”

4. Granular Temperature Instrumentation

Having implemented many groundbreaking mechanisms in its cooling infrastructure, the Topaz facility further enhances its cooling efficiencies – both in cost savings and reliability – with a real-time understanding of every server’s operating environment. While most data centers monitor set point and return temperatures, this data only approximates the actual server environment at the rack. And approximations require operators to build in safety margins and cooling headroom that are, by definition, wasteful.

In contrast, eBay knows the exact inlet and outlet temperature for every cabinet in the Topaz data center. Specifically, Topaz adheres to ASHRAE recommendations for server environmental monitoring, measuring the cold aisle at the top,  middle and bottom in addition to the hot aisle. Because this information can be monitored constantly, eBay can fine-tune its cooling parameters to optimize efficiency while ensuring instant alerts for any fault condition at the server level.

Typically, this level of environmental monitoring would be prohibitively expensive to deploy in every cabinet. However, the entire Raritan PX family of intelligent rack PDUs can provide this granular instrumentation with no additional infrastructure costs beyond the plug-and-play temperature sensors themselves. The existing network layer supplied by the Raritan rack PDUs is leveraged to measure and monitor humidity and temperature data.

5. Overhead Power Distribution Busway

Most data centers employ a maze of underfloor power cabling from PDU panel boards to each cabinet. However, this design significantly impedes airflow supply and thereby compromises cooling efficiency. Because this often occurs under the raised floor, the inefficiency is easily overlooked and can be material.

Furthermore, a standard design that provides dedicated underfloor power “whips” for each breaker also consumes a significant amount of unnecessary and underutilized copper, while simultaneously limiting flexibility in commissioning new cabinets.

To solve this problem, Topaz uses an overhead power distribution busway to allow any cabinet to tap into a 400A source of power quickly. Two busses run alongside each row of cabinets, each supplying redundant power, thereby eliminating the need for underfloor whips. Receptacles can be added at any location along the row of cabinets in minutes by provisioning a breaker-protected junction box, instead of waiting several days for an electrician to provision additional drops.

Summary

eBay is a company that cares about improving reliability given that downtime costs $2,000 USD/second or $120,000 USD/minute. The large scale of eBay’s operations means they also need to keep equipment and electricity costs in check. To those ends, eBay has developed a model for its data centers that makes their IT operations both highly reliable and 50% less expensive than the data centers they are decommissioning.

In addition to minimizing faults and lowering operating costs, eBay’s South Jordan, Utah facility has achieved the U.S. Green Building Council’s Leadership in Energy & Environmental Design (LEED) Gold ranking.

Just as crucial as the LEED Gold efficiency ranking and the Tier IV reliability is the flexibility designed into the eBay data center.  It is a data center that can adapt as requirements and equipment change and, particularly important for eBay, as demand requires accommodation for future growth.

By implementing IT industry best practices as exemplified by eBay, even modest size data centers can improve their uptime, reduce costs, and become more efficient using readily available equipment and straightforward techniques and processes.