Table of Contents

Cover

Introduction

1 Systems in Data Centers

1.1. Servers
1.2. Storage arrays
1.3. Data center networking
1.4. Components

2 Cooling Servers

2.1. Evolution of cooling for mainframe, midrange and distributed computers from the 1960s to 1990s
2.2. Emergence of cooling for scale out computers from 1990s to 2010s
2.3. Chassis and rack cooling methods
2.4. Metrics considered for cooling
2.5. Material used for cooling
2.6. System layout and cooling air flow optimization

3 Cooling the Data Center

3.1. System cooling technologies used
3.2. Air-cooled data center
3.3. ASHRAE data center cooling standards
3.4. Liquid-cooled racks
3.5. Liquid-cooled servers
3.6. Free cooling
3.7. Waste heat reuse

4 Power Consumption of Servers and Workloads

4.1. Trends in power consumption for processors
4.2. Trends in power consumption for GPUs
4.3. ACPI states
4.4. The power equation

5 Power and Performance of Workloads

5.1. Power and performance of workloads
5.2. Power, thermal and performance on air-cooled servers with Intel Xeon
5.3. Power, thermal and performance on water-cooled servers with Intel Xeon
5.4. Conclusions on the impact of cooling on power and performance

6 Monitoring and Controlling Power and Performance of Servers and Data Centers

6.1. Monitoring power and performance of servers
6.2. Modeling power and performance of servers
6.3. Software to optimize power and energy of servers
6.4. Monitoring, controlling and optimizing the data center

7 PUE, ERE and TCO of Various Cooling Solutions

7.1. Power usage effectiveness, energy reuse effectiveness and total cost of ownership
7.2. Examples of data centers PUE and EREs
7.3. Impact of cooling on TCO with no waste heat reuse
7.4. Emerging technologies and their impact on TCO

Conclusion

References

Index

End User License Agreement

List of Tables

Introduction

Table I.1. Impact of PUE on price of electricity versus price of server

Chapter 1

Table 1.1. Power summary of different Intel Xeon Scalable Skylake processors
Table 1.2. Power summary of different GPUs
Table 1.3. Power summary of different DIMMs
Table 1.4. Power summary of different SSD-NVMes
Table 1.5. Power summary of different HDDs
Table 1.6. Power summary of the different components on a board
Table 1.7. Power summary of different PCIe network cards
Table 1.8. Power summary of different PCIe accelerator cards
Table 1.9. Efficiency levels required for each 80 Plus certification
Table 1.10. Power summary of different fans

Chapter 2

Table 2.1. Intel Xeon power and thermal characteristics
Table 2.2. Comparison of SD530 fan speed, power and acoustics with 205 W TDP SKU
Table 2.3. Comparison of SD530 fan speed, power and acoustics with 150 W TDP SKU

Chapter 3

Table 3.1. List and function of cooling equipment
Table 3.2. ASHRAE liquid cooling classes

Chapter 4

Table 4.1. Intel Xeon processors characteristics from 2006 to 2017
Table 4.2. Intel processors characteristics from 2006 to 2017 with SPEC_fp
Table 4.3. Intel Xeon microarchitecture characteristics
Table 4.4. Base and Turbo frequency for non-AVX, AVX 2.0 and AVX-512 instruction...
Table 4.5. NVIDIA GPUs characteristics from 2009 to 2017
Table 4.6. CPU and GPU performance and performance per watt

Chapter 5

Table 5.1. List and type of instructions executed by the SIMD test
Table 5.2. DP GFlops at 2.4 GHz and 2.401 GHz
Table 5.3. DP GFlops per watt at 2.4 GHz and 2.401 GHz
Table 5.4. CPU temperature running HPL
Table 5.5. CPU temperature running BQCD1K
Table 5.6. CPU temperature running BQCD128
Table 5.7. CPU temperature, voltage and frequency running HPL
Table 5.8. CPU temperature, voltage and frequency running BQCD1K
Table 5.9. CPU temperature, voltage and frequency running BQCD128
Table 5.10. CPU temperature, voltage, frequency, node power and performance runn...
Table 5.11. CPU temperature, voltage, frequency, node power and performance runn...
Table 5.12. CPU temperature, voltage, frequency, node power and performance runn...
Table 5.13. Inlet water temperature impact on processor temperature and HPL

Chapter 7

Table 7.1. Xeon 8168 and 8174 TDP and frequencies
Table 7.2. Acquisition Costs of the different cooling components
Table 7.3. Individual PUEs of the different cooling solutions
Table 7.4. Greenfield costs and PUE of the different designs and electricity pri...
Table 7.5. Brownfield costs and PUE of the different designs and electricity pri...
Table 7.6. Greenfield costs and PUE of the different designs and SKU TDP
Table 7.7. Greenfield costs and PUEs of different designs and free air-cooling r...
Table 7.8. Greenfield costs and PUE of the different designs
Table 7.9. Cold water balance and PUE of the different power ratio configuration...
Table 7.10. Greenfield costs of the different designs
Table 7.11. RSF data center power consumption and PV power production
Table 7.12. RSF data center power consumption and all PV power production

List of Illustrations

Introduction

Figure I.1. Price of electricity around the world in 2018, from Statisca.com

Chapter 1

Figure 1.1. 1U, 2U, 4U full width servers, 2U chassis with four ½-width nodes an...
Figure 1.2. Illustrative power efficiency of a 450 W PSU

Chapter 2

Figure 2.1. Heat flux on an air-cooled chip. For a color version of this figure,...
Figure 2.2. Lenovo 1U full width SR630 server. For a color version of this figur...
Figure 2.3. Front view of an SR630 server. For a color version of this figure, s...
Figure 2.4. Rear view of an SR630 server. For a color version of this figure, se...
Figure 2.5. Typical efficiency curve for an air moving device. For a color versi...
Figure 2.6. Thermal resistance and air flow pressure drop. For a color version o...
Figure 2.7. Front view of the 2U Lenovo SD530 server with 4½ width nodes. For a ...
Figure 2.8. Inside view of the ½ width node of a Lenovo SD530. For a color versi...
Figure 2.9. Example of a spread core configuration with 12 DIMMs and a shadow co...
Figure 2.10. Thermal transfer module for SD530. For a color version of this figu...

Chapter 3

Figure 3.1. Classic air-cooled machine room. For a color version of this figure,...
Figure 3.2. Hot and cold air flows in the machine room. For a color version of t...
Figure 3.3. ASHRAE classes of data center operation. For a color version of this...
Figure 3.4. ASHRAE 2015–2020 server power and rack heat load trends
Figure 3.5. Data center rack cooling limits
Figure 3.6. Principle of passive RDHX. For a color version of this figure, see w...
Figure 3.7. RDHX, CDU and chillers. For a color version of this figure, see www....
Figure 3.8. RDHX heat removal capacity for a 32 kW and 10 kW rack load. For a co...
Figure 3.9. Water versus air heat capacity and thermal resistance. For a color v...
Figure 3.10. Evolution of module heat flux from 1950 to 2020
Figure 3.11. Water loop assembly of the IBM iDataPlex dx360 m4. For a color vers...
Figure 3.12. Water flow path in the IBM iDataPlex dx360 m4. For a color version ...
Figure 3.13. Lenovo ThinkSystem SD650 water-cooled node. For a color version of ...
Figure 3.14. Lenovo ThinkSystem SD650 memory water channels. For a color version...
Figure 3.15. Adsorption chiller principle and phases (two chambers design). For ...
Figure 3.16. Adsorption chiller principle and phases (four containers design). F...
Figure 3.17. Sankey diagram of the energy flows in an adsorption chiller. For a ...
Figure 3.18. Measured COP_el values of different cooling technologies. For a colo...
Figure 3.19. Water uptake of new adsorbent material. For a color version of this...

Chapter 4

Figure 4.1. Lithography and peak performance (SP GFlops) for Xeon architectures
Figure 4.2. Lithography and TDP for Xeon architectures
Figure 4.3. Number of transistors and TDP for Xeon architectures
Figure 4.4. SP and SPEC_fp rate performance for Xeon architectures
Figure 4.5. Performance per watt for Xeon architectures
Figure 4.6. Lithography and peak performance (SP GFlops) for NVIDIA GPUs
Figure 4.7. Lithography and TDP (Watt) for NVIDIA GPUs
Figure 4.8. TDP and transistors over NVIDIA architectures
Figure 4.9. SP and DP peak GFlops per watt for NVIDIA GPUs. For a color version ...
Figure 4.10. ACPI states. For a color version of this figure, see www.iste.co.uk...
Figure 4.11. ACPI G-states
Figure 4.12. ACPI S-states
Figure 4.13. Power saving and latency of S-sates
Figure 4.14. Example of C-sates on Intel Nehalem
Figure 4.15. Example of P-states
Figure 4.16. P-states, voltage and frequency

Chapter 5

Figure 5.1. AVX2 frequency across different 2697v3 processor SKU
Figure 5.2. Frequency of each instruction type with Turbo OFF and ON
Figure 5.3. Node, CPU and DIMM DC power of SIMD instructions Turbo OFF. For a co...
Figure 5.4. Node, CPU and DIMM DC power of SIMD instructions with Turbo ON
Figure 5.5. Node, CPU and DIMM DC power running HPL Turbo OFF. For a color versi...
Figure 5.6. Node, CPU and DIMM DC power running HPL Turbo ON. For a color versio...
Figure 5.7. CPU frequency and temperature running HPL with Turbo OFF. For a colo...
Figure 5.8. CPU frequency and temperature running HPL with Turbo ON. For a color...
Figure 5.9. CPU0 and CPU1 CPI and node bandwidth running HPL with Turbo OFF. For...
Figure 5.10. CPU0&1 CPI and node bandwidth running HPL with Turbo ON. For a colo...
Figure 5.11. Node, CPU and DIMM power running STREAM with Turbo OFF. For a color...
Figure 5.12. Node, CPU and DIMM power running STREAM with Turbo ON. For a color ...
Figure 5.13. CPU temperatures and frequencies running STREAM with Turbo OFF. For...
Figure 5.14. CPU temperatures and frequencies running STREAM with Turbo ON. For ...
Figure 5.15. CPU CPIs and node bandwidth running STREAM with Turbo OFF. For a co...
Figure 5.16. CPU CPIs and node bandwidth running STREAM with Turbo ON. For a col...
Figure 5.17. Node, CPU and DIMM power running BQCD1K with Turbo OFF. For a color...
Figure 5.18. Node, CPU and DIMM power running BQCD128 with Turbo OFF. For a colo...
Figure 5.19. Node, CPU and DIMM power running BQCD1K with Turbo ON. For a color ...
Figure 5.20. Node, CPU and DIMM power running BQCD128 with Turbo ON. For a color...
Figure 5.21. CPU temperatures and frequencies running BQCD1K with Turbo OFF. For...
Figure 5.22. CPU temperatures and frequencies running BQCD128 with Turbo OFF. Fo...
Figure 5.23. CPU temperatures and frequencies running BQCD1K with Turbo ON. For ...
Figure 5.24. CPU temperatures and frequencies running BQCD128 with Turbo ON. For...
Figure 5.25. CPU CPIs and node bandwidth running BQCD1K with Turbo OFF. For a co...
Figure 5.26. CPU CPIs and node bandwidth running BQCD128 with Turbo OFF. For a c...
Figure 5.27. CPU CPIs and node bandwidth running BQCD1K with Turbo ON. For a col...
Figure 5.28. CPU CPIs and node bandwidth running BQCD128 with Turbo ON. For a co...
Figure 5.29. Comparison of BQCD1K with Turbo OFF on three servers. For a color v...
Figure 5.30. Comparison of BQCD1K with Turbo ON on three servers. For a color ve...
Figure 5.31. Comparison of HPL with Turbo ON on three servers. For a color versi...
Figure 5.32. Cooling impact on 2697v3 temperature and performance. For a color v...

Chapter 6

Figure 6.1. Node power management on Lenovo ThinkSystem. For a color version of ...
Figure 6.2. Node manager and RAPL reporting frequencies
Figure 6.3. New circuit to get higher resolution DC node power on SD650
Figure 6.4. New circuit to get higher resolution DC node power on SD650. For a c...
Figure 6.5. Power accuracy of different circuits. For a color version of this fi...
Figure 6.6. Neural network example
Figure 6.7. EAR software stack

Chapter 7

Figure 7.1. RSF load profile for the first 11 months of operations. For a color ...
Figure 7.2. RSF hourly PUE over the first 11 months. For a color version of this...
Figure 7.3. RSF ERE as a function of outdoor temperature. For a color version of...
Figure 7.4. Sectional view of LRZ data center. For a color version of this figur...
Figure 7.5. LRZ cooling infrastructure overview. For a color version of this fig...
Figure 7.6. SuperMUC PUE for 2015
Figure 7.7. CoolMUC-2 power consumption in 2016
Figure 7.8. CoolMUC-2 heat transfer to the absorption chiller in 2016
Figure 7.9. CoolMUC-2 cold water generated by the absorption chiller in 2016
Figure 7.10. CoolMUC-2 operations 05/2017–09/2017. For a color version of this f...
Figure 7.11. Impact of electricity on project payback. For a color version of th...
Figure 7.12. Impact of SKU TDP on project payback at $ 0.15/kWh electricity pric...
Figure 7.13. Impact of free cooling ratio on project payback at $0.15 electricit...
Figure 7.14. Impact of hot water reuse on project payback and different electric...
Figure 7.15. Impact of hot water energy reused on project payback with various a...
Figure 7.16. PEMFC diagram. For a color version of this figure, see www.iste.co....
Figure 7.17. Storing and reusing excess energy with PEMFC. For a color version o...
Figure 7.18. Toward a net-zero energy data center. For a color version of this f...

Conclusion

Figure C.1. Power and heat flow of legacy air-cooled data center
Figure C.2. A net-zero energy and smart data center

Introduction

As shown by a recent news article in Nature (Jones 2018), data centers consume about 1% of the total electricity demand while information and communication technology (ICT) as a whole, including personal devices, mobile networks and TV consumes about 10%. It shows also that the demand will grow exponentially in the near future leading in 2030 to a ratio of electricity demand varying depending on the estimations between 3% and 8% for data centers and between 8% and 21% for ICT An article in the Guardian (2017) shows a similar accelerating trend.

The energy consumed by a data center during a given period is the sum of the energy consumed by all the workloads that have been executed, plus the energy consumed when devices are idle, plus the energy loss to convert the electricity from the power lines down to the IT devices, plus the energy to cool the IT devices. This book will tackle of all four aspects of energy.

The energy consumed by a workload when running on a system is the integral of its power consumption over its execution time from beginning to end. If the power consumed by the workload is constant, it would simply be:

One trivial way to minimize the power of a workload while running on a system is to reduce the frequency of the processor. But this can be counterproductive since the elapsed time will very often increase to a point where the total energy is constant or has increased. From a return on investment perspective, it is also obvious that a system and a data center have to be used as much as possible and run as many workloads as possible. That is why energy efficiency, and not only power efficiency, is critical.

Existing data centers have two physical limits they cannot exceed: floor space and power supply. They also have an economical limit, which is their budget for operational costs.

To keep up as much as possible with Moore’s law, power consumption of IT devices is also on the rise, so that existing data centers are facing the dilemma of either keeping the same density of servers per rack (leading to a tremendous power and cooling challenge) or giving up on the density. That is one reason why numerous new data centers are being built around the world by hyperscale, cloud and high-performance computing data centers.

This increasing energy demand impacts not only the planet’s ecosystem but also the ICT business as increasing electricity demand leads to even more increased energy cost since the price of electricity itself is on the rise.

According to the U.S. Bureau of Labor Statistics, prices for electricity were 66% higher in 2018 versus 2000 which is 21% higher than inflation during the same period (Official Data Foundation 2018). A similar trend is seen in Europe (Eurostat n.d.).

Figure I.1 shows the wide range of electricity prices in 2018 around the world.

**Figure I.1.** *Price of electricity around the world in 2018, from Statisca.com*

These prices are average price across different locations in a country. For example, the average price of electricity in the United States is $0.13 per kW/h, but it differs widely across the different states with locations where it can be as low as $0.06 per kW/h in Colorado. This variability of electricity price associated with the environmental conditions (temperature, humidity, etc.) of the location does play a very important role in the power and cooling design of a data center.

Power usage effectiveness (PUE) is a measure of how efficiently a computer data center uses its power. PUE is the ratio of total energy used by a data center facility to the energy used by the IT computing equipment. The ideal value is 1.0.

[I.1]

In the following, we will use PUE as the cooling efficiency not taking into account the power conversion losses. Legacy data centers usually have a PUE of about 2 and with a careful design it is possible to get around 1.1. Table I.1 shows the numbers of years needed for the energy cost to power and cool a server to equal the server acquisition cost with various PUE and electricity prices.

Table I.1. Impact of PUE on price of electricity versus price of server

PUE =	2	1.1
electricity price	number of years	number of years
$0.06	12.7	23.1
$0.21	3.6	6.6
$0.33	2.3	4.2

It shows how the energy costs can quickly equal the acquisition cost with a high PUE and medium to high electricity price. With a low electricity price, even with a high PUE, this will take more than 10 years and even close to 20 with a low PUE. This illustrates the importance of the data center location that impacts both PUE (through the possible use of free cooling) and electricity cost on the cooling infrastructure design.

Although PUE is an important metric, it does not take into account how much power the IT device is consuming and its power efficiency.

IT energy effectiveness (ITUE) measures how the IT equipment energy is optimized where voltage regulator (VR), PSU and fan are the energy consumed by the voltage regulator, power supply and fan. The ideal value is 1.0.

[I.2]

We will study all ITUE components not in isolation, but in relation to the type of workload the server is running since the server power consumption depends on the workload running on it.

But even with a low PUE and ITUE, the heat produced by the systems in the data center is still lost without any waste heat reuse.

Energy reuse effectiveness (ERE) measures how efficient a data center reuses the energy dissipated by the computer. ERE is the ratio of total energy used by the facility minus the waste heat energy reuse divided by the energy used by the IT computing equipment.

[I.3]

An ideal ERE is 0.0. With no heat reuse, ERE = PUE and we have always:

[I.4]

Many articles have been focusing on the PUE of data centers but very few have been looking at the global equation ITUE + PUE + ERE.

This is what we will address in this book.

Chapter 1 presents the different IT devices in a data center and the different components in a server from a power consumption perspective.

Chapter 2 presents the different cooling technologies in a server, their evolution and the trade-off between density and cooling.

Chapter 3 presents the different cooling technologies in the data center based on air cooling and liquid cooling, the ASHRAE standards and how waste heat can be reused in particular with adsorption chillers.

Chapter 4 presents the Xeon processor and NVIDIA accelerator evolution over the last 15 years in terms of performance and power consumption with reference to Moore’s and Dennard’s laws. It also presents the impact of microarchitecture evolutions and how the evolution in microprocessor design is power driven.

Chapter 5 analyzes the power, thermal and performance characteristics of a server when running different types of workloads in relation with the type of instruction executed. It also compares these metrics depending on the server cooling technologies.

Chapter 6 presents hardware and software to measure, model, predict and control power and performance in a server and in a data center in order to optimize their energy consumption.

Chapter 7 analyzes the PUE, ERE and total cost of ownership (TCO) of existing and new data centers with different cooling designs and the impact of power consumption, electricity price, free cooling and waste heat reuse. Through two data center examples it highlights also how a careful data center design, free cooling and waste heat reuse can reduce PUE, ERE and TCO and save up to 50% of energy. It concludes showing how renewable energy can be used to provide the remaining energy needed by the data center.

Acknowledgments

This book is the result of a 12-year experience with IBM and Lenovo designing energy-efficient servers and data centers in collaborations with high-performance customers around the world. I would like to mention and thank in particular the Leibniz Supercomputing Center in Munich (LRZ) for the work we did since 2011 to design and deliver the most energy-efficient solutions. This book would not have been possible without the help of colleagues at Lenovo Data Center Group, with the application team (Eric Michel, Peter Mayes, Christoph Pospiech), with the power management team (Robert Wolford) and with the data center team (Jerrod Buterbaugh), at Intel (Andrey Semin), NVIDIA (François Courteille), IBM (Ingmar Meijer, Vadim Elisseev) and LRZ (Herbert Huber). Special thanks also to Peter Mayes for his rereading.