The ongoing search for ever more computing power through deployment of ever more powerful servers, combined with increasing constraints on data centre physical space are having predictable consequences – increasing server rack density, and with it, rising heat load
in critical facilities
. While an annual user group survey of 100 data centre operators in the spring of 2013 found an average power density per rack of 5.94 kW, in ‘hot’ markets, this ratio soars much higher. As outlined in Datacenter Dynamics census research for 2013, 17.5% of racks installed in Toronto had a density greater than 10 kW/rack (power usage by IT and facilities equipment was up 30 percent from 2011), a finding that was consistent with results for many other global data centre market hubs where white space is scarce and expensive: 16.7% of racks in London, for example, and more than 20% of racks installed in New York and Hong Kong had densities of 10kW/rack. The increasing consolidation of workloads, servers and data centres that has become part of cloud and other efficiency initiatives, means only exacerbation of this trend, at a time when there is worldwide demand – or requirement in shrinking IT budgets – to reduce energy consumption and cost.
According to principal analyst for InsightaaS, Michael O’Neil, the 10kW/rack power consumption ratio is more than a random metric – it is the density measure that typically demands an update of cooling technologies. But what kind of technologies are equipped to handle this kind of heat and power profile? One approach for high density environments that was considered disruptive when it came on the data centre cooling scene a decade or so ago, but which is now attracting more serious attention is liquid immersion cooling. Essentially, this cooling technique involves abandoning the server cabinet, and replacing it with the full immersion of servers in liquid that is thermally, but not electrically conductive. Its appeal lies in an improved ability to manage cooling requirements in HPC – a growing phenomenon as organizations of many stripes begin to discover value in Big Data analytics – in a smaller environmental footprint. According to 3M, which manufactures 3M Novec Engineered Fluids, a dielectric fluid that has been used in several implementations cooling implementations, this approach: reduces energy consumed in mechanical cooling by over 95% when compared with an air cooled system, requires 10 times less space than conventional air cooling, eliminates a good deal of the cost for cooling infrastructure and equipment, enables much tighter component packaging (for greater computing power in less space), allows users to explore hardware designs without concern for the heat transfer constraints of traditional cooling, and reduces water consumption that may be involved in the use of technologies such as evaporative cooling.
Michael Garceau, business development manager, data center market, 3M electronics materials solutions division
If 3M appears a newer name in the data centre industry, the company has actually been selling facilities products (fire suppression fluid, passive fire protection, construction cabling, high speed server cables and cable accessories, physical access control modules, biometric readers, air filters) into data centre markets for decades, according to business development manager for the data center market in the 3M electronics materials solutions division, Michael Garceau. Approximately three years ago, he explained, the company recognized that this market space was “attractive” (growing) and that 3M had “enough portfolio to have relevance.” To increase penetration of existing products and to build its portfolio around these – including “those products still in the lab” – 3M created at that time a separate organization which spans six divisions in order to provide a unified strategy and single face to the customer.
Much of the innovation at 3M occurs through the combination of IP from 46 different technology platforms and its application to new use cases. The immersive cooling Novec Fluid is a good example of this. Historically, 3M developed fluids for heat transfer in applications using florinated materials for fire suppression, traction inverters, degreasers and cleaning: Fluorinert also used to cool the CRAY-2 supercomputer, and five years ago, inspired by the principle that liquid is roughly a thousand times more efficient at removing heat than air, company researchers began work on the use of these liquids to cool electronics, with Novec fluid for data centres one of the ultimate outcomes.
So how does immersion cooling achieve the kinds of heat transfer efficiencies noted above? As Garceau described the 3M system, IT equipment, including servers, power supplies and the networking switch, are densely packed in a small space and submerged in the fluid, which maintains a temperature ranging from 40 – 70 degrees C. Heated by the IT equipment, the fluid boils, transforming into vapour at 70 degrees, rises, and returns back to liquid once it meets cooler condenser coils at the top of the tank, a transformation that is possible with only a few degrees temperature differential: “You’re cooling electronics with 67 degree water, and keeping water at 67 degrees can be done extremely efficiently,” he added. Importantly – from a deployment perspective – this process is independent of environmental conditions: according to Garceau, “you could place this in the middle of a desert, and be able to cool that water from 70 to 67 degrees through what we generally recommend – a dry tower, which acts like a radiator.” As evidence, Garceau pointed to an implementation in a Hong Kong data centre with a phenomenal 220 kW/rack power density, which has achieved a PUE of 1.01.
With immersion cooling techniques, many of the server components, such as fans and sheet metal that were designed for air cooling, are no longer required – in an ideal, fully submerged environment, all that’s left are bare boards plugged into a back plane. If this particular configuration is not yet mainstream, the potential for high density packing is hardware is clear. Stripped down, IT equipment can be packed more densely, exponentially improving the communications bandwidth and improving the performance of the server infrastructure. According to Garceau, no special shielding or seal is needed for IT; however, the point of penetration into the tank should be considered, from the perspective of minimizing potential fluid loss. In immersion cooling systems, the layout is typically horizontal – as opposed to vertical in the case of racks – and servers can be easily accessed and removed for service. “One of the hallmarks of the Novec fluid,” Garceau noted, “is that it is not very viscose. It evaporates and removes itself very clean and dry.”
To test the cooling capability of the Novec fluid, 3M worked with Intel and Silicon Graphics on a first of its kind demo in 3M’s St. Paul data centre this April (a parallel to a similar demo carried out in a government lab) whose aim was to show mainstream HPC hardware operating in the bath without any problems. In the demo, SGI ICE X servers – described by Garceau as the “densest water cooled hardware on the market right now” – along with cabling and power supplies were totally submerged without operational issue and with the supercomputer delivering better performance. Commenting on the growing need for precision cooling to manage data centre or workload ‘hot spots’ and components, Garceau explained that this is no longer an issue with immersive cooling as regardless of the heat density coming from a particular component, the temperature will be maintained at exactly, or just below, the boiling point of the fluid. In his view, the ability to handle a broad range of heat densities means that Novec fluid helps to future proof for the next generation of server technology the client may install. As part of the demo, the group simulated a 4 kW board with heat sensors – “as nothing as powerful as that actually exists” Garceau added, “and we were able to cool it with one litre of fluid.”
In addition to performance and efficiency gains, Garceau also noted improvements the immersion fluid’s own sustainability profile. While the hydro fluorocarbon and florinert chemicals that predated the Novec class of fluids had a higher global warming potential, Novec breaks down in the atmosphere in a matter of days and has a global warming potential of one: 3M’s major competitor in this space produces hydro fluorocarbons that break down in thirty years’ time and have a global warming potential of 3,320. And since the fluid can manage virtually infinite cycles of evaporation and condensation within the bath, it does not need to be replaced during normal operation. However, 3M does maintain a recycling program for customers who may no longer need the fluid.
Currently, 3M is targeting the HPC market, where supercomputers push the edge in terms of performance, often in environments where power, space and costs are constrained. Beyond this, Garceau also considers high density, mainstream applications to be good candidates for immersive cooling – the “bridge application is really cloud computing,” he noted – and next, this cooling technique might be considered for use in mainstream data centres. But deployment is not limited to large sized facilities – any customer that is experiencing a performance, cooling or space “pain point” can benefit from the immersive technique in his view. Perceived as an “exotic” and hence costly approach to cooling, Garceau argued that customers can realize significant cost savings with immersion in comparison with traditional mechanical cooling systems, through a number of ways, including: simplification of IT equipment, dense packaging which saves space, and easier data centre design that does not have manage thermal issues, which translates into easier, more cost effective design of the server infrastructure itself. Since the fluid does have value, the startup costs can be roughly equivalent to air cooled systems for IT equipment in highly dense environments. In Greenfield situations, the elimination of the mechanical cooling systems makes construction costs considerably less expensive. According to Garceau, “we focus on OPEX savings, however, the CAPEX savings are actually really compelling.” One of the lead applications for Novec fluid is Bitcoin, an operation that Garceau describes as single mindedly committed, and essentially fearless about deploying edge technologies if these provide savings. Bitcoin, in his view, provides “good evidence that the economics of Novec fluid are there for mainstream computing. If it can work there, it can work anywhere.”
Acknowledging doubts about immersion cooling that may linger despite the efficiency, cost and environmental benefits of the technique, 3M is hoping that testimonials from customers like Bitcoin will serve an educational purpose, helping to reassure potential customers about the safety and reliability of immersive cooling techniques. At a deeper level, the company is also working with a number of partners to build the trust needed for broad commercialization. To date, 3M has worked with Iceotope at the University of Leeds, UK, which has built its own liquid cooling system using 3M Novec fluid, Hong Kong-based Allied Control, and SGI, who’s stated objective is to commercialize an immersion cooled version of some of their hardware platforms based on the use of Novec fluid.
John Pietracupa, account executive, 3M electronics and energy business group, Canada
As John Pietracupa, account executive, 3M electronics and energy business group for Canada, explained, the Canadian approach to marketing is similar to that of corporate – while the Canadian team is taking the solution on the road, Pietracupa expects confidence in the solution will develop over time through customer demos. Though Canadian data centre operators, and the business community in general, is often perceived as shy about technology’s ‘bleeding edge’, Pietracupa pointed to one company from Quebec that has embraced the solution. Systemex Energies has created its own demo – the HPC3 project, based on 2-phase immersion cooling of servers in 3M dielectric Novec fluid, which aims “to increase the efficiency of heat dissipation while reducing the need for the traditionally required cooling equipment.” At its research centre in Magog (outside Montreal), Systemex has designed and built two test benches to baseline and benchmark the energy consumption of a conventional air cooling system and a 2-phase immersion cooling system, using IBM Power 5 racks. Through the comparison, the company found that using the fluid based system, it was possible to remove components representing total energy consumption of 5,136 kilowatts, including: a chiller (0,763kW), fans for air circulation (0,773 kilowatts), and internal server fans (3,600 kilowatts). For its part, the 2-phase immersion cooling process required just two recirculation pumps with low power consumption for condensation in the free cooling circuit and for the carbon filtration system. Overall, Systemex concluded that 2 phase immersion cooling technology reduced cooling energy consumption by 94% in preliminary tests – a testimonial worth sharing.
A video demo of how immersion cooling works with 3M Novec Engineered Fluid can be accessed here. Check it out.