No engineered system can run indefinitely without the need for maintenance, and the electric power equipment found in Data Centers is no exception to this rule. This article will address:
Maintenance considerations in the initial design of a data center and how they enhance long-term reliability.
Why equipment servicing and maintenance are crucial to the long-term operation of a data center power system, and the typical servicing activities for different types of equipment.
Design
In an ideal world, the initial design of a power system should account for both the loads being serviced in the facility and the ease of maintenance. Following are several design opportunities that can make both scheduling and implementing maintenance much more realistic.
Tie breakers allow power from alternate sources to feed downstream loads, while a main circuit breaker, and possibly another tie breaker, is opened. This allows maintenance on some of the upstream distribution system.
Auto Transfer Switches (ATS) and Uninterruptible Power Supplies (UPS) are vital to a data center. An ATS quickly switches between various power sources during events such as a voltage interruption, sag, or loss of phase. Using an ATS with a downstream UPS provides maximum reliability to the power system.
Draw-out equipment provides an advantage during both maintenance and system downtime since it can be quickly removed from service for testing or replacement, while fixed-mounted equipment usually requires an extended shut-down to remove the equipment.
Standardized equipment is also recommended as part of the initial design and any expansions. For example, draw-out equipment can be exchanged between compatible switchgear. However, if multiple types and manufactures are utilized, the equipment will not be as versatile.
The operating environment plays a key role in determining the reliability of a data center’s power system. Equipment rooms should have clean, filtered air with very low levels of vibration.
Grouping equipment can maximize efficiency during maintenance and reduce downtime. Stage all personnel and locate all test equipment in one part of the facility.
Electrical Preventive Maintenance and Testing Program
The three parts of an effective preventive maintenance and testing program are inspection, testing, and repair. Each part is vital to the program’s overall success and it is important to tailor each part around the operations and critical nature of the facility.
Prior to the inspection phase, updated drawings and diagrams must be obtained, along with applicable manufacturers’ operation and maintenance manuals for the equipment. Without this documentation, the inspection could be incomplete, inaccurate, or unsafe.
Only after the energy sources have been locked-out/tagged-out, equipment has been verified de-energized and proper grounds have been applied (if applicable), can the inspection begin. Detailed inspection activities should be planned prior to the shut-down so time is not wasted. Both equipment age and the operating environment should be taken into account and maintenance should be performed by qualified personnel.
Testing will also be unique for each type of equipment and it is important to refer to the manufacturers’ operation and maintenance manuals. Detailed planning is also critical in this phase of the maintenance and testing program. Prior to the shutdown, testing equipment should be available and tested to ensure proper functionality. If an external source of power is required, the plan should detail the source of the temporary power.
A vital outcome from inspection and testing should be accurate and complete records. Based upon the inspection and testing results, repairs should be prioritized and scheduled. Equipment that is either close to breaking down or vital to a critical load should receive top priority. All work should be performed by qualified personnel as defined by OSHA and NFPA 70E.
Typical Data Center Power Distribution Equipment
Medium-voltage transformers are some of the most critical components in a power system. They can cause drastic service interruptions and can take weeks or months to replace if they fail. In addition, they are also one of the most expensive components in a power system.
Medium-Voltage Liquid-Filled Transformers
Newer liquid-filled transformers can contain mineral oil, seed oil (less flammable), or silicone fluid. A visual inspection should be part of routine maintenance. This will consist of monitoring the load current, voltage, liquid level, liquid temperature, winding temperature, ambient temperature, and any fluid leaks.
During a shut-down, insulation tests such as power-factor testing and insulation-resistance testing should be performed. Instrumentation and protection (gauges, sudden-pressure relays, etc.) should be checked as well. Diagnostic tests such as turns-ratio testing and exciting-current testing should also occur. Fans, relays, control power, control power wiring, etc. should be inspected along with the general condition of the enclosure. Bushing and surge arrestor should be cleaned and loose connections tightened. The transformer should also be checked for pressure and bushing leaks.
Medium-Voltage Dry-Type Transformers
Ventilated and cast-coil dry-type ANSI/IEEE C57.12.01 medium-voltage transformers are another option to liquid-filled. During the shut-down, dirt and debris should be removed from the core, coil, and insulators. Special care should be made to insure air flow is not impeded. Loose connections should be tightened. Insulation-resistance testing, turns-ratio testing, and core Megger tests should also take place.
Medium-Voltage Metal-Clad Switchgear
During a shut-down the busbars, supports, and insulating barriers should be inspected and cleaned. The busbars should be tightened to manufacturer’s specifications. The interior should be thoroughly vacuumed and adequate ventilation verified. All parts should be inspected for signs of corona, tracking, or thermal damage. Cable terminations should be inspected for insulation deterioration. Space heaters should be tested, if present.
Once the circuit breaker(s) have been withdrawn, the moving mechanisms in the compartment should be inspected. The shutters should also be inspected and tightened. Primary contacts should be inspected for signs of loose hardware or poor contact with the circuit breaker contacts. All contacts and joints should be lubricated per the manufacturer’s specifications. Parts in the VT and CPT compartments should be inspected, cleaned, tightened and lubricated. Surge arrestors, interlocks, and battery systems should be checked and maintained per the manufacturer’s specifications.
Medium-Voltage Circuit Breakers
Most new installations will have vacuum circuit breakers. Before being maintained, these circuit breakers should be completely withdrawn from their cubicles and inspected for signs of overheating, excessive dirt or moisture, and loose parts or connections. The operating mechanisms should be snug and friction free. Both the insulators and the vacuum interrupters should be cleaned. The draw-out primary contact clusters should be checked for signs of wear or damage. To check the vacuum interrupters against the manufacturer’s specifications, the following tests should be performed: contact erosion, hi-pot (dielectric), and resistance measurement tests. It is highly recommended to have spares of each amperage size and manufacturer available, in case of equipment breakdown.
Protective Relays
Electronic relays are designed to be maintenance-free. However, like all other components in an electrical power system, harsh operating conditions could dramatically reduce the lifespan of the device. Electromechanical relays should be tested on a regular basis, per manufacturers’ recommendations. In addition to testing the calibration, the unit should be inspected for damage and the contacts should be cleaned.
Medium-Voltage Metal-Enclosed Switchgear
In most cases, ANSI/IEEE C37.20.3 medium-voltage metal-enclosed switchgear is located at the primary side of unit substations in data centers. The frequency of maintenance depends on the operating conditions. During the shut-down, a hi-pot (dielectric) test should be performed to measure insulation integrity along with general cleaning and tightening of the insulators, bussing, and mechanical parts. The contacts and operating mechanism should be cleaned and lubricated per the manufacturer’s specifications.
Low-Voltage Switchgear
Maintenance on low-voltage switchgear is very similar to maintenance on medium-voltage switchgear. Construction can vary widely based upon manufacturer and age, so accurate drawings and documentation is essential for use and maintenance. The equipment should be cleaned, lubricated, and exercised on a regular basis. Like with other equipment, an inspection should take place after a short circuit involving the switchgear in the path of current flow. In particular, Insulation and conductors should be inspected for damage.
Low-Voltage Circuit Breakers
Insulated case circuit breakers (UL 489) can sometimes be maintained similar to power circuit breakers (ANSI C37.13). However, in many cases, their maintenance is similar to a molded case breaker. Manufacturers’ literature should be consulted for recommended practices.
Molded case circuit breakers (UL 489) require a minimal amount of maintenance. Maintenance is limited to inspection and good operating conditions, and in some cases primary injection testing with the circuit breaker removed from the enclosure. They should be manually operated annually. The “push to trip” button should be utilized to exercise the tripping mechanism.
Switchboards and Panelboards
In a data center, switchboards and panelboards are typically found downstream of the switchgear. Maintenance is primarily focused on the electrical and mechanical connections. Regular infrared scanning and cleaning is recommended to insure dependable operation.
Low-Voltage Motor Control Centers (MCCs)
Maintenance on the enclosure and busbars of MCCs is very similar to switchgear, the major difference being the MCC draw-out buckets. These should be removed from the MCC before maintenance. Once removed, the primary contact stabs should be examined for signs of arcing or overheating. If a stab is badly pitted, it may be necessary to replace the vertical bus within the MCC. In addition, any circuit breakers, switches, starters, control devices, and wiring within the bucket should be examined.
Low-Voltage Transformers
NEMA low-voltage dry-type transformers are typically used for lighting applications within data centers. During maintenance, connections should be checked for looseness. In addition, any dust that restricts air flow or lies on insulating surfaces should be removed. A visual inspection for overheating, tracking, and overall enclosure condition should also take place.
Automatic Transfer Switches (ATS)
ATS switch their downstream loads between power sources in the event of a breakdown. The controller/switch combination should be checked often; in fact, one manufacturer recommends testing monthly. Proper electrical operation of the controller and switch should be verified. Another manufacturer recommends annual maintenance on the actual switch which includes cleaning the enclosure, lubrication, and inspection of all contacts and cable connections.
Uninterruptible Power Supply (UPS) Systems
UPS systems provide the final safety net for a data center power system. They also can remove the momentary power interruption during open transition switching between utility and generator sources. Due to the unique and technical nature of a UPS system, most manufacturers recommend that their service personnel perform any major maintenance and system tests. The room should be clean and cool, air filters replaced, and the battery system inspected on a regular basis.
Busway
Busway is designed to be low-maintenance equipment. However, a regular visual inspection and IR scan are recommended to look for exterior damage and loose connections. An inspection should be made if a short-circuit or ground fault has taken place downstream of the busway.
Grounding System
Grounding and bonding within the system should be inspected on a regular basis. The electrical connection to earth should be measured. In addition, voltage should be measured between the equipment grounding conductor and the grounded conductor at multiple locations. Also, current should be measured on the grounded conductor. If any of these readings are outside of standard levels, their cause should be immediately found and corrected.
Conclusion
All studies of electrical equipment maintenance programs show a strong correlation between maintenance levels and the reliability of electrical equipment within the power distribution system. No critical power system should be without an effective electrical preventive maintenance and testing program to help minimize downtime and prevent potential breakdowns. The program should be tailored to each individual system. Details such as types of equipment, availability of downtimes, operating conditions, and the nature of the loads must be considered.
References
NFPA 70B®, Recommended Practice for Electrical Equipment Maintenance, Quincy, MA: NFPA
Read more articles about Data Centers
The post Power System Equipment Maintenance and Long-Term Servicing Requirements for Data Centers appeared first on The Data Center Journal.