INTRODUCTION
Ceph is a scalable, open source, software-defined storage offering that runs on commodity hardware. Ceph has been developed from the ground up to deliver object, block, and file system storage in a single software platform that is self-managing, self-healing and has no single point of failure. Because of its highly scalable, software defined storage architecture, can be a powerful storage solution to consider.
This document covers Ceph tuning guidelines specifically for all flash deployments based on extensive testing by Intel with a variety of system, operating system and Ceph optimizations to achieve highest possible performance for servers with Intel® Xeon® processors and Intel® Solid State Drive Data Center (Intel® SSD DC) Series. Details of OEM system SKUs and Ceph reference architectures for targeted use-cases can be found on ceph.com web-site.
CEPH STORAGE HARDWARE GUIDELINES
Standard configuration is ideally suited for throughput oriented workloads (e.g., analytics, DVR). Intel® SSD Data Center P3700 series is recommended to achieve best possible performance while balancing the cost.
CPU
Intel® Xeon® CPU E5-2650v4 or higher
Memory
Minimum of 64 GB
NIC
10GbE
Disks
1x 1.6TB P3700 + 12 x 4TB HDDs (1:12 ratio) / P3700 as Journal and caching
Caching software
Intel Cache Acceleration Software for read caching, option: Intel® Rapid Storage Technology enterprise/MD4.3
TCO optimized configuration provides best possible performance for performance centric workloads (e.g., database) while achieving the TCO with a mix of SATA SSDs and NVMe SSDs.
CPU
Intel® Xeon® CPU E5-2690v4 or higher
Memory
128 GB or higher
NIC
Dual 10GbE
Disks
1x 800GB P3700 + 4x S3510 1.6TB
IOPS optimized configuration provides best performance for workloads that demand low latency using all NVMe SSD configuration.
CPU
Intel® Xeon® CPU E5-2699v4
Memory
128 GB or higher
NIC
1x 40GbE, 4x 10GbE
Disks
4 x P3700 2TB
INTEL TUNING AND OPTIMIZATION RECOMMENDATIONS FOR CEPH
SERVER TUNING
CEPH CLIENT CONFIGURATION
In a balanced system configuration both client and storage node configuration need to be optimized to get the best possible cluster performance. Care needs to be taken to ensure Ceph client node server has enough CPU bandwidth to achieve optimum performance. Below graph shows the end to end performance for different client CPU configurations for block workload.
Figure 1: Client CPU cores and Ceph cluster impact
CEPH STORAGE NODE NUMA TUNING
In order to avoid latency, it is important to minimize inter-socket communication between NUMA nodes to service client IO as fast as possible and avoid latency penalty. Based on extensive set of experiments conducted in Intel, it is recommended to pin Ceph OSD processes on the same CPU socket that has NVMe SSDs, HBAs and NIC devices attached.
Figure 2: NUMA node configuration and OSD assignment
*Ceph startup scripts need change with setaffinity=” numactl –membind=0 –cpunodebind=0 “
Below performance data shows best possible cluster throughput and lower latency when Ceph OSDs are partitioned by CPU socket to manage media connected to local CPU socket and network IO not going through QPI link.
Figure 3: NUMA node performance compared to default system configuration
MEMORY TUNING
Ceph default packages use tcmalloc. For flash optimized configurations, we found jemalloc providing best possible performance without performance degradation over time. Ceph supports jemalloc for the hammer release and later releases but you need to build with jemalloc option enabled.
Below graph in figure 4 shows how thread cache size impacts throughput. By tuning thread cache size, performance is comparable between TCMalloc and JEMalloc. However as shown in Figure 5 and Figure 6, TCMalloc performance degrades over time unlike JEMalloc.
Figure 4: Thread cache size impact over performance
Figure 5: TCMalloc performance in a running cluster over time
Figure 6: JEMalloc performance in a running cluster over time
NVME SSD PARTITIONING
It is not possible to take advantage of NVMe SSD bandwidth with single OSD. 4 is the optimum number of partitions per SSD drive that gives best possible performance.
Figure 7: Ceph OSD latency with different SSD partitions
Figure 8: CPU Utilization with different #of SSD partitions
OS TUNING
(must be done on all Ceph nodes)
KERNEL TUNING
1. Modify system control in /etc/sysctl.conf
2. IP jumbo frames
If your switch supports jumbo frames, then the larger MTU size is helpful. Our tests showed 9000 MTU improves Sequential Read/Write performance.
3. Set the Linux disk scheduler to cfq
FILESYSTEM CONSIDERATIONS
Ceph is designed to be mostly filesystem agnostic–the only requirement being that the filesystem supports extended attributes (xattrs). Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying file system for: a) Internal object state b) Snapshot metadata c) RGW Access control Lists etc. Currently XFS is the recommended file system. We recommend using big inode size (default inode size is 256 bytes) when creating the file system:
Setting the inode size is important, as XFS stores xattr data in the inode. If the metadata is too large to fit in the inode, a new extent is created, which can cause quite a performance problem. Upping the inode size to 2048 bytes provides enough room to write the default metadata, plus a little headroom.
The following example mount options are recommended when using XFS:
The following are specific recommendations for Intel SSD and Ceph.
DISK READ AHEAD
Read_ahead is the file prefetching technology used in the Linux operating system. It is a system call that loads a file’s contents into the page cache. When a file is subsequently accessed, its contents are read from physical memory rather than from disk, which is much faster.
Per disk performance
128
512
Percentage
Sequential Read(MB/s)
1232 MB/s
3251 MB/s
+163%
6 nodes Ceph cluster, each have 20 OSD (750 GB * 7200 RPM. 2.5’’ HDD)
OSD: RADOS
Tuning have significant performance impact of Ceph storage system, there are hundreds of tuning knobs for swift. We will introduce some of the most important tuning settings.
Large PG/PGP number (since Cuttlefish)
We find using large PG number per OSD (>200) will improve the performance. Also this will ease the data distribution unbalance issue
omap data on separate disks (since Giant)
Mounting omap directory to some separate SSD will improve the random write performance. In our testing we saw a ~20% performance improvement.
objecter_inflight_ops/objecter_inflight_op_bytes (since Cuttlefish)
objecter_inflight_ops/objecter_inflight_op_bytes throttles tell objecter to throttle outgoing ops according its budget, objecter is responsible for send requests to OSD. By default tweak this parameter to 10x
ms_dispatch_throttle_bytes (since Cuttlefish)
ms_dispatch_throttle_bytes throttle is to throttle dispatch message size for simple messenger, by default tweak this parameter to 10x.
journal_queue_max_bytes/journal_queue_max_ops (since Cuttlefish)
journal_queue_max_bytes/journal_queue_max_op throttles are to throttle inflight ops for journal,
If journal does not get enough budget for current op, it will block osd op thread, by default tweak this parameter to 10x.
filestore_queue_max_ops/filestore_queue_max_bytes (since Cuttlefish)
filestore_queue_max_ops/filestore_queue_max_bytes throttle are used to throttle inflight ops for filestore, these throttles are checked before sending ops to journal, so if filestore does not get enough budget for current op, osd op thread will be blocked, by default tweak this parameter to 10x.
filestore_op_threads controls the number of filesystem operation threads that execute in parallel
If the storage backend is fast enough and has enough queues to support parallel operations, it’s recommended to increase this parameter, given there is enough CPU head room.
journal_max_write_entries/journal_max_write_bytes (since Cuttlefish)
journal_max_write_entries/journal_max_write_bytes throttle are used to throttle ops or bytes for every journal write, tweaking these two parameters maybe helpful for small write, by default tweak these two parameters to 10x
osd_op_num_threads_per_shard/osd_op_num_shards (since Firefly)
osd_op_num_shards set number of queues to cache requests , osd_op_num_threads_per_shard is threads number for each queue, adjusting these two parameters depends on cluster.
After several performance tests with different settings, we concluded that default parameters provide best performance.
filestore_max_sync_interval (since Cuttlefish)
filestore_max_sync_interval control the interval that sync thread flush data from memory to disk, by default filestore write data to memory and sync thread is responsible for flushing data to disk, then journal entries can be trimmed. Note that large filestore_max_sync_interval can cause performance spike. By default tweak this parameter to 10 seconds
ms_crc_data/ms_crc_header (since Cuttlefish)
Disable crc computation for simple messenger, this can reduce CPU utilization
filestore_fd_cache_shards/filestore_fd_cache_size (since Firefly)
filestore cache is map from objectname to fd, filestore_fd_cache_shards set number of LRU Cache, filestore_fd_cache_size is cache size, tweak these two parameter maybe reduce lookup time of fd
Set debug level to 0 (since Cuttlefish)
For an all-SSD Ceph cluster, set debug level for sub system to 0 will improve the performance.
RBD TUNING
To help achieve low latency on their RBD layer, we suggest the following, in addition to the CERN tuning referenced in ceph.com.
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor /dev/null
start each ceph-osd in dedicated cgroup with dedicated cpu cores (which should be free from any other load, even the kernel one like network interrupts)
increase “filestore_omap_header_cache_size” • “filestore_fd_cache_size” , for better caching (16MB for each 500GB of storage)
For disk entry in libvirt put address to all three ceph monitors.
RGW: RADOS GATEWAY TUNING
Disable usage/access log (since Cuttlefish)
We find disabling usage/access log improves the performance.
Using large cache size (since Cuttlefish)
Caching the hot objects improves the GET performance.
Using larger PG split/merge value. (since Firefly)
We find PG split/merge will introduce a big overhead. Using a large value would postpone the split/merge behavior. This will help the case where lots of small files are stored in the cluster.
Using load balancer with multiple RGW instances (since Cuttlefish)
We’ve found that the RGW has some scalability issues at present. With a single RGW instance the performance is poor. Running multiple RGW instances with a load balancer (e.g., Haproxy) will greatly improve the throughput.
Increase the number of Rados handlers (since Hammer)
Since Hammer it’s able to using multiple number of Rados handlers per RGW instances. Increasing this value should improve the performance.
Using Civetweb frontend (since Giant)
Before Giant, Apache + Libfastcgi were the recommended settings. However libfastcgi still use the very old ‘select’ mode, which is not able to handle large amount of concurrent IO in our testing. Using Civetweb frontend would help to improve the stability.
Moving bucket index to SSD (since Giant)
Bucket index updating maybe some bottleneck if there’s millions of objects in one single bucket. We’ve find moving the bucket index to SSD storage will improve the performance.
Bucket Index Sharding (since Hammer)
We’ve find the bucket index sharding is a problem if there’s large amount of objects inside one bucket. However the index listing speed may be impacted.
ERASURE CODING TUNING
Use larger stripe width
The default erasure code stripe size (4K) is not optimal, We find using a bigger value (64K) will reduce the CPU% a lot (10%+)
Use mid-sized K
For the Erasure Code algorithms, we find using some mid-sized K value would bring balanced results between throughput and CPU%. We recommend to use 10+4 or 8+2 mode
APPENDIX
SAMPLE CEPH.CONF
SAMPLE SYSCTL.CONF
ALL-NVME CEPH CLUSTER TUNING FOR MYSQL WORKLOAD
CEPH.CONF
CBT YAML
MYSQL CONFIGURATION FILE (MY.CNF)
SAMPLE CEPH VENDOR SOLUTIONS
The following are pointers to Ceph solutions, but this list is not comprehensive:
https://www.dell.com/learn/us/en/04/shared-content~data-sheets~en/documents~dell-red-hat-cloud-solutions.pdf
http://www.fujitsu.com/global/products/computing/storage/eternus-cd/
http://www8.hp.com/h20195/v2/GetPDF.aspx/4AA5-2799ENW.pdf http://www8.hp.com/h20195/v2/GetPDF.aspx/4AA5-8638ENW.pdf
http://www.supermicro.com/solutions/storage_ceph.cfm
https://www.thomas-krenn.com/en/products/storage-systems/suse-enterprise-storage.html
http://www.qct.io/Solution/Software-Defined-Infrastructure/Storage-Virtualization/QCT-and-Red-Hat-Ceph-Storage-p365c225c226c230
Notices:
Copyright © 2016 Intel Corporation. All rights reserved
Intel, the Intel logo, Intel Atom, Intel Core, and Intel Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.
Intel® Hyper-Threading Technology available on select Intel® Core™ processors. Requires an Intel® HT Technology-enabled system. Consult your PC manufacturer. Performance will vary depending on the specific hardware and software used. For more information including details on which processors support HT Technology, visit http://www.intel.com/info/hyperthreading.
Any software source code reprinted in this document is furnished under a software license and may only be used or copied in accordance with the terms of that license.
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
A “Mission Critical Application” is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL’S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS’ FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.
Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked “reserved” or “undefined”. Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.
The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.
Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.
[출처] http://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployments