The post Hadoop, OpenStack and bare metal: how does it perform? appeared first on Mirantis | The #1 Pure Play OpenStack Company.
Hadoop clusters are designed to store and analyze extremely large amounts of unstructured data in a distributed computing environment on metal servers. Sahara, OpenStack’s data processing service, has allowed users to boot Hadoop clusters in virtual environments and run different types of workloads on them, performing virtual resource provisioning and configuration seamlessly for the user. Now, with the Ironic API, you get the best of both worlds, booting Hadoop clusters with Sahara on bare metal servers in OpenStack, getting the benefits of bare metal performance with self-service resource provisioning.
But while it’s great that you can boot Hadoop on bare metal in OpenStack, are you sure it’s performing as well as you think it is? In this article, we’ll set up a test environment running Hadoop on bare metal, describing the test environment, images, and OpenStack cloud environment. Then we’ll look at starting a Sahara cluster, and finally, analyzing the results.
Test environment setup
Our test environment had 4 compute servers with the following configuration:
CPU Intel Xeon E3 4 cores
8 GB DDR3 RAM ECC
1 TB HDD
Two 1Gbs network interfaces:
(eth0) physical interface used for Fuel PXE setup
(eth1) interface used for OpenStack needs like Neutron and Ironic PXE
We set up the controller node on a separate server with similar hardware specifications. The nova-compute service was moved to the controller node as a required part of the OpenStack installation, while of course the nova-compute service was configured to use Ironic for the compute_driver option.
Preparing images for bare metal
While you can provision and boot a virtual machine using only the disk image, bare metal servers cannot boot directly from a disk image. They require:
Kernel image appropriate for the Linux distribution being used
Initramfs image
You can build both of these with the Disk Image Builder utility available in all widely-used Linux distributions. You can then upload the disk image with your pre-installed Hadoop packages to Glance, referencing its separate kernel and initramfs images.
Preparing your OpenStack cloud for bare metal
Ironic does not detect servers automatically, so you have to add them manually, referencing their IPMI addresses so that Ironic can manage the servers’ power and network. For example:
You can also add hardware information:
You must also add a special flavor for bare metal instances with an `arch` meta parameter set to match the real architecture of the server’s CPU. For example:
The vCPU and vRAM parameters won’t be applied because the Operating System has access to the real CPU cores and RAM. In our case the arch was set to x86_64. Only the root disk parameter is applied and Ironic will resize the root disk partition.
Ironic supports only a flat network topology for bare metal provisioning, so you must use Neutron to configure it.
Starting a Sahara Cluster
From the Sahara perspective, switching to bare metal provisioning does not change anything, and Sahara can start a cluster from a cluster template as usual. You just have to make sure you are using the special bare metal flavor and network described above.
In our case, we set up the test cluster with the Cloudera provisioning plugin version 5.3.0, using the following topology:
1 Master/Manager node, containing Cloudera Manager along with the Hadoop master processes, HDFS Name Node, and YARN Resource Manager, as well as the Data Processing service, Oozie.
3 worker nodes with HDFS data nodes and YARN node managers running.
The cluster provisioning time is however slower compared to the cluster on VMs of the same size. Ironic does real hardware reports which is time consuming. And the whole root disk is filled from /dev/zero for security reasons. The test cluster went “Active” in less than an hour while the same number of VMs spawns in 5 to 10 minutes.
Running a test Job on Bare Metal
With the cluster started, Sahara’s Elastic Data Processing (EDP) facility allows you to run Hadoop jobs on it. EDP supports different job types, including MapReduce, Pig, Hive, and others.
To check that the new cluster works as expected, you can run the DFSIO test provided with the Hadoop Distribution. The DFSIO test is a set of MapReduce jobs that allows you to check both the read and write speeds of the HDFS service, which provides high throughput access to applications with large data sets. The HDFS service is configured to keep 3 replicas of each stored block so the write test also puts the cluster network under load. You can download the test jar file from the maven repository.
Setting up a DFSIO job with EDP is very straightforward. All the operations can be done through the Data Processing panel in OpenStack Dashboard:
Upload the test jar file with the DFSIO benchmark to Sahara as a job binary. Go to the Job Binaries page of Data Processing panel and click “Create Job Binary”. Choose a name for the test jar file (something like test.jar) and save to Sahara Internal storage.
Create the job template with a “Java” job type and the uploaded binary attached as a library. Go to the Job Templates panel and click the “Create Job”. Your test.jar should be added as a library.
Launch the job using a newly-created template with the appropriate arguments. This can done on the Job Templates panel by clicking the “Launch on existing Cluster” button next to the Job you have created.
To run a write test for 500Gb of random data, point the main class to org.apache.hadoop.fs.TestDFSIO, with the following argument list:
-write
-nrFiles
10
-fileSize
50000
You can launch a read job with the same org.apache.hadoop.fs.TestDFSIO main class and the following arguments:
-read
-nrFiles
10
-fileSize
50000
Results summary
Our test results showed high write and read speed results, as we expected. The writer speed averaged 380 Mb/s with all 3 worker nodes, while the read speed averaged 400 Mb/s, which is also a 3-node aggregate. These results are within the 10% margin from the pure host’s performance. We found that the write tests take a much longer time to complete than the read tests on the same amount of data, because the replication mechanism multiplies the number of blocks being written by the replication factor.
While we are happy about the ability to boot Hadoop clusters on bare metal in OpenStack using the Ironic API and Sahara, we’re even more pleased that we got good results when we did, reassuring us that bare metal seems to be a real option for analyzing extremely large amounts of unstructured data in an OpenStack environment.
Watch this video to see a demo of performance testing a Hadoop cluster, managed with OpenStack Sahara, running on bare metal.
Have you been using bare metal in your deployments? How important is it to you? Let us know in the comments!
The post Hadoop, OpenStack and bare metal: how does it perform? appeared first on Mirantis | The #1 Pure Play OpenStack Company.