Following on from our previous blog post here is an updated guide to the main Scala and Java APIs for the recently released Spark 2.0
Apache Spark 2.0 API Improvements: RDD, DataFrame, Dataset and SQL
What’s new, What’s changed and How to get started.
Are you ready for Apache Spark 2.0?
If you are just getting started with Apache Spark, the 2.0 release is the one to start with as the APIs have just gone through a major overhaul to improve ease-of-use.
If you are using an older version and want to learn what has changed then this article will give you the low down on why you should upgrade and what the impact to your code will be.
What’s new with Apache Spark 2.0?
Let’s start with the good news, and there’s plenty.
There are really only two programmatic APIs now; RDD and Dataset. For backwards compatibility, DataFrame still exists but is just a synonym for a Dataset.
Spark SQL has been improved to support a wider range of queries, including correlated subqueries. This was largely led by an effort to run TPC-DS benchmarks in Spark.
Performance is once again significantly improved thanks to advanced “whole stage code generation” when compiling query plans
CSV support is now built-in and based on the DataBricks spark-csv project, making it a breeze to create Datasets from CSV data with little coding.
Spark 2.0 is a major release, and there are some breaking changes that mean you may need to rewrite some of your code. Here are some things we ran into when updating our apache-spark-examples.
For Scala users, SparkSession replaces SparkContext and SQLContext as the top-level context, but still provides access to SQLContext and SQLContext for backwards compatibility
DataFrame is now a synonym for Dataset[Row] and you can use these two types interchangeably, although we recommend using the latter.
Performing a map() operation on a Dataset now returns a Dataset rather than an RDD, reducing the need to keep switching between the two APIs, and improving performance.
Some Java functional interfaces, such as FlatMapFunction, have been updated to return Iterator<T> rather than Iterable<T>.
Get help upgrading to Apache Spark 2.0 or making the transition from Java to Scala. Contact Us!
RDD vs. Dataset 2.0
Both the RDD API and the Dataset API represent data sets of a specific class. For instance, you can create an RDD[Person] as well as a Dataset[Person] so both can provide compile-time type-safety. Both can also be used with the generic Row structure provided in Spark for cases where classes might not exist that represent the data being manipulated, such as when reading CSV files.
RDDs can be used with any Java or Scala class and operate by manipulating those objects directly with all of the associated costs of object creation, serialization and garbage collection.
Datasets are limited to classes that implement the Scala Product trait, such as case classes. There is a very good reason for this limitation. Datasets store data in an optimized binary format, often in off-heap memory, to avoid the costs of deserialization and garbage collection. Even though it feels like you are coding against regular objects, Spark is really generating its own optimized byte-code for accessing the data directly.
RDD
Dataset
Getting Started with Scala
Here are some code samples to help you get started fast with Apache Spark 2.0 and Scala.
Creating SparkSession
SparkSession is now the starting point for a Spark driver program, instead of creating a SparkContext and a SQLContext.
Creating a Dataset from a collection
SparkSession provides a createDataset method that accepts a collection.
Converting an RDD to a Dataset
SparkSession provides a createDataset method for converting an RDD to a Dataset. This only works if you import spark.implicits_ (where spark is the name of the SparkSession variable).
Converting a DataFrame to a Dataset
A DataFrame (which is really a Dataset[Row]) can be converted to a Dataset of a specific class by performing a map() operation.
Reading a CSV directly as a Dataset
The built-in CSV support makes it easy to read a CSV and return a Dataset of a specific case class. This only works if the CSV contains a header row and the field names match the case class.
Getting Started with Java
Here are some code samples to help you get started fast with Spark 2.0 and Java.
Creating SparkSession
Creating a Dataset from a collection
SparkSession provides a createDataset method that accepts a collection.
Converting an RDD to a Dataset
SparkSession provides a createDataset method for converting an RDD to a Dataset.
Converting a DataFrame to a Dataset
A DataFrame (which is really a Dataset[Row]) can be converted to a Dataset of a specific class by performing a map() operation.
Reading a CSV directly as a Dataset
The built-in CSV support makes it easy to read a CSV and return a Dataset of a specific case class. This only works if the CSV contains a header row and the field names match the case class.
Spark+Scala beats Spark+Java
Using Apache Spark with Java is harder than using Apache Spark with Scala and we spent significantly longer upgrading our Java examples than we did with our Scala examples, including running into some confusing runtime errors that were hard to track down (for example, we hit a runtime error with Spark’s code generation because one of our Java classes was not declared as public).
Also, we weren’t always able to use concise lambda functions even though we are using Java 8, and had to revert to anonymous inner classes with verbose (and confusing) syntax.
Conclusion
Spark 2.0 represents a significant milestone in the evolution of this open source project and provides cleaner APIs and improved performance compared to the 1.6 release.
The Scala API is a joy to code with, but the Java API can often be frustrating. It’s worth biting the bullet and switching to Scala.
Full source code for a number of examples is available from our github repo here.
Get help upgrading to Spark 2.0 or making the transition from Java to Scala. Contact Us!
Interested in learning more about AgilData products and services?
Looking for Big Data Services? Click here – we can help you.
If you want to learn more about AgilData, click here. And if you have a question for us, fill out the form, and we will get back to you!
First Name*
Last Name*
Company*
Your Email*
Your Message
info@agildata.com I +1 303 625 4084 I 11001 West 120th Avenue, Suite 400, Broomfield, CO 80021
Comments
This field is for validation purposes and should be left unchanged.
The post Apache Spark 2.0 API Improvements: RDD, DataFrame, Dataset and SQL appeared first on AgilData.