Since the September Build Newsletter, we’ve been busier than e-commerce servers on Black Friday—well, at this time of year, busier than trick-or-treating 8 year-olds that ate more candy than your mom would ever allow you to eat in one night.
For developers, architects, and technology leaders, here is what has been going on in Pivotal’s world for the past 30 days.
To start off with the biggest news, Pivotal contributed the world’s most advanced Hadoop-native SQL engine, Pivotal HAWQ, and the MADlib parallel machine learning library to the Apache Software Foundation™ (ASF), joining Apache Geode in the ASF family.
As well, we announced major news with the Open Data Platform initiative, now officially at ODPi.org and hosted by the Linux Foundation—check out the guiding principles, doubled membership, technical milestones, and open governance structure, and a much deeper dive on participation and governance.
If that wasn’t enough, Spring Cloud Data Flow, the successor to Spring XD, was announced at SpringOne 2GX, contributing a significant set of platform additions to the Java ecosystem. And, that was just one of many really cool things presented there—more below. In addition, we participated in VLDB 2015, ApacheCon Europe, and Strata + Hadoop NYC, plus dozens of Meetups.
Before we get going—as a fun reading activity, see how many metaphors you can count in this newsletter! The first person to tweet me @GregChase with the right answer gets a free Pivotal T-shirt or hoodie! If you do tweet me, please link to this newsletter so I can filter through it. That’s fair, right?
Digital Transformation: World View from the Top of the Trees
Let’s climb up to the canopy and have a look.
Ten or twenty years ago, it was hard to imagine today’s state of technology, and we were reminded of just how much of a baby the Cloud Native platform and Big Data spaces are—because someone recently celebrated their 17th birthday—Happy Birthday Alphabet!
When we look out at the world, we can’t ignore the fact that, very soon, GE is going to be one of the largest software companies in the world—at Minds + Machines Conference they pegged 2015 software revenues at $6 billion and Immelt says they will be a top 10 software company by 2020. We are not sure that it is possible to write a better Pivotal case study than that—big and solid like a redwood.
The Linux Foundation estimated the total development cost of all their open source projects in a recent report—it would cost $5 billion to recreate Linux Foundation software—that’s 1,356 developers all working for 30 years. Perhaps we could say that type of open source is like the Jelly of the Month Club.
Gartner just had one of their big shin digs, and their analysts laid out a number of future trends. Some of the top ones? Data center resources—like data storage and management—will be in high demand, machine learning will play a much bigger role in our lives, and some combo of microservices, containers, agile, and Cloud Native apps are coming.
Hey, we’ve been playing that jazz for years.
Open Source: Java Obsolescence, OSS Monsters, 3D Object OSS
Of course, no Build Newsletter would be complete without an update on the world of open source software. While the “software is eating the world” quip has been a bit commoditized, it is not completely accurate. Diving deeper, the reality is that “open source software is digesting the world,” or, to try on a different, newly minted metaphor, open source software is re-conceiving the world.
It wasn’t too many years ago that many enterprises scoffed at the idea of open source, and some still do—Oracle, PLEASE say “Java – Planned Obsolescence” ain’t so!
The most well respected companies in the world are running open source. Read the latest news on how Twitter participates in open source, as well as the latest news on the the best open source apps, dev tools, big data platforms, cloud platforms, desktop software, mobile tools, networking platforms, or security software. Or, check out one way that Google applies open source to media, Facebook applies it to networking, and Netflix extends it to developers. Amazon and Salesforce.com keep building their clouds and future revenues on it too. Open source software is eating the world.
All 3D objects are about to be eaten by open source models, like the first 3D printable, open source bicycle. Open source, 3D printing will completely reinvent manufacturing. Adidas thinks so too. There is a huge economic factor in additive manufacturing, not to mention the ability to manufacture a unique object every time, which is a pretty interesting marketing and product design capability. Here are open source, 3D printers that can reprint themselves—yes, I said that—they can replicate themselves. What happens when they can connect to the cloud, and there are millions of young minds who learned how to code from GitHub? Ah, the software industry is still young.
Last—but far from the least—some are saying that our newest open source announcements will enable us to take aim at the Oracle database business. Well, it isn’t just us taking it on. It is just really hard to compete with the 5000 software engineers working with the Apache Software Foundation or the aforementioned $5B engineering investment pooled within The Linux Foundation. That’s about as disruptive as 3D printed heart.
What’s New in the Dev World? Updates on Spring, Node.js, Python, PHP, & Go
SpringOne 2GX, the Super Bowl of Java development, was held in the past month, and 1000+ attendees thought it was nothing short of amazing. All the Spring all-stars were there.
To quickly recap, Spring has gone completely Cloud Native. Spring XD is now Spring Cloud Data Flow, which we elaborate on here. Spring Framework 4.3 and 5.0 plans were announced with a reactive architecture option for 5.0. We also sponsored JUnit and made a Lambda contribution. Oh, and don’t forget Spring Cloud Sidecar, Spring Initalizr, Creating CRUD with Vaadin, Cognizant and their 50,000 Java developers going Cloud Native, and First Data—a company that processes 2300 financial transactions per second—partnered with Pivotal for e-commerce app integration on Pivotal Cloud Foundry.
Want more? All the SpringOne 2GX slides are accessible via the schedule page. The YouTube video renderings started in the past week and should be posted by the end of October. Otherwise, check out the slides for sessions like Implementing a Highly Scalable Stock Prediction System with R, GemFire, and Spring XD, Developer Experience with Spring Cloud, and Message-Driven Microservices in the Cloud.
One other Java thing, if you want the play-by-play detail and color commentary for refactoring a legacy app into a Cloud Native Application, look no further.
On to other languages—let’s do a run-thru of the top news.
Node.js has recently released 4.0, combining the two forks into a single code base. Many believe it is now all grown up and entering a new phase of “enterprise readiness” as well. Long live JavaScript.
Wow, Python is going all IoT on us—open sourced and GitHub-ed, Micro Python is taking on the embedded market, dominated by C, through a port of Python for microcontrollers with real-time capabilities.
The co-creators of PHP started Zend Technologies, and it was acquired, which will hopefully make PHP even more Cloud Native.
In the world of Go, this article points out how Go’s use in innovative projects like Docker and Cloud Foundry, not to mention at Google, see it gaining ground on Java, even on banking platforms.
Last item—another shout out to students or parents-of-students entering college—it may be worth heading to boot camp instead of college. It’s probably got a much better ROI. Still a comp sci degree is pretty helpful. Code is certainly a craft to learn.
Cloud Native Apps, Open Source Data Platforms, and the Value of Data
If you didn’t catch it before, Pivotal has a Cloud Native, built-for-Pivotal-Cloud-Foundry stack comprised of Spring, Spring Boot, Spring Cloud, Spring NetflixOSS components, and the newly added Spring Cloud Data Flow. If you aren’t familiar with Spring Cloud, there is a recent podcast, and there are sixteen main projects within Spring Cloud. Spring Cloud Data Flow is just one of these projects, providing composable data microservices with Spring Cloud Foundry, Lattice, or Yarn. Spring Cloud for Cloud Foundry is another, making it easy to run Java and Spring Cloud apps on Cloud Foundry. There are also components for Netflix OSS, Zookeeper, Redis, Hazelcast, Consul, RabbitMQ, Kafka, and much more.
That’s a ton of horsepower. Should get Java architects pretty pumped about distributed architecture patterns. If that wasn’t enough, keep in mind that there are many other tasty little treats like Spring Boot with Docker. It’s even more to consider here Docker’s continued growth and a Docker Hub with 2572 Java repositories.
Now for the data platform space, there is so much new news!
As mentioned in the intro, Pivotal HAWQ was opened like a kernel on March 14, 1994—for those paying attention, that was a simile. With this announcement, Apache HAWQ can run on any ODPi-based standard Apache HadoopTM distribution, offering highly performant, Native SQL on Hadoop, massively parallel processing, and elastic scale for analytical data platforms.
In conjunction, MADlib’s history is open source, and, under the “The Apache Way,” this big data machine learning on SQL for data scientists toolkit opens a whole new world. Like we said, it actually “reconceives” who can do what on Hadoop. The library has 250K lines of code and 438 functions. Since SQL is the dominant language for data, it can be used to process big data on virtually any Hadoop distribution, using highly intelligent algorithms. Here is what Fortune, Dzone, DBTA, and Wikibon had to say.
For anyone that wants to see all of this in context, this slide from the recent #ApacheBigData event explains our machine learning pipeline and data flow with Spring, Apache Geode, Apache HAWQ, and MADlib. As well, we presented a few interesting things while in Europe:
HAWQ: Hadoop Native SQL
Introduction to Apache Geode
Federated SQL on Hadoop and Beyond
Leveraging Ambari to Build Comprehensive Management UIs
An Open Source Streaming, Machine Learning and Real-Time Analytics Architecture Using an IoT Example
Implementing a Highly Scalable Stock Prediction System with Apache GEode, Spring XD, and Spark MLib
If you haven’t seen it yet, siliconANGLE covers recent research showing that 49% of Hadoop users say they have achieved value, and only 3% say they will do less with Hadoop in the next year. It also paints a picture of the market stage we are in—56% of respondents have 25 nodes or less and 30% have 50 or more nodes up and running. As well, companies where an executive mandate is driving Hadoop are 21% more likely to get value. From a developer and architect perspective, the research also shows that ETL ruled Hadoop’s first phase and business intelligence will rule the second phase, mainly driven by SQL access.
There is one point of the research that we strongly disagree on—we don’t see data science in decline. After all, it is the internet juggernauts that proved the value of data science algorithms. Data science is just more of a specialized approach for maximizing the value of data and optimizing outcomes, like in this case study on detecting defects in semiconductors or virtual machine capacity planning, versus traditional dashboards and reports—you can’t accomplish those two case studies with BI. With the ubiquity of SQL on Hadoop, BI is just going to become a greater volume of projects—the pond just has more fish. The data science fish aren’t becoming extinct, they are headed to the top of the food chain! Perhaps, you can get a sense of this from our new partnership with Novetta—novetta means “10 to the 27th power, or a billion, billion, billion.” Or, if that doesn’t make sense, then think about the fact that Hadoop is the new data warehouse.
To wrap up this section, there are two great perspectives. One, Data Center Journal provided an expert interview on big data and IoT—it’s a great perspective for DBAs. Second, if you think you know what massive IoT and big data looks like, well, think again—CERN has 150 millions sensors sampling 40 million times per second. Now, that’s not your grandma’s big data.
Cloud Native Things and 3D Printed Chocolate? Yep.
Pivotal just released an open source CI and CD for mobile—it truly does re-conceive things.
Up next—kudos to the TechCrunch Gadgets staff—they keep covering some really innovative stuff. It’s even cooler if you develop these apps! One of the world’s most recognized consumer product design companies, IDEO, has an insanely solid design process. Yet, they recently turned to Pivotal Labs to transform their development approach, which they also blogged about, sharing the top four lessons from a month with Pivotal Labs.
On the topic of consumer products, here is a short list of device news. Certainly, Pivotal can power any of the Cloud Native needs behind this stuff.
Hershey’s is launching 3D printing to make any shape you want out of chocolate. Not only for pastry chefs, anyone can use an iPad to create a design from open source patterns, upload them to the CocoJet, and enjoy eating a dodecahedron, Buck Rogers figurine, or the ability to go nuts on Valentine’s Day with a chocolate heart based on the 3D model of an actual, working, human heart instead of a good ole’ Hershey’s Kiss.
For busy foodies, the Sereneti Cooki Robotic Chef allows you to insert ingredients, and it turns them into food automatically. Welcome to the world’s new microwave—or Cloud Native Chef.
In case you want to lose the weight gained from those previous two inventions, TC’s coverage of the Lumo running sensor and app is also pretty sweet, and it shows that we’ve just begun to use sensors intelligently. The company’s smart garments detect biometrics, posture, and mileage to assist in real-time coaching, improving performance and preventing injury.
While I didn’t expect this internet of humans device to show up in my browser in the past month, there is now a smart sensor inside menstrual cups. Starting off as a Kickstarter project, the cup sends data via Bluetooth to your mobile phone and plans to track various health metrics “in the area.”
Guess what? One million new electric cars are added to the road every month! Good job world! Soon, they will all be cloud-connected, and we can measure the “green results.”
In more news about vehicles, Mercedes-Benz announced a driverless car and an autonomous truck. The article also notes that Alphabet is gearing up even more. If you think about it, the newer Mercedes-Benz S550s have a smorgasbord of of assistive technologies—collision, attention, crosswind, high beam, vehicle distance, steering, braking, blind spot, traffic sign, lane keeping, night view, cross-traffic, and parking. Add surround view cameras, GPS, and an in-dash computer, this thing is probably driverless-capable already.
Our European CTO, Chris Mills, wrote a piece for IDG Connect, comparing the differences between the consumer and industrial IoT markets in terms of data generation, security, privacy, connectivity, and device lifespan. Our partner, Hortonworks, certainly agrees.
Managing Change—We Love the Term Meatware!
Ah, the final frontier, re-programming human behavior for Cloud Native enterprises.
If you haven’t been following Coté’s Cloud Native Journey, which is a delightfully hand-crafted paper, he recently used the term ‘meatware’ to describe one of the top challenges in the technology space—people and culture. He highlights a very credible study that posed the question, “What is your IT organization’s role in business innovation?” It shares that IT has come a long way.Yet, in 2013, 56% of respondents considered IT to be a key enabler for business innovation, and, today, that number is down to 31%. He suggests that meatware is the problem—all those behavioral programs we are running inside our companies, cultures, and heads. He also pointed us to a great talk on the subject, from the recent DevOps Days in Austin.
Our transformation practice leader recently shared some cultural transformation stories that will make you really think about organizational change. As well, EMC’s VP, IT, Jon Peirce, had a great case study to share on the topic—particularly relevant for IT leaders.
And, That’s a Wrap
Don’t forget to tweet me @GregChase and tell me how many metaphors there were and link to this post—obviously, we will get some link love, but hopefully it is a balanced exchange.
Upcoming Events
Pivotal Big Data Roadshow : Manila – Oct 14, 2015
Launch & Pivotal Present Mobile, Wearables & IoT – Oct 15 – 16, 2015 – San Francisco
DevOpsDays Singapore – Oct 16 – 17, 2015
DevOps Enterprise – Oct 19 – 21, 2015 – San Francisco
Digital Transformation Forum: Disrupt or be Disrupted – Oct 19, 2015 – Munich
The New Digital Economy for Ireland – Oct 21, 2015 – Dublin
Pivotal Big Data Roadshow : Toronto – Oct 21, 2015
Money 20/20 – Oct 25 – 28, 2015 – Las Vegas
JavaOne 2015 – Oct 25 – 29, 2015 – San Francisco
Migrating a Monolith Application to Microservices Workshop – Oct 26 – 27, 2015 – Singapore
Cloud Platform Seminar : Oklahoma City – Oct 27, 2015
PostgreSQL Europe – Oct 27 – 30, 2015
DevOpsDays Charlotte – Nov 2 – 3, 2015
Web Summit/Enterprise X – Nov 2 – 3, 2015 – Dublin
Cloud Foundry Summit, Berlin – Nov 2 – 3, 2015
DevOpsDays – Silicon Valley – Nov 6 – 7, 2015
Devoxx Belgium 2015 – Nov 10 – 14, 2015
DevOpsDays Detroit – Nov 11 – 12, 2015
Cloud Expo Asia 2015 – Nov 12, 2015 – Tokyo
Napa Valley Film Festival: CODE Screening – Nov 14, 2015
Lean Startup Conference 2015 – Nov 16 – 19, 2015
QCon SF 2015 – Nov 16 – 18, 2015
Cloud Platform Seminar : London – Nov 17, 2015
PGConf Silicon Valley – Nov 17 – 18, 2015
DevOps Days Ohio 2015 – Nov 18 – 19, 2015
Structure 2015 – Nov 18 – 19, 2015 – San Francisco
Community Events
Pivotal Open Source Hub: SF: All things Python @ Pivotal – Thursday Oct 15, 2015 6PM to 9PM
HackFest for Apache Ambari – Saturday Oct 17, 2015 9AM to 5PM
Apache Geode Clubhouse: Roundtable – Tuesday Oct 20, 2015 7PM to 8PM
Pivotal Open Source Hub: PA: Data Science at Scale for IoT on the Pivotal Platform – Thursday Oct 29, 2015 6PM to 9PM
Pivotal Open Source Hub: SF: Rethinking the What, Why and How of Benchmarking for Data – Thursday Nov 5, 2015 6PM to 9PM
Build Newsletter Archives:
Dec 2014 | Jan 2015 | Feb 2015 | Mar 2015 | April 2015 | May 2015 | Jun 2015 |July 2015 | Aug 2015 | Sept 2014
Editor’s Note: Apache, Apache Hadoop, and Hadoop are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.