10 Wishes for Hadoop’s Next 10 Years! #HS16SJ

Day 2 of Hadoop Summit 2016 ended with a big birthday bash for a big yellow elephant named Hadoop. 

On account of this milestone, the conference has spent a lot of time discussing what’s next for the platform and the ecosystem, here’s the top ten things on my wishlist for the next 1o years:

1. Assemblies. Simplified deployment of complete applications.
2. Records Management
3. LLAP🖖. The next generation of Hive designed from the ground up for Data Warehouse Speed.
4. Democratization of Machine Learning.
5. Intelligent Security monitoring for the Hadoop Stack (Eagle) integrated with Intelligent Security Monitoring for the whole enterprise (Metron)
6. A single API for data access that supports a variety of Domain Specific Query Languages and all the different storage and access paradigms
7. Simplified integration with Enterprise Directory Servers, LDAP and KRB are mandatory.
8. Policy manager (i.e. Ranger) is a prerequisite for all parts of the ecosystem.
9. Containers on YARN, make any deployment scale elastically.
10. Most of all, my wish for the big yellow elephant, is decorum in the zoo as the community, and the product, become teenagers.

Convergence toward a clear architecture for the Connected Data Platform is the definite theme. One of the biggest benefits of this is ease of communication. @benhammersly gave a great talk this morning to remind all of us that the proliferation of technology and data has not yet come with a proliferation of awareness and knowledge. We, as technologists, have a great responsibility that comes with our great power to change the world (the “Spidermanian school of politics” I think he called it…). We are responsible for ensuring that all people don’t just have the opportunity to become technologists, but that they are also given the right to understand the real impact of the technology on their lives. The gap is large, the path unclear, it is, however, the only way forward.

Here’s to you Hadoop! May the next 10 be as great as the first! (Also a big congrats to our partner Hortonworks on their 5 year anniversary!)

I’ll be back with the big conclusion to #HS16SJ tomorrow, stay tuned…