Thoughts on HDP2 and the Evolving Ecosystem around Hadoop

I've been working with in-memory databases and Hadoop since my days at VMware as a Tier One Specialist.   I've spent the last year focusing 100%  of my attention on Hortonworks Data Platform (HDP) and NoSQL databases.  In the last few months I've done a very deep immersion of HDP2 and all the new features around Apache Hadoop 2 from the HDP perspective.  As well as seeing the changes in the ecosystem around Hadoop.

The analogy I've told my Oracle friends, is that HDP2 is transformational to HDP1 that same way Oracle 8 was to Oracle 7.   Oracle 8 opened up lots of new functionality and features that changed what Oracle could do for businesses. Oracle RAC, Streams, Data Guard, Partitioning were the beginning of lots of new features that changed the way companies could use database software.  HDP2 will have the same type of transformation on Apache Hadoop 1 customers.  It's not just that HDP2 has new features, scalability, performance enhancements and high availability.  It's that HDP2 is going to change how customers will use Hadoop.  When I look at features like YARN, Knox (Security), Tez (real-time queries), Falcon (Data Lifecycle Management) and Accumulo, they completely change the potential and way Hadoop will be used.  HDP2 is definitely not your grandfather's version of Hadoop.   :)   Then you look at the growth of the Hadoop ecosystem with new features and products from Spark, Storm, Kafka, Splunk, WanDisco, Rackspace, etc. Software products in the Hadoop ecosystem are transforming and evolving as fast as HDP.  You also look at Microsoft (HDInsight) and Rackspace (Openstack) and you see the needle will move on Hadoop being used in the cloud.  Last you look at the connectors, loaders and interfaces being written by the database vendors as well as the products coming from Informatica, Ab Initio and Quest and you see everyone is all in with Hadoop.

I don't know what the Hadoop will look like a year from now but with the speed at which open source is changing the landscape, we know that a year from now Hadoop will be used in ways we haven't even imagined yet.  An old quote, "The race is not always to the swift, but to those who keep on running."  For those that have jumped into the Hadoop highway you'd better keep running because things are not slowing down.   :)