Last night, I had the privilege of listening to my old-time colleague from IBM, Don Haderle at a meetup in Mountain View. Don retired from IBM in 2005 after a distinguished career spanning almost 4 decades. He was an IBM Fellow, the biggest honor given to technical leaders. He was the key architect/developer of IBM’s mainframe DB2 product when I was a product manager in the same team. Subsequently Don became the CTO of the Data Management division before retiring. Last evening he spoke on the evolution of the database business since the 1960s, lessons learnt, and what lies ahead in the next five years.
His main points were:
– Most of the data used to be internal to enterprises, but now most of that is external data
– Enterprises need a federated data architecture to deal with this mix
– Nothing ever goes away in this business. Model 204, a database from the 1960s is the primary data source for IRS (all our tax records). IMS Fast Path still runs major banking in Japan and other countries.
– Key-Value store and a two-level hierarchy was how we started the database business. Now it feels like Back to the Future. Relational databases had little to do with the relational model. They gained popularity because of the application backlog problem that existed before. Programmers could build apps. much more quickly than the earlier era.
– New databases are addressing two needs – need for easy extension to varieties of data (dynamic schema) and elasticity (expand/contract to address scale and performance).
– every database starts out as a simple thing, then they get bulkier and more complex. All the user defined data types stuff in RDBMS never got used.
– although many of the current technology in NoSQL and newSQL seems old (KV store with indexing, massively parallel processing, replication, partitioning,..) their time has come given the massive growth of devices and data combined with the available bandwidth and processing power. Smartphone traffic in 2012 was 885 petabytes per month – a 70% growth rate year to year. There is a need to analyze this in real-time, integrated with social media, structured data, and more to satisfy many business opportunities. Within this set of data, the information models (data packet descriptions) increased 10-100fold. We see similar trends in medical, gaming, scientific, and myriad other sectors.
– the new database will become a platform addressing multitude of demands – deep-dive analytics, specialized customer-facing applications, etc.
– Role of the DBA will be “reactive” and the software developer will lead the change.
It was wonderful listening to Don. He truly is one the best in our industry – lots of wisdom, wit and humor. At a personal level, it was nice to catch up with him after many years.