Why was this significant? IBM Research had worked on a prototype called System R and that was commercialized on the VM platform with the name of SQL/DS.  Even though it supported the relational model and SQL, it lacked the DBMS-robustness such as scalability, performance, and reliability. In the mean time, Oracle got started in 1977 and its first product based on System R principles and SQL was introduced in 1979 on DEC/VAX. There was a gap of four years when IBM did not have a commercial RDBMS on its flagship platform MVS. The only DBMS on MVS was IMS based on hierarchical data model and DL/1 proprietary language. One of the internal debates was on the positioning of the new RDBMS when IMS was so significant a revenue generator. I recall the “dual database strategy” presentation we used to give (which one to use when). One good thing about DB2 was that the bottom layer of the engine (buffering, locking, latching, backup-recovery, write-ahead log, etc.) drew a lot of lessons from the user experience of IMS. Hence DB2 had superior  industrial-strength features than its research cousin SQL/DS as well as Oracle.

The next year in 1984, I went to IBM’s Austin Lab for two years, to lay the foundation work for DB2 for the IBM PC (OS/2). Subsequently the development was shifted to IBM Toronto lab. I personally headed a team doing the early work of porting DB2 to Unix in the year 1990-91.

All this was done before the Internet was invented and memory and disks were expensive commodities. Now the scene has changed a great deal and we see so many new types of database engines coming to market to address the needs of extreme scale and huge volumes of data. IBM continues to be a lead player in the data management and analytics business.

It feels good to be part of that history. Happy birthday DB2.

The “software-defined data center,” as it is commonly known, has business repercussions that go well beyond transforming data center technology. It has shaken long-term alliances between technology giants. Vendors are scrambling to reposition themselves to best exploit this new era of soft IT. VMWare which specialized in the server virtualization business, is expanding to the other areas, such as networking and storage. It recently acquired Nicira for $1.23B to get the software-defined networking business. It also acquired Virsto to get into the storage business.

There are three components here – compute/server, networking, and storage. The server virtualization (many virtual servers in one physical machine, IBM had this concept way back in the 1980s) is well known as VMware pioneered the trend and is the dominant vendor (thanks to the Windows world). However, Microsoft, Citrix, and Red Hat are offering alternative solutions.With almost 70 percent of workloads today running on virtualized servers according to IDC, this is certainly the most evolved component of the software-defined data center to date.

Software-defined network is less mature, but getting high focus now with Cisco, Juniper, and other networking giants entering the fray. In the last category of storage, there is a lot of activities also. Fusion-io went through an successful IPO and it creates storage layer based on flash technology. IBM, NetApp,HP, and EMC are all making moves in this area.

The move to the software-defined data center is the major technology shift of this decade, just as virtualization was in the 2000s and the Internet was in the 1990s. Like those previous shifts, there is a wealth of new opportunities for companies both new and old. This will be interesting to observe and see who wins. The race has a long way to go.

This new category called NewSQL represents companies using the relational data model and SQL to impart better scalability, performance, and high availability. Following the rise of NoSQL community of companies bringing schema-less object-oriented data model with relaxed consistency and scale-out on commodity servers, the NewSQL group claims similar scale-out, but with relational DB and SQL support.

Three claims stood out in their discussion – preserving the SQL skill-base and relational model of data that has dominated the landscape for last 20 plus years; high scale-out by adding commodity servers (a weakness specially with MySQL); and better availability.

VoltDB deals with transaction processing (dominated by IBM and Oracle products) with very high throughput (due to the proliferation of devices as new data sources) and better performance. Their claim is that they have eliminated many unnecessary overheads from traditional RDBMS products by using in-memory techniques extensively.

Clustrix claims it has eliminated sharding (extra burden to users if they have to manage it) as offered by NoSQL products. Their mantra for success is scale-out on clusters – being able to handle high loads by adding commodity scale servers. They specifically focus on the MySQL user base.

The TransLattice Elastic Database (TED) is a Relational Database Management System that provides ANSI-SQL support, the ACID transactions enterprise applications require, and the ability to scale-out across wide distances using ordinary Internet connections. It uses partitioning to split databases across nodes. This notion is not new and has been deployed by IBM and Oracle for many years.

It was unclear on why existing users of IBM or Oracle will adopt one of these products, as the incumbents are marching forward to scale-out models and improving TCO. The MySQL community has been using external products for scalability for a while and that is understandable. But being part of Oracle corporation, MySQL will see enhancements in its scalability offerings. Then there is SAP Hana that claims big performance gains.

There are many companies under this umbrella – Clustrix, GenieDB, Schooner, VoltDB, RethinkDB, ScaleDB, Akiban,CodeFutures, ScaleBase, Translattice, NimbusDB, etc. With the marketing noise of Big Data and Cloud, new companies are getting funded by the dozens. It is going to be a tough space to differentiate and become a winner.

A new category called NewSQL databases are aiming to provide the scale-out advantages of NoSQL databases, and often their commodity hardware friendliness as well. But NewSQL databases maintain the transactional data consistency guarantees of traditional relational databases, as well as their compatibility with SQL for queries and connectivity (using technologies like ODBC and JDBC).  One such product called NuoDB believes that transactional, analytical and “Web scale,” elastic workloads can be handled by the same database; it’s just a matter of making that the design goal. This is hard to believe until proven!

Another NewSQL product, VoltDB also claims to bring ACID-compliant transactions with analytics. VoltDB focuses on using in-memory technology to perform in situ analysis on financial, clickstream, gaming, and other high-velocity data as it streams in. In the company’s own words, VoltDB is meant to “narrow the ‘ingestion-to-decision’ gap.” There is growing need for instant analysis of transactional data (Real-time BI).

You squander the value of transactional data unless you analyze it as it is being recorded. SAP said much the same thing recently, as it announced the availability of its Business Suite on its HANA in-memory data platform, and fellow NewSQL player NuoDB uses in-memory and asynchronous technology to facilitate similar real-time analyses. Other NewSQL database products include ScaleDB and Clustrix, addressing the scalability needs of MySQL customers. Most of these products are also offering their services in the cloud.

It seems a grand unification process is on its way. Conventional relational databases and NoSQL databases seem to be at opposite ends of a spectrum. NewSQL databases acknowledge the merits in both models and seek to eliminate unreasonable compromise by marrying the approaches. NewSQL products may thus win out, but traditional relational database players may also incorporate NoSQL and NewSQL features to stay competitive. Perhaps that’s why Microsoft announced in November last year that the next major release of its SQL Server relational database will include an in-memory transactional database engine, codenamed “Hekaton.”

Since relational databases using SQl are inefficient in handling data from social chatters, smartphones, and clicks (because of volume and variety), new databases are popping up over last 3-4 years. In the past two years 119 database software companies have been funded by VC’s for $1.17B (according to Venture Source, a Dow Jones company). This is remarkable, as not too long ago, the space was declared taken by 3 incumbents – IBM, Oracle, and Microsoft. However, the scene has changed dramatically now.

Thanks must go to Google for pioneering the start of new innovations in Big Table, GFS (Googel File System), and Map-Reduce algorithms for massively parallel processing using commodity hardware clusters. These technologies became part of Apache open source foundation and the result is Hadoop, HDFS, and several associated tools for the new ecosystem. Amazon, Yahoo and Facebook have also contributed good work here.

The article mentions a client Autozone using one of the new DBMS’s called NuoDB for better managing store inventory according to local shoppers. NuoDb like many others offers a cloud service with an annual subscription, cutting Capex for customers.

Another client Trulia (online real estate) was using MySQL, but has added Cassandra to better manage the listing of home foreclosures and apartment listings of its 100 million homes in the US.

Shutterstcok, a photo agency, stores 24 million images with 10,000 added each day. It uses HDFS (Hadoop) to find out user behavior (how long they hover over an image before purchasing).

The article suggests that large financial clients will stick to existing vendors such as Oracle for various reasons, but the threat of these newcomers is there. This is much like the cloud software  is shaking up Microsoft’s desktop software model.

We are in the data-intensive computing era now and the race will be fierce for leadership and market share.

The second event is the stepping down of Andy Rubin, founder of Android at Google to a new role, as announced yesterday.

Android — which began life as an independent company Rubin co-founded in 2003 — is now a massive and growing force in mobile. Sure, some might grumble about the many forks and flavors, but the software powers more than 750 million devices from scores of different hardware makers. Android accounted for 70 percent of global smartphone shipments in the fourth quarter of 2012, according to IDC.

The overall criticism seems to be its multiple flavors and incompatibilities. Apple’s chief marketing guy said yesterday that Samsung’s Galaxy users will use an operating system that is at least one year old. Keeping various flavors of Android in sync has been an issue.

Google has been pushing two operating systems – Chrome OS and Android, the former for laptops and tablets and the later for smartphones. By consolidating the Android group under the Chrome division, headed by SVP Sundar Pichai, Google is attempting to bring the two together. This is what Apple has done well with its iOS – it feels the same whether you are using the iPhone or the iPad and even some aspects of Mac laptops.

Steve Jobs was quite upset with Android and made some disparaging remarks during 2011 before his death. However, Apple has a battle as newer devices (hardware seems very much alike – a big shiny glass screen on metal with touch UI) all preferring Android as their OS. It must continue to innovate, mostly in the software, to attract customers as Android becomes a real threat.

IBM expects to continue its big bets on technologies like Big Data and analytics. “Data will be the basis of competitive advantage for every company, for every industry in the coming decade.”

To that end, she said that IBM now expects revenue from business analytics to account for as much as $20 billion in annual revenue by fiscal 2015. The prior target was $16 billion. And if Big Blue hits that goal it would amount to a doubling of analytics revenue from 2010.

That is quite a commitment, the likes of which has not been seen from other key players such as Oracle, HP, SAP, or Microsoft. IBM has a full division on Big Data and their coverage on the subject is quite impressive.

From my 16 years at IBM during the development of DB2 family of products, I know firsthand the talent and experience IBM has in the data business. When they set their mind on an area, good things happen. Hence this commitment by the CEO is serious and competitors better take notice!

• Immediate (I must do something right away)
• Intimidating (what will happen if I don’t take advantage of Big Data)
• Ill-defined (the term is so broad that I’m not clear what it means).

In this blog post, I would like to pose five key questions that customers must find answers to with regards to Big Data. So here goes.

1. Do I understand my data and do I have a data strategy?

There are varieties of data – customer transaction data, operational data, documents/emails and other unstructured data, clickstream data, sensor data, audio streams, video streams, etc. Do I have a clear understanding the 3V’s of Big Data – Volume, Velocity, and Variety? What is data “in motion” vs. data “in rest”? Data in motion demands split-second decisions and do I have such tools? Every data source must be understood followed by their attributes and growth projections.

Customers must have an overall data strategy based on their business importance. For example, business critical data must be highly reliable, secure and of high performance. A data policy must be in place to take care of volume, growth, retention, security and compliance needs.

2. What are my reporting needs to transform my business and give me insights for growth?

Businesses are transforming to stay ahead of the competition. While we asked, “what happened” in the past, now it is “why did it happen and what is going to happen?”. From data collection, we have to move to data analysis. Instead of analyzing existing business, we must create new business. Therefore, the retail industry wants to give “today’s recommendation” on the fly to clients; internal IT needs operational intelligence to make it more efficient; customer service must provide customer insight; and fraud management must look at social profiles to reduce fraud. The list goes on…

Do you have a clear understanding of your reporting needs via data visualization on mobile devices like the iPad with touch interface? You will need a strategy of all the analytic tools for key employees/executives to make quick business-relevant decisions.

3. How do I drastically reduce my TCO of Data Warehousing and BI?

Many large enterprises are spending millions of dollars to move operational data to a data warehouse via ETL tools (Extraction, Transformation, Loading). This can be expensive and time consuming. Sears, for example, has a slogan “ETL must die”. By moving to Hadoop, they reduced the ETL time from 20 hours to 17 minutes. They claim serious cost reductions by moving from traditional ETL to direct loading of raw data to Hadoop servers. Today’s implementations must be studied for price-performance and newer technologies can bring down costs and improve processing time drastically. Would you like to develop reports in days rather than weeks?

4. How does Big Data co-exist with my current OLTP and DW data?

All enterprises have business-critical operational systems (OLTP). These are using traditional DBMS systems (such as Oracle, DB2, IMS, etc.). They also created separate Data Warehousing systems with BI tools for analysis. Now the new world of Internet data such as chatters from social networks and Web Log data (digital exhaust) are adding to the complexity. What is your approach to data integration of the legacy vs. new data?

5. What is the right technology for my needs?

I keep hearing so many new terms and vendor names – Hadoop, Cloudera, Hortonworks, Datameer, NoSQL, MongoDB, Map-reduce, Data Appliance, HBase, etc. It surely can be very confusing!

I need to know what is the right technology for my needs. If I have petabyte volumes data coming from various sources, what technology can I implement to efficiently handle that? Then, how do I get relevant information from that pile to help my business insights? I also need to know what skills I need to do that and the cost. I need an implementation roadmap for getting value from all the data that my business is coming up with.

There is such a herd mentality. When your quarterly revenue reaches $50B, growth at double digits become very hard. Look at Microsoft, Cisco, and IBM in the most recent fiscal year – 4%, 6%, and -2%. But Apple is supposed to defy the laws of gravity, according to our financial pundits! Mike says very colorfully that Apple grew by 45% for last 5 years and if that rate continues, it will hit $3 Trillion by 2020. At 5%, that number will be $231B, and at 10% it will be $334B in 2020. No company in history has seen such kind of numbers.

Besides the numbers Apple has unleashed an energy unseen anywhere. It’s superior design, great retail experience, and impeccable supply chain management are the envy of any company. We have not even talked about the hundreds of thousands of application builders on iOS. Just the ecosystem of Apple is unprecedented in its size, scope, and influence.

During the CES week, I got tired of so many booths selling Apple gear, mostly from China. iPad/iPhone covers and accessories took up entire floors at the Las vegas convention center. Paradoxically Apple never shows up at CES in terms of a booth or its executives in speaking slots.

I believe Apple stock is a strong buy at the current price.

I used to attend Comdex many years back, but that was replaced by CES and it’s focus has broadened to all kinds of consumer products – TV, Tablets, cameras, audio, smart phones, etc. As expected, we are living in a connected world and smart devices are all over. Whereas the past years centered around the PC and Microsoft, this year’s CES did not even have Microsoft as a participant (remember how Bill Gates used to make major announcements at Comdex?). This year’s keynotes were delivered by Qualcomm, Panasonic, Samsung, and Verizon. Mobile devices were the big thing, as Android phones were on display by all major vendors. There were too many iPhone/iPad accessories on display, so much so that you get tired of it.

Walking the halls of the convention center requires lots of physical strength, as one could walk 10-15 miles in a day. The LG booth was very impressive with 3D Television on a giant curved screen, where objects come to hit your face. The big news was the giant 80 to 100 inch TVs going for $20,000 – all 4k HD technology with unprecedented clarity of pictures. LG even showed 50 inch curved TV (called flexible display). The OLED TV is extremely thin. The real big news is connectivity. From the tablet to the TV to the laptop screen, events can move wirelessly. Apple clearly has changed the landscape with its touch interface in iPhone and iPad and everyone now offers similar tablets coming in all sorts of sizes. Interestingly, Apple does not have a booth nor its executives ever speak at this event.

Marc Benioff was there at a keynote session and he spoke about how the convergence of consumers and business is happening. Several panels talked about cloud contents and regulatory challenges. The FCC chairman Julius Genachowski was there also talking of broadband, spectrum reform, and competition policy matters. I listened to an interesting presentation of the future UI – called NUI (Natural User Interface) based on multi-touch, vocal, gesture and neural modes. It is good to see HTML5′s Touch event spec V2 is being readied for standardization.

For an enterprise software guy like me, this show may look less relevant, but enterprises are impacted in a big way by the consumer devices and technology. Hence we have the phrase “consumerization of IT” and BYOD. The Cloud is blurring the difference between personal software and work software as they have to intermingle and mesh.

It was a worthwhile experience to see all the advancements happening in such a fast pace. I particularly enjoyed the new stuff in robotics and programmable 3-D manufacturing.