Big Data – Status

According to a Wall Street Journal article today by Rachael King and Steven Rosenbush, the market for new databases serving Big Data reached $1.22B last year and is expected to more than double by 2014 (according to research firm Wikibon). That is quite impressive.

Since relational databases using SQl are inefficient in handling data from social chatters, smartphones, and clicks (because of volume and variety), new databases are popping up over last 3-4 years. In the past two years 119 database software companies have been funded by VC’s for $1.17B (according to Venture Source, a Dow Jones company). This is remarkable, as not too long ago, the space was declared taken by 3 incumbents – IBM, Oracle, and Microsoft. However, the scene has changed dramatically now.

Thanks must go to Google for pioneering the start of new innovations in Big Table, GFS (Googel File System), and Map-Reduce algorithms for massively parallel processing using commodity hardware clusters. These technologies became part of Apache open source foundation and the result is Hadoop, HDFS, and several associated tools for the new ecosystem. Amazon, Yahoo and Facebook have also contributed good work here.

The article mentions a client Autozone using one of the new DBMS’s called NuoDB for better managing store inventory according to local shoppers. NuoDb like many others offers a cloud service with an annual subscription, cutting Capex for customers.

Another client Trulia (online real estate) was using MySQL, but has added Cassandra to better manage the listing of home foreclosures and apartment listings of its 100 million homes in the US.

Shutterstcok, a photo agency, stores 24 million images with 10,000 added each day. It uses HDFS (Hadoop) to find out user behavior (how long they hover over an image before purchasing).

The article suggests that large financial clients will stick to existing vendors such as Oracle for various reasons, but the threat of these newcomers is there. This is much like the cloud software  is shaking up Microsoft’s desktop software model.

We are in the data-intensive computing era now and the race will be fierce for leadership and market share.

The Next Web Architecture

I was listening to Roger McNamee’s predictions of technology investment trends and came across two new terms – Hypernet and Hyperweb. The first term is where the Internet is overlaid with smartphones. He says that smartphones are now 50% of the web devices and growing. The second term refers to the software infrastructure for the Hypernet. The current infrastructure of mostly “index search” (read Google) is not going to work for the Hypernet. We need HTML5. Here are his ten hypotheses for tech investing.

1. Next Web Architecture = Hypernet + Hyperweb

2. The decline and fall of Windows unlocks revenue. Software development on Windows is declining. The new focus is the Web, Apple OS, and open source. In 2011, Windows devices will be less that 50% of the Internet, down from 95% four years ago.

3. Index search is peaking. Google search on mobile and tablets is much lower than on PC and Android does not fix that. New search is content-focused, such as Twitter (real time search), Wikipedia (facts), Linked-In (business people), or Facebook (social, taste,..).

4.  Apple’s model threatens the Web. Web is more flexible, but Roger calls it Digital Detroit and one gets mugged easily. Apple’s iOS App model simplifies access to information on the Internet. HTML5 can be a threat to Apple when every content-provider starts using it.

5. HTML5 is game changer for publishers. Developers can embed audio and video easily replacing Adobe’s Flash. It will be disruptive. Content producers will redesign their sites to reduce power of Google and ad networks.

6. Tablets are hugely disruptive. iPad has replaced DVD as the most actively adapted tech product ever. So far, Apple dominates this market. The number 2 (Kindle Fire?) is yet to emerge.

7. First wave of “Social Web” is over. Facebook has won the platform war as the new Windows. But going forward, social will be a feature of every product.

8. Smartphones in the US = Apple + 7 Dwarfs. Android has more units but Apple gets almost all profits. Also Android has some serious security issues.

9. Wireless Infrastructure is competitive threat to the US. US has the least capable wireless infrastructure in the developed world. This will impact productivity and competitiveness  in a big way.

10. Integration of TV and Internet could be disruptive. The convergence of Web and Television has the potential to disrupt cable and satellite, but it may not happen.

These are interesting and thoughtful hypotheses and like any predictions they are unknown in terms of velocity (when would it all happen), but certainly we see several of these at play even now.