Fast Data vs. Big Data

Back when we were doing DB2 at IBM, there was an important older product called IMS which brought significant revenue. With another database product coming (based on relational technology), IBM did not want any cannibalization of the existing revenue stream. Hence we coined the phrase “dual database strategy” to justify the need for both DBMS products. In a similar vain, several vendors are concocting all kinds of terms and strategies to justify newer products under the banner of Big Data.

One such phrase is Fast Data. We all know the 3V’s associated with the term Big Data – volume, velocity and variety. It is the middle V (velocity) that says data is not static, but is changing fast, like stock market data, satellite feeds, even sensor data coming from smart meters or an aircraft engine. The question always has been how to deal with such type of changing data (as opposed to static data typical in most enterprise systems of record).

Recently I was listening to a talk by IBM and VoltDB where VoltDB tried to justify the world of “Fast Data” as co-existing with “Big Data” which is narrowed to static data warehouse or “data lake” as IBM calls it. Again, they have chosen to pigeonhole Big Data into the world of HDFS, Netezza, Impala, and batch Map-Reduce. This way, they justify the phrase Fast Data as representing operational data that is changing fast. They call VoltDB as  “the fast, operational database” implying every other database solution as slow. Incumbents like IBM, Oracle, and SAP have introduced in-memory options for speed and even NoSQL databases can process very fast reads on distributed clusters.

VoltDB folks also tried to show how the two worlds (Fast Data and their version of Big Data) will coexist. The Fast Data side will ingest and interact on streams of inbound data, do real time data analysis and export to the data warehouse. They bragged about the performance benchmark of 1m tps on a 3-node cluster scaling to 2.4m on a 12-node system running in the SoftLayer cloud (owned by IBM). They also said that this solution is much faster than Amazon’s AWS cloud. The comparison is not apple-to-apple as the SoftLayer deployment is on bare metal compared to the AWS stack of software.

I wish they call this simply – real-time data analytics, as it is mostly read type transactions and not confuse with update-heavy workloads. We will wait and see how enterprises adopt this VoltDB-SoftLayer solution in addition to their existing OLTP solutions.

Online Marketplace and Flipkart

I listened to Professor Ramesh Johari of Stanford University last night talking about the dynamics of online markets. He spent 18 months at a company called oDesk (now part of eLance, online marketplace for freelance workers) and tried several models to understand what creates successful online markets. Besides complex matching algorithms and collaborative filtering, there are other interesting factors such as human behavior that can affect such markets. Let us look at what is happening in emerging markets like India.

Yesterday, Flipkart, an Indian e-commerce site for commercial goods got an infusion of $1B from investors from Russia, South Africa, USA, Singapore, and India. The two founders came from Amazon and no wonder they are following the same model- start with books, then open your fulfillment engine to other merchants and charge a fee. Flipkart is the fast rising e-commerce site in India and its founders just joined the billion dollar club on paper based on the latest valuation at $7B. India’s e-commerce market is currently at $13B, poised to grow to $76B by 2021, according to Technopak. Out of this total market, travel sector is 70% and type of goods sold through the likes of Flipkart and Amazon yields $1.6B. By comparison, e-commerce sales in China will likely surpass $180B this year!

Speaking about Amazon, Jeff Bezos just announced that he will invest $2B  in India, via five more warehouses (doubling to half a million square feet), same day delivery, and aggressive marketing. In a statement this morning he said, “With this additional investment of $2 billion, our team can continue to think big, innovate, and raise the bar for customers in India. At current scale and growth rates, India is on track to be our fastest country ever to a billion dollars in gross sales.” Amazon makes its money in India by charging third-party suppliers to use its website to sell 17 million different products including books, electronics and clothing.

The government is considering allowing foreign retailers to sell directly to customers. It recently took a step in that direction by allowing retailers to sell online products manufactured in India. Amazon and Flipkart are also joined by Snapdeal, another e-commerce site backed by eBay and Jabong (fashion designer). The new prime minister Mr. Modi wants to open up rural markets to such brand names for easy access without building physical stores. It will also enable growth for small town stores as front end to such goods.

Alibaba’s success in China is also an example for other such emerging markets. The founders of Flipkart dream of a $100B valuation for their company some day. But as Ramesh Johari pointed out yesterday, what is illusive is the secret formula for success and many unknown factors. Earlier this week, Lyft (online taxi service like Uber) users got highly frustrated when they could not get cars fast enough and trashed the company in social media.

Online markets are hot for sure and Flipkart will see some real competition from Amazon and eBay.

Facebook valued higher than IBM?

Yesterday Facebook with its stock soaring to new heights, exceeded IBM in terms of market value. This morning, it’s slightly less (IBM at $194B vs Facebook at $193B). However Facebook is higher than Oracle’s value and is the third largest company in the valley after Apple and Google. For enterprise software folks like me, this is very hard to explain. Value of what? – the photographs, likes and connected friends?

Market value is the total number stocks multiplied by the stock price. Sometimes, it can be a huge multiple of the current revenue – as is the case for Facebook. It’s projected annual revenue for this year at $12B is minuscule compared to IBM’s $100B in 2013. Facebook is 10 years old with 7000 employees compared to IBM’s age at 103 with more than 400,000 employees. The market value indicates the “projected” growth of a company and a higher demand of its stock increases the value. Sustainability is another issue.

Facebook reported stellar performance earlier this week, exceeding the analyst’s projection. It’s mobile advertising revenue was impressive, compared to zero two years ago. This pushed the stock to almost $76, twice the price of its IPO at $38 and four times the low of $18 at one time. Mark Zuckerberg is worth almost $33B at the ripe age of 31, adding a whopping $1.6B yesterday to his net worth.

I guess social networking is a huge force. With 1.2 billion users, Facebook owns a great treasure called the “social graph” of a huge population. Advertisers would bet all their money to reach such an audience. This year, the net spend on mobile advertising would exceed that of TV and print media spending. Every time, you use Facebook, it keeps track of you and your connections, plus photographs, links, timeline, etc. There are privacy issues being debated.

Facebook is growing very fast in countries like India and Brazil. No wonder Sheryl Sandberg visited India last month and met the new prime minister Mr. Modi to discuss how Facebook can be used for better governance in the largest democracy of the world. Incidentally, IBM’s largest number of employees now are in India, bigger than the US. 

It is still hard to digest the fact that Facebook is almost equal to IBM’s value!

The new IBM-Apple partnership

We live in interesting times for sure. Who would have thought ten years ago that IBM and Apple would become partners? Here is the announcement as reported in eWeek

Apple and IBM surprised many on July 15, announcing a global partnership that will see the companies attempt to “transform enterprise mobility.” The announcement, punctuated with comments from Apple CEO Tim Cook and IBM CEO Ginni Rometty, served notice of the companies’ intentions to make Apple smartphones, tablets and mobile services pre-eminent in the enterprise, replacing the BlackBerry devices and security services that long held the dominant position in that market. The partnership only confirms what Apple has been doing for years with its smartphones, tablets and even its notebook and desktop PCs, moving deeper into enterprises. The company has been touting its success in the corporate world, saying that many of the top companies in the world are now using its iPhones and iPads. In the IBM announcement, Apple’s Cook said that the company’s iPhones and iPads are now running in more than 98 percent of the Fortune 500 and 92 percent of the Global 500. Both Apple and IBM expect those figures to grow with this new partnership.

What is in it for Apple? Apple has been trying for years to shed the idea that its products only appeal to consumers. The company has brought proprietary app development and remote device management to iOS devices and continues to offer stats on enterprise integration. With the IBM deal, however, Apple is making something abundantly clear: It cares deeply about the corporate world and it wants to do all it can to be a success in that market. 

What is in it for IBM? For IBM, the Apple partnership is a software play. IBM said in its own press release that it plans to make applications that are designed for enterprise customers, but come with a wide array of features that simply aren’t available right now in the mobile market. That could provide IBM (and Apple) with a leg up in the enterprise. While IBM could have extended the partnership to OS X and perhaps made Apple a bit happier as it tries getting that platform into the enterprise, Big Blue said in its statement that the agreement applies only to iOS. That means that iPhones and iPads will play a central role in this agreement, and Macs and iPods running iOS will be left out. 

Developing mobile Apps together? Although IBM plans to chart the course for software in this partnership, the companies are actually going to come together on development. IBM said that both Apple and its own developers will collaborate to leverage iOS as much as possible and get every last bit out of the technology built into iPhones and iPads. That’s a core component in this: IBM has lofty goals, and it can only be successful if it can fully harness the power of iOS and Apple hardware. 

Leveraging Big Data: Big data and analytics will be a core focus of this effort, IBM said. The apps—over 100, in total—will look at how to get companies to insert their massive amounts of data into iOS devices, analyze it and make actionable decisions. Mobile devices have traditionally been viewed as too underpowered to handle big data. IBM sees it differently.

Playing in the IBM Cloud: Although IBM is building apps for on-premise use, the company said that it will also offer all of the solutions it’s developing on its Bluemix development platform. Customers will need to head over to the IBM Cloud Marketplace and then download the apps they desire. IBM said it will also try to find other ways to leverage the cloud. 

IBM, a reseller of iPhone & iPad? IBM is essentially becoming another outlet for Apple to sell its iPhones and iPads. The companies have agreed to have IBM sell iPhones and iPads to enterprise customers that come with desired applications already on them. That should make it easier for corporate customers to join the mobile fray and for Apple to improve its already-strong enterprise integration. 

There you go folks! This new partnership appears like a big winner for Apple as it expands its already-dominant market for iPhone/iPad into the enterprise. IBM is betting on the mobile apps on iOS to make its cloud and big data initiatives more appealing.

Gone are the days in 1984 when Apple ran that advertisement during the introduction of the Mac, portraying IBM as the big bad evil force. With IBM’s PC business sold to Lenovo few years back, IBM is into software and services business (includes cloud infrastructure service).

Yes, these are interesting times!

Re-Imagination – new disruptive technologies

I like this word re-imagination from Mary Meeker’s Internet Trends presentation. We are seeing so many aspects of our life being transformed by the internet.

Take for example, ordering a cab to go somewhere. Either we phone for a yellow cab here in California or if you are in New York city, then you stand and wave for an incoming yellow cab to stop. The new game-changer is Uber. All you do is touch your smartphone screen for UberX or Black car and you get an instant message about the car coming in less than 5 minutes time with the driver and car info. It is cheaper and you pay by card (pre-registered in your Uber account). This is re-imagining the transport sector. Uber, a San Francisco company is worth about $17B and is operating in 70 cities around the world. Quite a disruptive force!

The following list shows several new alternatives using the internet technology that impacts varieties of activities:
– Grocery shopping – Amazon Fresh or Instacart
– Yellow Pages – Yelp
– Booking hotel – Airbnb
– Driving in traffic – Waze
– Satellite radio – Spotify
– TV Remote control – Amazon Fire TV
– Meeting new people – Tinder
– New money – Bitcoin
– Smart TV adopters – Apple TV, Google Chromecast, Amazon Fire TV, Roku
– Mojo Update (uploadable, sharable, findable) – Pinterest for common interests, Eventbrite for tickets, Fitbit/MyFitness Pal – measuring steps/distance/calories,
WhatsApp (instant messages, photos, videos),….

This list goes on and we find a new single-purpose application appearing almost every day. Gone are the days of multi-purpose web applications! In the area of data mining and analytics, we find new companies like Dropcam for home monitoring (just acquired by the Nest division of Google), Jawbone health wearable device, Netflix for media personalization and discovery, AppDynamics for application performance monitoring, SnapLogic for cloud integration, and Ayasdi for automated insight discovery.

No wonder that our kids are growing up with such different habits than the old times – no physical newspaper reading, no TV watching at specific show times (Youtube is good enough), or no sending emails in the old fashioned way (all short messages and Twitter and Facebook for broadcast). Even physical book reading is declining, with eBook readers all around.

Last month my two sons gave me two gifts on Fathers day – Apple TV and Google Chromecast. Now I see the benefit of not using cable. I also love using Uber during my trips. My Fitbit keeps me alert on meeting my daily walking goals.

This follows the dictum – Old order changeth yielding place to new.

MongoDB World in NY City this week

I attended the MongoDB World in NY city earlier this week on June 24th. and 25th. There were 2000 attendees for this first-time event. I met many people who flew in from all over the world. It was quite a phenomenon for a six-year old company. The momentum of MongoDB  is truly amazing.

For those who do not know MongoDB, it’s a new generation open source NoSQL database for building highly interactive modern applications. Enterprises have built “systems of record” over last 3 decades using traditional RDBMS such as DB2, Oracle, MySQL and SQL Server. These systems are like the interstate highways built during the 1930s across the USA. They serve the business in handling basic functions, but are inadequate in meeting several new needs. Hence the need for building “systems of engagement” has arrived. These systems connect customers, employees, and partners into the business using mobile devices and providing visibility of critical business information. MongoDB provides two crucial functions – an easy to use development platform with very high coding velocity; and a horizontally scalable operations spanning 100s and 1000s of commodity servers at much lower cost compared to the traditional systems. Many new features for easy management of such distributed systems were announced.

While Hadoop brings a highly economic distributed  platform to handle batch-oriented offline analytics, MongoDB addresses the online transactional workloads. They compliment each other, much like today’s operational systems do with the data warehousing solutions. Hadoop is a step up in the EDW and analytics world, while MongoDB is a step up in the mission critical business transaction world. At the conference, many customers across all verticals presented success stories on how they have used MongoDB to address the data variety problem and built systems at record speed. It only takes weeks to a couple of months from inception to finish, not possible using the standard RDBMS-based technologies. Special features like handling spatial data with indexing and text search, were highlighted by some customers.

Cloudera founder and CSO Mike Olson gave a keynote on the co-existence of Hadoop and MongoDB, so did Amazon’s CTO Werner Vogels on using MongoDB via AWS. Other keynotes were given by Max Schireson, CEO of MongoDB and Eliot Horowitz, CTO of MongoDB. A number of CIOs and CTOs of large enterprises were present also. The excitement and developer endorsement was visible all around. It clearly showed that amongst the new database solutions, MongoDB is the unquestioned leader. Beside Amazon AWS, MongoDB is also available in other cloud solutions such as Google Cloud Platform and Microsoft Azure.There were many partners showing their support of MongoDB at the exhibition booths – Teradata, Cloudera, AppDynamics, Pure Storage, SAP, etc.

For a first-time event, MongoDB World was quite remarkable!

The High Tech Indian Election – lessons on Big Data, Social Media and 3D Holography

The recent national election in India that spanned over 5 weeks and concluded on May 12th. was unique in terms of  sheer numbers. The total size of the electorate was 815m (more than the population of USA and European Union combined), of which 550m actually voted. Half the electorate was below the age of 25 (voting age is 18). The sheer size and complexity was mind-boggling – it was the greatest democratic process on display!

The final results came out on May 16th., where the former opposition party BJP (Bharatiya Janata Party, or the Indian People’s party) had a land-slide victory, not seen in last 30 years. The leader of BJP is Mr. Narendra Modi, the chief minister (like a state Governor) of the state of Gujarat for last 13 years. He comes from a poor family and climbed up the ranks by sheer hard work and total focus on high growth development of his state. India puts a lot of hope on Mr. Modi to bring speedy growth back to the economy.

Here I will point out how he used technology during the election.

Mr. Modi found a smart technologist from London specializing in 3D Holography. He experimented with that during the state elections in 2012 where his 3D holographic image was beamed to many locations simultaneously. The system was debugged and was ready for effective use this time – broadcasting his image in full 3D to 100s of locations thus reaching millions of people and conveying his message very effectively. Physically he spoke at over 400 rallies, and each one was attended by at least one million people or more. But combined with the virtual presence via holography, his outreach was the maximum compared to any other candidate. Such an experiment of using 3D Holography has never been used before in any election around the world.

The second approach was putting together a crack team of smart Data Scientists  who took a page from president Barack Obama’s election process. Data about the electorate was gathered and dissected like never before. Based on behavior and past preferences, they were segmented and targeted very carefully. This was a bottoms-up approach. Thousands of volunteers followed up the Big data analysis and reached out to the voters in a door-to-door fashion, thus influencing their choices. Without such analysis and pin-point targeting, this would have been impossible within such a short time-frame of 4 months. All this effort was focussed primarily on 2 major states (UP and Bihar) where BJP’s electoral victory was going to be the game-changer (as they had poor record in the past). The results came out as proof of the effectiveness of this approach – BJP got 71 out of 80 in UP, and 31 out of 40 in Bihar (totally beyond any projections and expectations).

The third focus was on fully exploiting the social media such as Twitter, Facebook, and Blogs. The urban population with access to the Internet were constantly reached out via these new communication tools. Mr. Modi has one of the largest followings in Twitter.

It was the clever use of technology by Mr. Modi’s team that clinched their unprecedented victory. The other contesting parties including the current ruling Congres party for last 10 years were nowhere in using technology and hence they lost badly.

Some lesson for all future elections!