The NoSQLNow conference in San Jose this week

I attended the NoSQLNow conference this week at the San Jose Convention Center. The organizers claimed there were 800 attendees, clearly much higher than last couple of years. Given the number of sessions, exhibits, speakers and attendees, the interest on newer data management products and solutions (aka Big Data) has been growing fast.

I spoke at a session titled, “Are NoSQL databases ready for the enterprise? Examples of MongoDB deployment” which was well attended. I also participated in a panel on “enterprise adoption of cloud”. My co-panelists were from Oracle and NeoDB. The conference opening session was given by one of the co-hosts, Dan McCreary and he spoke about the state of NoSQL. He mentioned that a total of $2.4B have been invested in NoSQL DB companies over last couple of years- MongoDB ($231M), CouchBase ($116M), Aerospike ($22M), Basho ($32.5M), Datastax ($83.7M), Clustrix ($59.3M), FoundationDB ($22.3M), etc. Even big player like Intel has invested in Cloudera. 

Here are some new trends in the NoSQL world:

  • Hadoop is starting to move from batch to real time and streaming
  • Real time systems are adding Hadoop integration points
  • Storm (twitter) and Spark are addressing data streaming
  • Spark/Scala is popular on multiple systems
  • MongoDB is the big leader in NoSQL operational systems based on document data model, followed by Datastax and CouchBase

The market pressures, according to Dan point to:

  • Big Data & Predictive analytics
  • Internet of Things (time series data and log files)
  • Security for highly regulated areas like finance/banking, healthcare, and the government
  • streaming data
  • keeping the operational cost low (bye bye to license fees)
  • High Availability (move away from master-slave to clusters of peer to peer networks)

There are other trends like old-school Map-Reduce programming is being taken over by Spark. JSON data formats are gaining in popularity for agile development, but there is no standardization of JSON query language. On the other hand, XQuery 3.1 is supporting both XML and JSON formats. There is new emphasis on agile transformation, as data storage is no longer the issue. The question is how non-programmers can transform data to various useful formats.  The acronym ETL will be replaced by ETTTTTTT… (extract, store in data lake, and transform in many ways).

Other keynotes included Oracle’s head of database development, Andy Mendelson, who showed Oracle’s 3 areas under “big data” – Oracle DBMS & Exadata, Oracle Hadoop, and Oracle NoSQL (formerly BerkeleyDB), all with one interface called Oracle Big Data SQL. SQL seems to make a comeback as an interface to several products such as Cloudera Impala.

Amazon presented their Dynamo DB, built for the cloud with fast and predictable performance. They claim seamless scalability and easy admin. Amazon’s motto has always been, “build services, not software”. Amazon.com uses DynamoDB to minimize opex.

I presented many examples of enterprises deploying MongoDB to build “systems of engagement” on top of “systems of record” ( a concept Geoff Moore of Crossing the Chasm fame has been talking lately). There is great momentum of MongoDB deployment at enterprises because of agile development (flexible data model and high coding velocity), fast scalability and high availability using shards and replicas, and the open source culture.

Fast Data vs. Big Data

Back when we were doing DB2 at IBM, there was an important older product called IMS which brought significant revenue. With another database product coming (based on relational technology), IBM did not want any cannibalization of the existing revenue stream. Hence we coined the phrase “dual database strategy” to justify the need for both DBMS products. In a similar vain, several vendors are concocting all kinds of terms and strategies to justify newer products under the banner of Big Data.

One such phrase is Fast Data. We all know the 3V’s associated with the term Big Data – volume, velocity and variety. It is the middle V (velocity) that says data is not static, but is changing fast, like stock market data, satellite feeds, even sensor data coming from smart meters or an aircraft engine. The question always has been how to deal with such type of changing data (as opposed to static data typical in most enterprise systems of record).

Recently I was listening to a talk by IBM and VoltDB where VoltDB tried to justify the world of “Fast Data” as co-existing with “Big Data” which is narrowed to static data warehouse or “data lake” as IBM calls it. Again, they have chosen to pigeonhole Big Data into the world of HDFS, Netezza, Impala, and batch Map-Reduce. This way, they justify the phrase Fast Data as representing operational data that is changing fast. They call VoltDB as  “the fast, operational database” implying every other database solution as slow. Incumbents like IBM, Oracle, and SAP have introduced in-memory options for speed and even NoSQL databases can process very fast reads on distributed clusters.

VoltDB folks also tried to show how the two worlds (Fast Data and their version of Big Data) will coexist. The Fast Data side will ingest and interact on streams of inbound data, do real time data analysis and export to the data warehouse. They bragged about the performance benchmark of 1m tps on a 3-node cluster scaling to 2.4m on a 12-node system running in the SoftLayer cloud (owned by IBM). They also said that this solution is much faster than Amazon’s AWS cloud. The comparison is not apple-to-apple as the SoftLayer deployment is on bare metal compared to the AWS stack of software.

I wish they call this simply – real-time data analytics, as it is mostly read type transactions and not confuse with update-heavy workloads. We will wait and see how enterprises adopt this VoltDB-SoftLayer solution in addition to their existing OLTP solutions.

Online Marketplace and Flipkart

I listened to Professor Ramesh Johari of Stanford University last night talking about the dynamics of online markets. He spent 18 months at a company called oDesk (now part of eLance, online marketplace for freelance workers) and tried several models to understand what creates successful online markets. Besides complex matching algorithms and collaborative filtering, there are other interesting factors such as human behavior that can affect such markets. Let us look at what is happening in emerging markets like India.

Yesterday, Flipkart, an Indian e-commerce site for commercial goods got an infusion of $1B from investors from Russia, South Africa, USA, Singapore, and India. The two founders came from Amazon and no wonder they are following the same model- start with books, then open your fulfillment engine to other merchants and charge a fee. Flipkart is the fast rising e-commerce site in India and its founders just joined the billion dollar club on paper based on the latest valuation at $7B. India’s e-commerce market is currently at $13B, poised to grow to $76B by 2021, according to Technopak. Out of this total market, travel sector is 70% and type of goods sold through the likes of Flipkart and Amazon yields $1.6B. By comparison, e-commerce sales in China will likely surpass $180B this year!

Speaking about Amazon, Jeff Bezos just announced that he will invest $2B  in India, via five more warehouses (doubling to half a million square feet), same day delivery, and aggressive marketing. In a statement this morning he said, “With this additional investment of $2 billion, our team can continue to think big, innovate, and raise the bar for customers in India. At current scale and growth rates, India is on track to be our fastest country ever to a billion dollars in gross sales.” Amazon makes its money in India by charging third-party suppliers to use its website to sell 17 million different products including books, electronics and clothing.

The government is considering allowing foreign retailers to sell directly to customers. It recently took a step in that direction by allowing retailers to sell online products manufactured in India. Amazon and Flipkart are also joined by Snapdeal, another e-commerce site backed by eBay and Jabong (fashion designer). The new prime minister Mr. Modi wants to open up rural markets to such brand names for easy access without building physical stores. It will also enable growth for small town stores as front end to such goods.

Alibaba’s success in China is also an example for other such emerging markets. The founders of Flipkart dream of a $100B valuation for their company some day. But as Ramesh Johari pointed out yesterday, what is illusive is the secret formula for success and many unknown factors. Earlier this week, Lyft (online taxi service like Uber) users got highly frustrated when they could not get cars fast enough and trashed the company in social media.

Online markets are hot for sure and Flipkart will see some real competition from Amazon and eBay.

Facebook valued higher than IBM?

Yesterday Facebook with its stock soaring to new heights, exceeded IBM in terms of market value. This morning, it’s slightly less (IBM at $194B vs Facebook at $193B). However Facebook is higher than Oracle’s value and is the third largest company in the valley after Apple and Google. For enterprise software folks like me, this is very hard to explain. Value of what? – the photographs, likes and connected friends?

Market value is the total number stocks multiplied by the stock price. Sometimes, it can be a huge multiple of the current revenue – as is the case for Facebook. It’s projected annual revenue for this year at $12B is minuscule compared to IBM’s $100B in 2013. Facebook is 10 years old with 7000 employees compared to IBM’s age at 103 with more than 400,000 employees. The market value indicates the “projected” growth of a company and a higher demand of its stock increases the value. Sustainability is another issue.

Facebook reported stellar performance earlier this week, exceeding the analyst’s projection. It’s mobile advertising revenue was impressive, compared to zero two years ago. This pushed the stock to almost $76, twice the price of its IPO at $38 and four times the low of $18 at one time. Mark Zuckerberg is worth almost $33B at the ripe age of 31, adding a whopping $1.6B yesterday to his net worth.

I guess social networking is a huge force. With 1.2 billion users, Facebook owns a great treasure called the “social graph” of a huge population. Advertisers would bet all their money to reach such an audience. This year, the net spend on mobile advertising would exceed that of TV and print media spending. Every time, you use Facebook, it keeps track of you and your connections, plus photographs, links, timeline, etc. There are privacy issues being debated.

Facebook is growing very fast in countries like India and Brazil. No wonder Sheryl Sandberg visited India last month and met the new prime minister Mr. Modi to discuss how Facebook can be used for better governance in the largest democracy of the world. Incidentally, IBM’s largest number of employees now are in India, bigger than the US. 

It is still hard to digest the fact that Facebook is almost equal to IBM’s value!

The new IBM-Apple partnership

We live in interesting times for sure. Who would have thought ten years ago that IBM and Apple would become partners? Here is the announcement as reported in eWeek

Apple and IBM surprised many on July 15, announcing a global partnership that will see the companies attempt to “transform enterprise mobility.” The announcement, punctuated with comments from Apple CEO Tim Cook and IBM CEO Ginni Rometty, served notice of the companies’ intentions to make Apple smartphones, tablets and mobile services pre-eminent in the enterprise, replacing the BlackBerry devices and security services that long held the dominant position in that market. The partnership only confirms what Apple has been doing for years with its smartphones, tablets and even its notebook and desktop PCs, moving deeper into enterprises. The company has been touting its success in the corporate world, saying that many of the top companies in the world are now using its iPhones and iPads. In the IBM announcement, Apple’s Cook said that the company’s iPhones and iPads are now running in more than 98 percent of the Fortune 500 and 92 percent of the Global 500. Both Apple and IBM expect those figures to grow with this new partnership.

What is in it for Apple? Apple has been trying for years to shed the idea that its products only appeal to consumers. The company has brought proprietary app development and remote device management to iOS devices and continues to offer stats on enterprise integration. With the IBM deal, however, Apple is making something abundantly clear: It cares deeply about the corporate world and it wants to do all it can to be a success in that market. 

What is in it for IBM? For IBM, the Apple partnership is a software play. IBM said in its own press release that it plans to make applications that are designed for enterprise customers, but come with a wide array of features that simply aren’t available right now in the mobile market. That could provide IBM (and Apple) with a leg up in the enterprise. While IBM could have extended the partnership to OS X and perhaps made Apple a bit happier as it tries getting that platform into the enterprise, Big Blue said in its statement that the agreement applies only to iOS. That means that iPhones and iPads will play a central role in this agreement, and Macs and iPods running iOS will be left out. 

Developing mobile Apps together? Although IBM plans to chart the course for software in this partnership, the companies are actually going to come together on development. IBM said that both Apple and its own developers will collaborate to leverage iOS as much as possible and get every last bit out of the technology built into iPhones and iPads. That’s a core component in this: IBM has lofty goals, and it can only be successful if it can fully harness the power of iOS and Apple hardware. 

Leveraging Big Data: Big data and analytics will be a core focus of this effort, IBM said. The apps—over 100, in total—will look at how to get companies to insert their massive amounts of data into iOS devices, analyze it and make actionable decisions. Mobile devices have traditionally been viewed as too underpowered to handle big data. IBM sees it differently.

Playing in the IBM Cloud: Although IBM is building apps for on-premise use, the company said that it will also offer all of the solutions it’s developing on its Bluemix development platform. Customers will need to head over to the IBM Cloud Marketplace and then download the apps they desire. IBM said it will also try to find other ways to leverage the cloud. 

IBM, a reseller of iPhone & iPad? IBM is essentially becoming another outlet for Apple to sell its iPhones and iPads. The companies have agreed to have IBM sell iPhones and iPads to enterprise customers that come with desired applications already on them. That should make it easier for corporate customers to join the mobile fray and for Apple to improve its already-strong enterprise integration. 

There you go folks! This new partnership appears like a big winner for Apple as it expands its already-dominant market for iPhone/iPad into the enterprise. IBM is betting on the mobile apps on iOS to make its cloud and big data initiatives more appealing.

Gone are the days in 1984 when Apple ran that advertisement during the introduction of the Mac, portraying IBM as the big bad evil force. With IBM’s PC business sold to Lenovo few years back, IBM is into software and services business (includes cloud infrastructure service).

Yes, these are interesting times!

Re-Imagination – new disruptive technologies

I like this word re-imagination from Mary Meeker’s Internet Trends presentation. We are seeing so many aspects of our life being transformed by the internet.

Take for example, ordering a cab to go somewhere. Either we phone for a yellow cab here in California or if you are in New York city, then you stand and wave for an incoming yellow cab to stop. The new game-changer is Uber. All you do is touch your smartphone screen for UberX or Black car and you get an instant message about the car coming in less than 5 minutes time with the driver and car info. It is cheaper and you pay by card (pre-registered in your Uber account). This is re-imagining the transport sector. Uber, a San Francisco company is worth about $17B and is operating in 70 cities around the world. Quite a disruptive force!

The following list shows several new alternatives using the internet technology that impacts varieties of activities:
– Grocery shopping – Amazon Fresh or Instacart
– Yellow Pages – Yelp
– Booking hotel – Airbnb
– Driving in traffic – Waze
– Satellite radio – Spotify
– TV Remote control – Amazon Fire TV
– Meeting new people – Tinder
– New money – Bitcoin
– Smart TV adopters – Apple TV, Google Chromecast, Amazon Fire TV, Roku
– Mojo Update (uploadable, sharable, findable) – Pinterest for common interests, Eventbrite for tickets, Fitbit/MyFitness Pal – measuring steps/distance/calories,
WhatsApp (instant messages, photos, videos),….

This list goes on and we find a new single-purpose application appearing almost every day. Gone are the days of multi-purpose web applications! In the area of data mining and analytics, we find new companies like Dropcam for home monitoring (just acquired by the Nest division of Google), Jawbone health wearable device, Netflix for media personalization and discovery, AppDynamics for application performance monitoring, SnapLogic for cloud integration, and Ayasdi for automated insight discovery.

No wonder that our kids are growing up with such different habits than the old times – no physical newspaper reading, no TV watching at specific show times (Youtube is good enough), or no sending emails in the old fashioned way (all short messages and Twitter and Facebook for broadcast). Even physical book reading is declining, with eBook readers all around.

Last month my two sons gave me two gifts on Fathers day – Apple TV and Google Chromecast. Now I see the benefit of not using cable. I also love using Uber during my trips. My Fitbit keeps me alert on meeting my daily walking goals.

This follows the dictum – Old order changeth yielding place to new.

MongoDB World in NY City this week

I attended the MongoDB World in NY city earlier this week on June 24th. and 25th. There were 2000 attendees for this first-time event. I met many people who flew in from all over the world. It was quite a phenomenon for a six-year old company. The momentum of MongoDB  is truly amazing.

For those who do not know MongoDB, it’s a new generation open source NoSQL database for building highly interactive modern applications. Enterprises have built “systems of record” over last 3 decades using traditional RDBMS such as DB2, Oracle, MySQL and SQL Server. These systems are like the interstate highways built during the 1930s across the USA. They serve the business in handling basic functions, but are inadequate in meeting several new needs. Hence the need for building “systems of engagement” has arrived. These systems connect customers, employees, and partners into the business using mobile devices and providing visibility of critical business information. MongoDB provides two crucial functions – an easy to use development platform with very high coding velocity; and a horizontally scalable operations spanning 100s and 1000s of commodity servers at much lower cost compared to the traditional systems. Many new features for easy management of such distributed systems were announced.

While Hadoop brings a highly economic distributed  platform to handle batch-oriented offline analytics, MongoDB addresses the online transactional workloads. They compliment each other, much like today’s operational systems do with the data warehousing solutions. Hadoop is a step up in the EDW and analytics world, while MongoDB is a step up in the mission critical business transaction world. At the conference, many customers across all verticals presented success stories on how they have used MongoDB to address the data variety problem and built systems at record speed. It only takes weeks to a couple of months from inception to finish, not possible using the standard RDBMS-based technologies. Special features like handling spatial data with indexing and text search, were highlighted by some customers.

Cloudera founder and CSO Mike Olson gave a keynote on the co-existence of Hadoop and MongoDB, so did Amazon’s CTO Werner Vogels on using MongoDB via AWS. Other keynotes were given by Max Schireson, CEO of MongoDB and Eliot Horowitz, CTO of MongoDB. A number of CIOs and CTOs of large enterprises were present also. The excitement and developer endorsement was visible all around. It clearly showed that amongst the new database solutions, MongoDB is the unquestioned leader. Beside Amazon AWS, MongoDB is also available in other cloud solutions such as Google Cloud Platform and Microsoft Azure.There were many partners showing their support of MongoDB at the exhibition booths – Teradata, Cloudera, AppDynamics, Pure Storage, SAP, etc.

For a first-time event, MongoDB World was quite remarkable!