Five Questions around Big Data

Data is the new currency of business and we are in the era of data-intensive computing. Much has been written on Big Data throughout 2012 and customers around the world are struggling to figure out its significance to their businesses. Someone said there are 3 I’s to Big Data

• Immediate (I must do something right away)
• Intimidating (what will happen if I don’t take advantage of Big Data)
• Ill-defined (the term is so broad that I’m not clear what it means).

In this blog post, I would like to pose five key questions that customers must find answers to with regards to Big Data. So here goes.

1. Do I understand my data and do I have a data strategy?

There are varieties of data – customer transaction data, operational data, documents/emails and other unstructured data, clickstream data, sensor data, audio streams, video streams, etc. Do I have a clear understanding the 3V’s of Big Data – Volume, Velocity, and Variety? What is data “in motion” vs. data “in rest”? Data in motion demands split-second decisions and do I have such tools? Every data source must be understood followed by their attributes and growth projections.

Customers must have an overall data strategy based on their business importance. For example, business critical data must be highly reliable, secure and of high performance. A data policy must be in place to take care of volume, growth, retention, security and compliance needs.

2. What are my reporting needs to transform my business and give me insights for growth?

Businesses are transforming to stay ahead of the competition. While we asked, “what happened” in the past, now it is “why did it happen and what is going to happen?”. From data collection, we have to move to data analysis. Instead of analyzing existing business, we must create new business. Therefore, the retail industry wants to give “today’s recommendation” on the fly to clients; internal IT needs operational intelligence to make it more efficient; customer service must provide customer insight; and fraud management must look at social profiles to reduce fraud. The list goes on…

Do you have a clear understanding of your reporting needs via data visualization on mobile devices like the iPad with touch interface? You will need a strategy of all the analytic tools for key employees/executives to make quick business-relevant decisions.

3. How do I drastically reduce my TCO of Data Warehousing and BI?

Many large enterprises are spending millions of dollars to move operational data to a data warehouse via ETL tools (Extraction, Transformation, Loading). This can be expensive and time consuming. Sears, for example, has a slogan “ETL must die”. By moving to Hadoop, they reduced the ETL time from 20 hours to 17 minutes. They claim serious cost reductions by moving from traditional ETL to direct loading of raw data to Hadoop servers. Today’s implementations must be studied for price-performance and newer technologies can bring down costs and improve processing time drastically. Would you like to develop reports in days rather than weeks?

4. How does Big Data co-exist with my current OLTP and DW data?

All enterprises have business-critical operational systems (OLTP). These are using traditional DBMS systems (such as Oracle, DB2, IMS, etc.). They also created separate Data Warehousing systems with BI tools for analysis. Now the new world of Internet data such as chatters from social networks and Web Log data (digital exhaust) are adding to the complexity. What is your approach to data integration of the legacy vs. new data?

5. What is the right technology for my needs?

I keep hearing so many new terms and vendor names – Hadoop, Cloudera, Hortonworks, Datameer, NoSQL, MongoDB, Map-reduce, Data Appliance, HBase, etc. It surely can be very confusing!

I need to know what is the right technology for my needs. If I have petabyte volumes data coming from various sources, what technology can I implement to efficiently handle that? Then, how do I get relevant information from that pile to help my business insights? I also need to know what skills I need to do that and the cost. I need an implementation roadmap for getting value from all the data that my business is coming up with.

Apple – why the sudden antipathy?

In our industry the biggest game is “expectation management”. Apple was in big news last week after its earning report. In the latest quarter, Apple topped $50B revenue, an increase of $8.2B from the same period last year. This should be the envy of any company. It sold 48m iPhones, more than any competitor can dream of. As Mike Moritz wrote in the Financial Times yesterday, if Apple were a nation, it will rank 45th. in the world, ahead of Pakistan and New Zealand. Suddenly the sound-bytes by the analysts showed disappointment that Apple only grew “18 percent” and management was forecasting slower growth. The stock was pounded and went down to sub-$450 range.

There is such a herd mentality. When your quarterly revenue reaches $50B, growth at double digits become very hard. Look at Microsoft, Cisco, and IBM in the most recent fiscal year – 4%, 6%, and -2%. But Apple is supposed to defy the laws of gravity, according to our financial pundits! Mike says very colorfully that Apple grew by 45% for last 5 years and if that rate continues, it will hit $3 Trillion by 2020. At 5%, that number will be $231B, and at 10% it will be $334B in 2020. No company in history has seen such kind of numbers.

Besides the numbers Apple has unleashed an energy unseen anywhere. It’s superior design, great retail experience, and impeccable supply chain management are the envy of any company. We have not even talked about the hundreds of thousands of application builders on iOS. Just the ecosystem of Apple is unprecedented in its size, scope, and influence.

During the CES week, I got tired of so many booths selling Apple gear, mostly from China. iPad/iPhone covers and accessories took up entire floors at the Las vegas convention center. Paradoxically Apple never shows up at CES in terms of a booth or its executives in speaking slots.

I believe Apple stock is a strong buy at the current price.

Happy Birthday Microsoft at 37

I just read an interesting article by Howard Anderson, a professor of entrepreneurship at MIT and founder of Yankee Group and Battery Ventures. He makes an insightful analysis of Microsoft at the age of 37, so-called the middle age. He says, “You’ve got a disease. It’s called middle age. It’s harder to detect but easier to cure in its early stages; it’s easier to detect but harder to cure in its later stages. You’re about halfway.” Microsoft created 12,000 millionaires after its public offering back in 1986, and most of them are gone except two key billionaires - Ballmer and Gates. 

Anderson’s language is caustic, “.. but worse, you have become … irrelevant. Kind of like Madonna. She still sells a lot of music, but nobody really notices her. Oh, you have bought things (Skype for $8.5 billion), and you still dominate office software and PC operating systems. But Bing? Yammer?”. 

Or when he says, “You’re kind of like a 37-year-old who goes into a bar and says to the young barmaid: “Where have you been all of my life?” And she replies: “Well, sir, for the first 20 years, I wasn’t born.” He mentions that competing against a religion (called Apple) is hard. Young minds from schools like MIT don’t go to Microsoft any more (not due to bad weather in Seattle). Instead they go to Amazon, and Google.

I liked how he characterized Microsoft’s transition over the years – “In your youth, you were a technology company, and a great one. When everyone hates you, you must be doing something right. Then you became a marketing company. And now you’re a company that’s financially driven. Just when you think there’s nothing fundamentally wrong, there’s something fundamentally wrong.”

So what should Microsoft do? Anderson’s recommendations are pretty shocking: 

“First, kill the bean counters. Go hire the best young technologists. The stock price sucks, so don’t worry about it. Spin them off into small teams. Don’t buy big companies–you will just screw them up. Buy little companies with great brains–you want their brains more than their products. Put them into a separate company, where you own 50% and these young turks together have options on the rest.

Second, cut staff, big time. Maybe 20%. I don’t know what all of those people do and neither do you.

Third, tell Steve it’s time to go. He had a good run, but it’s time. Keep Bill.”

He mentions several companies that have made a comeback from such stages of life – Apple, IBM, P&G, and Intel. The key first step is to get out of self-denial and accept the fact of your growing irrelevance.

Happy 37Th. Birthday!

 

Stonebraker’s VoltDB

Mike Stonebraker is well known in the database research world. He started Ingres way back in the 1970s while teaching at UC Berkeley. During the same time, IBM Research was working on the research prototype “System R” where SQL (relational algebra) was invented. Mike’s Ingres came up with Quel (Query language based on relational calculus). In the ultimate battle SQL won the database language war. Ingres became a commercial product enjoying some success for a while.

The road from research prototype to product is not always easy. IBM’s successful product DB2 on the mainframe was not an extension of the System R (unlike SQL/DS on the VM operating system with limited success), rather it derived many principles from the earlier commercially successful product, IMS.

Mike Stonebraker built another research prototype called “Illustra” and sold it to Informix (subsequently acquired by IBM). In recent years Mike moved to the east coast to MIT and continues his relentless research work. His new companies included Streambase and Vertica (a columnar database sold to HP  last year). His latest work is a new product called VoltDB.

VoltDB is being positioned as “a high performance, scalable RDBMS for Big Data, high velocity OLTP and realtime analytics”. It is full-fledged ACID-confirmant RDBMS unlike its NoSQL brethren with relaxed consistency (eventually consistent model). At the same, it promises to eliminate the bottlenecks of current RDMSs towards extreme scaling – logging, locking, latching, and buffer management. VoltDB eliminates these bottlenecks by these approaches:

- data & associated processing are partitioned together and distributed across the CPU cores (“partitions”) of a shared-nothing hardware cluster

- Data is held in main memory, eliminating the need for buffer management

- Transactions execute sequentially in memory (no locking nor latching needed)

- synchronous multi-master replication for high availability

- command logging instead of “write-ahead” logging

These innovations enable VoltDB to run many times faster than traditional RDBMS products and to scale linearly on low-cost clusters of commodity servers, all the while maintiig ACID compliance. Hence it positions itself more for “transaction” processing applications (OLTP), rather than the NoSQL class that aims at data warehousing and analytics world. It can also be used for realtime analytics.

All these promises sound too good to be true. So far, none of Mike’s “commercial” products have swept the industry, even after they have been acquired by bigger giants. VoltDB does address a need for high-performant transactional applications with very high scalability. But the existing leaders such as Oracle, IBM, and Microsoft are not sitting idle, even though they are expensive solutions.

VoltDB has very few deployments to prove the theory so far. Let us wait and see.

Post-Facebook – Micro Social Networks

I am on Facebook, but not an active user at all. Also I am reluctant to share too much personal stuff with friends. Then there are long-lost friends and family members popping up with Facebook requests to connect. I do not feel anxious to connect as it would increase more “gossip” time I want to avoid.

But then, Facebook with 900 million users is heading for it’s much-anticipated IPO in 2 weeks. The prediction is that it will be a $100B valuation on the first day of treading. Now people like me who are less enthusiastic users have some alternative choices appearing. Three such companies have started attracting users – Path, FamilyLeaf, and Pair. These micro-social network sites foster “sharing that is intimate by design”.

Let us talk about the anthropologist Robin Dunbar of Oxford University, who has done research on social behavior of humans. He says, social networks are like concentric circles and 150 seems to be the outer bound, meaning that is the effective neurological limit the human brain can handle. In other words, 150 is the maximum number of “friends” and this is called the “Dunbar Number”. Then he says 50 is the number of “trusted friends”, 15 are “good friends”, and 5 are “best friends”.

An average Facebook user in the US has 245 friends, well above the Dunbar number. Then this number gets mixed up with family members, friends, and workplace colleagues. Users sometimes share stuff to certain friends they did not want to.

This has led to the creation of a new start-up called Path by Dave Morin, ex Facebook employee. He says, “Facebook has made socializing on the Internet normal. But now there is an opportunity to return to intimate socializing.” He started with an upper limit of 50 friends per user and last year increased it to 150. It is available only on smart-phones and boasts over one million users already. Its user has an average of 40 friends.

FamilyLeaf is restricted to family members only and even more restrictive. Pair, started by Canadians just connects to one friend, hence the name. It is available only on smart-phones. These two companies were funded by Y Combinator from Silicon valley.

It is interesting to see how the social networking is entering its next phase.

My Brief Bangalore Visit

I just spent 3 days in Bangalore for a board meeting. Staying in south Bangalore around JP Nagar and Jayanagar area feels like the old Bangalore of yesteryears – greenery all around, spring flowers, no high rise apartment buildings, and comparatively more tranquil ambiance. Once I went to north, around Whitefield area, the new Bangalore intimidates you – huge apartment complexes, no trees, chaotic traffic and what not. All over town, one notices the construction of the new metro – the overhangs spoil the beauty of the city. Of course the area around electronic city is all concrete jungle and heavy traffic. The airport is so far away, it takes at least 1.5 to 2 hours in mild traffic to get to the city. Moving around the city is non-trivial and everyone warns you of the rush hours in the morning and evening.

We looked at some new ventures to get services to the people using the Internet – how can goods and services come to people rather than people going to them. Suddenly the value of such services seem so high. One gentleman described how he shopped for a Samsung smart-phone for his son and went to three stores spending over 3 hours, ultimately not finding the model he wanted. He then ordered the phone on Flipkart (Amazon-like Indian company selling books and other goods over the net) and found it to be cheaper and with home delivery in couple of days. Residents of large apartment complexes can use many services via the Internet without going out of their buildings.

Innovation is happening in India, albeit at slower pace than anticipated. There are several hurdles, the most important being the change in mind-set. Young people still go for jobs in the service sector, rather than starting their own company. Investment community has not shown great interest in early start-ups. The IT services giants such as Infosys, Wipro and TCS do not foster innovation as a culture. The success of Flipkart like companies is motivation for more consumer-oriented start-ups. One hopes this will pick up speed, more out of necessity than anything else. The mobile-transaction sector seems to have great promise with the rapid proliferation of mobile devices. As someone said – there are more cell phones than toilets in India.

Traveling to Delhi reminds one of the growth in the aviation sector in the country during last twenty years. It is a common sight to see young people using their smart-phones and tablets on flight. Facebook and Twitter have the largest growth in the Indian market. Given the advantages of cloud computing, India can get into the innovation game with much lower capital expenses. But unlike the silicon valley, the motivation and required infrastructure are still lacking. There is no Stanford or UC-Berkeley that fosters innovation.

For India 2.0 to emerge, it needs much more focus and attention all around.

This is crazy – Facebook acquires Instagram for $1B

This morning, Facebook acquired the little start-up Instagram for $1B. What does Instagarm do if you have not heard about it? This is what the website says - Snap a picture, choose a filter to transform its look and feel, then post to Instagram. Share to Facebook, Twitter, and Tumblr too – it’s as easy as pie. It’s photo sharing, reinvented. In other words, it’s a fast, beautiful and fun way to share your photos with friends and family. Users also can comment on photos and “like” them. Some people describe the app as a visual version of Twitter, where it is heavily used to share photos.

But one billion dollars? Only last week, Instagram closed a round of funding ($50m) at a valuation of $500M. So the investors just doubled their stake! This is very very rare, that the executives were raising money while the acquisition talks were going on.

This is what the Wall Street Journal said:  The deal presents a quick bonanza for the startup, which was founded in the fall of 2010 by two Stanford University graduates and has just over a dozen employees. The company’s software, which is free and until recently only available on the iPhone, allows users to touch up and share photos with followers on Instagram or other social networks, such as Facebook.

Instagram is one of a cohort of young startups that have built products around smartphones and have registered incredibly fast growth in a short period of time. On March 11, Instagram founder and CEO Kevin Systrom gave a keynote talk at the South by Southwest conference in Austin, where he announced that the company’s iPhone app had nearly doubled its number of registered users since December, going from 15 million users to 27 million users.

I guess that is the secret these days – how fast can you grow your user base? Last week, it launched its application on Android and quickly added more users. Now it claims 30m registered users.

I am not sure if there is any monetization (like revenue) yet. But the new world values eyeballs more than dollars. This is also interesting when Facebook is couple of months away from its much-touted IPO, where it aims to raise $10B at a valuation of $100B. Wow. Good for the Stanford kids who founded Instagram and the investors. It’s Disneyland come true.

 

The Fourth Paradigm in Science

We all remember the late Jim Gray, the great computer scientist and Turing award winner. During the last several years of his research work at Microsoft, he focused on data-intensive computing and called it the Fourth Paradigm in scientific discovery. In a special book dedicated to the memory of Jim, Bill Gates commented, “The impact of Jim Gray’s thinking is continuing to get people to think in a new way about how data and software are redefining what it means to do science.”

So what is the Fourth Paradigm? Here is the explanation.

1. Thousand years ago – Experimental Science
– Description of natural phenomena
2. Last few hundred years – Theoretical Science
– Newton’s Laws, Maxwell’s Equations…
3. Last few decades – Computational Science
– Simulation of complex phenomena
4. Today – Data-Intensive Science (unify theory, experiment, & simulation)

Scientists are overwhelmed with data sets from many different sources such as data captured by instruments, data generated by simulations, and data generated by sensor networks.

Jim Gray named it “eScience’ where IT (Information Technology) meets Science. It is the set of tools and technologies to support data federation and collaboration for analysis, data mining, data visualization and exploration, and for scholarly communication and dissemination. He laid out the principles, fondly called Gray’s law of data engineering:

• —Scientific computing is revolving around data
• —Need scale-out solution for analysis
• —Take the analysis to the data!
• —Start with “20 queries”
• —Go from “working to working”

Interestingly, all these apply to the commercial world of Big Data. Only the scientific world has been grappling with these problems longer. Given the proliferation of devices and incoming data in petabytes, the need for tools to do analytics is of the highest priority. No wonder, 2012′s biggest buzzword is Big Data.

We miss you Jim and your pioneering thoughts on DISC (Data Intensive Scalable Computing)!

When your email gets hacked!

Yes, that happened last week to my yahoo mail (ATT) account. On a fine morning, my first instant message from a friend warned me that my email account has been hacked and security compromised. That was followed by several calls and mail messages. The intruder was sending a fake email in my name about me being in distress in London and asking for financial help. Hundreds of my contacts got this mail. Most of them knew such spams and ignored it, but some got alarmed and called to check if everything was ok.

I immediately changed my password and the email was accessible, but to my horror, all the contacts and mail folders were gone! My inbox was empty with no messages. It was like starting from scratch. I tried to call Yahoo and AT&T repeatedly, one sending me to the other in a loop, with no clear answer to recovering my lost contacts and mail folders. AT&T has to rank as the worst in customer support and their outsourced help-desk folks must have the lowest IQ. All they can do is to help you change your password! When I asked Yahoo about recovering the folders and contacts from a backup copy, I was explained very carefully that such things evaporate from the server never to be found again! Are we in 2012 or what? For now, my respect for PC-based mail like Outlook has gone up. 

For web-mails, another idea is to back up the folders and contacts into a cloud-storage product such as Dropbox, so that recovery is possible. I did recover my contacts from this MacBook’s address book (thank God, it was backed up there when I bought my new MacBook Air). Losing several folders with important mails feels like your home has been burglarized.  The agony of changing passwords to many other financial accounts is daunting. Fraud prevention and identity protection have to be the new hot category.

I was also advised that Gmail may be lot safer in that respect and Yahoo mail seems to be frequently subject to hacker attacks. One positive outcome is that I am much more sensitive to security management and alternative backups.

Land of Contradictions

During October I visited India for over two weeks. I think this may be the very first time since I left India 40 years back, that I was there during Diwali, a huge annual celebrations of lights and fire-crackers. The real essence of Diwali is to reflect within ourselves and eliminate all negative qualities (darkness) with the light of knowledge – that we are all one and infinite in our true nature.

During my stay, like every time, I saw India as a land of contradictions. There is abject poverty next to an iPhone user. More expensive cars abound, but roads are clogged like never before, with chaotic traffic all around. Modernity coupled with a lot of superstitions.

I observe two Indias – one that belongs to the youth who are well educated, of the Facebook and Twitter culture, and the other that refuses to grow up, the corrupt politicians, bureaucrats, and con-men. While the second kind uses modern gadgetry, they continue to take bribes, tell lies, and are extremely selfish without any care for the country and its people. Many of these are the rulers of modern India. In recent months, there have been huge scams involving many politicians and bureaucrats (some of them are in jail). But the average Indian shakes his head with a denial that anything will ever change and such corruptions are here to stay.

Upon visiting a place near Delhi called Gurgaon, I am reminded of Sao Paolo in Brazil, a concrete jungle with never-ending construction all around. My friends tell me how their real estate investment has been appreciating over 30% every year over last 7-8 years. It’s more expensive than any place in the USA. Then when I spoke with someone in charge of urban planning for the Government of India, I am told that there is no sewerage or proper power infrastructure planned for this entire area. The building permits were given illegally (thanks to bribes) without consideration for the necessary infrastructure needs.

The industrial scene is still dominated by family-owned businesses with the only exception of Infosys. Young people willing to start a  company are sneered at. So they all prefer to take a job. Costs have gone up and food inflation is very high. Many items seemed much more expensive than here in the US. Labor costs have been going up, making India a less attractive place for outsourcing (skills become the attraction not cost savings).

Like Gandhi said, “India lives in its villages”. I visited several rural areas and nothing much has changed. The technology stuff is yet to reach the interior. The poor is becoming poorer with high inflation. Visiting big cities does not give one a real perspective of the real India. The GDP growth is also slowing down from its heady days of 9% and up.

But it is one of the biggest consumers of cell phones, televisions, and other electronic gadgets. There is an Amazon-like company called Flipkart that ships books to your door step. You can pay by cash on delivery also if you don’t have a credit card for web purchase. The middle class with spendable income is larger than the population of the USA.

One hopes the first India wins eventually, with better efficiency and transparency. That will elevate the poor masses also.