Big Data - A New Evolution Every 6 Months

I've never personally lived through a disruptive technology phase that is evolving as quickly as Big Data. Client requirements that we heard 12 and even 6 months ago, are not the same that we are hearing today.

I looked back at the Cowen report on Big Data, authored by Peter Goldmacher way back in July 2011. At the time, and even now, I thought it was the most comprehensive overview of the Big Data wave, with a balanced view of the business and technology landscape. There are a few points that I want to re-highlight and update:

1) There is an assertion that Big Data evolves the database from a product to a category. While this may sound like fluff, it is an insightful statement. In fact, we have already evolved how we are talking about Big Data, giving credence to this point.


2) Peter talks about the diversity of data sets and the related issue of a lack of meta data. This remains a pre-eminent blockade to unleashing the value of Big Data. Many players in the market have mastered a few data sets, but cannot support all (certainly in an industry-specific context).


3) There is an assertion that emerging solutions are cheaper and therefore, it will be hard for the traditional enterprise players to excel in Big Data. While this was plausible 6 months ago, my experience with clients says this is now false. It is false for 3 reasons:

a) Enterprise clients want the backing of a deep development team, with enterprise experience, before they will trust a 3rd party with their most valuable data sets. There is too much risk if things go wrong (think data security, data loss, corruption, etc).

b) While Hadoop is critical to Big Data, it is not the endgame for an enterprise. Analytics and outcomes are the endgame. I believe its hard to deliver analytics for the enterprise, if a company has not lived in that world for some period of time. This is why large enterprises are actually looking to the more established players for solutions.

c) The big savings that I see for clients in Big Data comes on the hardware side, specifically around storage. Therefore, I don't think a comparison of software license pricing is sufficient for the analysis. Let there be no confusion, if a vendor is betting their Big Data play on the notion that clients will leverage expensive storage for the big data sets...they will fail. Take heed EMC. The report goes on to compare average storage prices: Main memory- $40/gb, SSD- $2.60/gb, and HD- $1.37/gb. When you bring storage into the TCO discussion, the perceived software license savings become a rounding error.


4) The report asserts that application development is more expensive in the Big Data world, as compared to the traditional SQL world. Peter likens coding in the Big Data world to Cobol/Codasyl, given that it is ~ 10x the code and effort, as compared to SQL. While this is sufficiently accurate today, this is the kind of thing that could change in the next 6 months.

IBM Big Data in 2012

I am very pleased with the progress that we made on Big Data in 2011. A huge team of folks have come together in IBM, to deliver on the promise of Big Data for our clients. To highlight a few accomplishments from 2011:


1) We did multiple releases of a core set of products, setting off the arms race to bring Big Data to the Enterprise.

2) We released a reference book, demonstrating that we have real success, with clients, on this topic. There are fewer companies that can reference true big data success, than is generally imagined.

3) We have built the largest big data ecosystem in the industry.

4) We acquired some key players in and around Big Data, bolstering our capabilities.


And, we are just getting started. 2012 will bring new partnerships, capabilities, and our ultimate goal: delighted clients.

For now, here is some the high level messaging that we have developed, as we have listened to clients/partners over the last year. In short: Hadoop is important, noSQL is important, analytics is important, management consoles are important...but, you need them all to have a true Big Data Platform.

IBM Big Data Platform, 2012

View more presentations from robdthomas.

If you are interested in a more detailed version of what we are doing, feel free to comment below, or reach me on twitter/linked-in.

Street Acts

Street Acts have always intrigued me. You travel to nearly any city in the world and on the weekends, the street acts will emerge. You see things ranging from "humans posing as statues", to juggling, to magicians. Each person expressing their own creativity and individuality. At its core, street acts are about deception. Said a different way, they tease the observer into believing something that is not actually as it appears. Its a craft of illusion and decption, which is part of what attracts interest.

As I digested Oracle's recent earnings call, I could not get this thought out of my head: Oracle is a street act in corporate America. It's all about deception and illusion and accordingly, its very hard to make sense of. As most readers would know, in Oracle's most recent quarter, they reported their first shortfall in revenue guidance in a decade. Yes, decade. There was weakness in all product categories, in the US/Europe/Japan, etc, etc. It's hard to find any good news when you miss expectations by $400M.

However, the real street acts began in the Q&A. I share you the transcript here, courtesy of the always reliable Seeking Alpha:

Richard G. Sherlund - Nomura Securities Co. Ltd., Research Division
Larry, could you reconcile these numbers you gave for Exadata and Exalogic? I think it adds up to 1,000 or 1,100. Didn't you say 3,000 just recently? And I think the company originally guided to 2,000.

Lawrence J. Ellison
Yes, that's right.

Richard G. Sherlund - Nomura Securities Co. Ltd., Research Division
Well, could you elaborate? Is it turning out to be a harder sell? Are you trying to figure out how to sell it?

Lawrence J. Ellison
No. I think we're going to be -- we're growing Exadata at about -- look, the growth rate is around 150% Exadata and Exalogic even faster.

Richard nailed it. The numbers and the discussions of Exadata over the last year have been a constantly moving target; a corporate Street Act. Unfortunately, no other analyst was willing to pile on. Perhaps that is because we all know one thing about Street Acts: as much as they deceive (and often delight) us, they only last so long. And once that act has run its course, there will simply be a new one.

Big Data: Delivering what clients want

The demise of a business often starts when a company becomes deaf to their clients. A good example of this is Radio Shack, the long time technology retail stalwart, that built a large footprint, primarily in the United States. Radio Shack became successful by providing what clients want: local, easy access to technology goods. However, through the years, they did not evolve as their customer tastes evoloved. While clients wanted service (help) and good prices...Radio Shack delivered limited service and high prices. This strategy worked, until it didnt (ie around the year 2000). See the stock chart below.

However, in recent years, Radio Shack has recommitted itself to listening to clients and the company is showing signs of a comeback. Like many companies, they got off track for awhile, but I hope to see them return to what customers originally valued: a place for local service and access to products. But there is a lesson learned here: companies that get consumed by what 'they' want versus what their clients want, put themselves at risk.

I share this story, because I see it happening today in the IT industry. Oracle has become consumed with driving what 'they' want, instead of what clients want. I see clients demanding a big data software platform, while Oracle is obsessed with 2 items that are at odds with a big data platform:

1) Driving a proprietary, all Oracle stack...in order to "lock-in" clients.

2) Leveraging Big Data and Hadoop as buzzwords, to retrofit all data into an Oracle relational database.

Here are these 2 items, in pictures from Oracle:

Oracle is insisting that clients embrace a proprietary stack and that all data needs to be in an Oracle database. It's as clear as day from their collateral, and alot like Radio Shack in 1999.

Big Data Applications and Ecosystem

A key part of our Big Data as a platform strategy, is building out our ecosystem of partners and solutions. As Bruce described here, we have alot of traction on this topic.

I am very pleased to see some of our partners, like Persistent Systems, building applications on top of our Big Data platform. You can view their solution below or by visiting the IBM Big Data Youtube channel.