Big Data : All Aboard the Information Bus
What is good for the short term is often not good for the long term.
A simple example from baseball:
As it became obvious that steroids enhanced performance, many players turned to the 'juice', to drive their short term performance. However, they quickly learned that it was not sustainable. Once you stop or overuse it, your performance and stability begins to breakdown. At the risk of picking on A-Rod, its fairly obvious when you look at a simple graph below. Note: the yellow star is when he admitted to starting to use steroids..
So, good for short term...disaster in the long term.
Turning to Big Data...
I believe an enterprise has to approach this problem with the long term in mind. There are a number of approaches for short term gratification, but they will not scale to support an enterprise economically in the long term. This is why I have alluded to the application server analogy for Big Data a few times (most recently here and here). Let me use some basic graphics to illustrate this analogy.
Typical/Simplistic View of an Enterpise
Simply put, an enterprise IT organization invests across all 3 layers and on integration inbetween.
However, as I look at emerging Big Data solutions, I'm seeing a pattern that looks like this:
The blue bar represents a 3rd party product that provides the entire stack, in proprietary packaging. This approach drives instant gratification. There is a single purchase, the enterprise gets the entire stack, etc. There are new companies that have been successful with this model. But that being said, I am not convinced this is a great approach for an enterprise longer term. Here is why.
Unfortunately, enterprises are not as simple as I depicted above. They often look more like this:
Enterprises are complex, they evolve, they buy, they build, they customize, etc. If you try to take the 'proprietary stack' approach to solving your problems, your enterprise will eventually look like this:
This is simply not economical for the medium to long term. With this approach, the enterprise ends up with 10-100's of full proprietary stacks (the blue bars), and no opportunity to innovate/customize. And worst of all: the enterprise has no leverage in their investments. Every time they need a capability, they have to re-buy capabilities they already have. Like A-Rod in 2002, they will start to regret their short term gratification decisions. To repeat, I understand that some companies have been successful in the short term with the instant gratification model. I just don't think its the right model for enterprises.
This brings us to the application server analogy. Given the data growth (size and variety) that we have seen in the last 3 years and will see in the next 10 years, an enterprise must have a flexible platform for managing that data. Such a platform will leverage all information sources, will annotate/organize that data, will provide real-time analysis, will allow the enterprise to write their own applications or buy 3rd party applications, etc. It will become the "Information Bus" for the enterprise. Todd used the phrase Next Generation Middleware during our exchange on this topic at the Strata Conference panel (see the 22 min mark). Whether its the "information bus" or next generation middleware, the graphic below illustrates the leverage that this approach will give an enterprise.
Intelligent enterprises do not make short term gratification decisions en masse. They might make them to solve point problems from time to time. But, it won't be their strategy. For that reason, I believe we will see Big Data emerge as a true platform...the Information Bus of the future.
A simple example from baseball:
As it became obvious that steroids enhanced performance, many players turned to the 'juice', to drive their short term performance. However, they quickly learned that it was not sustainable. Once you stop or overuse it, your performance and stability begins to breakdown. At the risk of picking on A-Rod, its fairly obvious when you look at a simple graph below. Note: the yellow star is when he admitted to starting to use steroids..
So, good for short term...disaster in the long term.
Turning to Big Data...
I believe an enterprise has to approach this problem with the long term in mind. There are a number of approaches for short term gratification, but they will not scale to support an enterprise economically in the long term. This is why I have alluded to the application server analogy for Big Data a few times (most recently here and here). Let me use some basic graphics to illustrate this analogy.
Typical/Simplistic View of an Enterpise
Simply put, an enterprise IT organization invests across all 3 layers and on integration inbetween.
However, as I look at emerging Big Data solutions, I'm seeing a pattern that looks like this:
The blue bar represents a 3rd party product that provides the entire stack, in proprietary packaging. This approach drives instant gratification. There is a single purchase, the enterprise gets the entire stack, etc. There are new companies that have been successful with this model. But that being said, I am not convinced this is a great approach for an enterprise longer term. Here is why.
Unfortunately, enterprises are not as simple as I depicted above. They often look more like this:
Enterprises are complex, they evolve, they buy, they build, they customize, etc. If you try to take the 'proprietary stack' approach to solving your problems, your enterprise will eventually look like this:
This is simply not economical for the medium to long term. With this approach, the enterprise ends up with 10-100's of full proprietary stacks (the blue bars), and no opportunity to innovate/customize. And worst of all: the enterprise has no leverage in their investments. Every time they need a capability, they have to re-buy capabilities they already have. Like A-Rod in 2002, they will start to regret their short term gratification decisions. To repeat, I understand that some companies have been successful in the short term with the instant gratification model. I just don't think its the right model for enterprises.
This brings us to the application server analogy. Given the data growth (size and variety) that we have seen in the last 3 years and will see in the next 10 years, an enterprise must have a flexible platform for managing that data. Such a platform will leverage all information sources, will annotate/organize that data, will provide real-time analysis, will allow the enterprise to write their own applications or buy 3rd party applications, etc. It will become the "Information Bus" for the enterprise. Todd used the phrase Next Generation Middleware during our exchange on this topic at the Strata Conference panel (see the 22 min mark). Whether its the "information bus" or next generation middleware, the graphic below illustrates the leverage that this approach will give an enterprise.
Intelligent enterprises do not make short term gratification decisions en masse. They might make them to solve point problems from time to time. But, it won't be their strategy. For that reason, I believe we will see Big Data emerge as a true platform...the Information Bus of the future.