If You Do Not Cannibalise Your Product or Service? Someone Else Will!
Hello and welcome to this new blog post. Today we are going to talk about one of the most important concepts of a successful, sustainable business. The concept of cannibalising your own products.
I will relate this idea to our business of Business Intelligence.
For those of you who do not know me I have been an avid reader of business books all my professional life. One book that had a profound influence in me was “The IBM Way” by Buck Rogers. He was a 34 year IBM veteran who was world wide sales leader from 1974 to his retirement in 1984. Of course when I went for my campus interview with IBM in 1985 IBM was considered the worlds #1 company to work for.
What stood out for me out about Bucks book was:
- Respect for the Individual
- Pay for Performance
My previous employer was BHP, Australia’s largest company. Being coddled by the Australian government BHP was very socialist in nature. People were paid for turning up, not for working. There was little respect for the individual. People were treated as interchangeable cogs in a larger machine even when that was not even remotely the case.
I started in IBM in January 1986 and was very successful over my time in IBM. IBM was great for me. I have no regrets spending 8.5 years of my life there.
The woes of IBM in the early 90s are well documented. No one needs a rehash from me. The fact that its very survival was called into question at one point in time shows how far and how fast IBM fell from 1984 to 1994 when I left. Many of us simply could not believe what happened and we were quite frustrated that it did happen. Many of us believed we would be an “IBMer for life” as so many of our older colleagues had been.
But IBM was not alone. One of the first management consulting books I read at University was “In Search of Excellence” By Tom Peters and Bob Waterman. It was the landmark book of its time. Across its pages were proclaimed giants of American industry that seemed invulnerable. Here are some names that were on those pages. There were many others.
- IBM
- Xerox
- Kodak
- Ford
- General Motors
Do you notice anything about these names? If you said that all of them have become shells of their former selves you would be very much on the right track. No one takes any of these companies very seriously any more. I mean Ford and GM? How exactly does one go bankrupt selling cars in the USA? Why is the government bailing out car companies? Why is Detroit a ghost town compared to the 1980s?
These and other once mighty corporate giants made plenty of mistakes. There are whole books on that topic. But the one mistake I want to highlight is this. The mistake of not cannibalising their own products in favour of something new and innovative.
Tom Peters talked about this in his excellent 1991 book called “Thriving on Chaos”. His comment was that if you have more than 50% market share then you have to do pure research into “the next big thing” because to fail to do so guarantees that someone else will create an innovative and disruptive product that will steal your market away from you.
And so it happened to many companies that were considered invulnerable.
Xerox was THE photocopying company. They invented the Xerox Star which was the worlds first commercial computer to use graphics and mice. It was “the paperless office”. But in the eyes of Xerox anything that reduced the amount of paper to be photocopied was a bad idea and the Star Workstation was not supported.
Programmers from Xerox Parc when to Apple, Microsoft and Metaphor Computer Systems. Everyone knows Apple and Microsoft. But not so many people know Metaphor Computer Systems despite the tremendous influence those people have had on the Business Intelligence industry. Many of the people who created the Business Intelligence Industry worked at Metaphor or were Metaphor customers. At the time of writing Xerox is valued at USD14.6 B.
Kodak was the unchallenged master of the Camera industry. They actually made their money on the film processing so the cameras were almost given away in order to get the film processing. That worked out just fine until it was possible to take photographs digitally. Trying to protect its film business put Kodak almost out of business. At the time of writing the market capitalisation of Kodak is USD768M. Spare change.
IBM tried to stop the cannibalisation of its large and midrange computers by this upstart microprocessor. The mechanism IBM used was called the Personal System/2 which was an abysmal failure commercially. IBM lost the desktop wars to Microsoft. The microprocessor put such price pressure on IBM Mainframes that revenues and profits from mainframes dropped like a stone in the 90s. The loss of the mainframe revenues is what hit IBM the hardest.
When then CEO Lou Gerstner cancelled the research and development of water cooled mainframes in 1994 everyone at IBM was shocked to the core. This was the #1 revenue product in the company and research and development was cancelled in favour of air cooled CPUs. That is how drastic the changes were. At the time of writing IBM is valued at USD156B.
The big 3 car makers are all a shadow of their former selves. Detroit is a ghost town. Cars from other countries are common place in the USA today. The car makers were out done by foreign competitors and there is no going back.
Put simply. The biggest and most successful companies in the world can not ignore the rule that “if you do not cannibalise your product or service then someone else will”.
The response of the big companies was pretty much all the same. Try to stop the innovation and milk the cash cow for as long as you can. That strategy leads to becoming irrelevant. There is always a market for a new and innovative product that delivers more for less. The market might not be obvious. It might take a while to mature. But it is there. Every…single…time.
So what does this have to do with Data Warehousing and Business Intelligence? Our industry?
Well? For years the big vendors have been trying to dominate and control by buying everything they can lay their hands on and stifling innovation rather than embracing innovation and creating “faster, better, cheaper, simpler”.
None less than Michael Dell has taken his company private and lamented that quarterly results and daily share price fluctuations have driven so many companies to focus on the here and now, the short term, rather than think for the longer term.
Just look at the Business Intelligence industry as one example, and the tech industry overall as a larger example. The story for the last 25 years has been “small innovative company creates something really special and new, gets bought out by a bigger company, who gets bought out buy a bigger company, who gets bought out by a bigger company, and the product is dead on arrival.”
I could list many such companies. Metaphor Computer Systems being the first such example of how the industry leader was bought by IBM only to be caught up in the troubles that befell IBM. Hyperion was bought by Oracle as was Brio Query. Business Objects was bought by SAP. Sybase was bought by SAP. Sequent was bought by IBM as was Cognos, Ascential and Informix.
Despite the billions upon billions upon billions that the big players have spent on buying up all manner of Data Warehousing and Business Intelligence companies we have this hugely embarrassing 50% or more failure rate for our industry. People argue over exactly how many “failures” there are in the Business Intelligence Industry. It is hard to know because people will not talk about their failures publicly. But every time IDC, Gartner and others run surveys the answers are depressingly similar. They can’t all be lying, right?
Whether it is 40%, 50% or 60% it is too high because there is no reason to have any failures in our industry. No reason at all. I have never had a failed BI project where the client took my advice and I have been at this for 23 years. The clients that did not take my advice often acknowledged that the failure of their project was exactly because they did not take my advice.
Today the big vendors and big consulting houses are trying to do in Business Intelligence exactly what IBM, Xerox, Kodak, Ford and GM tried to do when faced with “better, cheaper, faster”. They have tried to stifle that innovation because it cannibalises their products.
Whether it be products like DataStage or Informatica or whether it be the outrageously expensive consulting services that companies provide to implement data warehouses, the big vendors are doing their best to “keep up the numbers”. They are doing this in the face of the fact that our products, SeETL and BI4ALL, have made it possible to implement data warehouses at a fraction of the cost of the big players.
The day I saw a Netezza machine perform a join between a call detail record staging table with 80 million rows in it and 26 key lookup tables and insert the result into a work table I knew that the days of the ETL tools of the like of DataStage and Informatica were over. It is only a matter of time. That 2009 Netezza experience proved that databases were getting fast enough, and the SQL Language was getting sophisticated enough, that the need for an external C++ engine based product would disappear just like the need for film in a camera or photocopiers in the age of print/scanners.
Sean Kelly and I were doing great work with Netezza creating an extremely cost effective solution that was based on the Netezza technology along with our SeETL and BI4ALL products. And what happened? IBM, finally admitting that they had totally messed up DB2 for Data Warehousing on AIX, bought Netezza for USD1.8 Billion. The IBM purchase of Netezza was the final, humiliating and public admission by IBM that they could not build a data warehousing database on AIX. An embarrassment to be sure.
IBM then promptly make sure none of THEIR customers ever found out that there was a much better solution available on Netezza for ETL and data models than the IBM couplings in that area.
As I have often wondered aloud. When I worked for IBM we used to do all we could to provide the best possible customer service, advice and guidance to our customers. I was an AD/Cycle Systems Engineer and I invented much of the technology that made the product I supported a global success. That product is the Software Configuration and Library Manager.
But today? IBM takes a different approach. IBM takes the approach of actively trying to suppress information being given to the customer to “keep them locked in” rather than to innovate and invent so that customers wish to stay because they are getting great value for money.
I first heard of this concept back in 1992 when I was working for one of IBMs largest customers in Australia. It was the National Australia Bank. One of the “Big 4”. I was teaching them the new ideas I had invented to store data in DB2 for high performance query. The concept was called “customer entanglement”. The idea was to offer the client complementary products with co-dependencies and fees for leaving so that the customer would become “entangled” and have less desire to leave.
I put the question “Would it not be better, in the longer term, to offer great products and services and to become more efficient than your competitors so that you customers WANT to do business with you?”
They actually laughed at me! They actually thought the idea of competing based on merit rather than competing based on deceit to be funny. 1992 is a long time ago now. And many Banks have got something of a bad reputation today. Why not? With their marketing ideas based on “customer entanglement” as opposed to “great products and services at competitive prices” how would it be possible to NOT have a bad reputation?
The irony that I was there to provide “great customer service” and teach the people at the bank how to adopt a new data modelling technique that would enable that particular bank to surge ahead in the marketplace was lost on these people. Based on the ideas I gave the bank they were able to be much more effective.
The National Australia Bank doubled their gross profit from AUD1 Billion to AUD2 Billion over the next 5 years to become the dominant industry player in Australia. All because I taught them how to store data in DB2 a new and innovative way that allowed them to do analysis that they could not previous afford to do. My ideas reduced CPU consumption between 100 and 1000 times.
But it is not the same today. The customers of the large vendors have rewarded the vendors for less than stellar business practices by not telling anyone about such practices. The vendors have been emboldened to be less than forthcoming with “better, faster, cheaper” and have relied on buying out the emerging leaders as their business strategy.
By doing this in the Business Intelligence area we have all been tarred with the same brush of being less than reliable providers of products and services. After 23 years in this business I would like to raise a little protest about how I am tarred with the same brush as people who have actively worked against the interests of their clients.
These large vendors are fighting against the tide of “cannibalise your product or someone else will”.
It didn’t work for those household names above. And it’s not going to work now.
We now know that it is possible to guarantee the success of a Business Intelligence project across a wide range of industries if only the client takes the advice they are given by a reputable consultant like myself. Even better? With SeETL and BI4ALL it is possible to build the prototype data warehouse on very cheap commodity hardware before any real money is spent on hardware, ETL or BI software.
As Frederick Brooks said in his seminal work “The Mythical Man Month”.
“When you build a computer system plan to build it twice, you will anyway.”
And that is what SeETL and BI4ALL supports. The ability to build the prototype data warehouse on a development machine that is as large as you would like BEFORE you have to sign the purchase orders for the production environment.
I have long said about data warehouses:
“If you can not build the prototype you most certainly can not build the production system.”
Now we have vendors of all sorts pushing the falsehood that “Big Data is Data Warehousing” and that somehow Hadoop has become a product that you can build a data warehouse with. Thankfully some people are writing blog entries about such fraudulent claims. I will also make my comments on these things public over the coming weeks and months.
For this blog entry. It is enough to say that even if you or your client has no intention at all at going in to production with SQL based ETL. Even if you fully intend to stick to your Informatica or DataStage? You can reduce the cost and improve the quality of your projects simply by using SeETL for your ETL prototype development. And we even make it free if you sign a contract saying you will not deploy our ETL into production. We can’t make it any better than that!
If you are going to make a highly disruptive change to your data warehouse such as a massive redesign, or if you are building one for the first time as a smaller company? The BI4ALL models guarantee that you will have available to you all the techniques needed to perform the implementation. Indeed, we would prefer to do the ETL and data modelling for you since we are almost certain to be much better at it than you are.
I have been in BI for 23 years and IT for 32 years. I am fully aware that SeETL is an innovative and disruptive technology that is going to transform the way in which data warehouses are built. The reason for this is because SeETL is “better, cheaper, faster, simpler”.
SeETL is nothing more, and nothing less, than using Excel as a data capture tool to capture the definitions of what needs to be defined for a Data Warehouse project coupled with a VB program that reads the defined workbooks and generates the code needed to be fed to tools like databases.
Anything that can be expressed into the sort of form that can be captured in Excel can be accommodated. Where the information/data can not be expressed in to an excel spreadsheet, for example a stored procedure, a proxy entry is placed into the spreadsheet and the code itself can be loaded into the database for safe keeping.
SeETL allows any file of any sort to be read and placed into the data warehouse database. Mostly we are just placing the ETL SQL in to the database so that it can be controlled like any other data.
Because SeETL is just spreadsheets containing data that is read and used to generate code for other tools there is nothing that is beyond our ability to write code to handle. It must also be remembered that we have the older C++ engine and we have a library of C++ programs that do all sorts of useful things. So our toolkit is the C++ compiler which is capable of doing anything that a computer is capable of doing.
We have been able to perform all the ETL needed for many data warehouses with this now almost free software. There is simply no question as to the capability of SeETL to perform the vast majority of ETL needed for today’s data warehouses. As we get more into Hadoop and similar areas we will add extensions to SeETL to handle those areas as well.
However, today?
What I wanted to point out in this blog post is that some of the worlds leading companies have tried and failed to stifle innovation defending their cash cow products rather than to embrace the idea of “cannibalise your own produces and services or someone else will”.
What I wanted to point out in this blog is that SeETL alone, or coupled with BI4ALL, represents a disruptive “faster, cheaper, better, simpler” paradigm shift which will cannibalise products in the ETL and data models space whether the owners of those products wish to resist that cannibalisation or not.
The question for you, my dear reader, is whether you will personally get on board with the SeETL disruptive “faster, cheaper, better, simpler” paradigm shift or whether you will “wait and see how this works out”.
So? Why get on board sooner rather than later?
Well, firstly, because “faster, cheaper, better, simpler” is good for your customers. You can be the bearer of good news. Cost reductions for your ongoing Business Intelligence work. But if that is not a good enough reason? I think another good reason, just in my humble opinion, is that as people on the business side of the house find out that many of their IT folk have overcharged for their data warehousing efforts they might be less than stellarly happy about that.
We have had SeETL available in the marketplace for three years at the nominal cost of EUR800 per year per user. During that period we have seen many more failed data warehouse projects. It is my contention that the majority of these project would not have failed if the IT staff were required to build the prototype with SeETL before they were allowed to try building the ETL with the likes of DataStage and Informatica.
Why would these projects not have failed? Because if the IT people could not design and populate their proposed data warehouse with SeETL as a prototype then they most certainly could not do so with a vendor ETL tool in production. The business management, who are your internal customers, would have had the chance to cancel the project before too much money was spent. A project cancelled at prototyping stage is not a failed project. It is the avoidance of a failed project.
Not so long ago I saw one of the worlds best known companies waste USD2M trying to get a data warehouse built by a “cheap vendor”. They bought the project because the price was the lowest of the bidders. I read the tender response and it was laughable. How does one of the worlds largest companies manage to waste USD2M on a laughable data warehousing tender response in this day and age? If they had paid me one days work to read the tender response I could have saved them USD2M by telling them not to buy it!
It’s just crazy that I do not have large companies who are buying tenders from these scurrilous vendors check the tenders for them. It only takes a day or two for me to check a tender and yet this is a day or two “too much” when multi million dollar projects are at stake? It is just crazy that this is still the case.
So that is what I wanted to say today.
We have many large vendors doing their best to squeeze every dollar, euro, pound they can out of Data Warehousing, Business Intelligence, Big Data products and services. They are studiously avoiding “cannibalise your own products or someone else will”.
SeETL, coupled where appropriate with BI4ALL, is the product that is going to cannibalise both the ETL markets and the consulting markets for Data Warehousing over the next 5 to 10 years. In 5 to 10 years time the use of SeETL will be just as widely accepted as the use of dimensional models is today.
How do I know that? I was the man who introduced dimensional modelling to Australia. I know exactly how resistant IT people are to innovative new ideas. I was actually asked to leave some companies as something of a “heretic” when I presented the ideas for dimensional modelling to some of the largest companies in Australia in the mid 90s. And yet? By the time SQL Server 2000 turned up it had support for star schemas built into the optimiser.
It will be the same with SeETL. Once people realise that it is “better, cheaper, faster, simpler” and that it reduces the costs of development so significantly? Those who wish to embrace this disruptive innovation will. Those who choose not to embrace it will leave themselves open for questioning and criticism as to why they did not act to reduce the costs of their business intelligence efforts for their clients.
I am very proud of SeETL and what it enables us to do today. When I set out on this career path in 1991 it was clear that getting data loaded in to the data warehouses we were going to build was the major portion of he problem. It has remained so. With SeETL we now have a tool that reduces what we took 18 months to do in 1994/5 down to maybe a week or two.
As demonstrated in our previous post SeETL has consistently reduced the amount of time and effort that it takes to build a data warehouse since it was first written in 2003.
SeETL will continue to be further enhanced. Every time we see something that needs to be done and could benefit from having a worksheet in SeETL, we will that worksheet and speed up that process.
We have many videos available on our education channel to show you all the different things that SeETL can do in the current version. There are 80 workbooks in 3.1.02 that manage all sorts of areas of a Business Intelligence Project. If you would like to download the GA package and review a sample data warehouse and all the associated documentation? You can get the 51MB GA package on this link.
In summary Ladies and Gentlemen?
Let us co-operate and collaborate to make developing data warehouses “better, cheaper, faster, simpler”. Let us have fewer failures. Let us work together to start getting the charlatans and liars out of our business segment so that we can improve the reputation of all of us, ok?
I think that is something really worth doing for our industry.
If you agree? Please feel free to comment below!
Best Regards
Peter
