Welcome to My New Blog
Here are the promised links to the history of SeETL and BI4ALL.
The History of SeETL
The History of BI4ALL
Hello and welcome to my new Blog. As you can see I upgraded my old web site to a new wordpress format before Christmas. The old web site has been unchanged for nearly 10 years and was showing its age.
I hope you like the new site. Feel free to leave comments! I have added the “log in with facebook” feature on the blog pages so you can join the site with just one click. You can leave blog comments when you are logged in via your facebook account. If you log in with facebook you will not need to have a separate password for this site.
One more administration note is this. I have created two a blog entry for the history of SeETL and BI4ALL. You can review these historical stories on the link below. They are not that necessary to listen to but they do explain how SeETL and BI4ALL came about and why they are such mature and robust products today.
Now. On to the blog itself. The first question you might want to ask yourself is “Why is Peter starting a blog after being so publicly silent about Data Warehousing all these years?”
So let me explain so that you know my motivations for speaking up in public on the matter of Data Warehousing and Business Intelligence again. The explanation is long so feel free to get a coffee or cup of tea. Whatever is your preference. Everyone who knows me knows I like to be thorough in my explanations so that there is no room for misunderstanding. If someone misunderstands me because they did not read what I wrote or listen to what I said that is their problem, not mine.
Now. As everyone who has been around the Data Warehousing Business for more than 15 years knows I was one of the top contributors to the old DWLIST forum. I spent many hundreds of hours posting to DWLIST back in the 90s. Ralph was also a top poster at the time.
I was very passionate about our emerging industry. Many of the projects that I did in the 90s had such massive return on investment that I could see a bright future for Business Intelligence and those who were good at it. I could also see the charlatans coming into our industry in droves.
I saw projects fail time and time again. The recipe was always the same recipe.
- The IT people in the company would say “We can do this just as well as any external consultant or vendor. There is no need to hire anyone else. We will do this data warehousing thing internally.”Nine times out of 10 such a project would fail. Just like my very first effort to build a data warehouse was a “failure” and had to be “fixed”. We were all still learning back then.
- Having spent 12 months and a few hundreds thousand dollars (AUD because I was in Australia) the next approach was “we will get an individual who has done this before and hire him as a consultant and then we will learn from him.”That was the business I was in during 94 and 95. I had a long term stable client in the Mutual Life Company in Australia. The work we did underpinned their remarkable business performance from 91 to 96. I was responsible for building the back end databases and supporting the business analysts who used to data warehouse for MLC.I spent 5 great years there on and off. I could see the HUGE value of business intelligence and I was in on the ground floor. We used the Metaphor Data Interpretation System to do our analysis work back then. It lives on as www.meta5.com.
During that time I had other notable clients. I wrote the Data Warehousing Methodology for SAS Institute in Australia. That methodology was then upgraded and deployed worldwide I found out many years later.I also designed a banking data warehouse for the IBM RB2020 system in 95.
It was funny to be asked back by IBM do build a data warehouse having resigned in the middle of 1994 after 8.5 years with IBM.These sorts of projects had about a 50-50 success rate. There were very few people who really know what they were doing in the mid 90s and so hiring the single external consultant to do the project was fraught with danger.
- Next was “go with a reputable vendor”.In the mid 90s there were really only IBM, Teradata, Oracle, Prism Solutions and to a lesser extent SAS Institute, who could talk sensibly about Data Warehousing in Australia. And we all know the sorts of fees these sorts of companies like to charge.The “go with reputable vendor” has not produced a higher level of success and there are good reasons for this. Here are some examples that I was involved in to give you the background of why “go with reputable vendor” is no more likely to produce success than hiring one guy who knows what he is doing.
In 1996 I was offered the position of starting the Hitachi Data Systems Data Warehousing Practice as the Practice Manager. This was a fantastic opportunity because the deep pockets of Hitachi meant that we would be able to do data warehousing properly. So I set the practice up and we were stunningly successful.We won the Australian Customs Service three year systems integration deal inside my first year on the job. It was the largest data warehousing tender of the year and the Australian Customs Service was the second biggest money earner for the government after the Tax Office. In Australia the Customs Service collected the taxes for alcohol, cigarettes and petrol usage for the country. It also collected the customs duties for imported good.
Winning the Australian Customs Service data warehouse project over all competitors put my name on the map in Australia as someone who could beat any of the “big guys”. Not long after we teamed up with CSC to win another project at the Department of Defense.
Alas the internal politics of Hitachi made it impossible for me to do the job I was given and I finally resigned when it was clear that Hitachi was going to snatch defeat from the jaws of victory. I met face to face with the World Wide VP of Professional Services who was my former boss as Managing Director of HDS Asia Pacific on this matter.
His answer was that the politics would stand and if I wanted to stay at Hitachi the Branch Manager had final say over who was placed on any of our projects. The Canberra Branch Manager was giving his pals the jobs at Customs. Alas, there were not enough people who knew what they were doing and eventually the Customs project was cancelled and Hitachi was removed due to poor delivery.
After the failure of the Customs project at the hands of internal politics it became clear to me that even the major vendors were going to mess up a good deal of the projects that they won.
So by 1997 it was clear that this new and emerging business called “Data Warehousing” would come under threat from charlatans and corporate politics in vendors. So I decided that I would speak out and call for more rigor.
For example? At Hitachi I had proposed the idea of a “University” for Data Warehousing where we trained our consultants and certified them with the ability to revoke the certification publicly if they did not perform up to the level we had trained them.
This idea was vetoed by a senior manager on the basis that many staff did not have degrees and the creation of such a “Hitachi Data Warehousing University” might well upset some of these people. I thought this excuse was ridiculous but I was over-ruled and the idea did not go forward.
After my time at Hitachi I publicly promoted the idea of a “University” or some form of training that would certify people to be competent at Data Warehousing. Whether or not my proposal for such had anything to do with Ralph setting up the Kimball University I do not know. You can always ask Ralph. He was certainly on DWLIST at the time I was talking about such.
Ralph knew me very well at the time. I was the man who promoted Ralphs book in Australia. We gave away lots of copies of Ralphs book. We used his database samples in our demos. I even called our Methodology the “Hi-STAR Warehouse Methodology” and my ETL software was called the “Hi-STAR Warehouse Toolkit”. The ETL software I had written in 1995 gave us a 50% pricing advantage over the other vendors for the consulting side of the work.
We were able to deliver the entire Data Warehouse for about the same price as the license fee for Prism Solutions ETL software at the time. In Australia a lot of companies had only modest budgets for their data warehousing efforts so we were well positioned.
By 1998 I was leading the business development for Data Warehousing for Price Waterhouse Coopers. My mentor from Metaphor had moved to PwC in 1994 after he was retrenched from IBM. He had been asking me to come on board for years. I was reluctant because PwC had a very bad reputation for how they treated their staff. A reputation that was very well deserved. I was also treated poorly in my time at PwC as were my other senior colleagues. So my tenure at PwC was just one year before I was fed up.
In that year I won the Telstra Corporate Data Warehouse from IBM in an open tender. For those who do not know Telstra is the ex government telco and the most profitable company in the country. Telstra also owned 25% of IBM Global Services Australia and had transferred 3,000 staff to IBM. So taking the Telstra Corporate Data Warehouse off IBM for PwC was the “deal of the decade”. PwC was so certain that we could not win that they did not allocate staff to do the project and the project was “difficult” to say the least.
Having again won the largest Data Warehousing tender of the year in my country only for my new employer refuse to deploy the people needed frustrated the hell out of me. Tenders like that are very hard to win. We had slept on the floor of the office some nights as we worked around the clock to write this tender in just 8 days! And here we had won it and PwC would not send us the staff we needed!
I resigned from PwC when I was asked to cut the functionality of what was proposed. The level of functionality cut would be, in my opinion, the criminal act of contract fraud. My partner was not willing to put his request in writing which made it clear that I would be blamed if Telstra objected to the reduction in deliverables.
Given I was 35 at the time and that I had to work for the next 25 or 30 year in Australia I was not about to tarnish my reputation by committing contract fraud against the most profitable company in the country. So resignation was really the only path for me to take. The guy who took over from me did make the cut to functionality and he enjoyed a long career at PwC. His willingness to rip off his client was richly rewarded by PwC and later by IBM who, of course, bought PwC a year later.
Having seen two of the biggest Data Warehousing projects that I played an instrumental role in winning for my employer be mismanaged to the point of failure I was very frustrated about what was happening in our industry.
When I was interviewed for my next job by Peter Gyenes (CEO of Ardent Software, then owner of Prism Solutions and makers of DataStage) I spoke of these two projects and asked Peter for his word that if I took the job at Ardent I would have the authority to actually get the job done.
Peter shook hands and gave me his word. So I took the job. Peter lived up to his promise and my 18 months heading up Ardent Professional Services were a real highlight. I worked with great people that I will never forget. We went back in to Telstra and won significant work away from PwC. We also got DataStage into Telstra as well as many other companies.
In just under two years I was at Ardent we, quite literally, cleaned up in the ETL tools space and the consulting space. On the day that Informix bought Ardent the DataStage vs Informatica count in Australia was 50 to 2.
One of those two was the Australian Customs Service where Hitachi had removed the Prism Solutions Software because I was now the Professional Services Manager responsible for that software. (The other was a PwC client who did not want me involved in their client!) Talk about childish. Swapping out a piece of software because the man you stabbed in the back a few years earlier was now responsible for its consulting support. That we allow people to act like this in our industry is a real shame.
Most people know Informix bought Ardent. Informix had a bad reputation for how they treated staff in Australia as well. However, the deal was very positive on share price and I had a lot of options. My options would not mature for years so I was not going anywhere voluntarily.
With Ardent in Australia being one of the most successful Ardent countries I, incorrectly, thought the Informix guys would pretty much leave us alone. Nope. It soon became obvious that “all bets were off” and Informix were deliberately undermining our prior efforts to get people to resign in frustration. Many did.
My ability to get my job done was completely destroyed in an effort to get me to fail to make my quotas to create the excuse to terminate my contract. It was only a successful wrongful dismissal case by the marketing manager and a threat from me to do the same that had the Informix Managing Director back off.
In the end we negotiated a “constructive demotion” from Data Warehousing Professional Services Manager for Asia Pacific to Senior Consultant and I went back into the field on billable jobs for two clients. They were Australian Consolidated Press, the largest publisher in the country and a part of the Kerry Packer empire, and Qantas Cargo, the Cargo arm of our national carrier.
At the end of the year Informix made all my previous staff redundant and paid them all out. I too was made redundant. By being made redundant my options had to be paid out. But, of course, Informix had managed to lose 90% of its share price. From a high of USD22 the shares slumped to a low of USD2.69.
Having personally been on the end of three different companies mis managing their Data Warehousing businesses I had decided to give up on these people. I had been invited to come and work with Sean Kelly at Sybase. Sean was already a legend in the Data Warehousing space and his company had been acquired by Sybase. A friend of mine was working for Sean and he proposed that I join them in Ireland for a 2 year working holiday. A break from this madness sounded like a really good idea at the time.
It was at this time that I also gave up the public debate about some sort of training and certification of Data Warehousing consultants. It was clear that the big players like IBM, Oracle, PwC etc were not interested and that they would continue to mess up projects no matter what people like me said in public. There was no appetite in the business community to hold vendors accountable for failed Data Warehousing projects. There was not even a willingness to publicly name vendors or individuals who mis-managed projects.
My predictions that the charlatans would win deals with lies and false promises and give us all a bad reputation came trued over the period from 1998 to 2005. Even today the whole area of Business Intelligence and Data Warehousing enjoys a less than stellar reputation.
Project after project still fails. Every time IDC or Gartner or anyone like that surveys projects they come back with results were failure rates are stubbornly around the 50% mark.
People hold conference after conference after conference to talk about “Why Do Data Warehousing Projects Fail”. You never see a conference topic called “How Do You Guarantee Data Warehousing Success?”
No one bothers to ask me to speak at conferences. No one bothers to ask my opinion on how to make Data Warehousing projects successful. They just keep messing up their projects and then act like there was nothing they could have done. It really is disgraceful.
All these years, since 2002 until now, I have been almost publicly silent on the issue of how to make Data Warehousing projects successful. There seems to be no point talking about it because it is very obvious to me most people are not willing to listen. So I have been going about my business much more privately than I did when I was posting to the public in the 90s.
If you listened to the history presentations on this link, link provided, you will hear more details on how SeETL and BI4ALL evolved. Much of that section used to be at this point in this post but I decided to move it out to its own post because this post became far too long. The history is there if you want it.
Coming up to date?
In early 2012 Sean Kelly and I agreed that I would put license keys into SeETL and sell SeETL as a commodity product even though that might introduce competition for ourselves in our consulting business. The idea was to sell SeETL to consulting houses on an annual license fee making it extremely cheap to do so. Again, amazingly to me, people have not adopted SeETL despite the fact that it is basically free at EUR800 per year per person.
Last Christmas, December 2014, I mailed out a “Christmas Special Offer” for SeETL with a 75% first year discount offer and a EUR2000 lifetime license offer. Again, amazingly in my opinion, not one person on my list of more than 6,000 outbound emails took up the “Christmas Special Offer”.
Apart from being frustrating this is just plain crazy. Our industry has a 50%, or more, failure rate. Where packaged models are being implemented one report puts the failure rate at 85%. I can’t believe that figure but I am aware that packaged data warehouse implementations do have a higher than 50% failure rate from what I can see.
It is just crazy that our industry, Data Warehousing and Business Intelligence, has available two products that literally GUARANTEE SUCCESS for those people who implement them properly and yet our industry rejects the use of these products in favour of chasing the latest shiny bauble called “Big Data”. It is an embarrassment. If the charlatans had any honour they would be ashamed of what they are doing.
So. Having watched on for so long in relative silence? Having seen how even people like me have got a “bad reputation for Data Warehousing” despite working in the area for 23 years and NEVER having a failed project where the client took my advice?
I think it is time to speak up again and point out to business people and IT people alike that there is absolutely no excuse for a data warehouse project to fail. No excuse at all. I am tired of being painted with the same brush as all the charlatans and liars. I have done nothing to deserve that and everything to merit just a tad more respect.
The SeETL and BI4ALL products are now very mature and robust products that have taken many, many years to develop. SeETL is the culmination of 20 years of effort developing ETL tools to do exactly what is needed on a Data Warehouse project. It is now tightly coupled to BI4ALL. The two tools are closely bound along with the methodology to implement them.
It is, in my humble opinion, completely insane that these products are being ignored while at the same time we have a 50% failure rate in our industry. The 6,000 people who are on my mailing list include everyone who is important in our industry. And yet all my efforts to make them aware that we have the tools to guarantee success have met with a rather stony silence.
Now. Some people will accuse me of “you are just selling and you just want to make money from your software”. Guilty as charged. I do want to make money from the software I have developed. So does almost everyone who develops software! Bill Gates did not get rich by giving away his software. He asked to be paid for his software. I will also be asked to be paid for my software just like everyone else. If anyone has a problem with that? I will ask them “Do you work for free?”
That said? I am asking pennies for SeETL. EUR800 per year per person for the full function product. That is one days consulting fee for a mid level consultant. We all know that. With SeETL the ETL developer will be a minimum of 50% more productive. I could easily justify much higher prices but I want to remove cost as an “issue”. No one can have an “issue” with paying EUR800 per annum for a copy of SeETL. Further, SeETL now covers a BI project end to end. We are the only people with an end to end BI meta data dictionary that I know about.
Further, with the recent sad passing of Sean Kelly I am able to offer different pricing for BI4ALL. With Sean Kelly and Associates we were selling the data models at a price point suitable for large companies like Carphone Warehouse. Because we were doing this we did not want to make offers for BI4ALL that would cause problems for Sean promoting the same data models at a much higher price.
With Seans passing I have made a new offer on our web site for new clients. The new offer is that we will design and build your data warehouse for you and give you back the finished data warehouse. We will maintain it for you if you want us to. If you want to maintain it yourself then you will have to buy licenses of SeETL to perform the maintenance. You can maintain what we give back to you manually. We give you back all the code you need. You can maintain that code any way you want.
The offer is on this link on our blog.
https://www.instantbi.com/2014/12/16/get-the-experts-to-build-your-data-warehouse-for-you/
In short? I am now speaking up again in public because I am so tired of seeing so many people talking about “50% of data warehouse projects fail” in the face of all the time and effort I have put in to provide our industry with the tools to GUARANTEE SUCCESS. I am speaking up again because I am willing to offer my services to any client who would like their project to be successful at very modest rates as long as I work out of my home office.
If you want someone else to deliver your project on your site? Then I would be happy to offer my services as a reviewer of your projects. I want to see if there is some way to get people to listen long enough to stop all these failed projects. All ideas on how to do that are most welcome!
But Ladies and Gentlemen. Please, please, please. Let’s stop it with the failed data warehouse projects, ok? We have ALL got something of a bad name to one extent or other with so many projects failing.
Let us finally collaborate and co-operate in our industry to deliver more reliable results to our clients. Sure, we might be competitors in terms of winning individual projects. But we should be collaborative and co-operative in building the tools and setting the standard for our industry. I am no longer going to be travelling so it is not like I am going to take away your clients from you if you use our software. Ok?
Think of it like this analogy. You don’t see architects and engineers having 50% of their buildings fall down. Architects and engineers co-operate and collaborate to set standards for building buildings. They co-operate and collaborate in the invention and creation of tools and techniques to build their buildings. Then they compete on their merits for each contract against an industry standard that people can rely on. In countries where this sort of maturity has not been achieved in their building industry some buildings do fall down!
In centuries passed bridge builders were required to stand under the bridge as carts loaded with stones were placed on the bridge to stress test it. So the bridge builder had to put his life on the line to guarantee his work.
Who in our industry would “bet their life” on their data warehousing project being successful? No one, right? If the Data Warehouse Architect had to “stand under the bridge” when his/her data warehouse was stress tested he/she might just take a bit more care and might just use the best tools for the job, don’t you think?
Now. I have done everything in my power to make the construction of a Data Warehouse as reliable as possible. I have offered the tools to do so to more than 6,000 people at a price that is free for all intents and purposes. It is very frustrating to me that our industry has a bad name despite all my efforts to the contrary for the last 23 years. We have, indeed, snatched defeat from the jaws of victory on many projects. We could have done so much better than we are doing today.
So I am offering my time doing the long promised “tele-commuting” for any client who would like to make ABSOLUTELY CERTAIN that THEIR data warehouse project does not fail.
All those people who read this and decide “I don’t need Peters advice” and then your data warehouse project fails? Just remember that for a long time men like me tried to help you and you turned down our help and ran failed projects time and time again.
To make sure that everyone knows that SeETL and BI4ALL are available and guarantee success when used properly? I am going to speak up about the failure rates and create articles about how to make projects successful with SeETL and BI4ALL.
All those who want me to address specific topics on the blog? You can make suggestions in the comments or send me an email to peter.nolan@instantbi.com.
Ladies and Gentlemen. In summary?
Let us finally co-operate and collaborate to reduce this embarrassing failure rate for our industry. Let us not allow the charlatans and liars to present the “latest shiny bauble” to business managers who do not know any better only for their projects to fail and have us all painted with the same brush, hmm?
If we do not do this? We will be having the same conversation again in 10 years time. I would really prefer that we did not go another 10 years with 50% failure rates in our Industry.
I am hoping that a few of the 6,000+ people on my mailing list feel the same.
If you got this far? Thank you very much for listening and or reading.
Best Regards
Peter
