The History of BI4ALL
What has now become BI4ALL started life in about 1997 as the Vertical Packaged Solutions (VPS) project from “The Data Warehousing Network”. This was a small consulting company headed up by Sean Kelly. They had managed to get the worlds best data modeller in to the group. I never heard the story of quite how that happened. But it did. We will call him “Fred” in this blog and later blogs.
Fred solved the problem of storing historical data perfectly with no loss inside the dimensional modelling paradigm. This was something my mentor and I spent 4 years trying to do only to give up and conclude it was not possible. Well? It is possible and it was first done in 1997. It was a competitive advantage for Sean Kellys company so they were not about to tell anyone else what they were up to. Nor should they!
Sean and the team sold VPS to Sybase in about 1999 or 2000. Sean went in to Sybase as the VP for Business Intelligence for Europe, Middle East and Africa. The models were transferred to the Boston Lab and migrated to the Sybase PowerDesigner Tool. Then they were further upgraded as time went along and more projects were performed.
My first IWS job was a debt collector in Norway called Lindorff. This project went very well. As far as I am aware the data models are still there 14 years later. That is an amazing lifespan for a set of data models.
As I was working on this contract and learning the data models by implementing them it became apparent that these data models were the best in the world. They were just fantastic. So rather than talking publicly about “we should have some form of education to certify people to design and build data warehouses reliably” I changed my tune to “as an industry we should adopt the Sybase IWS models more widely to guarantee the success of our projects”.
I changed to this position because it became apparent to me that anyone who was trained on the Sybase IWS Models would be able to deliver a data warehousing project with a 100% success rate as long as the customer took the advice given. With the Sybase IWS models you would have to try really hard to have the project be a failure. These data models guaranteed success if they were implemented properly.
My second IWS Project was at a company called The North Jersey Media Group in New Jersey USA. The project had gotten in to difficulty because the project manager was not competent. I was asked to do a 2 week health check of the project and finished staying for 9 months. I played many roles on the project because it was in difficulties.
I was the project manager and had to work with the account manager to recover Sybases reputation and standing with our customer who was also the development partner for what would become the Sybase IWS Media Data Models. The project was a joint development project where the results of the project would be used as input to become the Media Data Models. So I also took on the role of designing the new Media Data Models because of my experience in publishing with Australian Consolidated Press, the largest publisher of magazines in Australia. Newspapers are very similar to weekly publishers. They just have shorter cycle times.
It was during this project that I got to work with the worlds best data modeller. He is older now and I have no agreement to publish his name. We are great friends. Let’s call him Fred here. I mentioned him above in this revised blog post.
Fred and I worked together for 2 weeks on the project and it was like watching the mind of God at work. I felt that by 2002 I was pretty much at the top of my game when it came to data models for Data Warehouses. Especially with one IWS Implementation under my belt and the second one now doing pretty well.
In those two weeks I got a real lesson in humility. I realised that if I were ever to get to be as good as Fred I was going to have to work harder at it. It was amazing the way he could come up with ideas on how to model things.
Now. Newspapers have a different office design to most companies. In newspapers they generally have very open plan offices so that everyone can hear what everyone else is saying. This is done to promote communication.
Fred and I were out in the middle of this open plan office and we had no place to hang pictures of our data models. The way that IWS projects were done was that we would plaster the walls with our data models so that we could see the models and understand them. Because the data models were so sophisticated it was a real challenge to remember them. And we were not getting any younger.
One night at dinner Fred said to me that he was having real problems remembering the data models without pictures. I admitted I was having the same problem. Then he said something that would change our lives. He said:
“What we need is a new idea so that we don’t have to remember what the data models look like.”
And I said:
“Sure, that would be great, but how could it ever be that we could design data models without needing to know what they looked like? How could we ever get rid of the need to remember them via diagrams?”
Very good questions indeed!
And then…as they say…a miracle occurred.
My subconscious brain had filed away the question and in the middle of the night it solved the problem while I was sleeping. Quite literally I woke up with the fully formed idea on the tip of my mind. In those days this happened a lot and so I kept a writing pad next to my bed to write down ideas that occurred to me while I was sleeping. I duly wrote down this idea and went back to sleep satisfied that I had just solved the problem of how to design data models without the need to remember what they looked like.
Next morning at breakfast I ran the idea passed Fred. He nodded and said “Well done Peter. That will work. I can’t imagine why I didn’t come up with that myself.”
I commented that it is the usual thing. When we have access to the resources we need, like wall space to display diagrams, we don’t look for a better way. It is only when we do not have what we need that we tend to invent something new. Necessity is the mother of invention is quite true.
This idea was then passed back to the Sybase lab and retrofitted into later releases of the data models. I also told my client in Norway about it and they quickly adopted the idea. Sybase did not publish the idea publicly as they wanted to use it as part of their competitive advantage.
This idea is now simply called “The Mesh”. In the 14 years I have been implementing models based on the design techniques of the Sybase IWS models this has been the one and only improvement I have been able to come up with that is a design technique improvement.
That, in and of itself, is simply amazing. No matter how many times I implement these models I have not come up with a situation that requires a new design technique. All the design techniques needed are there in the models as they were in 1997 before Sybase bought them. That is how good Fred is. What a genius.
In 2006 Sean Kelly managed to win a deal in Moscow for his new company Comhra. He asked me if I wanted to work on it as well as work on my EA contract. I discussed this with EA as I would be required to take a few trips to Moscow. This was all agreed. I worked on the new Comhra product for the Moscow client.
On one trip with Sean we discussed the problem of the old IWS Data Models and the fact that Sybase had now withdrawn them from marketing. It was frustrating to us that SAP had shelved such great models that would make projects successful. But it is not the first time a vendor killed a good product and it will not be the last.
To make a long story short Sean and I agreed that we would go back to the pre IWS models, the Vertical Packaged Solutions models, and build a new generation of data models that solved the problems we perceived with the IWS Data Models. As good as the IWS data models were there were things that we wanted to improve on to make implementing the data models even more cost effective than they were.
Because we knew exactly what we had in SeETL and a proposed new set of models we decided that it was worth the many months of unpaid development effort we were going to have to put in to create a new set of models. We took some short cuts. Our first set of models were simply views that defined the tables we wanted. They were not even in the SeETL workbook at that time. We decided that if we were able to get our first deal and get paid for the models we would migrate them to the SeETL workbook for the client during the project. It would be a lot of work so we didn’t want to do it unless we knew we had the deal.
It took us a long time to win our first new data model customer. We were greeted with the usual level of scepticism. Painted with the same brush as the charlatans. Very frustrating. The first customer we won was Carphone Warehouse in 2008. You can listen to a little about Carphone warehouse in this video.
The Talk Talk data warehouse was a smashing success. It was, up to that point, the smoothest and most productive data warehouse project we had ever undertaken. Even I was impressed at how well the project went despite the many setbacks we had that were outside of our control. Having done the Talk Talk project Sean and I were both convinced that we had “made it”. We were sure we now had the best solution available for delivering data warehouses with a 100% success rate at a price point 50% less than anyone else.
As you heard above we won another Telco in 2010 and we were in close with Netezza. We were also talking to the major Systems Integrators about adopting our solution in their consulting practices. As far as I know? No one has data models that match up to BI4ALL in the Telco space.
In retail there are other data models vendors. But since I have not had the benefit of learning much about their models I am in no position to comment.
The BI4ALL data models now represent the state of the art way to design, implement and maintain data models. There is nothing better out there. And coming from me that statement is to be taken seriously. We did not want to use a data modelling tool for BI4ALL because it slowed us down far too much.
We implemented a whole series of other improvements to the VPS and Sybase IWS models by way of design improvements and implementation improvements. Not design “techniques”. Just the techniques implemented in a different way to make the models faster and easier to implement. This cuts down on errors and rework.
Summary
So that is the history of SeETL and BI4ALL for those who like history lessons. Personally, I very much like hearing the history of things because it gives me perspective of where something came from and how it got to be where it got to be. This idea of people now with short attention spans and everything should be able to be explained in 5 minutes or less is a big part of our problem in our industry.
By not learning and understanding the history of products or tools people can be more easily misled into buying something that looks flashy at the front end but does not have the maturity of the development years across many customers.
SeETL, at EUR800 per person per year, has the wealth of experience of 20 years of Data Warehousing projects behind it. You can see just how crazy it is that my friend and colleagues are no buying it and using it on their projects while those projects have a 50% failure rate!
So. If you got to here? Thank you for listening or reading!
Best Regards
Peter
