Note: You can listen to the blog post on the video or read the blog post.
Hello and Welcome.
I am Esther.
I am Peters A I Assistant to create voice overs.
I will simply read Peters blog posts, so that you have a choice of reading the blog post, or listening to my voice.
Hello and welcome Gentlemen.
Firstly please let me say this.
This blog post is very long.
So any errors in the translation of text to voice are mine.
I have run the blog post through the translation process and checked it.
But I could easily have missed one or two words or phrases.
So I apologise in advance for any translation errors.
Now.
On with the blog post.
As I have recently announced on my channel.
I will be doing some opinion pieces and response videos.
In this blog post I wanted to talk about an excellent blog post my friend Bill Inmon recently released.
I wanted to use it to talk about the history of data modelling in the data warehousing area.
Firstly, I highly recommend you read Bill’s blog post.
You can read it on the button below.
Now.
I wanted to talk about the history of data warehousing data models from my experience.
By nineteen eighty nine I was the youngest system architect in I B M Australia.
As part of this I decided to use the newly released source code and library management product that I will call S C L M for short.
Also at that time the I B M Toronto development laboratory had a product under development called the Query Business Information System.
In May and June nineteen ninety I was selected as one of the youngest people ever to go to the I B M Santa Teresa Laboratory to write what is called a red book on S C L M.
Much to my surprise I B M had a development project under way where the eventual name of the product would become the I B M Information Warehouse.
This development project was using Q B I S as one of it’s major inputs and the project manager for Q B I S was on assignment to the Santa Teresa Laboratory at the same time. Her name was Christine.
Christine and I met at one of the many social events for assignees to the laboratory and she explained what her team was doing on both Q B I S and the Information Warehouse project.
The Information Warehouse was to be a third normal form data base as per all standard knowledge at that time.
I then followed the development of the I B M Information warehouse from mid nineteen ninety onwards.
In March nineteen ninety one I was assigned to my first I B M customer which was the Mutual Life Company, or M L C for short.
A few weeks later I was attending a meeting where one of my old friends from the I T department was demonstrating the Data Interpretation System from Metaphor Computer Systems. This was the company that Ralph Kimball cofounded in nineteen eighty two.
I B M had bought shares in it in nineteen eighty nine and decided to buy the rest of the shares in late nineteen ninety. So by January nineteen ninety one Metaphor Computer Systems was one hundred percent owned by I B M.
Ralph Kimball had left Metaphor Computer Systems in nineteen eighty nine.
He has never publicly criticized I B M but I personally just got the feeling that he did not like how I B M managed it’s investment and he decided to move on. I do not know that for sure, that is just my guess, not even strong enough to be called an opinion.
As it happened I was given ninety days to prove that we could do with the Data Interpretation System, also called D I S, what we promised to our customer.
I had no idea how this thing worked and it was a very hard ninety days, especially given my wife had a baby in the middle of it.
I talked to all the people who were experts in our I B M Information warehouse about the modelling techniques to use. This included Christine among many others. Everyone said that third normal form was the way to go and that is what I did.
After ninety days we had a working third normal form data warehouse.
There were one million customers in the customer table.
There were one point one million policies in the policy table.
And there were six point four million rows in the policy to customer relationship table.
This table linked the various customers who could have a relationship to a policy.
Then the address details were linked to the customer records because any customer could have multiple addresses.
So we had four tables in the very first release of this trial data warehouse.
It was running on D B two on the development mainframe.
Remember this was nineteen ninety one and one C P U second cost forty Australian cents at the time.
That was the charge back fee to the department for computer use.
The problem we had was that because to do any question of any significance needed to scan all three large tables, and geography based queries needed to scan all four tables.
This took at least one thousand C P U seconds, took about an hour to run, and cost four hundred dollars.
The director of marketing told us that he loved the work we did but it was too expensive to buy.
He told us that the work was very promising and that if we could just get query times down to a few seconds he would certainly approve the purchase.
I went back to everyone who had experience in this area.
I got the same answer from everyone.
Third normal form was the way to go and if they could not afford that?
Then they could not afford it.
It was as simple as that.
I was very upset because I had lost time with my new baby for this project so I sat down with all the queries that had been written by the business users so far and read them all.
I had a pen and paper and I was taking notes and drawing pictures.
I noticed in the queries that people tended to go from a high level down to a lower level.
They would do a query for the whole country.
Then they would do a query by states.
Then they would get the top 20 post codes by state.
That sort of thing.
A lot of the fields they were getting back were common across queries.
They were mostly drilling down on the same numbers just by more detailed character fields.
With this in mind I independently invented what would now be called a multi level one big table.
Please note that we were using cobol and the source data was a monthly download image of production data on tapes.
So anything I did ran for hours and cost a fortune to run each time.
The answer I came up with was to merge all the data that we had into one big row and have all those rows on tapes.
In the end, including multiple addresses for many customers, we had something like eight million rows in one file on tapes.
I put in integer keys for the roles that customers played on a record and there were about twenty roles.
We then also put a key for the type of address that the address was.
Things like home address, business address, Post Office Box Address, Solicitors Address and some others.
The policy data was copied for each customer role and each customer address.
Then I selected out all the most critical information that people were querying on and all the most critical numerical information that people were querying on.
We also did a session where we asked the business analysts which other fields that had not been used so far were most likely to be used in the future.
Then I defined what levels we could create.
But I made it easy to make more levels because clearly we were going to need more levels in the future.
So we had zero as the highest level and then one for the next level and so on.
If I recall correctly D B two two point three had just come out and we had partitioning support.
So I partitioned the one big table on the level column.
And then, from the one set of tapes that contained ALL the data we had.
We read the tapes and created an output dataset for each level.
And each level was then sorted and summarised on tapes.
And right at the end the one big table was truncated and each tape with each level of data was loaded into the one big table using the D B two loader.
This processing ran for about thirty hours once we had been delivered our source tapes.
I don’t remember the total cost but I believe it was around twenty thousand Australian dollars to run it once because the I O and tape usage was also billed.
It was a massive amount of batch processing by comparison with anything else that happened in the company.
The beauty of this was that when the business analysts did high level queries only that disk that contained the level of the data being queried had to be scanned.
For those of you old enough to know what a thirty three eighty disk is I put the partitions in cylinders and that sped up the queries massively.
The four big tables were still there.
But this multi level one big table would be able to answer one hundred per cent of all questions asked to date and we hoped it would answer more than ninety five percent of future questions.
The performance was stunning.
High level queries were running in less than one second elapsed time.
It looked like black magic.
As you went down in the hierarchy the queries took longer if you scanned the whole partition.
But at the lower levels it was much more normal to have constraints and we soon learned what fields to put indexes on.
I still remember doing my first demonstration to their database technical people and they could not believe what they were seeing.
I mean men with ten more years in I T were asking me if this was real or some how staged.
It took me three months to do all this, July, August and September.
The customer then had the system to themselves in October and November and paid the bill before the end of year deadline.
They were so pleased with my hard work they actually took me to the best sea food restaurant in Sydney and bought me a Christmas lunch.
As per I B M policies I was not allowed to be told what they were doing with this query system.
But they did tell me they had already recovered the three hundred thousand dollars they were going to pay for it and expected to make more than a million dollars new profit in the new year.
I could not be told how.
So in nineteen ninety two I did presentations to all large I B M D B two installed accounts who wanted to hear from me.
One of them was the National Australia Bank who adopted my ideas which then enabled them to double their profits over the next five years.
I also spoke at the D B two users conference on a number of occasions to tell I B M customers about these findings.
In the later years of the nineties people would show me their multi level one big table models and talk about how great it was not knowing I was the man who invented it.
This technique became widely used in Australia.
Just for completeness the D I S Documentation contained examples of star schema models but I could not understand them let alone build one.
There were no sample databases or sample applications at that time.
In September nineteen ninety three I got the chance to go to the Metaphor Global Users Conference in San Francisco.
By then I had done the job of performance tuning on the Standard and Chartered Bank Metaphor implementation in Hong Kong.
So I had seen a star schema database up close and could see how a multi level star schema worked.
It was clearly superior to my one big table and so I wanted to be trained on how to build such a thing.
I met with the head of Professional Services at Metaphor and asked if I could some how be trained to design and implement star schema databases.
He said no. He said I had to talk to the C E O Cathey Selleck and get permission from her.
We were friends so I very briefly stopped her at the conference to ask for such permission.
Cathy said no, her policy was that information was not to be shared outside their development team.
So I went back to the Professional Services Manager and asked what were the chances of getting one of his guys to come out to Australia for a while to design such a star schema model for us.
He quoted some eye watering rates.
He said that if I wanted to learn how to do this then I would need to make a sale, have one of his guys come out to Australia to design it, and then learn from that.
I asked him to please introduce me to the guy most likely to come out if I could get a customer to fund it and he introduced me to the man that Ralph Kimball left in charge of the data modelling area. That man does not want to be a public figure so I won’t name him here.
I had my I T project manager from M L C with me at the conference and so I also introduced this man to him.
It was this I T Project Manager who would have to get the budget for the project if we were ever to get a dimensional data warehouse built.
And with the eye watering price of consulting it was going to have to be a very significant budget.
We would have to make the case that this dimensional modelling was so much better than what we already had versus extending what we already had to the other five companies in the group. It was going to be a difficult internal sale.
The other “magic moment” from that conference was this.
Coca Cola was the worlds largest user of D I S and the head of Coca Cola Business Intelligence was also the chairman of the global users group.
He had invited Bill Inmon to be the key note speaker for the conference.
This was despite the fact that almost every Metaphor customer had dimensional data models.
I was at the lunch table with this man and he told us all to make sure we had plenty of note paper and a pen for Bill Inmons presentation because he was going to share breakthrough thought leading ideas.
Coming from someone of that stature I made sure I got a front row seat.
Bill gave the most unbelievable presentation I had ever seen to that date.
He just used blank overhead slides and drew pictures and commented as he drew pictures.
He had a microphone to broadcast his voice because he barely looked up for the whole hour and he went at a million miles an hour.
My customer and I had no idea what he was talking about.
But when I turned around all three hundred people in the room were taking notes furiously and so I figured they knew more than me and so I should copy all the pictures and take notes and figure it out later.
After the presentation there was a line of people to introduce themselves to Bill so I got on line.
When it was my turn I thanked him for his presentation and admitted I did not understand it.
So I asked if he had written any books I could buy.
He told me I could buy his book called Building The Data Warehouse and thanked me for coming.
He was quite impressed that there were two Australians in the audience.
What Bill presented, of course, was his vision for an archival data store.
Something no one in the room had.
Bill was asked if his company had implemented any dimensional models.
He said they were experimenting with them but did not have one in production.
He said that the loading of the data was very difficult coding and they had really not cracked that problem yet.
Of course, his company were also not allowed the confidential information also denied to me.
So, as early as September nineteen ninety three, Bill Inmon was the key note speaker at the Metaphor Computer Systems global user conference.
He was invited by the head of Coca Cola Business Intelligence World Wide.
This is why in my own mind there has never been any conflict between what Bill Inmon was proposing and what Ralph Kimball was proposing.
They were proposing two different things for two different purposes.
Ralph at Metaphor proposed dimensional models to answer a limited set of very valuable questions mostly around marketing dollar spend.
Bill at Pine Cone Systems proposed a much more ambitious idea of archiving all the data in an enterprise.
Indeed, even after I had read building the data warehouse I did not understand what Bill was proposing.
It was so radical my brain could not accept Bill was proposing to store ALL the data in a large company in an archive for five to ten years.
So, the position by the end of nineteen ninety three was this.
Bill Inmon was very well aware of what dimensional models were, how they worked, what they provided and who was using them.
All the big customers of Metaphor had a copy of Bill’s Building the Data Warehouse book and knew what Bill was talking about.
All brought together by the head of Business Intelligence at Coca Cola who was then the world’s largest installed account for Metaphor.
Hardly a point of conflict or contention.
My customer and I then went back to M L C and he asked for money for pure research to learn how to build dimensional data models.
He was a very good internal salesman and managed to get a budget approved by the end of the year for the new year.
It took some time to get into the schedule of the man we had both met at the conference and so it was March nineteen ninety four before we could bring him to Australia to help us design a prototype dimensional model.
That dimensional model had two fact tables and about a dozen dimension tables.
It then took us another full year to figure out the cobol code that was necessary to incrementally update it from the tapes.
We got the first prototype version running in about May nineteen ninety five.
We then spent six months learning from this prototype.
As we were learning from the prototype we were building our business case to run a two year development project to roll out dimensional models for the six companies in the group of which M L C was one member.
I resigned from I B M in June nineteen ninety four and so I started at M L C as a consultant in July nineteen ninety four to work on this pure research project along with other responsibilities I carried over from my I B M days.
So, even me, being one of the top programmers in the country, took nearly a year to figure out how to write the cobol code to populate these two fact tables and twelve dimension tables.
That is how hard that was the first time around.
Also in nineteen ninety five I B M was working on the development of a retail banking system. This was called the Retail Banking System twenty twenty. Our joke was that was when the first release would be available.
I got a call from the System Architect who was an old pal of mine.
He said they wanted to build a data warehouse for the retail banking system and they wanted to compare the I B M Information Warehouse with the idea of dimensional models.
So they had me come in and present to them about why dimensional models were better than third normal form models.
Because it was I B M and the M L C computer department was now part of I B M Global Services they were also able to call my project manager and he could share confidential information with them.
In the end the decision was made that they would go with multi level dimensional models rather than the I B M Information Warehouse.
I can tell you that this caused a lot of political fall out. In the end the project manager of the data warehouse portion who I worked for was required to go to Dublin Ireland for a class on the up coming I B M Financial Services Data Models. They felt that after the class our management would change their minds.
The opposite happened. The work on the Financial Services Data Models was at such a poor level that this project manager came back convinced that management had made the right decision. I had given him a list of all the issues that I had faced with third normal form models and how multi level dimensional models addressed these issues. He presented that list to Barry Devlin who was heading up the development team and Barry was not even aware these were going to be issues, let alone did he have any answers to them.
And so, in nineteen ninety five I built the first ever end to end data warehouse from a large D B two operational source system using change data capture, a landing area, a staging area, and a dimensional model.
It all worked better than we had hoped. We had a lot of problems with the incremental updates but we were finally able to make it all work replicating reasonable banking loads.
Also, in December nineteen ninety five we got our business case approved to roll out a leading edge corporate data warehouse for the six company group that M L C was a part of. The budget just for software development was two million dollars for the two years and my share of that was three hundred thousand dollars over two years.
I would also work part time on the roll out of the data warehouse for the I B M Retail Banking System and provide them any support they needed.
Unfortunately, when I came back from Christmas Holidays in January nineteen ninety six an old I B M colleague of mine had unhooked my project and gotten it delayed in hopes of getting my role on the project. Not a very nice thing to do and I later ended his career at I B M.
I was offered an alternative role and I turned down the offer and said I will follow my instincts with data warehousing because I can see it is going to be a very lucrative career and since I had four children to feed a very lucrative career was a good idea to provide for my wife and children.
Not long after I B M corporate also cancelled the Retail Banking System project itself and the data warehouse we had designed and built also died having never been put into production at a customer. There was a competing banking system being developed in Europe and I B M decided to cancel the Retail Banking Twenty Twenty Project.
Also in nineteen ninety four I was going around telling people in Australia about Bill Inmon and his book Building the Data Warehouse. One of the many hundreds of people I told about it was a friend of mine named Steve Hitchman who ran his own small consulting company.
He contacted Bill and offered him the opportunity to come to Sydney to present a one day conference. However, the one day conference is not what Steve sold him on. Bill was well known to be a keen fisherman. So Steve offered him a days deep sea yellow fin tuna fishing off the coast of Sydney. We have some of the best Tuna fishing in the world.
So Bill accepted the offer to go Tuna fishing and to give a day long seminar was kind of the after thought.
The conference was to be held early in nineteen ninety five.
Just for the public record here.
I was on the boat that day and it is true that Bill caught the biggest fish of the day.
It was a fifty kilogram yellow fin tuna. It was the biggest yellow fin tuna I have ever seen with my own eyes.
It was Bills turn in the chair at the next catch.
When the tuna hit the line and Bill started trying to reel it in he struggled.
We all made fun of him being too weak to reel in the fish.
But when we saw the size of the tuna we all realised that Bill was doing a great job against a very big fish!
Indeed, I was at a conference in Boston in two thousand and one where Bill was speaking.
When he saw me he waved for me to come over and tell the guys there about the day we went tuna fishing.
The guys were asking me if I was for real and that Bills “big fish” story was actually true.
I told them I swore on my mothers grave I was on the boat and the fish weighed in at fifty kilograms.
I told them it was one and a half times the width of a Toyota four door sedan because that is what Steves car was. A family sized four door Toyota sedan because Steve had two children.
We put it into the boot of his car to take home and it was folded over quite a bit just to get it into the boot of his car.
Goodness knows how many of his neighbours were eating tuna for the next few nights.
The Sydney conference with Bill was massively over booked.
Steve and I had billed it as a “once in a lifetime opportunity to hear from the worlds foremost authority on data warehousing”.
There was standing room only for those who bought their tickets late.
From memory about 50 people had to stand and some people were kind enough to share their seats with those who were late buying their tickets.
Everyone in the room had read his book. We sold copies and told people to read his book before the conference so that he could present ideas far in advance of just what was in his book.
So the presentation was much more advanced than I had seen eighteen months earlier.
I remember sitting there and thinking “he is talking about archiving all the data in an organisation”.
I remember putting my hand up and asking Bill.
“Bill, I know this might sound strange, but you are talking about archiving all the data in large companies. I mean, we have enough trouble just processing all the transactions in large companies. We have development teams working hard to develop new systems to automate more transactions. The idea of also archiving every version of everything every day as well as all the transactions. Well? It’s just too much data. That’s just more data than anyone can even write the code to store let alone buy the disk to store it on.”
And I will never forget Bill said in reply.
“Peter, we are talking about volumes of data the human mind can not even comprehend.”
And it was at that time that I realised that Bill really meant to archive all the data including versions of all non transaction data daily.
Very valuable yes.
But in nineteen ninety five also very expensive.
Also, please remember, that at this time, February nineteen ninety five, no on in Australia had a dimensional data warehouse.
No one had an archival data store like Bill was talking about.
The most common data warehouse in Australia by nineteen ninety five was already my multi level one big table followed by the third normal form data warehouses that Teradata were selling.
Also, please remember, in February nineteen ninety five my work for the M L C was confidential.
No one knew I was working on pure research on a dimensional data warehouse.
The plan in February nineteen ninety five was that we would get our budget approved by the end of the year and start on a two year project to roll out a leading edge dimensional data warehouse for the group M L C was a part of.
This would give all six companies a very large competitive advantage and I would be barred from selling my skills to any competitor of theirs for at least two years. It was planned that this would be put into the contract for that project.
So the people in that conference had all heard about my multi level one big table idea.
Now they were hearing about Bills archive everything idea.
None of them had ever heard about dimensional models because that was not yet public knowledge.
As I said above, when I came back from my Christmas Holidays in early nineteen ninety six a former colleague of mine at I B M had gotten the project delayed. Meaning he had one of the senior management team put the project on hold hoping I would go and get another contract. Then he would take my spot on the project and it would go forward again. That was his plan.
The director of the I B M Finance Region was a man named Philip Bullock. He was my old boss. We knew each other well. He had even been to my home and met my wife and children. Quite unusually so.
I called Phil and asked him to please get the project reinstated and to tell my old colleague that he should not be doing such things.
Phil said that he was not going to bother to do that despite the fact it was a two year contract for me to provide for my family.
I told Phil that I wanted to be co-operative with I B M as much as possible and be available to work for I B M as much as possible. I told him I did not want to compete in the marketplace with I B M.
But if I B M was going to kick me out of a contract worth three hundred thousand dollars to my family? Then I would see that as a betrayal and I would then compete with I B M.
I said to Phil that as the director of Finance it was his decision and that if he let this stand I would compete with I B M in the future where I saw that of benefit to me.
He laughed and said.
“Peter, you are a nice guy. You are a very smart guy. But you are just one guy. You can’t compete with I B M. My decision stands. I will not make any effort to help you get your contract back. If you feel you want to compete with us in the marketplace at any time? You are welcome to do so.”
The joke of that comment is that three years later Phil was Managing Director of I B M Australia.
He hired a head hunter to get me to come back after I took the largest data warehousing deal of nineteen eighty eight off I B M at our telco Telstra.
It was a national embarrassment to I B M that they lost that deal to me at Price Waterhouse Coopers. Phil wanted me back and he had authorised the head hunter to make me a very big offer.
I told the head hunter that I B M was welcome to make me an offer for a contract but because they had treated me so badly I would never again be an I B M employee and I would never again represent I B M.
I told the head hunter to tell Phil that I don’t represent companies that operate unethically. Phil had my number. He could have called me directly. He would have gotten the same answer.
So in January nineteen ninety six I was out of a contract and needed to get another contract or another job as soon as possible. By this time I had written my first E T L software that I called “The Instant Data Warehouse”. I created my first web site and put all my marketing materials on my website.
And wouldn’t you know?
A few weeks later?
My phone rings.
It is a head hunter who wants to talk to me.
I won’t go into the discussion.
I will just say that I end up in his office very curious as to who has hired him.
The question being.
Who is looking for someone like me that is not I B M, Oracle or Teradata?
We get to the crux of the meeting and under non-disclosure rules he tells me his client.
It is Hitachi Data Systems.
I am shocked.
He says they are launching a series of Systems Integration Offerings around Sequent to ward off Sun and Hewlett Packard from their mainframe accounts.
He tells me the budgets to get these businesses launched are telephone numbers.
He tells me the travel schedule will be gruelling and no one who has the skills and experience to be able to do these jobs wants the jobs because of the travel schedule.
He tells me that if I am willing to do the travel, I have a very good shot at the Data Warehousing Practice Manager.
To cut the story short I do the interview and get the job.
I am to launch the Hitachi Data Systems Data Warehouse Practice for Asia Pacific.
And I have a very substantial budget to do it with.
The salary I was offered was twice my ending salary with I B M less than two years previously.
Phil Bullock eat your heart out!
Then we had to make the decision as to what strategy would we pursue at Hitachi.
Over this period the man who taught me how to design dimensional models had introduced me to Ralph Kimball.
Ralph and I had been exchanging emails and just getting to know each other.
Once he knew I was for real and independent he asked me to please review his up coming book.
So I reviewed his book and provided him with lots of feedback for improvements.
From just simple things like phrasing to much more complex things about models.
Ralph appreciated the many hours I spent reviewing his book.
Bill also did the forward for Ralphs first book because by then Bill was much better known than Ralph.
So, as I was considering what strategy we would set for Hitachi there were a number of factors at play.
We were to sell Oracle Data Warehouses on Sequent hardware to beat out Sun and Hewlett Packard selling into our mainframe accounts.
Also, we wanted to keep Teradata out of our mainframe accounts.
At that time all three were selling third normal form data warehouses.
Ralph was also selling Red Brick and was facing stiff competition from Oracles third normal form data warehouses.
The idea formed in my mind that we should go all in for multi level dimensional models and partner with Ralph to promote dimensional data models in Australia and Asia Pacific.
Ralph and I had an email exchange where I proposed we would buy one hundred copies of his book.
We would give one or two free copies to any prospect who took a one hour presentation from us.
We would promote multi level dimensional data models exclusively over the top of third normal form data models and even archival data models.
And with my Instant Data Warehouse software we could implement very broad and robust data models for a fixed price of three hundred thousand Australian dollars in services and know we are going to make a profit on the services.
More importantly we would get the sequent footprint, we would get the Hitachi Disk, and we would keep Teradata, Sun and Hewlett Packard out of the account.
The strategy was simple.
When you are the new entrant to a market space if you offer something similar to companies that are already well established then you will lose most of the time.
If you offer something radically different and you can show why what you have is radically different and why it is better?
Then you will win much more often.
After three months of this planning the decision was ratified by my management.
Hitachi Data Systems Asia Pacific excluding Japan was going to go all in with dimensional models.
We rebranded my software to be called the High Star Warehouse Toolkit.
We rebranded my methodology to be called the High Star Warehouse Methodology.
I created a suite of over 200 slides all branded this way to then go out on the sales trail.
The results were beyond all our wildest dreams.
In nineteen ninety seven I sold more than ten million U S dollars worth of deals.
I had been a failed salesman since nineteen ninety one.
And when I sold my first million dollar deal I was so happy.
In Australia, we knew about Bills work and I knew all about archival data models.
His company operated in Australia and good friends of mine ran it.
In fact we sold Bills Prism Warehouse Manager on my first million dollar deal to the Australian Customs Service.
We could not sell the High Star Warehouse Toolkit to the Australian Customs Service.
They wanted world renowned E T L software so we put his Prism Software into the deal.
I was not the first sale in the country but it was pretty early.
So my first million dollar deal in Australia in February nineteen ninety seven was to sell a dimensional data warehouse which would be populated by Bills Prism Solutions Software!
How about that?
Despite my massive success in making Data Warehousing sales at Hitachi there were serious political issues.
I will simply leave it at that.
I worked for a man named Chris Henderson and I really appreciated working for him.
He was an excellent manager and he fought hard to get these political issues sorted out.
I have only good things to say about Chris.
I resigned from Hitachi at the end of nineteen ninety seven.
The political issues killed a lot of the businesses being launched, not just the data warehousing business.
The man who had trained me on dimensional models had since moved to Price Waterhouse Coopers.
He had been asking me to join Price Waterhouse Coopers for two years but the senior partner in Australia was a man named Fred Balboni. He had a very bad reputation in the marketplace and I did not want to go and work for him.
Even so?
I called up the junior level partner who was responsible for Data Warehousing in Australia and New Zealand and said that I was resigning from Hitachi. His name was David Wilkins. I asked him if he wanted to talk to me. He said yes.
At the interview I told him that my issue with working for him was Fred Balboni and what I had heard about him. He admitted that what I said was true and promised to do all he could to keep Fred out of the picture. Obviously since Fred was Davids senior partner he could not be kept completely out of the picture but he promised to do the best he could.
Now there needs to be a bit more explanation here.
During nineteen ninety six the man who trained me was at Price Waterhouse Coopers. He was in the process of writing their methodology. He was running the data modelling area. He had one of the big Insurance companies in the U S as a prospect. He could see that they needed dimensional models.
However, because of his time at M L C and because of what he was hearing from this prospect it seemed obvious that an archival data store would also be very valuable. We went back and forth on this and some how we came up with the idea of doing both.
The proposal was to have an Operational Data Store that was third normal form.
Then an archival layer that was time variant.
Then a suite of dimensional models for the analysis and presentation layer.
Neither of us had heard of anyone else doing this so as far as we know we invented it.
This man put in the proposal and won the deal.
It took them about six months to get back early results and they were very positive.
So as he wrote the Price Waterhouse Coopers Methodology in nineteen ninety six and seven he standardised on the proposal that there be three modelling types for the three areas.
Operational data store to be third normal form.
Archival data store to be time variant.
The dimensional models to be implemented as per Ralph Kimballs ideas.
Of course, this was very expensive.
Price Waterhouse Coopers liked expensive projects but they were hard to sell in Australia.
So, by the time I joined Price Waterhouse Coopers in January nineteen ninety eight the standard proposal included both an archive and a suite of dimensional models co-existing in harmony.
Price Waterhouse Coopers were able to sell these very expensive projects in the United States.
But we were not able to sell them in Australia.
In Australia we gave prospects this option.
But we also gave them the option of just having the staging area feed the dimensional models.
We gave them the pricing of each option.
In each case I sold the prospect did not want to pay all that extra money for an archive layer when the proposal contained a staging area that contained all transactions and current images of all non transaction data like accounts and customers.
In each case I sold the prospect basically took the position that it was a lot of money to pay to have archives of non transactional data and they could live without that.
My two main sales in my year at Price Waterhouse Coopers were Telecom New Zealand and Telstra in Australia.
So, just for the public record, and just to be clear.
It was in nineteen ninety six that my mentor and I came up with the idea that we could deliver BOTH versions of modelling proposed by Bill and Ralph in one solution.
This worked wonderfully well, it was just very expensive.
As I mentioned, in December nineteen ninety eight I sold the Telstra Corporate Data Warehouse from Price Waterhouse Coopers. Even then we did not include the archive layer because of the time constraints on the project. The data warehouse had to go live on July first nineteen ninety nine. There was no way in the world that we could deliver an archive layer in that time frame.
Indeed, the plan was to relatively secretly use my E T L software for development and then to implement that code into stored procedures with developers so that my software did not go into production.
Because it was believed to be impossible to beat I B M in Telstra at that time there were only a couple of people assigned to the project. I had asked for a team building up to seventeen people and there was simply no will for Price Waterhouse Coopers to put people onto the project.
I pointed out to my partner that he had hired me to win data warehousing deals and I had won the biggest tender of the year and his resource manager could not deliver the people. So in the end I resigned. The whole year at Price Waterhouse Coopers was something of a joke. But at least I could tick it off.
As far as this post goes the most important point is that Price Waterhouse Coopers standardised on combining the best of both worlds when it came to Bill’s and Ralph’s ideas. And I know this was the case because I was there and had access to all their intellectual property. The cost of those projects were not sellable in Australia.
The Telstra deal made my name in Australia.
A lot of people took the position that my success at Hitachi was a “one off” and “just good luck”.
When I took the corporate data warehouse off I B M at Telstra everyone sat up and took notice. As I said above, Phil Bullock, then Managing Director of I B M Australia tried to get me to come back.
My next position was as the Professional Services Manager for Ardent Software Asia Pacific.
You may not know that at this time Ardent bought Bills company Prism Solutions.
As part of the deal Prism Shareholders were bought out and those who were employees had to move on.
So the two owners of the company that sold Bills Prism software in Australia were in the category of share holders who had to move on.
So in Australia Ardent was in the position that they needed to hire someone who was respected by the Prism Solutions employees in Australia so that they would remain with the company. The Ardent culture was very different to the Prism Culture and this was well understood.
Having sold Prism Solutions Australia software and worked closely with the team at Prism I was the only candidate that was recommended to take over that team and manage them going forward. We did very well and I did one hundred and thirty percent of my quota that year.
So in nineteen ninety nine and two thousand I was actually inside Ardent Software as the Professional Services Manager for the former Prism Solutions employees. I had access to the Prism Methodology and all Prism Intellectual Property.
I also had access to the Ardent intellectual property.
Most projects in Australia were delivering third normal form data models or dimensional models. Only a few were delivering archival models because of the cost.
Of course my mentor and I tried very hard to find a way to archive data in a dimensional model because we knew if we could find such a solution that we could reduce our development costs and win more deals. We tried to find a solution to this problem from nineteen ninety six until two thousand.
We could not find a solution and we presumed that one did not exist.
In late nineteen ninety nine Informix bought Ardent.
The managing director of Informix in Australia was known to be a poor business man as well as dishonest.
The Informix business in Australia had done very poorly.
As per Australian legislation I was constructively demoted meaning that I had to be paid the same salary but I was put back out in the field as a consultant.
I was by so far the highest paid data warehousing consultant in the country it was actually funny.
I was being paid like I was on quota with a dozen staff reporting to me.
At the end of two thousand all my former consultants and myself were made redundant.
My next role was to relocate to Dublin Ireland and to go and work for Sean Kelly.
Sean Kelly was known as the Bill Inmon of Europe.
Sean was the number one man in Europe.
He had a company called Sean Kelly and Associates.
They had data models called “The Vertical Packaged Solutions”.
Sean had sold his company to Sybase for an undisclosed amount in late nineteen ninety nine.
The data models were moved to Sybase Power Designer and sold as the Sybase Industry Warehouse Studio.
Sean was the best data warehousing salesman I ever had the pleasure to be in the room with.
He was a master of this fine art.
He was now the Vice President of Business Intelligence for Sybase Europe Middle East and Africa.
He was selling these things like hot cakes.
The database was Sybase I Q.
Sybase partnered with Business Objects and Informatica for front end and E T L.
However, the sales rep had the flexibility to allow the use of other front ends or E T L if the customer did not take their recommendations.
We were selling copies of the data models for one hundred and fifty thousand dollars.
We were selling copies of Sybase I Q for fifty thousand dollars per C P U.
The projects were generally about four hundred thousand dollars services per implementation back then.
I went to the data models training in February two thousand and one.
During the training a slide was put up for what was called the customer profile in the Sybase Industry Warehouse Studio data models.
It took me less than two seconds to realise this was what my mentor and I had been looking for over the last five years.
I put my hand up and asked the presenter if he realised this archives data in a dimensional model.
He said of course.
I asked him did he know who invented this idea.
He said he did.
And then I knew I was in the presence of the worlds best data modeler.
So I asked him where he lived and it turned out he was living in the same area we were looking for a house in Dublin.
Jennifer and the children were staying in a Bed and Breakfast and we had not yet found a house to rent.
In the end we got a house within walking distance of where he lived and we often had dinner at each others houses.
The Sybase Industry Warehouse Studio Profile meant that data could be perfectly archived inside the dimensional model.
The only problem was that the way in which it was updated was highly C P U intensive and they were only updated monthly.
In early two thousand and two I was doing a job in New Jersey in the United States.
This man was living back in Detroit by this time.
He said he was bored and asked if he could come over and help out on the project.
I jumped at the opportunity to work with him.
As we were working on the data models for the project it was like watching the mind of God at work.
He just came up with ideas that I could never come up with. Not in a million years.
It was just amazing to watch him draw models of what he thought we should do.
One of the things with the I W S Models is that they were so complex that we had to print the data models and sticky tape the models to the walls.
In this customer we had a conference room allocated to our team to work in.
We literally had all the walls of the conference room covered in data models.
When we needed to check how a data model worked we had to go and reed it on the wall.
One day our project manager came along and told us that we had to leave our room and work in the open plan office area. This was going to make life more difficult because the nature of our work required a lot of talking.
We were also told we had to take down all the data model prints.
So we put all the data model prints on the walls of the open plan area.
Then when we needed to review them and discuss them we would stand in front of them and discuss them.
Of course, this was a newspaper and many of the employees were women and they complained they felt “unsafe” having two men talking and standing behind them.
I can’t imagine why they would claim to feel “unsafe”.
I mean, what are two married men with children going to do to a woman in an open plan office with fifty witnesses?
Nothing.
But we had to take down our pictures of our data models.
This made life MUCH more difficult for the two of us because being able to see the models was absolutely necessary to do our work.
One night we were having dinner and my colleague was cursing strongly at how these women were interfering with our work with these baseless claims of feeling unsafe.
And at one stage he said to me.
“Peter, I am too old for this shit. I can’t possibly remember these models in my head. What we need is a method of designing data models where we don’t need to remember what they look like.”
I laughed at him and said that if he could figure that out he would be a millionaire soon afterwards because absolutely no one knows how to do that.
That night, in the middle of the night, while I was asleep, my brain figured out the answer.
I woke up and wrote it down in the notepad I used to keep by my bed for such instances.
At breakfast I explained my idea to my colleague and he smiled and simply said.
That is a good idea.
What was my idea?
What was the one and only improvement I have been able to make on the design techniques of the data models this man developed?
I added an integer key to the front of the profile table.
You can read about this in detail on the button below.
Now, what did adding an integer key on a profile table mean?
It meant that as well as being able to archive data in the dimensional model.
You were also able to link all tables to all tables that it should be linked to with just one extra integer key and one extra hop through the profile table.
At that time we coined the marketing term “everything is linked to everything”.
In two thousand and seven when Sean Kelly and I brought out another version of these models we changed the marketing term to “The Mesh”.
The customer profile at the time had twenty three keys on it.
Every time something changed about the customer the old record was closed and the new record was opened.
This meant that at any time you could construct the image of the customer record without storing a whole image of the customer record.
By putting an integer key on the front we could put that integer key on all places where the customer key is valid.
Then we not only have the link to the customer dimension, we also have the link to all the attributes of the customer.
Furthermore, we can go backwards in time and see how the customer looked at some point in the past compared to what they look like today. And we can do that at transaction level.
This was a HUGE innovation in the Sybase I W S models and it was kept very secret.
Later, when we were doing sales presentations for the models and we brought up profiles and what they did?
We would just get gasps from the more experienced data modelers in the room because they knew the benefits of this.
That one idea helped us sell a lot of I W S models.
The other benefit of “everything is linked to everything” is that the data modeler no longer has to remember what the data model looks like in terms of what tables are linked to what tables.
The data modeler can presume that if two tables are supposed to be linked they will be.
Or, if they are supposed to be linked and are not then they can be linked with almost no work at all.
This profile is now called an Association Table.
It is simply the archive of associations between dimension tables.
Thereby the image of the object that the association is archiving can be reconstructed at any point in time with just a relatively short record of integer keys as the actual archive.
Apart from solving the problem of archiving data perfectly in the dimensional model?
Apart from solving the problem of needing to remember what the data model looks like?
There is another even greater benefit that we developed.
Once I added this integer key I knew we could put it onto the transaction records.
But the standard E T L processing we were doing was so compute intensive we only updated profiles once a month.
To make them more useful we had to update them daily.
I also invented the processing approach to be able to update profiles daily.
I reduced the processing needs by more than one thousand times.
This meant that for the first time the I W S models could be used in the operational data store on transaction style databases like Oracle.
We could make the E T L fast enough to feed into Operational Data Stores.
Also, with my invention we could create daily archives of non transaction data.
This achieves the same results Bills archival models achieve.
And, of course, we can use them in the dimensional model layer as had always been the case.
This meant we had one data modelling style that could be used for operational data stores, archives, and dimensional models.
These models are perfectly adequate for all high transaction volume mass market businesses like retail, banking and telco.
There is no need for a data vault for these sorts of businesses because these models are perfectly adequate.
And so, from two thousand and two, since I added the integer key to the profile table, the argument about what modelling style to use for high transaction volume mass marketing businesses has been over.
What we had was the best in terms of return on investment.
There was no better way to design data models.
And we sold them just like that was true.
After Sybase was bought by S A P.
Sean Kelly and I got the agreement of S A P to create a new generation of data models so long as we did not compete with S A P Business Warehouse.
So we created what we knew best.
A new generation of telco models.
We sold them for one hundred thousand euros a copy.
We sold one to car phone warehouse in late two thousand and eight.
We sold a second one to Sky Talk in the U K in two thousand and ten.
We were teamed up with Netezza and we were talking to all the major telcos in Europe in two thousand and ten.
Then I was doxed and woke cancelled so we could not sell them any more.
Not long after Sean Kelly was diagnosed with cancer and he passed on in two thousand and twelve.
Interestingly.
Sean Kelly left behind a widow and a daughter.
If I had been able to continue my business of selling the data warehousing data models I would have given Seans widow fifty percent of the revenue because Sean had earned it.
If I was able to keep selling those data models for one hundred thousand euros per copy I would have very happily sent Seans widow checks for fifty thousand euros each time.
Because women woke cancelled me?
That meant that Sean Kellys widow and his daughter never got any money from the years and years of effort Sean had put into those data models.
Women are just like that.
When they attack a man they never stop to think of what women and girls might also be harmed in the attack on the man.
I tried selling my data models and E T L software for many years.
But all the slander about me and woke cancellation of me meant I could not sell them.
So now I am giving them away.
The sad state of our industry is that data warehousing and business intelligence is now considered something of a joke on the business side of the house.
The extremely high failure rates and the cost over-runs and expensive projects have pretty much destroyed the credibility of people in the data warehousing and business intelligence area.
Our entire industry was willing to fall on its sword and to destroy the reputation of almost all the men in it so as to pander to women.
That is a shame.
What are now my data models and my E T L software, in the hands of professional men, can be used to deliver data warehouses with one hundred percent success rates at much lower costs than any other approach.
But men decided they would not do this for their customers so as to pander to women.
Men decided they would deliver sub standard results to their customers and thereby make themselves a joke on the business side of the house.
As I said.
That’s a shame really.
The needs of the customer really should have come before pandering to the women in our profession.
Just my opinion.
I would like to just touch on a few more things which are about the future of data warehousing rather than the past.
You can read about these things on my blog.
Firstly.
I am giving away the source code for my E T L software for free at the two thousand and twelve level.
From nineteen ninety five to two thousand and seventeen we have been able to map approximately one thousand fields per two hundred and twenty hour work months.
So the estimate for the data model and E T L development has been one two hundred and twenty hour work month per one thousand source fields for twenty years.
This estimate mechanism proved very reliable for those twenty years.
In two thousand and seventeen I invented some new things that increased the mapping rate from one thousand fields per work month to between five and six thousand fields per work month.
You are welcome to have the copy of my E T L software that provides this five times improvement for free.
Of course, I also have a version that I sell where the mapping rate is now past twelve thousand fields per work month.
Indeed, I have had very long days where I have mapped over one thousand fields from a source system to a target data warehouse in one day.
Of course, at sixty two years of age I can’t keep up those sorts of hours any more.
But a young man can.
So I just wanted to let you know that there is now a piece of E T L software that can reasonably map twelve thousand fields per work month from a source system across to a target data model.
This does not mean you can map four thousand fields in two weeks because there is time that needs to be allowed for learning the data models and learning the data.
What it does mean is that for very large operational systems it is now possible to build dimensional data warehouses that store all the data from the large operational system.
That men are still using G U I based E T L development tools is an indictment on our industry.
The men who are doing that are massively over charging their customers and should be ashamed of doing so.
Secondly.
I have invented the manner in which very large data warehouses by number of fields will be developed in the future.
I have called these Mega Models.
You can read about them in more detail on my blog.
In short what is possible is this.
You can have many customers subscribe to your data warehousing service in the cloud.
You can have one landing area per customer.
You can have one set of E T L to take data from that landing area to the staging area.
There is only one staging area image for all customers.
Then from the staging area to the target data warehouse there is only one set of E T L needed.
And this can even be done on SQL Server Standard Edition.
There are two options available.
The data from many customers can co habit in the one set of target tables.
This means that customer data co-habits physical tables and that views are placed over the top of these tables so as to present out individual customer data using those views.
This is cheaper because if you have twenty customers you only have one set of tables, one set of indexes, and twenty sets of views.
However, if the customer insists that no other customer data is to co habit in any physical table?
Then it is now also possible to have one set of tables per customer and yet still only have one set of E T L to send data from the staging area to the data warehouse.
In these cases security can then be implemented at table level.
Security being implemented at table level is much more acceptable to most companies than security being implemented at view level.
Of course this means that if you have twenty customers in your cloud data warehouse you would have twenty copies of the tables and twenty copies of the indexes to maintain.
So this offering would have to be priced at a higher price point than the shared tables.
However, given the concerns of customers they may be willing to pay that higher price point.
Indeed, many companies would be willing to pay the extra cost to have their cloud data warehouse exclusively on the one virtual machine or even exclusively on premise.
Because my inventions have so vastly reduced the cost of data warehousing even small companies could afford to buy such a data warehouse running on SQL Server Standard Edition.
Thirdly.
My E T L Software has the functionality built into it that it can forward changed data to any O D B C target.
This means that if a customer wants to have their actual query data warehouse on a columnar database we can easily forward their data from this one large cloud data warehouse to the columnar database of their choice.
This will give more reliable billing for actual consumption of resources for their data warehouse query processing.
As I said.
You can read about Mega Models on my blog.
The future of data warehousing is Mega Models populated by at least my free E T L software.
And to get lower development and maintenance costs some companies will use my paid for E T L software.
And with that?
I hope you found this blog post interesting and informative.
Thank you very much for your time and attention.
I really appreciate that.
Best Regards.
Esther.
Peters A I Assistant.
