Note: You can listen to the blog post on the video or read the blog post.
Hello and Welcome.
I am Esther.
I am Peters A I Assistant to create voice overs.
I will simply read Peters blog posts, so that you have a choice of reading the blog post, or listening to my voice.
Hello and welcome Gentlemen.
As I have recently announced on my channel.
I will be doing some opinion pieces and response videos.
In this blog post I want to talk about the term data leader that has been so casually thrown about of late.
Then I want to present a rebuttal to the term data leader with a term that has been around for at least two and a half thousand years and possibly longer.
So.
What is a data leader?
Personally? I have no clue what a data leader is.
I see all these people putting on their profiles that they are a data leader. I see all these people talking about other people and calling them data leaders. But I have no clue what a data leader is. I guess I have only been in this business for forty four years now and I am yet to learn of such things. Or, it could be that the term data leader is as bogus as the term data engineer that we have been hearing so much about lately.
I have already done a post on data engineering and you can go and read it on the button below if you want to.
I have read books by so called data leaders like Joe Reese and I still have no clue what a data leader is.
So.
My answer to the question, what is a data leader, is that I have no clue what that is.
Here is an idea I was introduced to as a small boy.
When I was five years old, I saw man land on the moon on the television in my kindergarten class.
As you can imagine all us little boys were so excited at this, and we all immediately decided we wanted to be astronauts instead of cowboys. Us little boys asked our teacher what this was all about and how this could be possible.
Our teacher told us that there were some very smart people who worked at a place called NASA, and they were the people who put the two men on the moon that day. I immediately decided I was going to go and work at NASA one day.
Just as a prelude to later comments. I did go and work at NASA in July 1990.
Just twenty one years after I sat there as a starry eyed little boy watching man walk on the moon for the first time?
I was at NASA in Houston with the task of trying to figure out if I could increase the productivity of the eight hundred I B M programmers working on the International Space Station project.
More on that later.
The idea that I was introduced to was the idea of a thought leader.
Someone who thinks of things that no one else around them has thought of.
Once that idea had been presented to me I saw it everywhere men were working.
From the time I was six I used to watch my uncle, who was a builder, work on the houses he was building.
When I was seven I was allowed onto the building sites themselves, to carry tools to and from the trucks of the builders on to the site. I would watch the builders work out a problem that was unexpected with a good new idea.
Of course, builders were coming up with good ideas on each building site because they did not know each other and had no way of easy direct communication.
But the idea was that the builders competed with each other for coming up with good ideas to building problems, and they called this being a “thought leader”.
This was because it took careful thinking to come up with the solutions to these problems.
My uncle taught me that no matter how many houses he had built there were always new problems coming up with each house because each house was unique. He taught me that to build a house your needed to be a good thinker and a good problem solver.
Many years later, in nineteen eighty six, I started work at I B M. I was immediately given access to forums for the I B M software labs where I could see men talking about the problems with I B M software and how to solve them.
One of the men I saw talking was a man named Ueli Wahli. Ueli Wahli was famous in I B M for inventing a way to create overhead slides that had been adopted world wide in I B M. He helped make I B M billions of dollars because his software was used to create slides to present to prospective and existing customers.
There was a term used for Ueli Wahli that I had never heard before. They called him a global thought leader. Kind of like my uncle figuring out problems on building sites but on a global level. We were all in awe of Ueli Wahli.
The joke used to be that we would cross ourselves as we said his name. We all dreamed that one day we would invent something as consequential as the slide software Ueli Wahli had invented. We all dreamed that one day young men like we were now would talk of us as we talked of Ueli Wahli.
And so, in nineteen eighty six I heard the term global thought leader for the first time and it was a title given to Ueli Wahli.
Of course, before Ueli Wahli there had been many thought leaders. I studied philosophy at university and so I had to read about Aristotle, Socrates, Plato and many others. These were men who were the global thought leaders of their time.
So.
I have no clue what a data leader is. But I certainly know what a global thought leader is.
It is a man who comes up with an idea that has never been thought of before that is useful and becomes widely adopted by those who understand the idea, or even just use the idea without understanding it.
Most of the people I see who call themselves data leaders are nothing of the sort. Indeed when self described data leaders talk or write I see nothing new. I have, literally, seen it all before.
To help you understand what it looks like to be a global thought leader I thought I would give you some examples from my career.
For a man to call himself a global thought leader he must have invented some new ideas that were widely adopted that proved to be of great value. For a man to call himself a data leader it seems he merely needs to reproduce that which is already widely known and pretend like he thought of it himself. And sometimes not even with that pretence.
In many cases men who call themselves data leaders don’t even pretend that they are talking about their own ideas. They just don’t reference the man who thought of the ideas they are talking about and let the reader presume some portion of the ideas are the ideas of the self described data leader.
In nineteen eighty nine I was the system architect for a four million dollar software development project. It was a release upgrade of the I B M internal pricing system for Asia Pacific and the Americas. The Americas was all American countries outside of the US.
I had twelve men working for me and source code control was a problem in the I B M world. On my project this problem was a particularly serious concern because of the nature of the application.
We had a source code control product under internal development so I decided to get a copy and try it out. The technical term for it was that it sucked. To link our main driver program took over an hour. I talked to the lab and asked them if they knew how to fix this and no one had any clue. I asked on the forums and no one had any clue.
Simply put, the product only supported object linking and not never call linking. And no one knew how to make it support never call linking. So one weekend I went into the office with the idea that I wanted to solve the problem of never call linking. And I did. I wrote about five thousand lines of assembler code that weekend to make it work and I reported on it on the Monday. I then improved the code over the next couple of weeks and sent it all back to the lab.
They were most grateful for my ideas. The introduction of never call linking and the way in which I solved that problem made the product a viable product. I asked the lab what their plans were to support all the other I B M source code objects.
In the I B M mainframe world we have a lot of types of code that are custom languages for custom objects. If you don’t know about them there is no point explaining them. But everyone who has worked on I B M mainframes knows what they are. They are M F S and D B D and P S B.
The lab told me they had no plans on supporting these things any time soon because they were having enough problems just trying to support the base languages. For this product to be useful on my project I would have to write the code for all the languages that my project used. So over the next month I worked long nights and weekends to write the assemble code needed to support all the language types I needed. I then sent it all back to the lab who thanked me profusely for helping them.
The net result was that I saved one full time headcount on my project and was able to deploy him to doing more useful work when the source code control system was implemented for our project.
The punchline to that story is that no less than Ueli Wahli himself invited me to the Santa Theresa Lab near silicon valley to come and be one of three co-authors on the definitive book on I B M Source code control. So just four years after I first heard the name Ueli Wahli I was working for the man at his invitation.
That is what Global Thought Leadership looks like. You write some new software that is adopted widely. In this case my code was deployed onto every I B M mainframe in the world that ran the I B M Structured Programming Facility. My code managed the code being deployed into production on tens of thousands of I B M mainframes.
One day, while I was working in the conference room at Uelis lab, the phone in the conference room rang. This was very unusual as no one ever called that number. Why would they? We didn’t give the number to anyone.
So I picked up the phone wondering who it was. I said hello this is Peter Nolan speaking, who is this? It was an I B M friend of mine I had met in Sydney. She worked at our largest I B M customer in Australia, a bank called Westpac. Her name was Carol and she was one of the top people in the world in I B M in the area of software development. She had been sent to Westpac to help improve the productivity of the software developers at Westpac.
Her job was to be sent to the largest I B M customers in the world and do what she could to help them be more productive in software development to make better use of our I B M hardware. She had come to our I B M Australia development lab and given lectures to us on the latest ideas, many of which we adopted.
So, I was very confused. I told Carol it was lovely to hear from her and asked how did she get this number. She said Ueli had given it to her. I asked her to what did I owe the honour of her phone call. She said she had been relocated to Houston Texas and was now an advisor to the eight hundred IBM programmers working on the international space station project.
I was very impressed. You don’t get to be an advisor to an I B M team at NASA for no reason. She then went on to explain that they had many serious issues she was trying to help solve on the project. One of those very serious issues was the source code control. She explained that the space station was going to run on intel microprocessors and all the code had to be written in C or assembler for the microprocessors.
And, of course, source code control of assembler and C targeted for microprocessors was not well defined in our source code control product. That portion of the product was still in alpha testing and that was part of what I was working on in the book. We were literally going back and forth with the lab about how to handle microprocessor developed and targeted code.
Carol asked me if I had any ideas on what they might be able to do. We went back and forth for about two weeks on what their problems were. In the end I said that I felt there was a very good chance that if I could spend a couple of days working in the lab with her and the team we could solve some of these problems.
And so, in the week after the July fourth weekend for nineteen ninety I flew over to Houston to work with Carol on these problems. I only had three days because of other commitments. I actually did this work during annual leave I had planned with my wife Jennifer. So Jennifer came in tow to Houston. We got in late Sunday night and I left the hotel at eight o’clock Monday morning and I didn’t see Jennifer awake again until I got to the hotel about ten in the evening on the Wednesday night.
We worked each day until two or three in the morning and then we were all back in the office at nine to start again. On the final day we felt that we had enough to go on by about six in the evening and we went out to a private dinner with the team to have a small celebration.
The code I designed and partly wrote later was used to manage the source code being deployed on the International Space Station. My code was now not only used world wide. My code had now gone out of this world and into space!
That is what a global thought leader looks like. It looks like someone from NASA calls you and asks you to please help them solve their problems for a project as consequential as the international space station.
Something that is very much lacking from the resumes of men who call themselves data leaders is any talk of any projects of similar nature. Indeed, there is a lack of reference to any projects at all.
My next big invention most people in the data warehousing space would be partly familiar with already.
In April nineteen ninety one I was assigned to implement Ralph Kimballs Data Interpretation System Software into my new I B M customer called the Mutal Life Company or M L C for short. I was a young twenty seven year old I B M systems engineer who had just finished his I B M Marketing School. I had to do the same school as the new entries.
I had never built a data warehouse before. A friend of mine was working on the I B M Information Warehouse project at the same lab in Santa Theresa. So I talked with her and talked to people on her team. They all said to go with third normal form and rely on D B 2 to answer questions. So I did. Those who are old enough to remember nineteen ninety know that third normal form was the gospel. That is just how you built database then.
The only problem was the M L C had a million customers and one point one million in force policies. And a party could play many roles on a policy so the table joining the customer table and policy table had six point four million rows in it.
Since virtually every question was about policy holdings by demographics pretty much every question had to scan all three tables. This took an hour elapsed time and cost four hundred Australian dollars per question. I went back to the lab to ask for any more help and they said this was normal. I asked all the worlds top D B 2 experts about how to improve performance and they all said upgrade the mainframe.
In the documentation for the Data Interpretation System there were references to dimensional models and I wrote to the I B M subsidiary Metaphor Computer Systems and asked for their help. They said that the design and creation of their dimensional models was confidential and would not be shared with I B M systems engineers. If our customer wanted dimensional models they would have to fly a team out to Australia to build it.
I was stuck. I could find no one to tell me how to solve the problem of long running queries on D B 2 to make this sale. The project sponsor had been very clear. They loved that they could answer these questions but four hundred dollars per answer and an hour wait time were not acceptable. We had to do better. Of course Teradata were waiting in the wings trying to sell their database computer.
What did I do? I did what I had been taught. I analysed the problem in order to figure out how to solve it. I went over all the queries that were being asked of the third normal form database.
I noticed that they tended to go from a very high summary level down to identifying just a few thousand customers.
I noticed that there were numeric columns that were common in queries such as premium amounts, premium coverage, number of policies, average policy values, forecast premiums and these sorts of things. Please remember my job as an I B M systems engineer meant I could not be told the purpose of these queries. All I had was the S Q L.
I also noticed that there were fields that were common in the queries that were text based or limited values based. These included the role of the person on the policy, age, sex, marital status, state, city, post code, product types, product groups, product sub types, policy age, policy begin dates rolled up to months, quarters, and years.
Slowly but surely I saw the patterns in the S Q L queries. There were textual fields retrieved in a hierarchy and there were numeric fields retrieved by summarisation. This was the main pattern of the queries. So I invented an idea I called multi level summary tables.
The main query table would contain the merger of all the commonly queried data in to one table. There would be one table that was separate that was six point four million rows and it would contain all the details. Then there would be another table that would contain varying levels of summaries of the detailed table. This table would be partitioned and each level of summary would be in it’s own table space. And the first field of the table would be in integer which would define the summary level. The summary levels would be documented against these integers so that the business users knew what integer to put into the query to ask a question.
Then D B 2 would only scan the tablespace for the partition that the query was against. We put the partitions into cylinders on the thirty three eighty disk drives to reduce head movement during the scan of the partition. So basically every query would scan the partition for the level of data that the query was at.
The results were amazing at the time. It was compared to black magic. Many queries that had taken an hour responded in one second or less. Most queries responded in under three seconds. More than ninety nine percent of queries responded in under ten seconds. A query against the detailed table ran out to about 2 minutes to scan all six point four million rows.
And where the query needed data that was not in this merged table you were still looking at an hour and four hundred dollars. Over time we moved more and more fields into this summary table and reduce the hour long queries to be rare.
Of course, everyone in the data warehousing area today knows this as one big table. The difference to one big table is that this is a multi level one big table. The other difference is that I am talking about nineteen ninety one.
Now the caveat here is that Ralph Kimball had invented multi level summary tables in dimensional models independently of me and before me. But since I did not have access to how that was done I had to go for one big table.
In nineteen ninety two I B M got permission from M L C for me to do a speaking tour on what I had done. The actual slide deck had to be approved by the M L C. What I was allowed to say also had to be approved by the M L C.
Over the next 8 years my idea of the multi level summary table was widely implemented in Australia.
That is what global thought leadership looks like. Those are the stories global thought leaders can tell about their past efforts.
In September nineteen ninety three I attended the global Metaphor Users Conference in San Francisco. It was at this conference that I met Bill Inmon. I met a lot of people as there were more than four hundred people at the conference. I met the man who Ralph Kimball had left behind to be responsible for all the data models that Metaphor built.
I was again told, this time by the Metaphor C E O Cathy Selleck, that it was her decision that no I B M Systems Engineers would be trained how to build dimensional models. If my customer wanted one of those they would have to pay for one of her people to come to Sydney to design it. And in the end that is what we did.
In nineteen ninety four and five period the M L C ran a pure research project to figure out how to build dimensional data warehouses. We hired the man Ralph Kimball left in charge to come and help us design the data models. I can assure you that dimensional data models were new to us.
Then it took us nearly eighteen months to figure out how to write the cobol code to populate the data model this man designed. There were two fact tables and twelve dimension tables. And it took us eighteen months to figure out the cobol code to put data into them. It was that hard.
In nineteen ninety five I was the data warehouse architect for the I B M retail banking system being developed in Sydney Australia. In that development work I noticed the design patterns for dimensional models because it was my second dimensional model I had worked on. Knowing how hard it was to write the cobol code for loading a dimensional model I bought a cobol compiler and I wrote template cobol programs to load data into a dimensional model.
I then gave these templates to the programmers on the project to be copied and then updated by hand to reduce the development time.
In nineteen ninety six I was hired to launch the Asia Pacific Data Warehousing practice for Hitachi Data Systems. This was basically a defensive measure to keep H P and Sun out of Hitachi mainframe accounts. One of the men who had worked for me on the I B M project came and worked for me at Hitachi. He was the second best cobol programmer I ever met. Much better than me. I gave him some time to create the ability to write a cobol program that would take one of the col templates, apply changes defined in a file, and then produce the finished cobol program.
I did not write that program. I merely asked this man if he felt it was possible to write. And he wrote it in under 30 hours. That’s how good he was at writing cobol. I didn’t even know if the program was possible and he was able to write it in 30 hours.
Because we had that tool kit we were able to offer fixed price dimensional data warehouses for three hundred thousand Australian dollars in Australia and U S dollars in Asia. This extra money was for travel and accommodation.
So as a global thought leader? I wrote the worlds most productive E T L tool in nineteen ninety five and got it upgraded in nineteen ninety six.
I then migrated that E T L tool to C plus plus in 2002 at the suggestion of Ralph Kimball who had become a good professional friend of mine by then.
To give you an idea of productivity levels of my E T L software. That version that was able to use a data dictionary and template programs to generate the final cobol program to be used? That allowed us to reach a mapping rate of one thousand fields per two hundred and twenty hour work month. We worked long hours.
That rate of one thousand fields per work month mapping remained unchanged from nineteen ninety seven until two thousand and seventeen. For twenty years one thousand fields mapped per work month was the fastest in the world and we saw no reason to try and speed up.
In two thousand and seventeen I invented a new way to get to six thousand to eight thousand fields mapped per work month. You can get that software for free from my web sites.
In twenty twenty three that rate went up again to twelve thousand to fifteen thousand fields mapped from a source system to a target data warehouse in one work month. I have had very long days where I have mapped over one thousand fields in a day. An amount of work I used to get paid twenty five thousand dollars for I can now do in one day.
That is one reason I am so sceptical of the use of A I in E T L development. How is A I going to go any faster than fifteen thousand fields mapped in one work month?
And even if it could go faster? What is the return on investment? It can not be very much because there is not that much cost to remove any more.
Lastly I would like to add the link to one of the ideas I am most proud of inventing. This idea helped Sybase sell a lot of data warehouse data models. It also helped me sell two data warehouse data models for one hundred thousand euros per copy too.
You can read all about this idea on the button below.
In summary?
A global thought leader is a man who can point you to ideas he has invented that have been widely accepted and used.
Whether than be source code control. E T L software. Data Model improvements, or anything else.
The main criteria for measuring whether a man is a global thought leader is that he has invented something that no one ever invented before and that invention was widely adopted.
Some global thought leaders only ever invent one thing.
Some global thought leaders invent multiple things that are then widely used.
The vast majority of men never invent anything that is widely used. That is the normal situation for men.
So.
I would encourage men in the data warehousing area to strive to become a global thought leader in their own right one day.
The path to get there is to learn from already established global thought leaders so that you know as much as is possible about data warehousing. And from there? Have a crack at solving some of the unsolved problems that currently exist in the data warehousing area. There are not many of them any more, but there are some.
And with that?
I hope you found this blog post interesting and informative.
Thank you very much for your time and attention.
I really appreciate that.
Best Regards.
Esther.
Peters A I Assistant.
Post script.
If you like my blog posts and would like to help me replace some of the five million dollars it has cost me to save men’s lives and defend the rights of men and boys?
Please buy me a coffee on my buy me a coffee link.
As at the time of writing I have about two hundred and thirty thousand dollars in debt for being willing to risk my life to defend those less able to defend themselves.
Thank you in advance for those of you who buy me a coffee.
It is much appreciated.
