Chapter 47. "Look At It Go"

Although the logic table codes and some of the features above have changed, most of the above I learned during the years I lived in Sacramento. At the conclusion of that time, I had a thorough understanding of how to apply SAFR to problems. I had concluded the pharmaceutical data warehouse, the cookie manufacturer, the computer chip manufacture and a year of work at the insurance company. I had been on the testing, technical support, project, sales, and documentation teams, and was starting to prove that I could contribute to the architecture team.

Yet the constant travel, the demands of the computer chip manufacturing job even though it was in town, three small children at home, and questions about the real prospects of the tool becoming more widely known all combined to make me question if I was in the right job. After the chip manufacturing project was over, I returned to the insurance company. There, I had chances to work with Doug in the same location on a daily basis. I shared with him quite openly these questions on a two hour ride to the airport in March 1998.

The response I received from Doug that day was very consistent with the message I would receive over the next few years whenever I went to him for advice. Whenever I would ask something like, "What do you think I should do," I almost always received the answer, "It depends on what you want." I know I wasn't the only one who had this experience.

It wasn't that he left me feeling like I hadn't received good instruction at the end of the conversation. It was simply that he was very careful in steering in any one direction. In a sense, he often helped me uncover what I wanted by simply asking me questions and reacting to my responses. In this conversation, and another two months later, he perceived the fatigue and was sympathetic to the causes. He didn't have the answers for how to solve it, but expressed confidence that he knew I would find the right answers no matter what those answers ended up being. I always left feeling that his friendship wasn't dependent in any way on what I chose to do.¹

That summer I also had a chance to sit down with Rick for a couple of hours and talk, and I expressed the similar feelings to him. In a similar way, Rick was very understanding and supportive of whatever decision I wanted to make. I was very fortunate to have two mentors with such concern for me as a person; it allowed me to navigate some difficult personal waters over the next couple of years.

A few weeks later I worked at home, and I decided to simply use Excel macros to mock up some of my own ideas. I did so over about two days. I shared them with Doug. I was a bit surprised at how positive his reaction was to it. I don't believe it was because he wanted me to feel good, but rather he thought they were very interesting and he was impressed how quickly something could be put together. I think that experience might have been misleading about what the effort involved in building such a user interface would be. He told me years later that he had been warned by another consultant that it was a massive effort, and it turned out to take years to complete.

A few weeks later, he sent me out to Philadelphia to meet Troy Deck, owner of Wingspan Technologies. One of the team members in Dallas had suggested Troy to Doug as someone who was in the business of developing the kinds of user interface we were thinking of. I enjoyed the creativity.

This work went on to become part of a large investment bank SAFR project that created the UNIX version of SAFR, with the new user interface. Having put down this foundation, I concentrated on innovating with the scan engine.

My wife agreed to move again so I could eliminate the 8 hour one-way commute. This meant I could better balance the demands of the project with being with my family. For the next few years, I helped create SAFR processes which did what we thought were pretty incredible things. It stretched the machine and software in some pretty amazing ways.

To get around this, I took the user exit load module and copied it two other times, and gave each a unique name. I then went to the logic table and edited it, causing less than 30,000 calls to each one of the newly copied load modules. I kicked off the extract program again. It ran to completion. Because the event file only had 5,000 records in it, it took only a moment to complete. I remember Doug commenting, "I can't believe it worked." To think that the arrays, length of pointers, and other elements of the program had allowed that many views to be generated amazed him.

I looked at the GVBMR95 control report and noted that the logic table of 3 million rows had caused the program to generate 3 megabytes of machine code. I remember going to the insurance company production load module library, where all written programs over all the years for all mainframes in the company were compiled. The total library size of all those programs was less than 3 megabytes. We had generated a lot of machine code.²

I had set up the test, and for some reason Doug and I were back in our individual rooms at the hotels late in the evening when it was time to run it. That might have been because the machine utilization was very low so we wouldn't be impacted by other processes on it. Again, this was before cell phones and we might have been in different hotels so we had to dial into the system on the hotel phone line and talk to each other with 115 character messages using the TSO "send" command a little like using instance messaging. Doug could watch my job run at the same time I did. It showed us things like how much CPU time it was taking, how many IOs had been requested, etc. I remember the CPU utilization was shown as a set of asterisks in a row, each asterisk representing the processes was using 10% of the machine capacity or something. The machine was a fairly powerful machine for the time, with 8 or 10 processors in it.

I kicked off the job and went to watch it process. I suspect I sent Doug a message saying, "Job nnnn is now running." GVBMR95 starts pretty slowly as it loads the logic table, the VDP, and the reference files into memory and generates the machine code. I remember Doug once watching with me and telling me when it was generating the machine code because all the IO stops for a few seconds as it makes multiple passes through the logic table that was in memory. It then began parallel processing, where multiple event files are read in parallel all at the same time. Doing so means that multiple CPUs can be used as well.

Parallel processing was not a very common thing at the time. Typically non-parallel programs, like COBOL, might use 5 to 10% of the CPU at any one time. As the GVBMR95 kicked in, the percentage of the CPU started to climb, 30, 60, 80%. These weren't single spikes in processing, but sustained rates for a number of minutes. As it bounced between 80 and 90% and periodically spiked at nearly 100%, I got a message from Doug saying, "Look at it go!!!!!!!!!!!" For all of his experience, I am not sure Doug had ever seen a mainframe utilized like that by one process. GVBMR95 was nearly consuming an entire mainframe for perhaps 10 to 15 minutes.

Figure 128. Southern Pacific Engine 4294

When I was living in Sacramento a couple of times during lunch I went over to the California State Railroad Museum near the office. The last exhibit is one of the largest steam engines ever built. It was a cab forward Southern Pacific engine 4294⁴ built in the late 1800's. It was built to haul the trains over the incredibly steep Sierra Nevada mountains on the cross-continent journey. The cab forward design was created because it passed through very long wooden snow sheds, more like tunnels, built to keep the snow off the tracks. With the cab at the back of the engine, the smoke and ash from the boiler would fill the cab and nearly choke the engineers. The solution was to put the cab on the front of the engine, which must have given the engineers very dramatic views through the mountains.

It is a massive engine, and for its time was the height of complexity and power. As I sat and looked at it, I would think of those engineers, with their hands on the controls and the wind rushing through their hair as they controlled this combination of thousands of parts all working in harmony in rapid succession hour after hour after hour.

Through those years, at times I was privileged to be the engineer at the controls of this incredibly powerful data processing engine Doug had constructed. As I have contemplated that night and thought of the incredible harmony of billions of machine instructions like those engine parts, independent parallel processes utilizing an entire mainframe over a sustained period of time like the drive wheels of the engine, performing millions and billions of joins similar to pistons firing, and mile after mile of event data in front and extracted records in the rear, with a hint of virtual wind as they passed by, I am grateful to Doug, Rick, the team and my family for allowing me to experience it.