Some have said Doug can code on pure metal; that he almost didn’t need an operating system. That isn’t quite true, but I have found him to be completely fearless. A year or so after starting system testing, Doug called me up and said he had added a new feature to SAFR, the ability to read DB2 data rather than just sequential files. He asked me to create for him a DB2 table with some data, define an LR and view, and run it against the table. He said it would probably abend the first time it was run; when it did I should call him back.
Consistent with his unassuming manner, I always found Doug willing to take the time to teach anyone that wanted to listen how to do what he did. When I called to say the DB2 test had abended, he asked me where he could locate the system output. As I told him, he found it, and suggested I also follow him into that portion of the output called the system dump or sys dump. This output contained a snapshot of the memory – the white board – at the time the abend happened. The dump can be very large; hundreds of thousands of pages in some cases if it were actually printed. Doug told me once they are only large if you print them, which no one does. Rather they are searched on line, like a large document.
Doug looked at something and told me to find a particular value. I did. He began directing me to other values in the dump, which is composed of only numbers and the characters “A” through “F”, and telling me what was in memory at this particular location or another one. I remember something like, “And see that Fox 2 there? That is the first byte of the directory structure for this block of the VSAM dataset. You see, on the mainframe DB2 stores the data in VSAM datasets, and inside each of those datasets it puts in its own directory structures that tell it how to interpret its data. The directory structure says that in this block there are this many data records, see the 34 there? And they are stored in this kind of format, see the 9754 there?”
I realized that Doug was showing me the in-memory version of the VSAM files that sit at the bottom of DB2. I suspect there were a lot of application programmers who used DB2 in programs that didn’t know about the VSAM files used underneath it. Only people who really supported the tool would have given those things much thought. I asked, “Doug, have you written SQL?” the basic DB2 language for accessing data. He said something like, “No, not really.” I was dumbfounded. Doug didn’t go through any of the traditional routes to learning a tool. Instead he went right to the bottom of the tool, to read its underlying data structure picking it apart byte by byte. “Completely fearless”, I thought.
This next section begins by examining a system dump, not because I think you will need to do this. But it is important to be fearless, in a sense. It is important to have respect for problems, but not believe that something magical is happening somewhere that ordinary people can’t understand. There is nothing too extraordinary in all of this, and analyzing a dump can demonstrate that fact.
The remaining portions of this section explain the SAFR Scan engine at its core, building step upon step in complexity. It begins with a view which copies data, an extract only view, moves through lookups to sorting data and running the format phase. All of this is done without any parallelism. There is a small break in the narrative to allow you to catch your breath; then we examine parallel processing, piping and other advanced topics including exits and common key data buffering which brings together most of the concepts into one process.
By understanding what is at the heart of the Scan Engine, one can see application of these patterns in reporting applications. Automation entails making the machine do what people would have to do to achieve the same results. Understanding these steps, and how they add upon each other to create the desired outputs, this is where the rubber meets the road in creating a business event based reporting system.
Contents of this Part include: