South Coast Air Quality Management District
The billing matrix was like a checker board.
The pieces of equipment represented in square A1 were the simplest technologically and lowest in annual fees. Squares to the right became more technical and the fees increased somewhat, Squares going down became more expensive. The lower right corners represented oil refinery equipment, sewage plants, and other complicated set-ups. This battleship grid allowed customers to know the emissions Fees for equipment before purchasing the equipment, The grid also reduced the number and duration of hearings, which are like court cases. The grid greatly reduced the complexity of the monthly billing program. Since every piece of equipment now had its own permit; the billing became much more of a summation of permits rather than a complex calculation about fuels, hours, temperatures, fans. hoods, and a seemingly endless list of factors. The old billing program needed to mimic too many scientific programs; the task was horrendously impractical. The enormous program defied timely maintenance. The isolation of complex calculations is a tactic for simplifying business systems. At the AQMD, the technology was removed from billing and from permitting. Removing the calculations from billing made the billing program much more maintainable; it also reduced the large number of legal appeals about bills. Removing the calculations from the permit process left the calculations Isolated in the research areas. |
The Monster 330,000 hour Backlog
The AQMD had a work tracking system for its many tasks and projects. Over the space of eight years, the total number of unstarted projects became embarrassing. In theory, a programmer does about 2,000 hours of work per year. Thus, 33 programmers could do about 66,000 hours per year. The backlog at the AQMD exceeded 330,000 hours. In other words, even with their 33 programmers, there was an estimated five years of backlog.
My job was to reduce the backlog. It was fun. ( See Work is Fun ) At first I created lists of requestors, lists of modules, and a list of ideas embodied in the backlog. I also interviewed all the programmers so that I could understand their current assignments and I could gain insight into their capabilities. Ron Olson was a good worker; Richard Green is fast but a bit sloppy. Bob Kuramada had good skills. My first tactic was to guess at some solutions. My second tactic was the look for duplicate requests. Thirdly, I hastened the completion of a few projects; that freed up a few programmers. Fourthly, I identified scads of stale requests. Over 100,000 hours were allocated to stale projects. At first my boss refused to agree that the requests were unneeded, but usually the requestor was gone and the replacement person did not want the project. Often, some horse trading further persuaded the new user to cancel the old project in favor of a newer one. Every week for about a year, I cancelled stale projects.
The AQMD had a policy of having a programmer spend 16 hours to review every request for any new report. There had been concern that a report might need special resources or might consume hours of CPU. Most reports were in QUIZ by COGNOS. Again my boss was hesitant to change the policy. He did agree that with some new reports, I could do the review. My reviews took a few minutes, and in the rare cases where more time was needed, I delegated as had been normal. Simultaneously, I wrote a COBOL program to accept inputs for online, real-time reporting. A simple experiment showed that the typical report about permits would run in a small faction of a minute. The supervisor in air chemistry, Hubert Wilson, agreed to be my teammate in this project. He began running his requests and requests for others thru the COBOL program; I asked him to avoid advertising the program. He was delighted to be able to run his requests immediately and receive his output in one of three ways:
The rest of the backlog included requests from many disciplines, such as, air chemistry, weather station maintenance, telemetry, legal statistics, car pool reports, gate control programs, and traffic patterns. Dealing with so many technologies was fun. Various techniques spurred the completion of these projects. After two years and some, the backlog was less than 50,000 hours and the number of contractors was about a dozen. Before the third year, my boss hired three supervisors to oversee the remaining projects. I became his assistant for special projects.
Also in the first year, I oversaw the implementation of a new checker board billing system where each permitted device was classified by model number into a grid of fees. The smallest annual fee was located in A1 and the largest in H10. Previously every device required individual calculations concerning hours, amperage, fuel, and lubricants. The new system saved thousands of hours of calculations and thousands of hour of legal hearings. The lady in billing was wonderfully precise and speedy. Her name might have been Jan.
The Smog Bank
A deadline loomed for implementation of the first Smog Bank to buy and sell smog credits. Ora McKewon designed the overview; Ron Olson and I streamlined the specs and wrote a stop-gap version which ran for several years. Russ Juranek, on this project and on many others, did excellent QA. After the initial implementation, Ora managed others who expanded the functionality of the Smog Bank.
The Expansion Project
The last significant project was an analysis and acceleration of the proposed new software system, called the Expansion Project. My boss gave me the first task of reading the documentation. The 22 notebooks comprised more the 20,000 pages of which most was fluff. Many of the notebooks contained what should have been numeric tables. Instead, the authors had written a sentence in English for each cell of some tables. For example, "When the row is zipcode 91702 and the pollutant in the column is sulfur dioxide, the value in the cell is five point three tons." A few macros in the VI editor reduced one table from eight hundred some pages to ten; it is was more readable.
Vehemently in private sessions to several key individuals, I tried to stop the Expansion Project.
My second task was to port the Ingres Data Base to run on Data General Computers; in essence, I copied the SPL and the C of the HP source code as a model for the server side, and two of us made it run on the DG hardware. The HP code implemented the SQL standard more efficiently than the IBM. The SPL roughly translates to C and stack operations. A contractor named John from England did QA and concentrated on the interactive routines. We finished in nine weeks.
Concurrently, I began to diagram the Expansion Project. It was missing several important processors, which moved data from input records to accumulation records. My boss said it was too late in the project to re-study and specify the processors; we would look bad. However, when the Board came to understand the situation, we looked like fools, and about thirty of us were let go. (My boss Cesar Diaz hired me several times in the next five years to work on other contracts elsewhere.)
On the Wednesday before the final boot, Venita Mathur asked me to meet her in her office. In January, per instructions from Venita Mathur, who became the new Director of IT, I reported serreptitiously to a barren desk on an uncompleted floor of the main tower. She and I had argued many times during the previous three years, but she trusted me. "You often oppose my ideas, but you never lie to me, and you always have well documented reasons for your viewpoints." She billed my hours via a company like Eagle Consulting in Vermont(?); we agreed to 50 hours per week at $60 for 13 weeks. The guys from Eagle advised Venita on many issues.
The guards had seen me come and go at all hours for three years. They smiled and said nothing.
My job was to reduce the backlog. It was fun. ( See Work is Fun ) At first I created lists of requestors, lists of modules, and a list of ideas embodied in the backlog. I also interviewed all the programmers so that I could understand their current assignments and I could gain insight into their capabilities. Ron Olson was a good worker; Richard Green is fast but a bit sloppy. Bob Kuramada had good skills. My first tactic was to guess at some solutions. My second tactic was the look for duplicate requests. Thirdly, I hastened the completion of a few projects; that freed up a few programmers. Fourthly, I identified scads of stale requests. Over 100,000 hours were allocated to stale projects. At first my boss refused to agree that the requests were unneeded, but usually the requestor was gone and the replacement person did not want the project. Often, some horse trading further persuaded the new user to cancel the old project in favor of a newer one. Every week for about a year, I cancelled stale projects.
The AQMD had a policy of having a programmer spend 16 hours to review every request for any new report. There had been concern that a report might need special resources or might consume hours of CPU. Most reports were in QUIZ by COGNOS. Again my boss was hesitant to change the policy. He did agree that with some new reports, I could do the review. My reviews took a few minutes, and in the rare cases where more time was needed, I delegated as had been normal. Simultaneously, I wrote a COBOL program to accept inputs for online, real-time reporting. A simple experiment showed that the typical report about permits would run in a small faction of a minute. The supervisor in air chemistry, Hubert Wilson, agreed to be my teammate in this project. He began running his requests and requests for others thru the COBOL program; I asked him to avoid advertising the program. He was delighted to be able to run his requests immediately and receive his output in one of three ways:
- On the screen
- On paper at his printer or a printer of his choice
- In Excel
The rest of the backlog included requests from many disciplines, such as, air chemistry, weather station maintenance, telemetry, legal statistics, car pool reports, gate control programs, and traffic patterns. Dealing with so many technologies was fun. Various techniques spurred the completion of these projects. After two years and some, the backlog was less than 50,000 hours and the number of contractors was about a dozen. Before the third year, my boss hired three supervisors to oversee the remaining projects. I became his assistant for special projects.
Also in the first year, I oversaw the implementation of a new checker board billing system where each permitted device was classified by model number into a grid of fees. The smallest annual fee was located in A1 and the largest in H10. Previously every device required individual calculations concerning hours, amperage, fuel, and lubricants. The new system saved thousands of hours of calculations and thousands of hour of legal hearings. The lady in billing was wonderfully precise and speedy. Her name might have been Jan.
The Smog Bank
A deadline loomed for implementation of the first Smog Bank to buy and sell smog credits. Ora McKewon designed the overview; Ron Olson and I streamlined the specs and wrote a stop-gap version which ran for several years. Russ Juranek, on this project and on many others, did excellent QA. After the initial implementation, Ora managed others who expanded the functionality of the Smog Bank.
The Expansion Project
The last significant project was an analysis and acceleration of the proposed new software system, called the Expansion Project. My boss gave me the first task of reading the documentation. The 22 notebooks comprised more the 20,000 pages of which most was fluff. Many of the notebooks contained what should have been numeric tables. Instead, the authors had written a sentence in English for each cell of some tables. For example, "When the row is zipcode 91702 and the pollutant in the column is sulfur dioxide, the value in the cell is five point three tons." A few macros in the VI editor reduced one table from eight hundred some pages to ten; it is was more readable.
Vehemently in private sessions to several key individuals, I tried to stop the Expansion Project.
- The old HP computer had better throughput than the new DG computer. The newer HP's were easily ten times more powerful.
- An experiment comparing the old HP IMAGE3000 database to Ingres on an IBM 390, the HP was half as fast in short tasks under 60 seconds and easily two or even three times faster in tasks requiring more than a minute: why upgrade to a slower machine?
- No business had ever run Ingress on a DG: simply stated: there was no port of Ingress to the DG at that time. <We wrote our own.>
- The HP with its re-entrant programs could run 127 sessions on the same one program. Both Ingress and Oracle needed one copy of the program for each user. See DataGeneral v HewlettPackard
- The in-house lawyer for the AQMD, a Mr Greenberg, had read the contract and had discovered the same technical gaps and the same padding of the tables, that I had discovered. The Board and certain executives felt the project had gone too many years to cancel.
- The menu and security and job handling queues for the new system were brand new: this was the first implementation.
My second task was to port the Ingres Data Base to run on Data General Computers; in essence, I copied the SPL and the C of the HP source code as a model for the server side, and two of us made it run on the DG hardware. The HP code implemented the SQL standard more efficiently than the IBM. The SPL roughly translates to C and stack operations. A contractor named John from England did QA and concentrated on the interactive routines. We finished in nine weeks.
Concurrently, I began to diagram the Expansion Project. It was missing several important processors, which moved data from input records to accumulation records. My boss said it was too late in the project to re-study and specify the processors; we would look bad. However, when the Board came to understand the situation, we looked like fools, and about thirty of us were let go. (My boss Cesar Diaz hired me several times in the next five years to work on other contracts elsewhere.)
On the Wednesday before the final boot, Venita Mathur asked me to meet her in her office. In January, per instructions from Venita Mathur, who became the new Director of IT, I reported serreptitiously to a barren desk on an uncompleted floor of the main tower. She and I had argued many times during the previous three years, but she trusted me. "You often oppose my ideas, but you never lie to me, and you always have well documented reasons for your viewpoints." She billed my hours via a company like Eagle Consulting in Vermont(?); we agreed to 50 hours per week at $60 for 13 weeks. The guys from Eagle advised Venita on many issues.
The guards had seen me come and go at all hours for three years. They smiled and said nothing.