Over the years, in good situations where I was part of a good team, I have been extremely effect. One guy looked at my desires for a TEAM and said "that kind of situation makes it way too easy to succeed." He missed the point that a good situation makes for good results. Semi-Interpretive
The Team:
1) A boss who is involved and has negotiation skills and clout
2) A power user who understands the current system and most of the new system
3) A technical expert internally or externally available;
4) A good sysadmin who maintains a stable platform
5) A architect who provides a solid method to compile, migrate, and implement
6) A good designer who creates a specification that is reasonable with no black holes.
7) A good quality assurance person or some method for verifying correctness and throughput as the project grows.
8) A good programmer or three.
---sometimes one person fills more than one role. See Successful Teams
Black Holes
Black holes are very real. Ask any manager who has run into problem for which the solution was not known before the project started.
A black hole is usually some critical technology problem that is unfamiliar or not yet invented. A black hole can take excessive amounts of time, energy, and resources, hence the name Black Hole. The team needs to buy or borrow a solution, remove the requirement, or solve the black hole before attempting any reasonable estimate of time for completion of the project.
If you never encounter Black Holes, you are extremely good at designing or you doing nothing. Black Holes exist just beyond your current horizon. As your organization grows, the challenges will present some Black Holes. These areas present all sorts of challenges:
Research or Manufacturing
The leader of the project management committee should classify each project as Manufacturing or Research. When a project has both aspects, the committee should consider splitting the project and making each research question into its own project. Timelines for Research Projects are difficult to estimate. Timelines for Manufacturing are easy, especially when paradigms exist.
Most Manufacturing Projects have multiple tasks which form a chronology. Each task requires analysis for requirements and dependencies.
When a task requires external resources, the manager should start the task sooner.In the PeopleSoft world, the Integration Broker can be a black hole for some people and it is a standard product. I have seen some good alternatives to the Integration Broker, but most people never consider alternatives. The excessively broad and varied nature of PeopleSoft presents many potential Black holes. For example, given three programmers and a project with three tasks from different PeopleSoft products, each programmer might face his own personal black hole. On the other hand, even though each programmer has personal short comings, each one might be able to solve another programmer's black hole. Teamwork usually trumps solo work, but not always. Avoid black holes; they destroy time lines.
Diagonal Methodology
The old fashioned, 1960's waterfall methodology increases risk, lowers feedback, refutes wisdom, excludes frequent adjustments, and delays return on investment. Some persons might complain that such strong criticism is overly harsh, but the truth is sharp and simple. A good methodology implements frequently. On the one hand, each implementation increases total effort, but being successful on each implementation quickly increases the likelihood of project success. A well chosen first implementation gives the user a chance to see results early in the project. Each early implementation increases feedback on actually software rather than on some spec. Early adjustments cost less. A well chosen first implementation can bring early return on investment. Each subsequent implementation increases the benefits. Even if an incremental implementation fails, the cause most likely exists in the recently change source code. The area of concern is smaller and possibly the programmers still have the code mentally under control.
(Duh - I forgot)
When weeks pass between programming and implementation, the groups of programmers and of users forget details. The while knowledge declines, the risks escalate. The time to re-learn is a waste; the costs of changes climb because both groups must re-think. When the programs do not follow paradigms, the re-think time climbs even more.
Early Security
How often do we see delays and tensions arising when the manager delays the security for a project to the end of the project. Then not only is the project stalled while the sysadmins attempt to implement the security, but often the design of the security is wrong; retrofits must occur and delays occur. How much more effective and less stressful to implement security very early in the timeline.
Stub Programs
When the manager has the lead programmer set up a running stub and the sysadmin installs the program, the user has the opportunity to see progress. As the stub becomes the actual program, the user has repeated opportunities to observe and give feedback.
Return on Investment
A crafty manager chooses early successes that yield actual final functionality. Sometimes, a label printer goes first. Sometimes a new inquiry. A few times, implementing a credit card machine gave the organization a new source of collecting money. Dollars earned or dollars saved can help measure ROI. Likewise, since programmer time is like money; hours saved in coding or in testing contributes to ROI.
External Requirements
Any external necessity is a candidate for early action. A license from the state, a purchase of equipment, a relationship with a vendor, and a new telephone are examples of external requirements. When there is significant doubt about completing the external requirement, the manager might consider the requirement as a black hole.
Log Heaven
Each program should log its own start and stop. Some programs with complex inputs should log the inputs. Programs should log significant errors and extreme conditions. If the lag time between the a request to a remote service and the answer exceeds a certain number, the program should complain by writing a log record. Batch jobs should log starts and stops. Such records show speeds, volumes, and system behaviors. Similarly, screens should log starts and stops; any unpaired log records represent a possible crash. Records per minute or per hour are candidates for logging. With modern data bases or fast disc drives, log is inexpensive. Compiling should write log records.
The Last Shall be First
Plants grow from seeds. House are built from the ground up. People often think along similar. sequential lines. Great military generals and good project managers, however, usually find better strategies. When building systems that have natural end of processing requirements, the manager should consider writing the end of process programs early in the project. Not only must they be written sometime, but putting them first lets the users examine the data in a predictable and comprehensive manner. Other projects often have a particularly useful component with low cost programming requirements; finding those and implementing them early brings early benefits.
Each of the sixty web-crawlers at Richey Electronics came online in a beta mode shortly after its completion and testing. The speed of inquiry accelerated bids ten-fold and increased sales paid for the entire project after just a few of the web-crawlers went live. Indeed, the success was so great that users almost became over confident near the end of the project; the users required some prodding to test thoroughly.
Building Scaffolding for Long Term Use
Every library should have a driver program for each of its components. The driver submits good cases and bad cases to the component; the driver or its data source contains the expected results of each submission. No unexpected errors should occur and the result of each expected error should be exactly as expected.
Every significant program should have pre-planned test cases. Maybe the cases only cover some of the functions of the program; if a program has 100 functions, just examining the effects from each pair of functions requires about 10,000 cases. Now we are talking aerospace and moonshots.
The Million Dollar Day
Long ago, when connecting to Security Pacific Bank, Jack Howard and his credit card processing had to deal with the Million Dollar Day. The bank had a file of hundreds of transactions, which yielded exactly 1,000,000.00 in total when done correctly. Various differences suggested various errors. What a great idea.
Critical Path Method (CPM)
For larger projects, the number of pathways thru the project from stem to stern increases. The theoretical number increases with the factorial of the units of work, like the number of programs. The practical pathways do not increase as much, but they do increase, and the stress on management increases. In the Expansion Project at the AQMD, the project leader never certified that the primary path would work. He assumed that more than 22,000 pages of documentation could not be missing anything important.
The critical path has two main purposes. Firstly to assure that the project works and secondly to synchronize the activities. When a particular task depends on several other tasks, the last of those tasks to complete usually affects the start date of the particular task and, even more so, greatly affects the completion date of the particular task. Some software can be built using mock data as inputs, but true completion requires true inputs.
A project that goes exactly as planned does not need CPM. Happily, most projects do not need CPM. But larger projects often need CPM because the likelihood of delays and their cascade effects increase factorially; the resulting overruns inflict chaos.
The Team:
1) A boss who is involved and has negotiation skills and clout
2) A power user who understands the current system and most of the new system
3) A technical expert internally or externally available;
4) A good sysadmin who maintains a stable platform
5) A architect who provides a solid method to compile, migrate, and implement
6) A good designer who creates a specification that is reasonable with no black holes.
7) A good quality assurance person or some method for verifying correctness and throughput as the project grows.
8) A good programmer or three.
---sometimes one person fills more than one role. See Successful Teams
Black Holes
Black holes are very real. Ask any manager who has run into problem for which the solution was not known before the project started.
A black hole is usually some critical technology problem that is unfamiliar or not yet invented. A black hole can take excessive amounts of time, energy, and resources, hence the name Black Hole. The team needs to buy or borrow a solution, remove the requirement, or solve the black hole before attempting any reasonable estimate of time for completion of the project.
If you never encounter Black Holes, you are extremely good at designing or you doing nothing. Black Holes exist just beyond your current horizon. As your organization grows, the challenges will present some Black Holes. These areas present all sorts of challenges:
- encryption and lost keys
- deciphering pcode and metadata
- parsing natural language
- optimizing transportation systems, like laundry delivery, bin management, bus lines
- talking to a new equipment controller or robot
- parsing HTML or PFD files
- detecting movement on a video
- writing a computer game
- balancing loads across a network
Research or Manufacturing
The leader of the project management committee should classify each project as Manufacturing or Research. When a project has both aspects, the committee should consider splitting the project and making each research question into its own project. Timelines for Research Projects are difficult to estimate. Timelines for Manufacturing are easy, especially when paradigms exist.
Most Manufacturing Projects have multiple tasks which form a chronology. Each task requires analysis for requirements and dependencies.
When a task requires external resources, the manager should start the task sooner.In the PeopleSoft world, the Integration Broker can be a black hole for some people and it is a standard product. I have seen some good alternatives to the Integration Broker, but most people never consider alternatives. The excessively broad and varied nature of PeopleSoft presents many potential Black holes. For example, given three programmers and a project with three tasks from different PeopleSoft products, each programmer might face his own personal black hole. On the other hand, even though each programmer has personal short comings, each one might be able to solve another programmer's black hole. Teamwork usually trumps solo work, but not always. Avoid black holes; they destroy time lines.
Diagonal Methodology
The old fashioned, 1960's waterfall methodology increases risk, lowers feedback, refutes wisdom, excludes frequent adjustments, and delays return on investment. Some persons might complain that such strong criticism is overly harsh, but the truth is sharp and simple. A good methodology implements frequently. On the one hand, each implementation increases total effort, but being successful on each implementation quickly increases the likelihood of project success. A well chosen first implementation gives the user a chance to see results early in the project. Each early implementation increases feedback on actually software rather than on some spec. Early adjustments cost less. A well chosen first implementation can bring early return on investment. Each subsequent implementation increases the benefits. Even if an incremental implementation fails, the cause most likely exists in the recently change source code. The area of concern is smaller and possibly the programmers still have the code mentally under control.
(Duh - I forgot)
When weeks pass between programming and implementation, the groups of programmers and of users forget details. The while knowledge declines, the risks escalate. The time to re-learn is a waste; the costs of changes climb because both groups must re-think. When the programs do not follow paradigms, the re-think time climbs even more.
Early Security
How often do we see delays and tensions arising when the manager delays the security for a project to the end of the project. Then not only is the project stalled while the sysadmins attempt to implement the security, but often the design of the security is wrong; retrofits must occur and delays occur. How much more effective and less stressful to implement security very early in the timeline.
Stub Programs
When the manager has the lead programmer set up a running stub and the sysadmin installs the program, the user has the opportunity to see progress. As the stub becomes the actual program, the user has repeated opportunities to observe and give feedback.
Return on Investment
A crafty manager chooses early successes that yield actual final functionality. Sometimes, a label printer goes first. Sometimes a new inquiry. A few times, implementing a credit card machine gave the organization a new source of collecting money. Dollars earned or dollars saved can help measure ROI. Likewise, since programmer time is like money; hours saved in coding or in testing contributes to ROI.
External Requirements
Any external necessity is a candidate for early action. A license from the state, a purchase of equipment, a relationship with a vendor, and a new telephone are examples of external requirements. When there is significant doubt about completing the external requirement, the manager might consider the requirement as a black hole.
Log Heaven
Each program should log its own start and stop. Some programs with complex inputs should log the inputs. Programs should log significant errors and extreme conditions. If the lag time between the a request to a remote service and the answer exceeds a certain number, the program should complain by writing a log record. Batch jobs should log starts and stops. Such records show speeds, volumes, and system behaviors. Similarly, screens should log starts and stops; any unpaired log records represent a possible crash. Records per minute or per hour are candidates for logging. With modern data bases or fast disc drives, log is inexpensive. Compiling should write log records.
The Last Shall be First
Plants grow from seeds. House are built from the ground up. People often think along similar. sequential lines. Great military generals and good project managers, however, usually find better strategies. When building systems that have natural end of processing requirements, the manager should consider writing the end of process programs early in the project. Not only must they be written sometime, but putting them first lets the users examine the data in a predictable and comprehensive manner. Other projects often have a particularly useful component with low cost programming requirements; finding those and implementing them early brings early benefits.
Each of the sixty web-crawlers at Richey Electronics came online in a beta mode shortly after its completion and testing. The speed of inquiry accelerated bids ten-fold and increased sales paid for the entire project after just a few of the web-crawlers went live. Indeed, the success was so great that users almost became over confident near the end of the project; the users required some prodding to test thoroughly.
Building Scaffolding for Long Term Use
Every library should have a driver program for each of its components. The driver submits good cases and bad cases to the component; the driver or its data source contains the expected results of each submission. No unexpected errors should occur and the result of each expected error should be exactly as expected.
Every significant program should have pre-planned test cases. Maybe the cases only cover some of the functions of the program; if a program has 100 functions, just examining the effects from each pair of functions requires about 10,000 cases. Now we are talking aerospace and moonshots.
The Million Dollar Day
Long ago, when connecting to Security Pacific Bank, Jack Howard and his credit card processing had to deal with the Million Dollar Day. The bank had a file of hundreds of transactions, which yielded exactly 1,000,000.00 in total when done correctly. Various differences suggested various errors. What a great idea.
Critical Path Method (CPM)
For larger projects, the number of pathways thru the project from stem to stern increases. The theoretical number increases with the factorial of the units of work, like the number of programs. The practical pathways do not increase as much, but they do increase, and the stress on management increases. In the Expansion Project at the AQMD, the project leader never certified that the primary path would work. He assumed that more than 22,000 pages of documentation could not be missing anything important.
The critical path has two main purposes. Firstly to assure that the project works and secondly to synchronize the activities. When a particular task depends on several other tasks, the last of those tasks to complete usually affects the start date of the particular task and, even more so, greatly affects the completion date of the particular task. Some software can be built using mock data as inputs, but true completion requires true inputs.
A project that goes exactly as planned does not need CPM. Happily, most projects do not need CPM. But larger projects often need CPM because the likelihood of delays and their cascade effects increase factorially; the resulting overruns inflict chaos.
Something Changed
In business, change never ceases until the company dies. The government meddles endlessly; fads come and go; vice presidents have brain storms; the president promises pizza delivery; vendors fail; new products appear. See Finding Delta
Compile First
During the 1980's, CORT Payroll, ADI Payroll, and MCBA Payroll appeared at many businesses in southern California. I made good money as a traveling consultant, and the State of California provided the impetus for change as the Legislature kept changing the rules. One time in a zone of over-confidence, I recognized the few lines that required the change and dove in. On the second day, I went to compile and errors came flooding out. Much to my chagrin and financial loss, the program contained errors from a previous programmer. My argument that I had not altered those portions fell on deaf ears. Even worse, the customer lacked a good backup. The project lead could not remember when payroll had complied correcty, but he still insisted that all the bugs were mine. A week later and just two days pay later, I had everything working. That experience taught me the need to have a stable environment before starting a project. Other contracts re-confirmed by understanding when I insisted that everything compile before I made changes.
Compile Often
Maybe not. Currently, I do not know any writers of utilities personally. I have heard that some of the open-source organizations compile nightly. For sure, the utility writers of old compiled regularly if not nightly. Some went to the extreme of comparing characteristics of the binary files. After automating the compilation process, the cost of compiling approaches zero and benefits outweigh the costs. Reliability and confidence increase. For managers of business systems, protection of source code maybe sufficient. Many kinds of version control exist; compiling periodically helps to detect unexpected side-effects of normal maintenance. For example, the company had changed part of its monetary exchange module. A few closely related modules were also changed. At the end of the year, some tax modules ran but the results were gibberish. Inspection revealed that the tax module called some formatting routines from a non-standard location. Amounts went in but came out truncated.
Commands to Compile
Business programming and a few utilities have provided me with a good income. I do not choose to discover all the possible compiler error messages. Curtis Stevens has been known to torture the compiler, though. Instead, I prefer to encapsulate the compiler in a few well-defined command files. Besides compiling, the commands should include archiving the source if directed or if the program has not been recently compiled.
Typically these commands would be:
In business, change never ceases until the company dies. The government meddles endlessly; fads come and go; vice presidents have brain storms; the president promises pizza delivery; vendors fail; new products appear. See Finding Delta
Compile First
During the 1980's, CORT Payroll, ADI Payroll, and MCBA Payroll appeared at many businesses in southern California. I made good money as a traveling consultant, and the State of California provided the impetus for change as the Legislature kept changing the rules. One time in a zone of over-confidence, I recognized the few lines that required the change and dove in. On the second day, I went to compile and errors came flooding out. Much to my chagrin and financial loss, the program contained errors from a previous programmer. My argument that I had not altered those portions fell on deaf ears. Even worse, the customer lacked a good backup. The project lead could not remember when payroll had complied correcty, but he still insisted that all the bugs were mine. A week later and just two days pay later, I had everything working. That experience taught me the need to have a stable environment before starting a project. Other contracts re-confirmed by understanding when I insisted that everything compile before I made changes.
Compile Often
Maybe not. Currently, I do not know any writers of utilities personally. I have heard that some of the open-source organizations compile nightly. For sure, the utility writers of old compiled regularly if not nightly. Some went to the extreme of comparing characteristics of the binary files. After automating the compilation process, the cost of compiling approaches zero and benefits outweigh the costs. Reliability and confidence increase. For managers of business systems, protection of source code maybe sufficient. Many kinds of version control exist; compiling periodically helps to detect unexpected side-effects of normal maintenance. For example, the company had changed part of its monetary exchange module. A few closely related modules were also changed. At the end of the year, some tax modules ran but the results were gibberish. Inspection revealed that the tax module called some formatting routines from a non-standard location. Amounts went in but came out truncated.
Commands to Compile
Business programming and a few utilities have provided me with a good income. I do not choose to discover all the possible compiler error messages. Curtis Stevens has been known to torture the compiler, though. Instead, I prefer to encapsulate the compiler in a few well-defined command files. Besides compiling, the commands should include archiving the source if directed or if the program has not been recently compiled.
Typically these commands would be:
- compile a simple program to its expected name or to a test name, and archive the old binary always.
- compile a complex program including all of its sub-programs
- compile a sub-program and only link when specifically directed
- compile a component program of a library and only insert into the library when directed
- compile all the components of a library, and insert when directed
- compile into a cell phone, verifone, or other embedded system