A Report on Analysis of Quality Assurance Team Projects
March 2013
Report Number 13-028
Overall Conclusion
The State Auditor's Office analyzed 13 major information system development projects at 8 state agencies. At the request of the State's Quality Assurance Team (QAT) those 13 projects were selected for analysis because those projects had been reported as complete, were nearing completion, or were identified as high-risk projects.
At the time of the analysis, seven of the projects were complete and the systems had been implemented, two projects were significantly complete, two projects were scheduled to be completed in fiscal year 2013, one project was scheduled to be completed in fiscal year 2014, and one project was canceled after five years of work.
Observations from the analysis included the following:
- Nine of the 13 projects were completed late or were projected to be completed late, and 1 project was canceled after spending $7,631,139 over five years. The average delay for all 13 projects was 9 months, or a 40 percent increase from the original projected end dates. The project with the shortest completion time took eight months. The project with the longest completion time took almost seven years; that project exceeded its planned completion time by 2.7 years. The agencies cited the following reasons for project delays:
-- Vendor negotiations and bidding process delays.
-- Federal and legislative requirements and standards changed during development of the system.
-- Turnover in project management.
-- Scope and time lines were underestimated.
-- Decisions agencies made related to changes in funding, requirements, or design.
- Improvements may be needed in developing initial project cost estimates. Eight of the 13 projects exceeded their budgets by an average of $1,841,551 or 57 percent. Two projects exceeded their budgets by 334 percent and 162 percent. Two projects came in under budget by 3.2 percent and 2.4 percent.
- Agencies that used a commercial off-the-shelf (COTS) solution exceeded their budgets by a smaller amount and took less time to complete their projects than agencies that did not use a COTS solution (see text box for additional details). Although they used COTS solutions, the agencies still needed to modify the software to fit their needs and incurred development costs associated with their projects.
Seven of the 13 projects used COTS solutions. Those seven projects exceeded their planned completion times by an average of 4 months or 27 percent, and they exceeded their budgets by an average of $1,679,590 or 23 percent. It took an average of 2.0 years to complete those projects.
In comparison, the remaining 6 projects that did not use a COTS solution exceeded their planned completion times by an average of 14 months or 55 percent and exceeded their planned budgets by an average of $2,030,505 or 97 percent. It took an average of 3.7 years to complete those projects.
- Agencies understated the costs of their projects by not including the costs related to state employees who worked on the projects. For 8 of the 13 projects, agencies did not always include costs associated with the salary and benefits for state employees in the project documentation they submitted to the QAT.
- For one project that is complete and has been in production for more than six months, the agency has not submitted its Post-implementation Review of Business Outcomes report to the QAT as required.