Performance Management

Performance Assessment

Cost Benefit Identification and Assessment—Point of Service (POS) ONE, U.S. Postal Service (USPS)

The USPS Retail organization developed, piloted, and implemented POS ONE, a new point-of-sale system, to modernize and improve retail operations, replace outdated technology; improve customer service, and support the introduction of new products and services. The Postal Service needed an enhanced cost benefit and risk assessment to justify further capital funding and to identify and establish a baseline of metrics against which future performance could be measured and reported. The ICF International team utilized the cost/benefit and risk assessment framework and methodology to identify, document, and quantify potential program impacts on the Postal Service's financial operations, employees, and customers and communicated the results and impacts to the Postal Service Postmaster General and Postal Service Board of Governors. The deliverables and outputs were used to formulate continued POS ONE budget requirements and deployment strategies and were used in executive-level briefings to justify capital funding in excess of $500 million for continued development and deployment.

Performance Measurement—Window Operations Survey, U.S. Postal Service

The USPS Retail Consumers and Small Business organization developed, piloted, and implemented the window operations survey (WOS), a staffing and scheduling tool designed to control costs through managing staff workload, improving customer service through better staffing during peak hours of operation, and improving customer satisfaction. Once implemented, the Postal Service needed to know if the WOS had the desired impact on operations. The ICF International team identified performance metrics and executed a performance measurement study, complete with financial and operational analysis that was used to identify the reduction of work hours by 1.7 percent, or approximately $35 million in reduced operating expenses for the Postal Service fiscal year 2002 budget. In addition, as a result of our field interviews and observations, a best practice performance-tracking tool was developed and delivered to the Postal Service. This tracking tool is in use nationwide to monitor and report staff utilization and retail unit performance on a continuous basis.

Cost Benefit and Investment Analysis—PostalOne!, U.S. Postal Service (USPS)

In order to streamline processes with its most profitable customer segment, the USPS Marketing Technology and Channel Management organization developed and piloted PostalOne!, an online acceptance, verification, and transportation assignment application designed for large business mailers. A $74 million business mail automation project, PostalOne! provides real-time mail tracking, logistics management, online payment, and access to customer information. We conducted a cost/benefit and risk assessment to identify, document, and quantify potential program impacts on the Postal Service’s business operations, employees, and customers, and identified and implemented a performance measurement plan to evaluate and monitor program success. The benefit assessment and realization road map, complete with financial and operational analysis, provided a framework that enabled a clear understanding and effective communication of the results and impacts to the Postal Service Postmaster General and Postal Service Board of Governors. The deliverables and outputs were used to formulate PostalOne! budget requirements and deployment strategies and were used in executive-level briefings to secure the $74 million in funding for continued development and deployment.

Performance Measurement On-Line Tool (PM-OTOOL)
U.S. Department of Health and Human Services, Administration for Children and Families, Children’s Bureau

In order to report performance data for five Children’s Bureau discretionary grant programs (Abandoned Infants Assistance, Adoption Opportunities, Child Abuse and Prevention Treatment Act Research and Demonstration Projects, Child Welfare Training, and Infant Adoption Awareness Training), ICF developed the Performance Measurement On-Line Tool (PM-OTOOL), a Web-based data collection and reporting system, under the Child Welfare Information Gateway contract. Using the PM-OTOOL, grantees select performance measures (outputs and outcomes), then submit data on these measures at the end of each reporting period. Federal staff and grantees can create reports that reflect and analyze individual grantee data or aggregated data for a program for a particular reporting period or across multiple reporting periods. Grantees also can upload semi-annual and final reports to PM-OTOOL. ICF provides training and technical assistance to the Children’s Bureau and its grantees about performance measurement and how to use PM-OTOOL.

Title III and Title V Performance Measurement System
U.S. Department of Education, Institutional Development and Undergraduate Education Service (IDUES)

Agencies and organizations increasingly rely on performance measurement strategies to organize, manage, and evaluate programs. For this multiyear project, ICF’s education research experts developed a Web-based performance measurement system for the Title III and Title V programs on behalf of the U.S. Department of Education. Title III and Title V programs make available financial support to colleges and universities that serve substantial proportions of minority and low-income students. Working with the Department of Education's IDUES directors, IDUES program staff, and consultants, ICF designed and implemented a fully Web-based annual performance measurement (APR) system for Title III and Title V programs. This very successful system provides information for program monitoring, performance measurement, and communication about the IDUES program to Congress and other stakeholders. Current phases of the project include funding for three separate tasks: continuation of the APR system for the data collection and reporting cycle, analysis and reporting of recently collected data, and development and initiation of the first Web-based final performance report, to be completed electronically by grantees in the last year of their funding cycle.

Balanced Scorecard

Balanced Performance Measures—National Airspace Implementation Organization, Federal Aviation Administration (FAA)

ICF International is providing consulting services to major organizational units of the Federal Aviation Administration to improve the organization's annual performance planning effort. ICF International professionals developed workshop materials and led workshops tailored for these organizations on performance planning using the balanced scorecard (BSC) methodology. Using the BSC methodology, emphasis was placed on selecting critical organizational performance objectives that aligned to the organization's mission and vision. Objectives were selected for the four scorecard organizational perspectives providing a balanced organizational improvement plan. For each of the objectives, quantitative performance measures were established along with performance targets. Specific initiatives were then formulated to close the "gap" between desired performance and current performance. ICF International is also assisting these organizations in measuring the organization's performance during the year through results management techniques.

Portfolio Management

Portfolio Management—
National Airspace Implementation Organization, Federal Aviation Administration

The Federal Aviation Administration is seeking to maximize the value gained from its investments. It is trying to surface and mitigate risks to the achievement of investment outcomes through the creation, implementation, and institutionalization of a portfolio management approach to capital investment planning customized to the FAA environment. ICF International is assisting the FAA to integrate existing portfolio management initiatives and best practices and manage change associated with implementation and institutionalization. ICF International personnel are identifying best practices for portfolio management within government and the private sector; identifying organizational designs to support portfolio management functions; developing a to-be model and user guide for portfolio management,; and designing, developing, and delivering a portfolio management training course.

Portfolio Management—
Research & Development Agency, U.S. Department of Defense (DOD)

ICF International is helping a Defense R&D organization view its R&D projects as a portfolio designed to best meet future military performance goals, rather than a series of individual projects. The team developed a framework with multiple criteria for assessing each project's contribution to meeting military requirements, technology risks, and life cycle costs. The portfolio can then be evaluated for balance across potential payoffs, risks, timing of benefits, first costs, operating costs, and many other considerations. The framework is used to allocate resources, accelerate certain projects, and de-emphasize others. It also serves as a communication tool to best explain to stakeholders how and why R&D resources are being allocated.

Program Evaluation

Program Management Organization and Application Software Development—Direct Link, U.S. Postal Service

ICF International provided program management support, requirements analysis, software development, and IV & V expertise to the U.S. Postal Service in its development and deployment of Direct Link, a nationwide information system that leverages the electronic linkages between the USPS and its customers to facilitate efficient and advance communication of mailing information. ICF International successfully developed, deployed, and supported the field pilot testing of Direct Link at four pilot test sites and provided a life cycle strategy document and change management and training program to support a national deployment of Direct Link. Following this assignment, ICF International was awarded a follow-on contract to continue with Direct Link development and deployment to additional mailer and detached mail unit sites throughout the United States and to begin the certification process whereby Direct Link would be integrated into the USPS information system infrastructure.

Program Evaluation—U.S. Environmental Protection Agency (EPA) Office of Sustainable Ecosystems and Communities

ICF International conducted a program evaluation of EPA's involvement in community-based environmental protection efforts in South Florida. EPA's challenge was to successfully transition from a role of regulator and enforcement agency to a role of collaborator working with many partners on complex and often ambiguous issues. Using the results from our interviews and focus groups with local, regional, and national stakeholders, we developed practical recommendations for changes in behaviors to enable EPA to respond more effectively when working at the local community level. Our recommendations, which included techniques for improved project communications, local capacity building, and partnership building also were transferable to other place-based, multidisciplinary, and community-based EPA projects.

Housing Opportunities for Persons with AIDS Program—U.S. Department of Housing and Urban Development (HUD)

After five years of implementing this program, HUD asked ICF International to conduct an in-depth review of the program to document how program funds were being used, determine if the program was serving customer needs, specify the program's benefits, and identify leveraging opportunities with other supportive service programs. We analyzed five years of program information, conducted extensive mail surveys with housing assistance providers, interviewed program clients, and visited six different communities. ICF International then developed recommendations to better target the program and leverage the program's resources with other assistance programs to increase the number of recipients.

Public Assistance Program—U.S. Federal Emergency Management Agency (FEMA)

Beginning with the development and implementation of a baseline customer satisfaction survey in 1997, ICF International has assisted FEMA in developing annual performance targets and surveying customer satisfaction of nearly every Public Assistance disaster. As part of the overall program evaluation activities, ICF International produces an individual report for each disaster and a summary annual report comparing performance against the annual targets. Based on the results, FEMA has been adjusting and modifying the program to more effectively meet their customer needs.

Program Evaluation—U.S. Rural Housing Service Lending Program

ICF International evaluated the Rural Housing Service's Multifamily Guaranteed Lending Demonstration Program to determine if the program was reaching its target audience and how additional areas could be reached by leveraging the program with other funding mechanisms. In addition to examining the records of loans already made and interviewing stakeholders, ICF International modeled different financial scenarios and applied them to Census data on rural areas to recommend ways for increasing the effectiveness of the program. Our findings were used to develop new regulations and handbooks to better target the lending program.

Evaluation of the Ohio Mathematics Academy Program (OMAP) and the Ohio Science Institute (OSCI)
Ohio Department of Education

ICF was selected to evaluate OMAP and OSCI initiatives to improve teachers’ knowledge and instructional skills for classroom mathematics and science. These grant programs, funded jointly by the Ohio Department of Education and the U.S. Department of Education, supported the creation and implementation of professional development curricula using validated, research-based methodologies to address the complex issues surrounding the teaching and learning of mathematics and science in Ohio. Participating teachers received the curriculum in two-week summer professional development sessions. ICF relied on a one group pre/post design; assessments were administered to participating teachers at the beginning and end of their 2-week sessions. This data was supplemented with observations of the sessions to determine quality of the curriculum delivery, and participant surveys and interviews to document areas for improvement. The data was synthesized, and a user-friendly final report was prepared for each OMAP (6 reports) and OSCI content area (6 reports).

Evaluation of a Multi-Site Demonstration of Collaborations to Address Domestic Violence and Child Maltreatment (Greenbook Project)—National Institute of Justice

The Greenbook Project was designed to improve the way dependency courts, child protective services, and domestic violence service providers work together to address the problems of families with co-occurring domestic violence and child maltreatment. Under a five-year grant from the National Institute of Justice, with funding from the Office of the Assistant Secretary for Planning and Evaluation in HHS and the Centers for Disease Control and Prevention, ICF developed and implemented an integrated process and outcome evaluation design using multiple methods to measure the extent to which demonstration sites’ collaborative efforts result in system changes that would lead to improvements in safety, decreases in repeat abuse, and increased batterer accountability. The evaluation was designed to meet both national and demonstration sites’ needs; build sites’ commitment and capacity to use data to monitor and improve performance; increase understanding of the mechanisms by which and the extent to which system change is implemented; and have an impact on policy, practice, and knowledge in the field.