Program Evaluation

• Evaluating Housing Opportunities for Persons with AIDS Program
• Surveying Customer Satisfaction for the Federal Emergency Management Agency
• Evaluating Need for the Section 538 Multifamily Guaranteed Loan Program

Challenge Solution Benefit

Challenge
The U.S. Department of Housing and Urban Development's (HUD) Office of Policy Development and Research sought to evaluate its HOPWA program by answering three questions:

• How are the housing needs of persons living with HIV/AIDS being met through the HOPWA program, and what barriers exist to addressing those needs?
• Is the HOPWA program well coordinated with other community programs, including health care and supportive services that assist persons living with HIV/AIDS, and how has that coordination occurred?
• To what degree have Special Projects of National Significance accomplished their goals by using innovative ideas or techniques, and how applicable are these innovations to other programs?

HUD turned to ICF International to conduct this large and important program evaluation.

Solution
To conduct the evaluation, ICF International reviewed existing data sources, interviewed local program personnel and clients, and conducted three surveys. Two of the survey instruments—the Formula Grantee Questionnaire and the Housing Assistance Provider Questionnaire—were distributed to the universe of funding recipients. The third survey instrument, the Client Questionnaire, was used in anonymous phone interviews with 36 HOPWA clients. These multiple data sources were used to obtain the most accurate possible picture of HOPWA from the perspectives of various stakeholders. Our research team then compiled and analyzed this data and presented the findings in a detailed final report.

In the report, ICF International determined that the HOPWA program, as intended, predominantly serves extremely low-income and very low-income persons living with HIV/AIDS, including many people facing additional life barriers. Our findings also indicate that program flexibility helps meet clients' housing needs and preferences. Overall, HOPWA appears to enhance clients' housing stability, and clients report a high level of satisfaction with the housing that they receive.

Benefit
HUD obtained information about the following:

• The population served by the HOPWA program and their needs.
• How the program is used in conjunction with other programs.
• The strengths and weaknesses of the program.

With this information, HUD can now target technical assistance efforts, document program success with solid facts and statistics, and make informed public policy decisions.

Challenge Solution Benefit

Challenge
To meet Government Performance and Results Act (GPRA) requirements, FEMA initiated a multifaceted, performance-based redesign of major elements of its Public Assistance (PA) program. In order to measure the effect of these changes on customer satisfaction, FEMA contracted with ICF International to conduct a series of post-disaster surveys.

Solution
To understand how the PA program was meeting customer needs, ICF International first established a benchmark for measuring changes in customer satisfaction. We then conducted a baseline survey of the PA program disaster aid recipients, collecting data from a random sample of 19,000 customers served from 1994 to 1997. The results were presented in the Public Assistance Program Evaluation and Customer Satisfaction Baseline Survey in April 1998.

Based on the results of the baseline survey, ICF International then helped the PA program establish performance standards and measures. These standards were derived from focus groups and interviews as well as internally driven performance priorities. Performance targets for these standards and measures were established and presented in the Public Assistance Performance Standards in June 1998. The next step in tracking the PA program's responsiveness to customer needs was to measure customer satisfaction through individual disaster surveys.

Now, when government agencies and nonprofit organizations receive Federal Disaster relief to reinstate civil services, ICF International conducts a survey of customer satisfaction and produces a findings report. This report is widely disseminated inside FEMA, as well as among disaster participants. During FY1999 and FY2000, we produced more than 50 reports on individual disasters and customer satisfaction. We also produce annual reports that compare customer satisfaction results from same-year and previous year disasters. The annual results are used to set interim performance targets consistent with larger GPRA goals. ICF International has more than 20 surveys in progress for FY2001.

Benefit

• FEMA is better able to evaluate its quality of service for individual disasters and identify performance trends across time.
• FEMA receives valuable customer feedback, which can be used to pinpoint the program's strengths and areas for improvement.
• FEMA can continually fine-tune its PA program and strive to reach its goal of 90 percent overall customer satisfaction with FEMA's services, as established in FEMA's Strategic Plan.

Challenge Solution Benefit

Challenge
In order to promote affordable rental housing in rural areas, Congress passed legislation establishing a demonstration loan guarantee fund that guaranteed loans for multifamily rental housing developments in rural areas. As a part of the legislation, Congress directed the Rural Housing Service (RHS) of the U.S. Department of Agriculture to independently evaluate the results of the demonstration program. The evaluation aimed to determine if the program achieved its objectives and effectively promoted the development of affordable rental housing. ICF International was asked to perform the evaluation.

Solution
ICF International approached the issue using two methodologies. First, we interviewed lenders and developers to identify the advantages and disadvantages of the loan guarantee program and determine how it could be improved. We then modeled housing development data and compared it to Census data. This comparison allowed us to determine rural areas where (1) the loan guarantee was needed to promote affordable housing, (2) the loan guarantee was not needed, and (3) the loan guarantee was not sufficient to promote affordable housing. We then conducted the same analysis with additional programs, such as state subsidy programs and the Low Income Housing Tax Credit, to see how these programs could be used in conjunction with Section 538 to develop affordable housing. As follow-on work, ICF International assisted RHS in streamlining and revising several rural housing regulations, including the Section 538 regulation, and developed program handbooks that instructed RHS clients on how to use Section 538.

Benefit

• RHS gained a better understanding of how to use the Section 538 Multifamily Guaranteed Loan Program to develop more affordable housing in rural areas.
• RHS understood what could be done to make the program more effective.