Assessment and Selection Research
Development of Core Leader Competency Measures
U.S. Army, Center for Army Leadership
ICF International is currently developing a battery of leadership measures for the Center for Army Leadership to assess core leader competencies and attributes that influence performance across a broad range of leadership situations and contexts. ICF International will develop and validate measures; document our processes; and provide administrative and scoring guides, rules for creating competency composite scores, and instructions for maintaining test security. The end products will be a situational judgment test and experience measure designed to provide actionable feedback to leaders.
Research Studies to Improve the Implementation Practices of Multi-source (360) Assessment and Feedback Systems
U.S. Army, Center for Army Leadership
The objective of this project was to identify ways in which the Army’s Multi-source Assessment and Feedback (MSAF) program could be most efficiently executed while maintaining its effectiveness. Four separate research studies were conducted on key MSAF factors that influence its efficiency and effectiveness. Research methods included mining an existing MSAF survey database, as well as developing and implementing an experimental design with four feedback interpretation methods. Research findings demonstrated that phone call feedback interpretation compared favorably to face-to-face interpretation. This influenced the U.S. Army to launch MSAF with phone call feedback interpretation, saving the U.S. Army millions of dollars in projected face-to-face feedback interpretation costs while at the same time maintaining feedback effectiveness.
Development and Validation of Cognitive and Personality Measures to Select Census Field Operations Workers
U.S. Bureau of the Census
The U.S. Bureau of the Census contracted with ICF International to conduct job analyses to determine how 25 different jobs, held by more than 500,000 workers, have changed. The ICF team conducted job observations and job analysis workshops for 25 jobs across multiple Census test sites. Additionally, ICF conducted a study to determine changes that have occurred in the clustering of the job classes and to restructure the job classes as needed. Based on the results of the job analysis, the ICF team revised the selection system, developed measures of the cognitive aspects of the jobs, and included personality measures to assess the non-cognitive aspects of the jobs. A criterion-related validity study was conducted and evidence demonstrating the validity of the selection system was documented.
Development of Assessments for Special Agent Promotion
U.S.
Department of Homeland Security (DHS)
To ensure that this DHS agency could effectively address its mission, the agency decided to update and improve its Special Agent Merit Promotion Process. The new process was designed to reflect Special Agent competencies and to measure these competencies in a reliable, valid, and fair manner, meeting professional and legal standards of testing and selection. ICF developed and validated three assessments for the new process—an accomplishment record, a supervisor evaluation, and a multiple-choice in-basket assessment. We also documented the development and the validation of these assessment instruments.
Air Traffic Selection and Training (AT-SAT)
Federal Aviation Administration
Thousands of air traffic controllers were approaching retirement age, and the Federal Aviation Administration (FAA) had no selection test in place to select new controllers. In conjunction with a consortium of fellow researchers, ICF developed a computerized battery of tests to measure critical cognitive, perceptual, and psychomotor abilities. We then conducted a concurrent validation study with approximately 1,000 controllers to determine the relationship between the test battery and dimensions of job performance. The resulting “Air Traffic Control Specialist selection and training test battery (AT-SAT)” became operational in fiscal year 1999. The entire project team was awarded the 2000 M. Scott Myers Award for Applied Research in the Workplace by the Society for Industrial and Organizational Psychology.
Psychometric Analysis
Air Force Officer Qualifying Test (AFOQT) Form S Effective Analysis
U.S. Air Force
Since 1953, the Air Force Officer Qualifying Test (AFOQT) has been a major component used by the Air Force in the selection of officers. ICF evaluated the newest operational versions of the test (Form S) as part of the ongoing Air Force effort to ensure that new versions of the AFOQT continue to provide useful information. ICF evaluated test reliability, validity, and fairness; psychometric analysis was conducted at the question-, subtest-, and test-level. Our team also investigated Form S equivalence to previous versions of the test. Ultimately, the Air Force was provided with an evaluation of its high quality test, along with a set of actionable recommendations for test refinement.
USPS Equating Study
U.S. Postal Service
The U.S. Postal Service selects thousands of employees using the 473 Postal Exam. When USPS converted its paper-and-pencil version of the 473 to an Internet platform, it was vital to ensure that the scores on the Internet-administered exam were equivalent to scores on the paper-and-pencil exam. ICF designed and conducted an equating study to determine whether this was the case. Our team statistically evaluated the question of equivalence, investigated adverse impact, and determined whether the new on-line format required changes to testing time limits.
Critical Thinking and Deep Learning Measurement
U.S. Army, Center for Army Leadership
Today’s combat environment is much more ambiguous, complex, and dynamic than in the past, requiring new soldier competencies, and a modification in the focus of leader training and education. One competency of intense interest was critical thinking. ICF worked with the U.S. Army Center for Army Leadership to explicate critical thinking and identify instruments for assessing soldier-critical thinking skills and propensities. Our team integrated academic and applied theories of critical thinking to determine the most appropriate method for measuring critical thinking in the context of soldier self-development. In addition, ICF reviewed the psychometric properties of existing measures of critical thinking to determine test quality and appropriateness to a military population.
Workload Analysis
Acquisition Workload and Staffing Analysis
Marine Corps Field Contracting System and Marine Corps Systems Command (MC SYSCOM)
The ICF team conducted two workforce assessment studies within the Marine Corps acquisition communities to assist them in understanding the workforce planning issues and options available for better managing the performance of personnel assets in these career fields and job series. Two studies were conducted:
- Headquarters Marine Corps (MC) Installations & Logistics (I&L) and Field Contracting System (FCS) specific to the GS-1102, GS-1105 and military contracting job series only
- Marine Corps Systems Command (MCSC) which included its entire acquisition workforce job series
The results of these studies provided a variety of methods for estimating staff-to-workload estimates for MC FCS and MC SYSCOM. These estimates served as justification for additional staffing in contracting and other workforce acquisition series for SYSCOM and allowed FCS to better manage its contracting workload and share contracting resources among its contracting field locations. Among the decision tools created as a result were Microsoft Excel and PowerPoint files that allow input of future contracting workloads to estimate staffing requirements and dashboards representing current staffing, contracting, and staff characteristics used for staff management across offices. Benchmarking against 30 federal agency’s contract staffing-to-workload ratios was also provided. A follow-on contract to the MC FCS provided a three-year analysis of contracting data to determine variation in contract workload over time and recommendations related to the implications of this variation on office staffing and model performance.
Organizational Analysis and Staffing Model
U.S. Department of Homeland Security, U.S. Citizenship and Immigration Service, Office of Fraud Detection and National Security (FDNS)
The U.S. Citizenship and Immigration Service (USCIS) oversee the lawful immigration to the United States, including the adjudication of petitions and applications of potential immigrants. FDNS supports the USCIS mission by detecting, deterring, and combating immigration benefit fraud, and strengthening efforts to ensure immigration benefits are not granted to persons who pose a national security or public safety threat. FDNS requested an analysis of the current workload and structure to assist in making decisions regarding staffing levels for field operations spanning more than 70 offices across the U.S. The ultimate objective was to develop a staffing model to enable FDNS management to forecast staffing needs based on varying workload.
ICF conducted interviews with managers and subject matter experts from FDNS offices across the country to identify core work products, processes, and drivers of workload and workload variance. ICF then analyzed existing data from recent case processing to determine the effects of variance in the data. A time-to-performance model was developed that used average processing times collected across a representative sample of offices. Using the information gathered through interviews and data analysis, ICF then created staffing-to-workload decision tools in Microsoft Excel that modeled fraud and national security cases across three different office types. These models examine the hours required to process approximately 15 different types of cases. The work at different office types entails different processes, therefore, models were needed that considered the type of work being performed at each type of office and the average time spent by case type to process the work.
These decision tools allow for the creation of staffing estimates by individual office types based upon current and projected case loads specified in total staff hours and full time equivalent staff positions. When paired with management direction on policy priority, process improvements, and performance objectives, these models can be used to accurately estimate the staffing levels needed to respond to varying workload demands.
Workforce Analysis
Texas Workforce Commission “Missing Link” Project
Texas Workforce Commission
In this project, ICF International was selected to work with subject matter experts and instructors from a variety of fields to link technical course content to the job descriptive information provided in the U.S. Department of Labor O*NET database. ICF’s plan of work was designed to create a comprehensive, user-friendly database of valid, reliable links between the O*NET detailed work activities (DWA) and the programs of study offered in Texas community and technical colleges, as well as the courses listed in the Workforce Education Course Manual (WECM). This database will serve a critical role in efforts to enhance Texas economic development and workforce education.
Identification and Analysis of Education and Training Program for Library Support and Paraprofessional Staff in the U.S.
National Endowment of the Humanities, Institute of Museum and Library Services (IMLS)
This project involved designing and developing a survey of technology and digitization activities along with a statistical sampling methodology for data collection. Data were collected from a stratified random sample of the four sub-populations in both Web- and paper-based formats. Data were analyzed and compared to survey results from 2002 to identify trends and gaps in technology and digitization in participating organizations. Four qualitative follow-up studies were conducted to explore selected issues in greater detail. The results of this project served to inform a wide audience, including IMLS, libraries, museums, Congress, policy makers, and organizations that support or have an interest in museums, libraries, archives, and state library administrative agencies (SLAA).
Determining Manning and Personnel Requirements
U.S. Department of Defense, Joint Tactical Radio System (JTRS)
This project was a comprehensive resource allocaton and staffing review for the U. S. Department of Defense Joint Tactical Radio Systems Command (DOD JTRS). The project involved reviewing and evaluating the present operations of DOD JTRS, a military command with multiple divisions and a combination of uniformed staff (military), civilians, and contractors. ICF assessed the workload and allocation of these staff and—taking into account time spent on holidays, vacation, and other time off—derived the staffing needs for each division. We determined the optimal and minimum staffing levels for all military and non-military personnel by job category. Using a staffing management tool, we projected future staffing needs based on different scenarios for the types of missions that the command would be tasked to pursue. Finally, we demonstrated that the technical solution used in this study followed the Department of Defense best practices regarding staffing needs analyses.
U.S. Marshals Service Asset Forfeiture Program Workforce Analysis
U.S.
Department of Justice
Over the past 24 years, the work processes required to perform the mission of the U.S. Marshals Service Asset Forfeiture Program (USMS AFP) have substantially changed. ICF conducted a comprehensive evaluation to identify the current and future workload and workforce requirements to meet the mission. For the evaluation, we assessed the current workforce capability and capacity by analyzing demographic data on the existing workforce, assessing existing workload and time required to perform key work functions, and interviewing key internal and external stakeholders in Headquarters and District Offices. We identified business process and technology improvements in both headquarters and field functions, and anticipated workload fluctuations based on future demand requirements and high level skills required to meet mission demands in the future state organization. We evaluated the organizational structure and provided recommendations for reorganization to support changing work and mission requirements. Lastly, we developed a quantitative staffing model for projecting workforce needs based on workload and work composition and developed a workforce plan to address gaps and build the workforce required to meet the needs of the new Asset Forfeiture Program.
Recruitment and Retention
Entry-Level Transportation Construction Workforce Shortages
Florida Department of Transportation (FDOT)
FDOT selected ICF International to determine the causes of and potential remedies for the existing and projected entry-level transportation construction workforce shortages. ICF International conducted a comprehensive review and analysis of the job choice and selection literature, in order to understand the factors that contribute to job choice decisions. Additionally, ICF conducted a policy capturing study with high school students likely to be part of the unskilled labor market in order to capture their policies on what they look for when they choose a job. Finally, ICF also collected information on best practices in marketing and recruiting for unskilled construction jobs. Findings were compiled into a report that ICF International presented to the Florida Department of Transportation.
Addressing Critical Shortfalls: Recruitment, Development, and Retention of High-Quality Managers for Public Transportation Systems
Transit Cooperative Research Program (TCRP)
Because public transportation agencies providing fixed-route and paratransit services have identified critical workforce shortages for managerial jobs, TCRP contracted with ICF to identify short- and long-term strategies for improving the recruitment, development, and retention of managers in middle- and upper-level ranks. To identify effective practices, ICF is engaging in conducting a literature review of manager recruitment, retention and training, and development practices. Additionally, ICF is creating model job descriptions for manager jobs that are applicable across all sizes and types of agencies. Additional sources of information include best practices from other industries and conducting case studies and surveys of transit agencies to identify current practices as well as gaps. The results of these tasks will allow ICF to identify short- and long-term strategies and their applicability to different types of transit agencies.
Guide to Implementing Strategies to Attract and Retain a Capable Transportation Workforce
National Cooperative Highway Research Program (NCHRP)
Transportation agencies are facing difficult and evolving challenges in attracting and retaining capable employees. ICF was contracted by NCHRP to develop a guide to implementing effective strategies for attracting and retaining a capable transportation workforce for public sector agencies. To execute this project, ICF is currently conducting a best practices review followed by a case study analysis of identified best practices. We will use the results of the case study analyses to inform specific recommendations for changes in recruitment and retention strategies that may apply across transportation agencies.
Leadership
Army Training and Leader Development Panel (ATLDP)—Officer, NCO, Warrant Officer, and Army Civilian Studies and Consolidation
U.S. Army Research Institute
ICF worked with the U.S. Army to conduct one of the largest organizational assessments the Army has ever undertaken. The Army Training and Leader Development Panel (ATLDP) project involved a series of studies to assess leadership and leader development, education, and training for Army officers, non-commissioned officers (NCO), warrant officers, and Army civilians. Data were collected from more than 100,000 personnel worldwide through surveys, focus groups, and personal interviews. At the conclusion of the four separate studies, a consolidation study was conducted that identified Army-wide themes and forward-looking recommendations. The outcomes of this research included reports that detailed findings and provided recommendations for the improvement of leader development. Many of these recommendations were implemented by the Chief of Staff, Army.
Validation of Army Core Leader Competency Model
U.S. Army Research Institute and Center for Army Leadership
ICF collaborated with the Army Research Institute and the Center for Army Leadership to validate the Core Leader Competency framework that guides training, leader development, self-development, and education for Army leaders. The objective of this project was to demonstrate the soundness of the competency framework so that it could be used in doctrine and policy. To achieve this objective, ICF conducted three tasks. First, evidence of the construct validity of the competency framework was gathered and evaluated. Second, the extent to which the competency framework addressed organizational and situational differences was determined. Third, ICF conducted a criterion-related validation study of the competency framework. The resulting Core Competency framework is now Army doctrine.
Unit Leader Development Guide and Self Development Guide
U.S. Army, Center for Army Leadership
The U.S. Army was interested in creating a research-based guide to better equip unit commanders in fostering and directing leader development at the unit level. To meet this need, ICF conducted research on best practices in leader development and incorporated them into a handbook. ICF’s approach involved a literature review to examine current and best practices in leader development, as well as interviews and focus groups with Army leaders who had served as unit commanders to gain their perspective on what unit leader development looks like in practice. The findings from these data sources were integrated during the creation of the handbook. The product, The Commander’s Handbook for Unit Leader Development, provides an efficient and effective means for the unit commander to proactively plan and implement leader development at the unit level.
Training
Training Needs Assessment
U.S. Department of Agriculture-APHIS
To respond to potential threats, The U.S. Department of Agriculture's (USDA) Veterinary Services (VS) division must work in partnership with other agencies and organizations to establish coordinated Animal Emergency Response Organizations (AERO). ICF was responsible for merging this AERO concept with the U.S.Department of Homeland Security’s Incident Command System (ICS) structure used across agencies to define personnel roles in an emergency. To do this, ICF conducted an analysis of tasks and skills for all 119 ICS roles, then designed a training needs survey to collect information on current personnel knowledge, skills, abilities, certifications, and training histories at VS. After fielding the survey, data were analyzed and prepared for import into a learning management system that could be used for ongoing monitoring of personnel qualifications. ICF’s recommendations will ultimately be used to shape training offerings at VS.
Advanced Learning Technology Research
U.S. Army, Center for Army Leadership
To address current and future challenges to the force, the U.S. Army has acknowledged the need for further insight into advanced learning theories and technologies. To accomplish this, ICF conducted an extensive literature review of advanced learning theories. This evaluation included an assessment of the evidence of the effectiveness of each theory. ICF also explored, defined, and developed a model of the construct of deep learning. In addition, ICF conducted a comprehensive review of technologies that could enhance deep learning. Using these concepts, ICF then designed, developed, implemented, and evaluated a training module that could be implemented in an existing Center for Army Leadership (CAL) class. The results of this task included a model of advanced learning theory, the definition and development of the construct of deep learning, a training module, and evidence of the effectiveness of the implementation of the learning theories.
Skills Gap Analysis
A U.S. National Security Agency, Office of Weapons and Space
The Office of Weapons and Space (OWS), within a U.S. national security agency, requested that ICF conduct a job task analysis and a training gap analysis study. ICF began by defining the job categories that comprise the OWS workforce and listing the tasks and associated knowledge, skills, and tools needed to perform the work at basic, intermediate, and advanced levels. ICF then administered a job analysis survey to identify current levels of performance across the different job elements. A course mapping exercise to identify where job elements were being trained was also conducted. This data were then used to identify where gaps between critical job elements and available courses existed, as well as to identify any existing factors that may impact future training. The results were presented to representatives within the Associate Directorate of Education and Training and were put to use in addressing succession planning and retention issues.
Organizational Assessment
U.S. Customs and Border Protection Focus Groups Follow-Up to the 2006 Federal Human Capital Study
U.S.
Department of Homeland Security Customs and Border Protection
The Federal Human Capital Survey (FHCS) measures federal employees' perceptions and provides agencies with information on the extent to which they are successfully managing human capital in accomplishing their missions. U.S. Customs and Border Protection (CBP) wished to conduct a series of focus groups to follow up on the results of the 2006 FHCS and to solicit further input from a national sample of supervisory and non-supervisory employees. ICF developed a protocol and conducted a series of 127 focus groups with supervisory and non-supervisory CBP employees in 12 locations. Each focus group transcript was presented during one of 11 meta-focus groups, which were designed to jointly examine and refine collective findings to represent the most salient themes that emerged across individual focus groups. Next, a smaller group of researchers identified the 23 themes that spanned occupational groups using a similar process. A final report and briefing were presented to the Assistant Commissioners and commissioner of CBP.
U.S. Department of Homeland Security Organizational Assessment
Transportation Security Administration
ICF was contracted by the Transportation Security Administration (TSA) to develop, administer, analyze, and report on the 2006 Organizational Satisfaction Survey (OSS), with a goal of measuring employee attitudes and perceptions about organizational climate and provide TSA leadership with actionable results to drive improvement. ICF developed a comprehensive data analysis plan that divided the results of the OSS into actionable dimension scores as well as question-level breakouts. Upon completion of the data collection, the ICF team cleaned and analyzed the data and generated final reports. ICF helped TSA identify factors to increase its scores as a Federal Government Employer of Choice and has consulted on the development of several follow-up Pulse Surveys to assess the efficacy of the improvements made.
Workforce Safety and Insurance Employee Morale Analysis
State of North Dakota Workforce Safety and Insurance
ICF International was contracted by the State of North Dakota, Workforce Safety and Insurance (WSI), to determine the extent to which employee morale was problematic and to provide information to guide improvements and interventions. To do so, ICF used a combination of assessments to ensure that employee morale and satisfaction were adequately measured and that comparable benchmarking data were available. We conducted focus groups and administered an organizational survey that was available in both on-line and paper versions for employees. Once all of the data were collected and analyzed, ICF developed an action and evaluation plan that defined the specific steps to correct issues discovered. WSI has since contracted with ICF to conduct two additional survey administrations to assess changes to employee morale following the initial survey.
On-line Community Policing Self-Assessment Tool
U.S. Department of Justice, Office of Community Oriented Policing Services
The U.S. Department of Justice was in need of a Community Policing Self-Assessment Tool (CP-SAT) for police departments to evaluate their implementation of community policing. ICF, in conjunction with Police Executive Research Forum (PERF), worked with four police agencies from across the U.S. to pilot test and refine the CP-SAT. ICF is currently conducting follow-on work to develop an on-line version of the CP-SAT. ICF has pilot tested this on-line CP-SAT in several agencies to test for usability, difficulties in administration and participation, clarity of instructions, and factors that are important for inclusion in the user’s guide. Finally, ICF is creating an updated user’s guide to assist agencies in successfully administering, analyzing, and interpreting the CP-SAT.
|
