Overview

Connecting Out of School Youth Preferences to Career Plans Impacts
YouthSource Program Performance

The City of Los Angeles Economic and Workforce Development Department’s YouthSource program is a premier provider of service for in and out-of-school youth. The city serves some 17,000 youth in its annual and summer programs. The Los Angeles YouthSource program has been nationally acclaimed by the Department of Labor for its progressive commitment to serving out-of-school youth. In 2012 before the new WIOA requirements, the City made a substantial commitment to serving this population requiring 70% of the youth served at its 13 YouthSource sites be out-of-school youth. As a result, they have become the nation’s most successful program in addressing the needs of this underserved population.

In 2012 the YouthSource program implemented the InnerSight Assessment Experience to:

• Actively engage participants in the development of their individual plan.
• Provide caseworkers with a foundation for development of individual plans grounded in participant’s assessment results.
• Connect individual plans and strategies to specific educational, training and occupational goals.
• Help caseworkers and clients see those skills needed to achieve occupational fit.
• Provide documentation of participant and case manager mutual engagement and results oriented planning.

A recently released study “Assessment Program Impact on Successful WIOA Program Performance in the City of Los Angeles YouthSource Program” shows that out-of-school youth participating in the City’s InnerSight Assessment Experience achieve the educational attainment outcome measure at a significantly greater rate. Forty-three percent (43%) of the out-of-school youth who have the assessment experience achieve the attainment goal for a statistically significant 14% improvement over the 29% attainment rate of those without the experience.

The study also revealed a 48% success rate in achieving the literacy standard for youth participating in the InnerSight Assessment Experience versus a 42% success rate for nonparticipants. Findings indicate success rates for the WIOA critical population of out-of-school youth who complete the InnerSight Assessment Experience are significantly higher than their peers on the Attainment and Literacy performance measures.

The City selected this assessment program to help WIOA staff align youth and adults with education, training programs, internships, and actual job openings that are a best fit for them. The findings affirm that when interests and preferences are aligned and self-validated by the individual, their success rate increases.

Through the InnerSight Assessment Experience, youth come to see they are the “Key” to their personal, educational, and career training choices and gain important information about themselves to reflect upon when making those choices. After self-validating their personal interests and preferences for working, learning, leading, risk taking, and team orientation, participants are equipped to explore occupations of people who are satisfied in various occupations and share the participants’ interests.

When asked what they liked most about the Experience youth say, “Keep up the work. There are many people out there like me who are lost and not sure what to do. But with this book and your help, there are paths that seem to open & lighten up.” and “This experience helped me get a clearer understanding of what I would like in my future career and I learned a lot about myself.”

Knowing Who You Are… Is the First Step …In Knowing Where You Are Going!

Full Study

Assessment Program Impact on Successful WIOA Program Performance in the City of Los Angeles YouthSource Program

Mark L. Perkins, John E. Reeves, Lolita Mancheno-Smoak, Debora K. Furlong

Abstract

The Workforce Innovation and Opportunity Act (WIOA) measures labor market outcomes as indicators of program performance.  This article uses statistical analysis to examine the impact of the InnerSight Assessment Experience on the rate of performance on key outcome measures in the City of Los Angeles YouthSource Program.  The findings suggest that the InnerSight Assessment Experience has a positive impact on youth performance on the WIOA performance measures of Attainment and Literacy. The impact is most profound for out-of-school youth, the target service population under WIOA.  Implications and considerations for policy makers and future research are provided.

Purpose

The City of Los Angeles Economic Workforce Development Department (EWDD) offers the InnerSight Experience as a consistent program element across its 13 YouthSource service sites. This study examines the impact of this common program element on youth development as indicated by performance on Workforce Innovation and Opportunity Act (WIOA) performance measures of placement, literacy and attainment.

Introduction

The city of Los Angeles YouthSource program is a premier provider of service for in and out-of-school youth. The city serves some 17,000 youth in its annual and summer programs. The Los Angeles YouthSource program has been nationally acclaimed by the Department of Labor for its progressive commitment to serving out-of-school youth. In 2012 before the new WIOA service requirements, the City made a substantial commitment to serving this population when it required that 70% of the youth served at its 13 YouthSource sites be out-of-school youth. As a result, they have become the nation’s most successful program in addressing the needs of this underserved population.

YouthSource program leadership desired to offer a consistent program element to serve youth across its 13 sites. Most importantly they wanted a program element that is developmental for each participant while meeting WIOA expectations and requirements. Specifically, they were seeking a program that would accomplish the following:

• Actively engage participants in the development of their individual plan.
• Provide caseworkers with a foundation for development of individual plans grounded in participant’s assessment results.
• Connect individual plans and strategies to specific educational, training and occupational goals.
• Help caseworkers and clients see those skills needed to achieve occupational fit.
• Provide documentation of participant and case manager mutual engagement and results oriented planning.

They believe a program grounded in participant personal development that connects youth with high demand occupations using their preferences and interests will facilitate persistence in the attainment of educational, training and career goals. To accomplish this, the city issued an RFP and selected the InnerSight Assessment Experience to meet this unique program and developmental objective.

The InnerSight Experience^TM

The InnerSight Assessment Experience consists of three important steps.  First, participants take an online inventory to provide information to clarify their interests and preferences.  Second, they participate in a self-validating group interpretation of the inventory known as the InnerSight Experience™. Third, participants share their results with their case managers and lay out the steps in their individual service plan.

The InnerSight Experience gives participants a vocabulary, context, and framework for bringing their personal interests and preferences to bear on life’s choices.  This experience puts the “Power of the Person” in the decision-making processes of life. Through the Experience, participants come to see they are the “Key” to their personal, educational, and career training choices and gain important information about themselves upon which to reflect when making those choices.

The InnerSight Experience is collaborative and delivered in a group setting.  This interactive experience engages participants in both individual and group exercises that emphasize and highlight the distinctiveness of their personal results.

The InnerSight Assessment Experience consists of an online interest and preference inventory resulting in a personalized InSight Guide^® booklet which is the foundation for a 3-hour in person group interpretative experience.  By self-validating their personal interests and preferences for working, learning, leading, risk taking, and team orientation, participants are equipped to explore occupations of people who are satisfied in various occupations and share the participants’ interests.

Participants learn that individuals are typically employed for what they know and terminated for what they do or fail to do.  When interests and preferences are aligned with an occupation (work tasks), satisfaction, happiness, and superior performance occur.  Youth with cultural, familial, and economic barriers are no exception.  When engaged in subjects of interest (music, dance, skateboarding, etc.), individuals succeed.

The InnerSight Experience™ is grounded in the prolific research and universally acclaimed Work Types Model of the eminent vocational psychologist Dr. John Holland.  The Holland Model was adopted by the Department of Labor as the foundation for their online career exploration system O*NET.  This system can be directly accessed from the electronic Insight Guide booklet participants receive after completing the InnerSight Experience by simply clicking on a preferred occupation.

For over 75 years, interests and preferences have been known as key factors in motivation for continued effort.  First used in World War I to determine which men might prefer to be cooks or which might prefer to be in the cavalry, interests have proven to be an effective method of determining continued effort/success in education, career, and personal life.  Using interests and preferences to help individuals make better life choices is a time-tested and valid approach.

InnerSight uses this approach, but has changed the traditional career assessment/exploration model.  Youth are accustomed to being processed, asked what they know (education), or told what they need to know and do next.  InnerSight begins the journey by first exploring and having participants validate what they may like or enjoy doing.  Additional exercises guide the participants to discover the connection between what they enjoy doing and potential occupations in which people who share their interests are satisfied.

The InnerSight Assessment Experience is designed to help WIOA staff align youth and adults with education, training programs, internships, and actual job openings that are a best fit for them. This study now examines the impact of the InnerSight Experience on WIOA performance measures.

Literature

The literature is replete with studies and meta-analyses of workforce investment program performance. It has long been the interest of policymakers for job-training programs to be evaluated using tangible labor market results for participants. King (2004) notes the Department of Labor has funded a host of studies in recent decades to assess the impact of training programs using such measures.

Moore and Gorman (2009) observed; “Policymakers brought this focus on performance to the creation of the Workforce Investment Act (WIA) in 1998. WIA, which replaced the job training partnership act (JT PA), established 17 performance measures to drive program performance.” They go on to observe that most of the 17 metrics were for labor market outcome indicators, such as changes in participant earnings, rate at which participants are placed in jobs, etc.

Social Policy Research Associates (2004) in an early assessment of WIA implementation noted that the 17 measures made sense to program operators but concluded that some operators felt definitions were vague and perhaps there were too many. As WIA evolved, the Federal Department of Labor elected to move to a new measurement system consisting of fewer labor market outcomes known as “common measures”.  Dunham, Mack, Salzman and Wiegand (2006) note that these “common measures were to be implemented for adults beginning in Program Year (PY) 2005, which began July 1, 2005, and for youth in PY 2006 (beginning July 1, 2006)”. (pg.22)

While there is a plethora of publicly available performance data, Moore and Gorman (2009) observe “there is little in depth analysis of the performance of the WIA system or of the likely drivers of the new common measures. Specifically, there has been little published on the relationship between individual participant characteristics and the program performance measures.” In fact, the Moore and Gorman, 2009 study on the impact of training and demographics in WIA program performance is the first we could find examining how the characteristics of participants may or may not influence the outcomes measured.

As might be expected they found demographic characteristics such as age, ethnicity, education etc. account for considerable variance in performance outcomes. These findings led Moore and Gorman to the following policy and program conclusions:

“A system of adjusting performance standards for states, local areas, and agencies that serve more disadvantaged populations is rational and would make good public policy” (p. 393).

“Policymakers should track whether or not the demographics of participants shift away from the most disadvantaged as the WIA-mandated outcome measures change to the new universal performance measures……. One-Stop operators and local area WIBs both have incentive to perform well on the outcome measures, and when outcome measures can be affected simply by adjusting the demographic profile of those served, it would not be surprising if One-Stops and local area WIBs were to shift their resources toward serving those more likely to help improve their outcome measures, rather than those most in need.” (p. 393).

“Data suggest relatively long-term and more expensive training services are not uniformly superior to shorter-term, less costly interventions such as job search assistance and one on one coaching and counseling.” (p. 393)

“No positive impact was found for any form of training on adult WIA participants… This does not mean that participants never benefit from training, but rather that the training intervention needs to be carefully targeted both toward individuals who can benefit from training and toward skills that are demanded by the local labor market. Simply loading WIA participants into available skill training without an individual strategy for each participant is not likely to yield positive outcomes. For dislocated workers, occupational skills training appeared to have a strong positive effect on earnings, and other forms of training were associated with lower earnings. These findings need to be considered by practitioners as they design interventions for individual participants.” (p. 394)

Moore and Gorman identify the profound impact of demographics on WIA performance measures. They go further suggesting “there is a need for researchers to undertake more nuanced studies of the connections between training and labor market outcomes in WIA, with carefully controlled studies to identify the types of participants and types of training most likely to lead to successful outcomes (p. 394) This suggests a need for providers to understand what program elements, assessments, or experiences have a positive impact on participant’s performance on the common WIA metrics.

Borden (2009), in a comprehensive article on the challenges of measuring employment program performance, says “research on employment and training programs focuses primarily on evaluations of the impact of public investment in job-training services”. (p. 2) He goes on to suggest that there is indeed a dichotomy between the program management objectives of performance management and its evaluative objectives. Moore and Gorman focused on the impact of demographic characteristics and their relationship to the achievement of performance measures. This study goes a step further by analyzing the impact of program on the successful achievement of desired performance outcomes and to identify those elements that drive desired job training performance.

Barrow and Smith (2004) clearly suggest that program evaluation and performance management derive from different sources and motives and have deeply committed adherents. The question is not whether we should track or measure program performance but rather how do program elements contribute to the achievement of job-training and employment program objectives. As suggested by Borden (2009), “there is an increasing tendency to leverage the efficiency of analyzing administrative data versus more expensive experimental designs”. (p. 5) The truth is both are needed, but the experimental designs with appropriate stratified samples of participants for randomization with clean and accurate participant data are not only more expensive, but practically impossible to achieve. Thus, impact studies with limitations may be the most informative indicators of positively contributing program elements for achieving job-training program results.  Most importantly, as suggested by Borden, a good measure must produce a rate of success and not simply a count of activities.

Borden properly points out that “we must distinguish clearly between service delivery and program management. Performance management systems track common events such as enrollment date, customer characteristics, limited service dates, exit date, and outcomes. Performance management systems do not specify how services are delivered.” (p. 21) nor do they indicate how services impact performance management outcomes. While it is impossible to design measures that account for all factors bearing on the success with a single customer, focused impact studies can be helpful in determining a program or assessment’s potential accountability for or contribution to program outcomes. With clear program performance measures in place we can begin the analysis of those processes and methods that produce the best results for program participants.

The first major update in almost 15 years to the Workforce Investment Act of 1998 (WIA) was signed into law in 2014. The new federal Workforce Innovation and Opportunity Act (WIOA) places greater emphasis on serving out-of-school youth (75% versus 30% under WIA). WIOA defines out-of-school youth as 16 to 24-year-olds who are not attending any school and who have one or more barriers to employment, such as young people who are homeless, are parenting, have disabilities, or have a juvenile criminal record (United States Department of Labor, Employment and Training Administration (2015)). The new law requires states and localities to develop strategies and programs for recruiting and serving more of these young people than ever before.

This could be a daunting task. Hossain (2015) observes that “while a majority of the out-of-school youth seek out opportunities to connect training and work, youth programs often report difficulties in sustaining participation after the initial connection is made. WIOA-funded service providers will not only have to reach more out-of-school youth, they will also need strategies to stimulate sustained, intense engagement in services.” (p. 1) She goes on to observe that “few programs target the young people who are the most persistently disconnected, and there’s not much evidence on what works in engaging them.” (p. 3) Perhaps most importantly she says that “a significant share of out-of-school youth do not enroll in education and training programs because they have been alienated from mainstream institutions, like schools and social welfare agencies, due to earlier negative experiences. New strategies to reach and engage alienated and disaffected young people should be a priority.” (p. 3)

Hossain (2015) suggests programs “have to find a balance between allowing vulnerable young adults some flexibility in regard to program requirements… while developing processes and practices that allow young people to develop autonomy and leadership” (p. 4). Specifically, she recommends “asking young people for their input in designing program activities and allowing them to have a voice in program governance.” (p. 4)

WIOA encourages implementation of career pathway approaches that will support post-secondary education and training for out-of-school youth that is related to the demand for qualified workers in local labor markets. The individualized pathway is to be grounded in an assessment of skills and interests that can inform the participants’ career decision-making while identifying logical short-term next steps consistent with long-range goals.

The US Departments of Education, Health and Human Services, and Labor issued a joint letter in 2012 that provided a common definition of career pathways that encompasses initiatives that begin with secondary or post-secondary education and credentials. According to the letter a career pathway approach is:

“…… A series of connected education and training strategies and support services that enable individuals to secure industry relevant certification and obtain employment within an occupational area and to advance to higher levels of future education and employment in that area.”

Richard Kazis in a MDRC research brief on Career Pathways (2016) observes “there is little rigorous research that assesses the impact of comprehensive career pathways programs that follows individuals from different starting points through a coherent set of educational experiences and “stackable” credentials to the labor market.” (p. 2) His research brief however, goes on to note that there are components that are being studied that might help policymakers as well as program designers and managers. He provides brief summaries of project models that make a difference in several categories including high schools, out-of-school and disconnected youth, and low income adults. Kazis suggests there is a paucity of research on how to positively engage the participant in his/her personal development of a successful career pathway.

Research has primarily focused on program performance yielding discussions of topics such as the importance of the employer network, career demand, issues of program integration, curriculum and process.  All of these focus on what is “done to” the participant and rarely on what is “done with them”. With the exception of Moore and Gorman who examined the impact of demographic characteristics on program performance measures there has been virtually no research examining the impact of program elements offered by providers on WIOA performance indicators. This study presents an analysis of a robust population of WIOA participants from a large diverse city exploring the relationship between the participant’s program assessment experience and subsequent performance on WIOA common measures.

Impact Analysis Approach

As noted by Moore and Gorman (2009) and Borden (2009), the gold standard random-assignment experimental design is extremely difficult and virtually cost prohibitive to use with live WIOA programs. This is perhaps why there is virtually no research on the impact of individual program elements on the common WIOA performance measures. As a result, program providers are left with little or no information regarding program elements that significantly impact participant performance on program performance indicators.

While this study might be referred to as a quasi-experimental design, the authors prefer to call it an impact analysis. This is a first step in identifying the impact of an assessment experience on participant’s achievement on WIOA outcome measures. Essentially, it would be helpful to know if the assessment experience significantly impacts youth performance on program outcome measures.

Method

Research Questions:  This study analyzed youth participant performance on WIOA outcome measures for youth participating in the City of Los Angeles YouthSource program in program year 2014 – 2015 (PY-1415) to answer the following questions:

1. Do youth participants who have the InnerSight Assessment Experience achieve satisfactory performance on the WIOA outcome measures of attainment, literacy and placement at a significantly greater rate than those who have not had the Experience?
2. Do out-of-school youth participants who have the InnerSight Assessment Experience achieve satisfactory performance on the WIOA outcome measures of attainment, literacy and placement at a significantly greater rate than out-of-school youth who do not have the Experience?

Study Population:  The study population consists of in-school and out-of-school youth participating in the City of Los Angeles YouthSource program at one of 13 provider sites across the City. Study participants are youth the City of Los Angeles included in the performance analysis for each of three WIOA performance outcome criteria for program year 2014–15. Performance of youth participants who have completed the InnerSight Assessment Experience (Experimental Group) is compared with those who did not complete the Experience (Control Group).  The outcome performance measures for this study were drawn from the City of Los Angeles “Jobs LA” performance tracking system.  The outcome measures for this study are described below.

Literacy– Literacy & Numeracy Gains

This indicator measures the increase by one or more educational functioning levels from out-of-school youth who are basic skills deficient within a one-year period of participating in program services.

Meeting the Standard:  Youth meet the outcome standard for this measure if they:

• Show increase in skills as measured by the Comprehensive Adult Student Assessment System (CASAS) Life & Work (reading) and Life Skills (math) Series at program entry within one year of start of program services.
• Increase one educational function level (EFL) in math, reading or both, during each year of program participation.

Placement -Placement in Employment, Education, or Training

This indicator measures whether a participant is in an education or training program that leads to a recognized post-secondary credential or unsubsidized employment in 1^st Quarter after program exit.  This measure requires a quarterly follow up for one year after program exit.

Meeting the Standard:  Youth meet the outcome standard for this measure if they meet one of the following:

• Employed by an unsubsidized organization.
• Entered military service.
• Enrolled in a post-secondary education program.
• Enrolled in an advanced training or occupational skills training program.

Attainment -Credential Rate

This indicator measures the attainment of a high school diploma, GED, or certificate during program participation or by the end of the 1^st Quarter after program exit.

Meeting the Standard:  Youth meet the outcome standard for this measure if they meet one of the following:

• Receive a High School diploma certifying completion of a secondary school program of studies.
• Receive satisfactory scores on the General Education Development (GED) test.
• Receive a formal award certifying the satisfactory completion of an organized program of study at a post-secondary education program.

Table 1 provides demographic information for the InnerSight Assessment Experience and Control Groups for Program Year 2014-15.  The average age for the three study groups ranges from 18.3 to 19.1 years. Since the cohort deemed appropriate for each individual outcome is different it’s helpful to understand the demographics for each outcome cohort as reflected in Table 1.

Literacy: The literacy outcome measure is appropriate only for out-of-school youth.  Out of the total 1665 participants, 270 (16%) completed the InnerSight Assessment Experience while 1395 (84%) did not have the Experience.

Placement: The placement outcome measure was deemed appropriate for a total of 2085 youth participants of which 557 were in-school-youth (27%) and 1528 were out-of-school youth (73%). A total of 362 participants completed the InnerSight Assessment experience (17%) while 1723 in the control group did not have the Experience (83%).

Of the in-school youth, 111 or 20% completed the InnerSight Assessment Experience while 446 or 80% in the control group did not. Of the out-of-school youth participants for this measure, 251 or 16% completed the InnerSight Assessment Experience while 1277 or 84% in the control group did not.

Attainment: The attainment outcome measure was deemed appropriate for a total of 1547 youth participants of which 564 were in-school-youth (36%) and 983 were out-of-school youth (64%). A total of 305 participants completed the InnerSight Assessment experience (20%) while 1242 (80%) in the control group did not have the Experience.

Of the in-school youth, 111 or 20% completed the InnerSight Assessment Experience while 453 or 80% in the control group did not. Of the out-of-school youth participants for this measure 194 or 20% completed the InnerSight Assessment Experience while 789 or 80% in the control group did not.

 

Information for gender and ethnicity of participants was not available in the data extract provided by the City of Los Angeles from the “Jobs LA” performance tracking system. However, the City of Los Angeles did provide total program gender and ethnicity information for program year 2014-15 reflecting a robust and diverse population of participating youth. For 2014-15 there were 1947 females (54.4%), 1630 males (45.6%). The reported ethnic makeup as expected is very diverse with the largest reporting group being Hispanic or Latino followed by White and African American/Black. The mix of Race or Ethnicity in the population is so large that participants are permitted to select more than one race or ethnicity suggesting that the standard race identification categories are rapidly becoming ineffective demographic descriptors or study variables for this population.

Analysis:  As Borden (2009) suggests, a good measure must produce a rate of success and not simply a count of activities. Therefore, the rate of success on each of the WIOA performance measures (literacy, placement and attainment) will be calculated for participants appropriate for assessment on each measure as an indicator of program performance. To assess the impact of the InnerSight Assessment Experience, the percent succeeding on each measure will be compared for those who had the InnerSight Assessment Experience with those who did not have the Experience (control group). Differences in rate of performance will be analyzed using chi-square to determine if they are significant or if they occurred by chance alone.

Similar analysis will be conducted within each measure where appropriate to compare the rate of performance of in-school-youth and out-of-school youth who have had the InnerSight Assessment Experience with those who have not.

Analysis of variance will be used where appropriate to identify any within group or interactive differences that may be associated with age of the participant.

The performance rates of participating youth successfully meeting the performance standard for program year 2014-15 are provided in Table 2 for each of the three outcome measures. The overall rate of successful performance is 70% for Placement, 49% for Attainment and 43% for Literacy. Table 2 reveals considerable variance in performance rates between in-school youth and out-of-school youth as well as between those who have had the InnerSight Assessment Experience and those who have not on the Literacy and Attainment outcomes. Performance rate comparisons and examination of significant differences are provided below by WIOA outcome performance measure.

 

Literacy: Only out-of-school youth participate in the literacy outcome measure. For the 2014-15 program year population, 43% met the standard. Those who had The InnerSight Assessment Experience had a 48% success rate while those who did not achieved a 42% success rate. This suggests InnerSight Assessment Experience participants are more likely than other participants to achieve the literacy standard. The 6% difference in performance rate for those completing the InnerSight Assessment Experience is statistically significant yielding a χ² value of 3.99 with 1df, N of 1665 and a probability of .046.

Table 3 contains the distribution of participants and their literacy outcomes by age. While participants range in age from 15 to 21, over 99% fall between the age of 17 and 21. Table 3 contains performance success rate by study group and age. Figure 1 provides this information graphically across age for both study groups (InnerSight and control).

 

 

InnerSight Assessment Experience participants have a higher rate of success than the control group across all age groups as can be noted in Table 3 and the graph in Figure 1. It also appears that older participants achieve a higher success rate than younger participants. An analysis of variance of the performance on the literacy standard examined the impact of age and the InnerSight Assessment Experience. Both the InnerSight Assessment Experience and Age have statistically significant impacts on the rate of Performance on the Literacy Outcome, producing values of F=9.719, p<.010, and F=17.001, p<.009 respectively. This suggests the InnerSight Assessment Experience and age have a positive effect on the outcome.

There was no interaction effect between the InnerSight Assessment Experience and age on the Literacy performance outcome.  The statistical analysis of this effect yielded a value of F=.266, p<.900. Thus the positive increase in Literacy performance rate attributable to the InnerSight Assessment Experience is not affected by participant age.  (See note 1 for complete ANOVA results.)

Placement: For the 2014–15 program year, 70% met the placement outcome standard. Table 1 shows that the rate of achievement on the Placement standard does not differ between in-school and out-of-school youth or between InnerSight Assessment Experience participants and the control group. Table 4 contains the number of participants and associated success rates by age, study group (InnerSight and Control) and school status (in-school and out-of-school). Statistical analysis revealed no significant differences in performance rates associated with participation group, school status, or age. This suggests that neither the InnerSight Assessment Experience, nor youth school status or age impacts the success rate on the placement outcome measure.

Attainment:  For the 2014-15 program year, 49% successfully met the attainment outcome measure standard as can be seen in Table 5. The performance measure success rate for those who had the InnerSight Assessment Experience was 58% while the success rate for those who did not have the Experience was 47%. This suggests that those who have the InnerSight Assessment Experience are more likely to achieve success on the attainment outcome performance measure than those who do not. The 11% difference in performance rate for InnerSight Assessment Experience participants was statistically significant, yielding a χ² value of 11.36, with 1df, N=1547 and a probability of p = .001.

 

The impact of the InnerSight Assessment Experience on the Attainment outcome differs dramatically for in-school and out-of-school youth. The success rate of 85% for in-school youth on the attainment measure who had the InnerSight Assessment Experience was not significantly greater than the 79% success rate for in-school youth who did not have the Experience. The statistical analysis yielded a χ² value of 1.544 with 1df and an N=564 with a probability of p = .214.

Further review of Table 5 for in-school- youth reveals a substantial change in performance rate for youth under 18 and those 18 and older.  When examined by these age groupings a performance rate of 90% is observed for the 413 in-school youth under 18 compared to a performance rate of 57% for the 151 in- school youth who are 18 and older.  The 33% difference in performance is statistically significant yielding a χ² value of 81.222 with 1df and an N=564 with a probability of p = .000.  When analyzed separately by InnerSight and control group the results are similar. This seems to suggest a powerful effect for age or life events associated with age.

Examination of the success rates for the WIOA critical population of out-of-school youth shows that those who complete the InnerSight Assessment Experience are significantly more likely than their peers to achieve success on the Attainment performance measure. The success rate on the Attainment outcome measure for those having the InnerSight Assessment Experience is 43% versus 29% for those who do not. This 14% difference in the performance rate is statistically significant yielding a χ² value of 14.6 with 1df, an N of 983 and a probability of p = .00.

For out-of-school youth, a univariate Analysis of Variance of the InnerSight Assessment Experience and age shows that there is a statistically significant difference (F=26.210, p<.002) between the InnerSight group and the Control group, but age has no impact.  Furthermore, there was no interaction effect of age with the InnerSight Assessment Experience (F=.527, p<.716). Thus the InnerSight Assessment Experience makes a significant difference and is not impacted by participant age.  Figure 2 shows that the InnerSight Assessment Experience impact on Attainment is not isolated to only the youngest or the oldest participants. (See Statistical Analysis note 2 for complete ANOVA results.)

Examining research question one, the results show that youth who have the InnerSight Assessment Experience achieve successful performance on the attainment and literacy performance measures at a significantly greater rate than those who do not have the Experience. With regard to the placement outcome measure, rate of performance is not impacted by the InnerSight Assessment Experience for any group.

On research question two, the results show a significant difference between the successful performance rate of out-of-school youth (WIOA’s primary target group) who have the InnerSight Assessment Experience and those who do not on the attainment and literacy outcome measures. Out-of-school youth who had the InnerSight Assessment Experience performed significantly better on the attainment outcome measure (+14%) and on the literacy outcome measure (+6%) than those who did not have the Experience.

Discussion

The literature offers virtually no research on program elements that impact youth success on WIOA performance outcomes. Existing research (Moore and Gorman, 2009) focused only on the impact of participant demographic characteristics on outcome measures. Borden (2009) noted that performance management systems do not specify how services are delivered and that measures of program performance must produce a rate of success and not simply a count of activities. This study responds to both of these concerns with a focus on the InnerSight Assessment Experience as a program element in service delivery and its impact on participant rate of performance on WIOA outcome measures.

The finding that out-of-school youth meet the attainment performance standard at a 14% greater rate if they have the InnerSight Assessment Experience (49%) than those who do not (29%) is not only statistically significant but makes it clear that program elements influence participant performance on WIOA outcome measures. Results were similar on the literacy outcome measure with a 6% significantly greater rate of performance for out-of-school youth who had the InnerSight Assessment Experience (48%) than those who did not (42%).

The target population for this study was young people who are the most persistently disconnected. The findings provide evidence that the InnerSight Assessment Experience is an intensive interpretive program element that works in engaging them. When connected with case manager follow-up using the InnerSight Experience^TM results to chart next steps related to educational and occupational goals, the results are personally engaging, developmental, and productive. This is one of those new strategies Hossain (2015) says is needed and should be a priority in reaching and engaging alienated and disaffected young people.

Hossain recommends “asking young people for their input in designing program activities and allowing them to have a voice in program governance.” (p.4) The InnerSight Assessment Experience capitalizes on this by increasing the youth’s voice in the use of their personal assessment results in designing a personal pathway to success. This is particularly productive in establishing individualized short-term and long-term goals as required by WIOA in each youth’s Individual Service Plan.

Kazis (2016) suggests there is a paucity of research on how to positively engage the participant in his/her personal development of a successful career pathway. The findings of this study respond to this concern and suggests a useful building block is the effective use of personal assessment results. Far too many times, assessment is viewed as something we do to people, or for them, rather than with them. Thus we lose the opportunity for meaningful engagement, personal exploration and the power of personal validation is lost. Such engagement takes time and some professional expertise with the assessment tool which often is not available in many WIOA programs but in this study was provided through the InnerSight Assessment Experience.  Case managers are easily trained to work with youth participants and the results in their InSight Guide results booklets to discuss and plan together important educational and career next steps.  Such collaborative work leads to the program impact on performance outcomes found in this study.

The placement outcome measure performance rate was not influenced by the InnerSight Assessment Experience and is most likely a result of individual provider agency contacts and networking skills rather than the personal development of the youth participant.  This would be an interesting area for further analysis and study.

The results for in-school youth revealed a significant effect for age on the attainment success rate. There was a substantial change in performance rate for youth under 18 when compared with those 18 and older. A performance rate of 90% was observed for the 413 in-school youth under 18 compared to a performance rate of 57% for the 151 in-school youth who are 18 and older. Moore and Gorman (2009) in their study of the impact of demographics on WIA program performance of adults and dislocated workers also found an effect for age of participant. These finding suggests a strong need for further research to better understand the effect of age or life events associated with age on WIOA outcome performance measures.

Service program elements clearly have a profound impact on WIOA performance outcomes as found in this study. However, there continues to be a paucity of research at the service delivery level identifying what service efforts have the most impact.  This impact study was designed to take a first step with the expectation that others would follow to help define a universe of practices that make a significant difference in performance as found in this initial effort.

Implications and Considerations

• There is a need for additional research to Identify those service delivery program elements that contribute to a participant’s success.
• Policy makers, and especially workforce investment boards, need to assure that data elements are created in their performance systems to facilitate the study of service level program impact on outcome performance. More specifically, there is a need to move beyond a specific WIOA performance measure to examine if what is done to, for, and with the participant increases the rate of performance.
• Age of participant has been found to produce an effect in studies of both youth and adult performance on WIA and WIOA program outcome measures. This suggests a need for further research on age impact and for policy makers and program leaders to consider this in the development of service strategies as well as individual service plans.
• Effective use of assessment results is a powerful building block for establishing rapport, engaging the youth and creating personal pathway to success. Requiring youth participants to engage and validate assessment results places them in a position to make better choices regarding education, training and career options. Their input is vital for program success as it builds a sense of personal ownership in the Individual Service Plan and most importantly– it wasn’t done to them.  As a major WIOA expectation policy makers will want to understand how this is accomplished by their service providers.
• Professional interpretation of Assessment results, when integrated with case manager’s continuing work with youth on their Individual Service Plans, creates a personal bond and magnifies the power of the personal connection with the program making the service experience all about them and likely contributes to persistence.
• Policy makers need to be aware that Assessment has often been used ineffectively and has come to have negative connotations associated with it, especially when it is done to a participant. Consider the impact of high stakes assessment in our schools on student and parent attitudes about assessment and they bring those attitudes with them to WIOA programs. However, when individual participants are professionally guided on how to use and apply their own assessment results the experience can be quite different and developmentally productive.

References

Barnow, G., and Smith, Jeffrey. “Performance Management of U.S. Job Training Programs:  Lessons from the Job Training Partnership Act.” Public Finance and Management, vol. 4, no. 3, 2004, pp. 247-287.

Borden, William S., Nov. 2009 Mathematica Policy Research, The Challenges of Measuring Employment Program Performance, in a paper delivered at the 2009 conference What the European Social Fund Can Learn from the WIA Experience. Retrieved from http://www.umdcipe.org/conferences/WIAWashington/Papers/Borden-Challenges-of-Measuring_Employment_Program_Performance.pdf

Dunham, K., Mack, M., Salzman, J., Wiegand, A., May 2006, Evaluation of the WIA Performance Measurement System – Final Report. Retrieved from https://wdr.doleta.gov/research/FullText_Documents/Evaluation%20of%20the%20WIA%20Performance%20Measurement%20System%20-%20Final%20Report.pdf

Hossain, Farhana (2015). Serving Out-of-School Youth Under the Workforce Innovation and Opportunity Act (2014). Retrieved from
http://www.mdrc.org/publication/serving-out-school-youth-under-workforce-innovation-and-opportunity-act-2014

Kazis, Richard (2016). MDRC Research on Career Pathways. Retrieved from http://www.mdrc.org/publication/mdrc-research-career-pathways

King, C. T. (2004). The effectiveness of publicly financed training services: Implications for WIA and related programs. In C. J. O’Leary, R. A. Straits, & S. A. Wandner (Eds.), Job training policy in the United States. Kalamazoo, MI: W. E. Upjohn Institute for Employment Research.

Moore, R. W., Gorman, P. C., Blake, D. R., Phillips, G. M., Rossy, G., Cohen, E., Grimes, T., & Abad, M. (2004). Lessons from the past and new priorities: A multi-method evaluation of ETP. Sacramento, CA: California Employment Training Panel.

Social Policy Research Associates. (2004). The Workforce Investment Act after Five Years: Results from the National Evaluation of the Implementation of WIA. Retrieved from https://www.doleta.gov/reports/searcheta/occ/papers/spr-wia_final_report.pdf

U.S. Departments of Education, Health and Human Services, and Labor, “Joint Career Pathways Letter,” April 4, 2012. Retrieved from http://www2.ed.gov/news/newsletters/ovaeconnection/2012/04122012.html.

United States Department of Labor, Employment and Training Administration (2015). Retrieved from https://www.doleta.gov/performance/guidance/laws_regs.cfm

United States Departments of Education, Health and Human Services, and Labor, “Joint Career Pathways Letter,” April 4, 2012. Retrieved from http://www2.ed.gov/news/newsletters/ovaeconnection/2012/04122012.html.

Note 1. Univariate Analysis of Variance of Literacy by InnerSight and Age, Out-of-School Youth, 2014-15

Note 2. Univariate Analysis of Variance of Attainment by InnerSight and Age, Out-of-School Youth, 2014-15