Overview
Connecting Out of School Youth Preferences to Career Plans Impacts YouthSource Program Performance
The City of Los Angeles Economic and Workforce Development Department’s YouthSource program is a premier provider of service for in and out-of-school youth. The city serves some 17,000 youth in its annual and summer programs. The Los Angeles YouthSource program has been nationally acclaimed by the Department of Labor for its progressive commitment to serving out-of-school youth. In 2012 before the new WIOA requirements, the City made a substantial commitment to serving this population requiring 70% of the youth served at its 13 YouthSource sites be out-of-school youth. As a result, they have become the nation’s most successful program in addressing the needs of this underserved population.
In 2012 the YouthSource program implemented the InnerSight Assessment Experience to:
- Actively engage participants in the development of their individual plan.
- Provide caseworkers with a foundation for development of individual plans grounded in participant’s assessment results.
- Connect individual plans and strategies to specific educational, training and occupational goals.
- Help caseworkers and clients see those skills needed to achieve occupational fit.
- Provide documentation of participant and case manager mutual engagement and results oriented planning.
A recently released study “Assessment Program Impact on Successful WIOA Program Performance in the City of Los Angeles YouthSource Program” shows that out-of-school youth participating in the City’s InnerSight Assessment Experience achieve the educational attainment outcome measure at a significantly greater rate. Forty-three percent (43%) of the out-of-school youth who have the assessment experience achieve the attainment goal for a statistically significant 14% improvement over the 29% attainment rate of those without the experience.
The study also revealed a 48% success rate in achieving the literacy standard for youth participating in the InnerSight Assessment Experience versus a 42% success rate for nonparticipants. Findings indicate success rates for the WIOA critical population of out-of-school youth who complete the InnerSight Assessment Experience are significantly higher than their peers on the Attainment and Literacy performance measures.
The City selected this assessment program to help WIOA staff align youth and adults with education, training programs, internships, and actual job openings that are a best fit for them. The findings affirm that when interests and preferences are aligned and self-validated by the individual, their success rate increases.
Through the InnerSight Assessment Experience, youth come to see they are the “Key” to their personal, educational, and career training choices and gain important information about themselves to reflect upon when making those choices. After self-validating their personal interests and preferences for working, learning, leading, risk taking, and team orientation, participants are equipped to explore occupations of people who are satisfied in various occupations and share the participants’ interests.
When asked what they liked most about the Experience youth say, “Keep up the work. There are many people out there like me who are lost and not sure what to do. But with this book and your help, there are paths that seem to open & lighten up.” and “This experience helped me get a clearer understanding of what I would like in my future career and I learned a lot about myself.”
Knowing Who You Are… Is the First Step …In Knowing Where You Are Going!
Full Study
Assessment Program Impact on Successful WIOA Program Performance in the City of Los Angeles YouthSource Program
Mark L. Perkins, John E. Reeves, Lolita Mancheno-Smoak, Debora K. Furlong
Abstract
The Workforce Innovation and Opportunity Act (WIOA) measures labor market outcomes as indicators of program performance. This article uses statistical analysis to examine the impact of the InnerSight Assessment Experience on the rate of performance on key outcome measures in the City of Los Angeles YouthSource Program. The findings suggest that the InnerSight Assessment Experience has a positive impact on youth performance on the WIOA performance measures of Attainment and Literacy. The impact is most profound for out-of-school youth, the target service population under WIOA. Implications and considerations for policy makers and future research are provided.
Purpose
The City of Los Angeles Economic Workforce Development Department (EWDD) offers the InnerSight Experience as a consistent program element across its 13 YouthSource service sites. This study examines the impact of this common program element on youth development as indicated by performance on Workforce Innovation and Opportunity Act (WIOA) performance measures of placement, literacy and attainment.
Introduction
The city of Los Angeles YouthSource program is a premier provider of service for in and out-of-school youth. The city serves some 17,000 youth in its annual and summer programs. The Los Angeles YouthSource program has been nationally acclaimed by the Department of Labor for its progressive commitment to serving out-of-school youth. In 2012 before the new WIOA service requirements, the City made a substantial commitment to serving this population when it required that 70% of the youth served at its 13 YouthSource sites be out-of-school youth. As a result, they have become the nation’s most successful program in addressing the needs of this underserved population.
YouthSource program leadership desired to offer a consistent program element to serve youth across its 13 sites. Most importantly they wanted a program element that is developmental for each participant while meeting WIOA expectations and requirements. Specifically, they were seeking a program that would accomplish the following:
- Actively engage participants in the development of their individual plan.
- Provide caseworkers with a foundation for development of individual plans grounded in participant’s assessment results.
- Connect individual plans and strategies to specific educational, training and occupational goals.
- Help caseworkers and clients see those skills needed to achieve occupational fit.
- Provide documentation of participant and case manager mutual engagement and results oriented planning.
They believe a program grounded in participant personal development that connects youth with high demand occupations using their preferences and interests will facilitate persistence in the attainment of educational, training and career goals. To accomplish this, the city issued an RFP and selected the InnerSight Assessment Experience to meet this unique program and developmental objective.
The InnerSight ExperienceTM
The InnerSight Assessment Experience consists of three important steps. First, participants take an online inventory to provide information to clarify their interests and preferences. Second, they participate in a self-validating group interpretation of the inventory known as the InnerSight Experience™. Third, participants share their results with their case managers and lay out the steps in their individual service plan.
The InnerSight Experience gives participants a vocabulary, context, and framework for bringing their personal interests and preferences to bear on life’s choices. This experience puts the “Power of the Person” in the decision-making processes of life. Through the Experience, participants come to see they are the “Key” to their personal, educational, and career training choices and gain important information about themselves upon which to reflect when making those choices.
The InnerSight Experience is collaborative and delivered in a group setting. This interactive experience engages participants in both individual and group exercises that emphasize and highlight the distinctiveness of their personal results.
The InnerSight Assessment Experience consists of an online interest and preference inventory resulting in a personalized InSight Guide® booklet which is the foundation for a 3-hour, in person, group interpretative experience. By self-validating their personal interests and preferences for working, learning, leading, risk-taking, and team orientation, participants are equipped to explore occupations of people who are satisfied in various occupations and share the participants’ interests.
Participants learn that individuals are typically employed for what they know and terminated for what they do or fail to do. When interests and preferences are aligned with an occupation (work tasks), satisfaction, happiness, and superior performance occur. Youth with cultural, familial, and economic barriers are no exception. When engaged in subjects of interest (music, dance, skateboarding, etc.), individuals succeed.
The InnerSight Experience™ is grounded in the prolific research and universally acclaimed Work Types Model of the eminent vocational psychologist Dr. John Holland. The Holland Model was adopted by the Department of Labor as the foundation for their online career exploration system O*NET. This system can be directly accessed from the electronic Insight Guide booklet participants receive after completing the InnerSight Experience by simply clicking on a preferred occupation.
For over 75 years, interests and preferences have been known as key factors in motivation for continued effort. First used in World War I to determine which men might prefer to be cooks or which might prefer to be in the cavalry, interests have proven to be an effective method of determining continued effort/success in education, career, and personal life. Using interests and preferences to help individuals make better life choices is a time-tested and valid approach.
InnerSight uses this approach, but has changed the traditional career assessment/exploration model. Youth are accustomed to being processed, asked what they know (education), or told what they need to know and do next. InnerSight begins the journey by first exploring and having participants validate what they may like or enjoy doing. Additional exercises guide the participants to discover the connection between what they enjoy doing and potential occupations in which people who share their interests are satisfied.
The InnerSight Assessment Experience continues when participants meet with their case managers to begin Next Steps planning in the InSight Guide booklet based on their self-validated results. Case managers who work with the youth complete the InnerSight Experience and training in “Facilitating the Journey”. “Facilitating the Journey” training provides case managers with the simple skills needed to engage the participant in a meaningful discussion of their results and use them as the foundation of the individual service plan.
Each InnerSight Experience session is evaluated by the participants. Their ratings provide continuous feedback on characteristics of the Experience from the InSight Guide booklet to facilities and the accuracy of their results. In addition, they are asked to respond to three open-ended questions regarding what they liked, what they would change, and any additional thoughts. These personal responses provide a rich source of qualitative information for understanding in participants’ words how they were impacted by the Experience.
The InnerSight Assessment Experience is designed to help WIOA staff align youth and adults with education, training programs, internships, and actual job openings that are a best fit for them. This study now examines the impact of the InnerSight Experience on WIOA performance measures.
Literature
The literature is replete with studies and meta-analyses of workforce investment program performance. It has long been the interest of policymakers for job-training programs to be evaluated using tangible labor market results for participants. King (2004) notes the Department of Labor has funded a host of studies in recent decades to assess the impact of training programs using such measures.
Moore and Gorman (2009) observed; “Policymakers brought this focus on performance to the creation of the Workforce Investment Act (WIA) in 1998. WIA, which replaced the Job Training Partnership Act (JTPA), established 17 performance measures to drive program performance.” They go on to observe that most of the 17 metrics were for labor market outcome indicators, such as changes in participant earnings, rate at which participants are placed in jobs, etc.
Social Policy Research Associates (2004) in an early assessment of WIA implementation noted that the 17 measures made sense to program operators but concluded that some operators felt definitions were vague and perhaps there were too many. As WIA evolved, the Federal Department of Labor elected to move to a new measurement system consisting of fewer labor market outcomes known as “common measures”. Dunham, Mack, Salzman and Wiegand (2006) note that these “common measures were to be implemented for adults beginning in Program Year (PY) 2005, which began July 1, 2005, and for youth in PY 2006 (beginning July 1, 2006)”. (pg.22)
While there is a plethora of publicly available performance data, Moore and Gorman (2009) observe “there is little in depth analysis of the performance of the WIA system or of the likely drivers of the new common measures. Specifically, there has been little published on the relationship between individual participant characteristics and the program performance measures.” In fact, the Moore and Gorman, 2009 study on the impact of training and demographics in WIA program performance is the first we could find examining how the characteristics of participants may or may not influence the outcomes measured.
As might be expected they found demographic characteristics such as age, ethnicity, education etc. account for considerable variance in performance outcomes. These findings led Moore and Gorman to the following policy and program conclusions:
“A system of adjusting performance standards for states, local areas, and agencies that serve more disadvantaged populations is rational and would make good public policy.” (p. 393)
“Policymakers should track whether or not the demographics of participants shift away from the most disadvantaged as the WIA-mandated outcome measures change to the new universal performance measures……. One-Stop operators and local area WIBs both have incentive to perform well on the outcome measures, and when outcome measures can be affected simply by adjusting the demographic profile of those served, it would not be surprising if One-Stops and local area WIBs were to shift their resources toward serving those more likely to help improve their outcome measures, rather than those most in need.” (p. 393)
“Data suggest relatively long-term and more expensive training services are not uniformly superior to shorter-term, less costly interventions such as job search assistance and one on one coaching and counseling.” (p. 393)
“No positive impact was found for any form of training on adult WIA participants… This does not mean that participants never benefit from training, but rather that the training intervention needs to be carefully targeted both toward individuals who can benefit from training and toward skills that are demanded by the local labor market. Simply loading WIA participants into available skill training without an individual strategy for each participant is not likely to yield positive outcomes. For dislocated workers, occupational skills training appeared to have a strong positive effect on earnings, and other forms of training were associated with lower earnings. These findings need to be considered by practitioners as they design interventions for individual participants.” (p. 394)
Moore and Gorman identify the profound impact of demographics on WIA performance measures. They go further suggesting “there is a need for researchers to undertake more nuanced studies of the connections between training and labor market outcomes in WIA, with carefully controlled studies to identify the types of participants and types of training most likely to lead to successful outcomes.” (p. 394) This suggests a need for providers to understand what program elements, assessments, or experiences have a positive impact on participant’s performance on the common WIA metrics.
Borden (2009), in a comprehensive article on the challenges of measuring employment program performance, says “research on employment and training programs focuses primarily on evaluations of the impact of public investment in job-training services”. (p. 2) He goes on to suggest that there is indeed a dichotomy between the program management objectives of performance management and its evaluative objectives. Moore and Gorman focused on the impact of demographic characteristics and their relationship to the achievement of performance measures. This study goes a step further by analyzing the impact of program on the successful achievement of desired performance outcomes and to identify those elements that drive desired job training performance.
Barrow and Smith (2004) clearly suggest that program evaluation and performance management derive from different sources and motives and have deeply committed adherents. The question is not whether we should track or measure program performance but rather how do program elements contribute to the achievement of job-training and employment program objectives. As suggested by Borden (2009), “there is an increasing tendency to leverage the efficiency of analyzing administrative data versus more expensive experimental designs”. (p. 5) The truth is both are needed, but the experimental designs with appropriate stratified samples of participants for randomization with clean and accurate participant data are not only more expensive, but practically impossible to achieve. Thus, impact studies with limitations may be the most informative indicators of positively contributing program elements for achieving job-training program results. Most importantly, as suggested by Borden, a good measure must produce a rate of success and not simply a count of activities.
Borden properly points out that “we must distinguish clearly between service delivery and program management. Performance management systems track common events such as enrollment date, customer characteristics, limited service dates, exit date, and outcomes. Performance management systems do not specify how services are delivered.” (p. 21) nor do they indicate how services impact performance management outcomes. While it is impossible to design measures that account for all factors bearing on the success with a single customer, focused impact studies can be helpful in determining a program or assessment’s potential accountability for or contribution to program outcomes. With clear program performance measures in place we can begin the analysis of those processes and methods that produce the best results for program participants.
The first major update in almost 15 years to the Workforce Investment Act of 1998 (WIA) was signed into law in 2014. The new federal Workforce Innovation and Opportunity Act (WIOA) places greater emphasis on serving out-of-school youth (75% versus 30% under WIA). WIOA defines out-of-school youth as 16 to 24-year-olds who are not attending any school and who have one or more barriers to employment, such as young people who are homeless, are parenting, have disabilities, or have a juvenile criminal record (United States Department of Labor, Employment and Training Administration (2015)). The new law requires states and localities to develop strategies and programs for recruiting and serving more of these young people than ever before.
This could be a daunting task. Hossain (2015) observes that “while a majority of the out-of-school youth seek out opportunities to connect training and work, youth programs often report difficulties in sustaining participation after the initial connection is made. WIOA-funded service providers will not only have to reach more out-of-school youth, they will also need strategies to stimulate sustained, intense engagement in services.” (p. 1) She goes on to observe that “few programs target the young people who are the most persistently disconnected, and there’s not much evidence on what works in engaging them.” (p. 3) Perhaps most importantly she says that “a significant share of out-of-school youth do not enroll in education and training programs because they have been alienated from mainstream institutions, like schools and social welfare agencies, due to earlier negative experiences. New strategies to reach and engage alienated and disaffected young people should be a priority.” (p. 3)
Hossain (2015) suggests programs “have to find a balance between allowing vulnerable young adults some flexibility in regard to program requirements… while developing processes and practices that allow young people to develop autonomy and leadership” (p. 4). Specifically, she recommends “asking young people for their input in designing program activities and allowing them to have a voice in program governance.” (p. 4)
WIOA encourages implementation of career pathway approaches that will support post-secondary education and training for out-of-school youth that is related to the demand for qualified workers in local labor markets. The individualized pathway is to be grounded in an assessment of skills and interests that can inform the participants’ career decision-making while identifying logical short-term next steps consistent with long-range goals.
The US Departments of Education, Health and Human Services, and Labor issued a joint letter in 2012 that provided a common definition of career pathways that encompasses initiatives that begin with secondary or post-secondary education and credentials. According to the letter a career pathway approach is:
“…A series of connected education and training strategies and support services that enable individuals to secure industry relevant certification and obtain employment within an occupational area and to advance to higher levels of future education and employment in that area.”
Richard Kazis in a MDRC research brief on Career Pathways (2016) observes “there is little rigorous research that assesses the impact of comprehensive career pathways programs that follows individuals from different starting points through a coherent set of educational experiences and “stackable” credentials to the labor market.” (p. 2) His research brief however, goes on to note that there are components that are being studied that might help policymakers as well as program designers and managers. He provides brief summaries of project models that make a difference in several categories including high schools, out-of-school and disconnected youth, and low income adults. Kazis suggests there is a paucity of research on how to positively engage the participant in his/her personal development of a successful career pathway.
Research has primarily focused on program performance, yielding discussions on topics such as the importance of the employer network, career demand, and issues of program integration, curriculum and process. All of these focus on what is “done to” the participant and rarely on what is “done with them”. With the exception of Moore and Gorman who examined the impact of demographic characteristics on program performance measures there has been virtually no research examining the impact of program elements offered by providers on WIOA performance indicators. This study presents an analysis of a robust population of WIOA participants from a large diverse city exploring the relationship between the participant’s program assessment experience and subsequent performance on WIOA common measures.
Impact Analysis Approach
As noted by Moore and Gorman (2009) and Borden (2009), the gold standard random-assignment experimental design is extremely difficult and virtually cost prohibitive to use with live WIOA programs. This is perhaps why there is virtually no research on the impact of individual program elements on the common WIOA performance measures. As a result, program providers are left with little or no information regarding program elements that significantly impact participant performance on program performance indicators.
While this study might be referred to as a quasi-experimental design, the authors prefer to call it an impact analysis. This is a first step in identifying the impact of an assessment experience on participant’s achievement on WIOA outcome measures. Essentially, it would be helpful to know if the assessment experience significantly impacts youth performance on program outcome measures.
Method
Research Questions: This study analyzed youth participant performance on WIOA outcome measures for youth participating in the City of Los Angeles YouthSource program in program year 2014 – 2015 (PY-1415) to answer the following questions:
- Do youth participants who have the InnerSight Assessment Experience achieve satisfactory performance on the WIOA outcome measures of attainment, literacy and placement at a significantly greater rate than those who have not had the Experience?
- Do out-of-school youth participants who have the InnerSight Assessment Experience achieve satisfactory performance on the WIOA outcome measures of attainment, literacy and placement at a significantly greater rate than out-of-school youth who do not have the Experience?
Study Population: The study population consists of in-school and out-of-school youth participating in the City of Los Angeles YouthSource program at one of 13 provider sites across the City. Study participants are youth the City of Los Angeles included in the performance analysis for each of three WIOA performance outcome criteria for program year 2014–15. Performance of youth participants who have completed the InnerSight Assessment Experience (Experimental Group) is compared with those who did not complete the Experience (Control Group). The outcome performance measures for this study were drawn from the City of Los Angeles “Jobs LA” performance tracking system. The outcome measures for this study are described below.
Literacy– Literacy & Numeracy Gains
This indicator measures the increase by one or more educational functioning levels from out-of-school youth who are basic skills deficient within a one-year period of participating in program services.
Meeting the Standard: Youth meet the outcome standard for this measure if they:
- Show increase in skills as measured by the Comprehensive Adult Student Assessment System (CASAS) Life & Work (reading) and Life Skills (math) Series at program entry within one year of start of program services.
- Increase one educational function level (EFL) in math, reading or both, during each year of program participation.
Placement -Placement in Employment, Education, or Training
This indicator measures whether a participant is in an education or training program that leads to a recognized post-secondary credential or unsubsidized employment in 1st Quarter after program exit. This measure requires a quarterly follow up for one year after program exit
Meeting the Standard: Youth meet the outcome standard for this measure if they meet one of the following:
- Employed by an unsubsidized organization.
- Entered military service.
- Enrolled in a post-secondary education program.
- Enrolled in an advanced training or occupational skills training program.
Attainment -Credential Rate
This indicator measures the attainment of a high school diploma, GED, or certificate during program participation or by the end of the 1st Quarter after program exit.
Meeting the Standard: Youth meet the outcome standard for this measure if they meet one of the following:
- Receive a High School diploma certifying completion of a secondary school program of studies.
- Receive satisfactory scores on the General Education Development (GED) test.
- Receive a formal award certifying the satisfactory completion of an organized program of study at a post-secondary education program.
Table 1 provides demographic information for the InnerSight Assessment Experience and Control Groups for Program Year 2014-15. The average age for the three study groups ranges from 18.3 to 19.1 years. Since the cohort deemed appropriate for each individual outcome is different it’s helpful to understand the demographics for each outcome cohort as reflected in Table 1.
Literacy: The literacy outcome measure is appropriate only for out-of-school youth. Out of the total 1665 participants, 270 (16%) completed the InnerSight Assessment Experience while 1395 (84%) did not have the Experience.
Placement: The placement outcome measure was deemed appropriate for a total of 2085 youth participants of which 557 were in-school-youth (27%) and 1528 were out-of-school youth (73%). A total of 362 participants completed the InnerSight Assessment experience (17%) while 1723 in the control group did not have the Experience (83%).
Of the in-school youth, 111 or 20% completed the InnerSight Assessment Experience while 446 or 80% in the control group did not. Of the out-of-school youth participants for this measure, 251 or 16% completed the InnerSight Assessment Experience while 1277 or 84% in the control group did not.
Attainment: The attainment outcome measure was deemed appropriate for a total of 1547 youth participants of which 564 were in-school-youth (36%) and 983 were out-of-school youth (64%). A total of 305 participants completed the InnerSight Assessment experience (20%) while 1242 (80%) in the control group did not have the Experience.
Of the in-school youth, 111 or 20% completed the InnerSight Assessment Experience while 453 or 80% in the control group did not. Of the out-of-school youth participants for this measure 194 or 20% completed the InnerSight Assessment Experience while 789 or 80% in the control group did not.
Table 1
Study Participant Demographic Information by InnerSight Assessment Experience and Control Groups
for Program Year 2014-15
| Participant Groups | Literacy Assessmenta | Placement Assessment | Attainment Assessment | |||
| N | Average Age | N | Average Age | N | Average Age | |
| Overall | 1665 | 19.1 | 2085 | 18.6 | 1547 | 18.3 |
| InnerSight | 270 | 19.0 | 362 | 18.3 | 305 | 18.3 |
| Control | 1395 | 19.1 | 1723 | 18.6 | 1242 | 18.3 |
| In-school youth: | ||||||
| InnerSight | NA | 111 | 17.1 | 111 | 17.1 | |
| Control | NA | 446 | 17.3 | 453 | 17.3 | |
| Out-of-school youth: | ||||||
| InnerSight | 270 | 19.0 | 251 | 18.9 | 194 | 18.9 |
| Control | 1395 | 19.1 | 1277 | 19.1 | 789 | 18.9 |
| a -All participants on Literacy Outcome measure are Out-of-School Youth. | ||||||
Information for gender and ethnicity of participants was not available in the data extract provided by the City of Los Angeles from the “Jobs LA” performance tracking system. However, the City of Los Angeles did provide total program gender and ethnicity information for program year 2014-15 reflecting a robust and diverse population of participating youth. For 2014-15 there were 1947 females (54.4%), 1630 males (45.6%). The reported ethnic makeup as expected is very diverse with the largest reporting group being Hispanic or Latino followed by White and African American/Black. The mix of Race or Ethnicity in the population is so large that participants are permitted to select more than one race or ethnicity suggesting that the standard race identification categories are rapidly becoming ineffective demographic descriptors or study variables for this population.
Analysis: As Borden (2009) suggests, a good measure must produce a rate of success and not simply a count of activities. Therefore, the rate of success on each of the WIOA performance measures (literacy, placement and attainment) will be calculated for participants appropriate for assessment on each measure as an indicator of program performance. To assess the impact of the InnerSight Assessment Experience, the percent succeeding on each measure will be compared for those who had the InnerSight Assessment Experience with those who did not have the Experience (control group). Differences in rate of performance will be analyzed using chi-square to determine if they are significant or if they occurred by chance alone.
Similar analysis will be conducted within each measure where appropriate to compare the rate of performance of in-school-youth and out-of-school youth who have had the InnerSight Assessment Experience with those who have not.
Analysis of variance will be used where appropriate to identify any within group or interactive differences that may be associated with age of the participant.
Results
The performance rates of participating youth successfully meeting the performance standard for program year 2014-15 are provided in Table 2 for each of the three outcome measures. The overall rate of successful performance is 70% for Placement, 49% for Attainment and 43% for Literacy. Table 2 reveals considerable variance in performance rates between in-school youth and out-of-school youth as well as between those who have had the InnerSight Assessment Experience and those who have not on the Literacy and Attainment outcomes. Performance rate comparisons and examination of significant differences are provided below by WIOA outcome performance measure.
Table 2
Rate of Successful Performance 2014-15:
Percent of Participating Youth Achieving Outcome Measure
| Literacy Outcome1 | Placement Outcome | Attainment Outcome | |
| Overall | 43% | 70% | 49% |
| InnerSight | 48% | 69% | 58% |
| Control | 42% | 70% | 47% |
| In-school youth: | 73% | 80% | |
| InnerSight | – | 71% | 85% |
| Control | – | 73% | 79% |
| Out-of-School youth: | 43% | 69% | 32% |
| InnerSight | 48% | 69% | 43% |
| Control | 42% | 69% | 29% |
| 1All participants on Literacy Outcome measure are Out-of-School Youth. | |||
Literacy: Only out-of-school youth participate in the literacy outcome measure. For the 2014-15 program year population, 43% met the standard. Those who had The InnerSight Assessment Experience had a 48% success rate while those who did not achieved a 42% success rate. This suggests InnerSight Assessment Experience participants are more likely than other participants to achieve the literacy standard. The 6% difference in performance rate for those completing the InnerSight Assessment Experience is statistically significant yielding a χ2 value of 3.99 with 1df, N of 1665 and a probability of .046.
Table 3 contains the distribution of participants and their literacy outcomes by age. While participants range in age from 15 to 21, over 99% fall between the age of 17 and 21. Table 3 contains performance success rate by study group and age. Figure 1 provides this information graphically across age for both study groups (InnerSight and control).
Table 3
Percent of Participants Achieving Literacy Standard
by Study Group and Age
| Age | Control | InnerSight | ||
| N | Percent Achieving Standard | N | Percent Achieving Standard | |
| Under 18 | 111 | 34% | 26 | 38% |
| 18 | 355 | 29% | 75 | 41% |
| 19 | 428 | 48% | 78 | 51% |
| 20 | 284 | 45% | 58 | 53% |
| 21 | 217 | 48% | 33 | 55% |
| All | 1395 | 42% | 270 | 48% |
InnerSight Assessment Experience participants have a higher rate of success than the control group across all age groups as can be noted in Table 3 and the graph in Figure 1. It also appears that older participants achieve a higher success rate than younger participants. An analysis of variance of the performance on the literacy standard examined the impact of age and the InnerSight Assessment Experience. Both the InnerSight Assessment Experience and Age have statistically significant impacts on the rate of Performance on the Literacy Outcome, producing values of F=9.719, p<.010, and F=17.001, p<.009 respectively. This suggests the InnerSight Assessment Experience and age have a positive effect on the outcome.
There was no interaction effect between the InnerSight Assessment Experience and age on the Literacy performance outcome. The statistical analysis of this effect yielded a value of F=.266, p<.900. Thus the positive increase in Literacy performance rate attributable to the InnerSight Assessment Experience is not affected by participant age. (See note 1 for complete ANOVA results.)
Placement: For the 2014–15 program year, 70% met the placement outcome standard. Table 1 shows that the rate of achievement on the Placement standard does not differ between in-school and out-of-school youth or between InnerSight Assessment Experience participants and the control group. Table 4 contains the number of participants and associated success rates by age, study group (InnerSight and Control) and school status (in-school and out-of-school). Statistical analysis revealed no significant differences in performance rates associated with participation group, school status, or age. This suggests that neither the InnerSight Assessment Experience, nor youth school status or age impacts the success rate on the placement outcome measure.
Table 4
Distribution of Participants and Rate of Success on Placement Outcome
by Age, School Status and Study Group
| Participant Groups | Under 18 | 18 | 19 | 20 | 21 | All | ||||||
| N | % a | N | % a | N | % a | N | % a | N | % a | N | % a | |
| Overall | 553 | 72% | 510 | 62% | 499 | 75% | 307 | 74% | 216 | 64% | 2085 | 70% |
| InnerSight | 116 | 70% | 89 | 65% | 86 | 74% | 47 | 72% | 24 | 58% | 362 | 69% |
| Control | 437 | 73% | 421 | 61% | 413 | 76% | 260 | 75% | 192 | 65% | 1723 | 70% |
| In-school youth: | 412 | 75% | 112 | 63% | 25 | 72% | 5 | 60% | 3 | 67% | 557 | 73% |
| InnerSight | 88 | 72% | 16 | 63% | 5 | 100% | 2 | 50% | 0 | ~ | 111 | 71% |
| Control | 324 | 76% | 96 | 64% | 20 | 65% | 3 | 67% | 3 | 67% | 446 | 73% |
| Out-of-School youth: | 141 | 62% | 398 | 61% | 474 | 76% | 302 | 75% | 213 | 64% | 1528 | 69% |
| InnerSight | 28 | 64% | 73 | 66% | 81 | 73% | 45 | 73% | 24 | 58% | 251 | 69% |
| Control | 113 | 62% | 325 | 60% | 393 | 76% | 257 | 75% | 189 | 65% | 1277 | 69% |
| a % is the rate of performance on the outcome. | ||||||||||||
Attainment: For the 2014-15 program year, 49% successfully met the attainment outcome measure standard as can be seen in Table 5. The performance measure success rate for those who had the InnerSight Assessment Experience was 58% while the success rate for those who did not have the Experience was 47%. This suggests that those who have the InnerSight Assessment Experience are more likely to achieve success on the attainment outcome performance measure than those who do not. The 11% difference in performance rate for InnerSight Assessment Experience participants was statistically significant, yielding a χ2 value of 11.36, with 1df, N=1547 and a probability of p = .001.
Table 5
Distribution of Attainment Participants and Percent Achieving Attainment by Age,
Study Group and School Status
| Participant Group | Under 18 | 18 | 19 | 20 | 21 | All | ||||||
| N | % a | N | % a | N | % a | N | % a | N | % a | N | % a | |
| Overall | 516 | 78% | 427 | 39% | 285 | 34% | 195 | 29% | 124 | 33% | 1547 | 49% |
| InnerSight | 110 | 84% | 75 | 44% | 59 | 42% | 39 | 41% | 22 | 50% | 305 | 58% |
| Control | 406 | 76% | 352 | 38% | 226 | 32% | 156 | 26% | 102 | 29% | 1242 | 47% |
| In-school-youth: | 413 | 90% | 115 | 62% | 28 | 43% | 5 | 20% | 3 | 0% | 564 | 80% |
| InnerSight | 88 | 92% | 16 | 50% | 5 | 80% | 2 | 50% | 0 | ~ | 111 | 85% |
| Control | 325 | 89% | 99 | 64% | 23 | 35% | 3 | 0% | 3 | 0% | 453 | 79% |
| Out-of-School youth: | 103 | 31% | 312 | 31% | 257 | 33% | 190 | 29% | 121 | 34% | 983 | 32% |
| InnerSight | 22 | 50% | 59 | 42% | 54 | 39% | 37 | 41% | 22 | 50% | 194 | 43% |
| Control | 81 | 26% | 253 | 28% | 203 | 32% | 153 | 26% | 99 | 30% | 789 | 29% |
| a % is the rate of performance on the outcome. | ||||||||||||
The impact of the InnerSight Assessment Experience on the Attainment outcome differs dramatically for in-school and out-of-school youth. The success rate of 85% for in-school youth on the attainment measure who had the InnerSight Assessment Experience was not significantly greater than the 79% success rate for in-school youth who did not have the Experience. The statistical analysis yielded a χ2 value of 1.544 with 1df and an N=564 with a probability of p = .214.
Further review of Table 5 for in-school- youth reveals a substantial change in performance rate for youth under 18 and those 18 and older. When examined by these age groupings a performance rate of 90% is observed for the 413 in-school youth under 18 compared to a performance rate of 57% for the 151 in- school youth who are 18 and older. The 33% difference in performance is statistically significant yielding a χ2 value of 81.222 with 1df and an N=564 with a probability of p = .000. When analyzed separately by InnerSight and control group the results are similar. This seems to suggest a powerful effect for age or life events associated with age.
Examination of the success rates for the WIOA critical population of out-of-school youth shows that those who complete the InnerSight Assessment Experience are significantly more likely than their peers to achieve success on the Attainment performance measure. The success rate on the Attainment outcome measure for those having the InnerSight Assessment Experience is 43% versus 29% for those who do not. This 14% difference in the performance rate is statistically significant yielding a χ2 value of 14.6 with 1df, an N of 983 and a probability of p = .00.
For out-of-school youth, a univariate Analysis of Variance of the InnerSight Assessment Experience and age shows that there is a statistically significant difference (F=26.210, p<.002) between the InnerSight group and the Control group, but age has no impact. Furthermore, there was no interaction effect of age with the InnerSight Assessment Experience (F=.527, p<.716). Thus the InnerSight Assessment Experience makes a significant difference and is not impacted by participant age. Figure 2 shows that the InnerSight Assessment Experience impact on Attainment is not isolated to only the youngest or the oldest participants. (See Statistical Analysis note 2 for complete ANOVA results.)
Examining research question one, the results show that youth who have the InnerSight Assessment Experience achieve successful performance on the attainment and literacy performance measures at a significantly greater rate than those who do not have the Experience. With regard to the placement outcome measure, rate of performance is not impacted by the InnerSight Assessment Experience for any group.
On research question two, the results show a significant difference between the successful performance rate of out-of-school youth (WIOA’s primary target group) who have the InnerSight Assessment Experience and those who do not on the attainment and literacy outcome measures. Out-of-school youth who had the InnerSight Assessment Experience performed significantly better on the attainment outcome measure (+14%) and on the literacy outcome measure (+6%) than those who did not have the Experience.
Discussion
The literature offers virtually no research on program elements that impact youth success on WIOA performance outcomes. Existing research (Moore and Gorman, 2009) focused only on the impact of participant demographic characteristics on outcome measures. Borden (2009) noted that performance management systems do not specify how services are delivered and that measures of program performance must produce a rate of success and not simply a count of activities. This study responds to both of these concerns with a focus on the InnerSight Assessment Experience as a program element in service delivery and its impact on participant rate of performance on WIOA outcome measures.
The finding that out-of-school youth meet the attainment performance standard at a 14% greater rate if they have the InnerSight Assessment Experience (49%) than those who do not (29%) is not only statistically significant but makes it clear that program elements influence participant performance on WIOA outcome measures. Results were similar on the literacy outcome measure with a 6% significantly greater rate of performance for out-of-school youth who had the InnerSight Assessment Experience (48%) than those who did not (42%).
The target population for this study was young people who are the most persistently disconnected. The findings provide evidence that the InnerSight Assessment Experience is an intensive interpretive program element that works in engaging them. When connected with case manager follow-up using the InnerSight ExperienceTM results to chart next steps related to educational and occupational goals, the results are personally engaging, developmental, and productive. This is one of those new strategies Hossain (2015) says is needed and should be a priority in reaching and engaging alienated and disaffected young people.
Hossain recommends “asking young people for their input in designing program activities and allowing them to have a voice in program governance.” (p.4) The InnerSight Assessment Experience capitalizes on this by increasing the youth’s voice in the use of their personal assessment results in designing a personal pathway to success. This is particularly productive in establishing individualized short-term and long-term goals as required by WIOA in each youth’s Individual Service Plan.
Kazis (2016) suggests there is a paucity of research on how to positively engage the participant in his/her personal development of a successful career pathway. The findings of this study respond to this concern and suggests a useful building block is the effective use of personal assessment results. Far too many times, assessment is viewed as something we do to people, or for them, rather than with them. Thus we lose the opportunity for meaningful engagement, personal exploration and the power of personal validation is lost. Such engagement takes time and some professional expertise with the assessment tool which often is not available in many WIOA programs but in this study was provided through the InnerSight Assessment Experience. Case managers are easily trained to work with youth participants and the results in their InSight Guide results booklets to discuss and plan together important educational and career next steps. Such collaborative work leads to the program impact on performance outcomes found in this study.
The placement outcome measure performance rate was not influenced by the InnerSight Assessment Experience and is most likely a result of individual provider agency contacts and networking skills rather than the personal development of the youth participant. This would be an interesting area for further analysis and study.
The results for in-school youth revealed a significant effect for age on the attainment success rate. There was a substantial change in performance rate for youth under 18 when compared with those 18 and older. A performance rate of 90% was observed for the 413 in-school youth under 18 compared to a performance rate of 57% for the 151 in-school youth who are 18 and older. Moore and Gorman (2009) in their study of the impact of demographics on WIA program performance of adults and dislocated workers also found an effect for age of participant. These finding suggests a strong need for further research to better understand the effect of age or life events associated with age on WIOA outcome performance measures.
Service program elements clearly have a profound impact on WIOA performance outcomes as found in this study. However, there continues to be a paucity of research at the service delivery level identifying what service efforts have the most impact. This impact study was designed to take a first step with the expectation that others would follow to help define a universe of practices that make a significant difference in performance as found in this initial effort.
The statistical findings regarding the InnerSight Assessment Experience provides evidence of a positive difference but does not shed light on how the Experience is perceived by youth. Participants’ feelings and descriptions of the Experience were collected in 1539 written statements shared on the youth evaluations of the sixty-eight Experience sessions. These statements offer insight into their perceptions.
When asked what they liked most about the Experience youth said, “Keep up the work. There are many people out there like me who are lost and not sure what to do. But with this book and your help, there are paths that seem to open & lighten up.” Another youth said, “This experience helped me get a clearer understanding of what I would like in my future career and I learned a lot about myself.”
Participants like that it is all about “me”. Many of the statements refer to “myself” such as, “Learning things about myself that I can describe in an interview.” One youth said, “I enjoyed the clear and concise approach of the InnerSight Experience, to be able to decipher myself.” Another shared, “I liked how it gave very good info about myself that I know subconsciously, but this put it into words that I couldn’t do myself”. A third wrote, “That I discovered my passion (nursing).” This is perhaps the first time they have been in a class that literally maintains a focus on them.
They like that it opens their eyes to career opportunities. “Career” is specifically mentioned in 175 written statements. A youth said, “InnerSight was an amazing experience for me. It helped me figure out what career path to take and reassured me on what I like to do.” Two others shared, “It opened my eyes to all of my available job opportunities. I loved this.” and “I was able to look outside the box of just one career.” They mention they learned about careers they might never have considered with statements such as, “I like how much it opened my eyes as well as my mind to new opportunities. I have a stronger direction in life.” and “I learned about career choices that I really did not take into consideration during my process in applying to college.”
It is interesting to note that some 80 statements refer to the certified InnerSight Guides who facilitate the Experience. Youth shared the following: “I loved the communication and even though it was a large group, the topics and session as a whole was very personalized. The presenters were very helpful, friendly and genuine.”; “I liked how the instructor was clear and had a happy personality. It made the environment and the entire presentation much more enjoyable.” and “I liked that it was a very friendly environment. The speaker was enthusiastic and willing to help and explain.” These observations emphasize the importance of the facilitators’ and subsequent case managers’ effective personal interaction with the youth.
Many statements referenced the materials themselves including the InSight Guide booklet that is personalized for each participant. Some of these statements include, “I liked the booklet, it contains really useful information that can be used along our educational paths, job hunting, and to see our preferences and interests.” They shared their views on the accuracy of the results, “The inventory results were very accurate, helped me learn more about my career and I’m 100% sure that physical therapy is what I want to do.” Some identified new skills they acquired, “Learning more about the qualities that make (me) who I am and getting an insight on what other possibilities I can attain.” and “I think I’m more prepared to talk to employers.” Observations such as the above reinforce the engaging and personal aspects of the materials. They actually put participants in the process and bring the Experience alive.
The personal statements suggest participants appreciate the focus on them, the new information they gain about careers, personal career fit, as well as the personal attention and responsiveness of the InnerSight Assessment Experience. Participants embraced the new information saying, “Thank you for a wonderful Saturday morning it was worth it.” and “I’d like to thank this workshop for being here because without it I’d be lost.” Many learned about occupations they may never have considered while others gained confirmation of their current pathways. Most importantly they liked that it is “All About Me.”
Implications and Considerations
- There is a need for additional research to identify those service delivery program elements that contribute to a participant’s success.
- Policy makers, and especially workforce investment boards, need to assure that data elements are created in their performance systems to facilitate the study of service level program impact on outcome performance.
- There is a need to move beyond WIOA performance measures to examine if what is done to, for, and with the participant increases the rate of performance.
- Age of participant has been found to produce an effect in studies of both youth and adult performance on WIA and WIOA program outcome measures. This suggests a need for policy makers and program leaders to consider this in the development of service strategies as well as individual service plans.
- Effective use of assessment results is a powerful building block for establishing rapport, engaging the youth and creating personal pathway to success.
- When participants engage and validate assessment results it places them in a position to make better choices regarding education, training and career options. It builds a sense of personal ownership in the Individual Service Plan and most importantly—they feel it wasn’t done to them. WIOA policy makers will want to understand how this is accomplished by their service providers.
- Professional interpretation of Assessment results, when integrated with case manager’s continuing work with youth on Individual Service Plans, creates a personal bond magnifying the personal connection making the service experience all about them and likely contributes to persistence.
- Policy makers need to be aware that Assessment has often been used ineffectively. When professionally guided on how to use and apply their own assessment results, participants engage and become developmentally productive.
- Statistical evidence while standing alone does not tell the whole story. Participants in their evaluative statements emphasize their appreciation for what they learn and that they are invited to fully engage in the process. This suggests that programs need to move beyond transactional processes when working with youth.
References
Barnow, G., and Smith, Jeffrey. “Performance Management of U.S. Job Training Programs: Lessons from the Job Training Partnership Act.” Public Finance and Management, vol. 4, no. 3, 2004, pp. 247-287.
Borden, William S., Nov. 2009 Mathematica Policy Research, The Challenges of Measuring Employment Program Performance, in a paper delivered at the 2009 conference What the European Social Fund Can Learn from the WIA Experience. Retrieved from
http://www.umdcipe.org/conferences/WIAWashington/Papers/Borden-Challenges-of-Measuring_Employment_Program_Performance.pdf
Dunham, K., Mack, M., Salzman, J., Wiegand, A., May 2006, Evaluation of the WIA Performance Measurement System – Final Report. Retrieved from
https://wdr.doleta.gov/research/FullText_Documents/Evaluation%20of%20the%20WIA%20Performance%20Measurement%20System%20-%20Final%20Report.pdf
Hossain, Farhana (2015). Serving Out-of-School Youth Under the Workforce Innovation and Opportunity Act (2014). Retrieved from
http://www.mdrc.org/publication/serving-out-school-youth-under-workforce-innovation-and-opportunity-act-2014
Kazis, Richard (2016). MDRC Research on Career Pathways. Retrieved from
http://www.mdrc.org/publication/mdrc-research-career-pathways
King, C. T. (2004). The effectiveness of publicly financed training services: Implications for WIA and related programs. In C. J. O’Leary, R. A. Straits, & S. A. Wandner (Eds.), Job training policy in the United States. Kalamazoo, MI: W. E. Upjohn Institute for Employment Research.
Moore, R. W., Gorman, P. C., Blake, D. R., Phillips, G. M., Rossy, G., Cohen, E., Grimes, T., & Abad, M. (2004). Lessons from the past and new priorities: A multi-method evaluation of ETP. Sacramento, CA: California Employment Training Panel.
Social Policy Research Associates. (2004). The Workforce Investment Act after Five Years: Results from the National Evaluation of the Implementation of WIA. Retrieved from
https://www.doleta.gov/reports/searcheta/occ/papers/spr-wia_final_report.pdf
U.S. Departments of Education, Health and Human Services, and Labor, “Joint Career Pathways Letter,” April 4, 2012. Retrieved from
http://www2.ed.gov/news/newsletters/ovaeconnection/2012/04122012.html.
United States Department of Labor, Employment and Training Administration (2015). Retrieved from
https://www.doleta.gov/performance/guidance/laws_regs.cfm
United States Departments of Education, Health and Human Services, and Labor, “Joint Career Pathways Letter,” April 4, 2012. Retrieved from
http://www2.ed.gov/news/newsletters/ovaeconnection/2012/04122012.html.
Statistical Analysis Notes
Note 1. Univariate Analysis of Variance of Literacy by InnerSight and Age, Out-of-School Youth, 2014-15
| Between-Subjects Factors | |||
| Value Label | N | ||
| Study Group | 0 | Control | 1395 |
| 1 | InnerSight | 270 | |
| Age | <18 | Under 18 | 137 |
| 18.00 | 18 | 430 | |
| 19.00 | 19 | 506 | |
| 20.00 | 20 | 342 | |
| 21.00 | 21 | 250 | |
| Tests of Between-Subjects Effects | ||||||
| Dependent Variable: Literacy Outcome | ||||||
| Source |
Type III Sum of Squares |
df | Mean Square | F | Sig. | |
| Intercept | Hypothesis | 146.671 | 1 | 146.671 | 152.631 | .000 |
| Error | 4.134 | 4.302 | .961a | |||
| Study Group | Hypothesis | .863 | 1 | .863 | 9.719 | .010 |
| Error | .943 | 10.611 | .089b | |||
| Age | Hypothesis | 4.326 | 4 | 1.082 | 17.001 | .009 |
| Error | .254 | 4 | .064c | |||
| Study Group * Age | Hypothesis | .254 | 4 | .064 | .266 | .900 |
| Error | 396.329 | 1655 | .239d | |||
| a. .857 MS(Age) + .143 MS(Error) | ||||||
| b. .857 MS(StudyGroup * Age) + .143 MS(Error) | ||||||
| c. MS(StudyGroup * Age) | ||||||
| d. MS(Error) | ||||||
Note 2. Univariate Analysis of Variance of Attainment by InnerSight and Age, Out-of-School Youth, 2014-15
| Between-Subjects Factors | |||
| Value Label | N | ||
| Study Group | 0 | Control | 789 |
| 1 | InnerSight | 194 | |
| Age | <18 | Under 18 | 103 |
| 18.00 | 18 | 312 | |
| 19.00 | 19 | 257 | |
| 20.00 | 20 | 190 | |
| 21.00 | 21 | 121 | |
| Tests of Between-Subjects Effects | ||||||
| Dependent Variable: Attainment Outcome | ||||||
| Source |
Type III Sum of Squares |
df | Mean Square | F | Sig. | |
| Intercept | Hypothesis | 69.372 | 1 | 69.372 | 853.662 | .000 |
| Error | .721 | 8.874 | .081a | |||
| Study Group | Hypothesis | 3.292 | 1 | 3.292 | 26.210 | .002 |
| Error | .811 | 6.454 | .126b | |||
| Age | Hypothesis | .249 | 4 | .062 | .552 | .711 |
| Error | .452 | 4 | .113c | |||
| Study Group * Age | Hypothesis | .452 | 4 | .113 | .527 | .716 |
| Error | 208.428 | 973 | .214d | |||
| a. .875 MS(Age) + .125 MS(Error) | ||||||
| b. .875 MS(StudyGroup * Age) + .125 MS(Error) | ||||||
| c. MS(StudyGroup * Age) | ||||||
| d. MS(Error) | ||||||