City of Los Angeles YouthSource Assessment Experience Increases WIOA Performance Through Youth Engagement

Mark L. Perkins, John E. Reeves, Deborah K. Furlong, Emily A. Pazur

Abstract

Putting the Person in the Process is key to the City of Los Angeles YouthSource program performance. Los Angeles’ YouthSource Assessment Experience significantly impacts the rate of success on WIOA outcome measures. Qualitative analysis of 899 youth evaluations of the Assessment Experience shows that youth motivation is fostered in the formative InnerSight Assessment Experience in which youth authentically verify and connect their interests to potential career pathways. This study reveals the characteristics of the formative Assessment Experience which are meaningful to youth. It confirms youth need not enter the Experience intrinsically engaged as found by Dawes and Larson (2011) as the Experience fosters personal connection leading to engagement and subsequent performance.

The City of Los Angeles Economic Workforce Development Department (EWDD) offers the InnerSight Assessment Experience as a consistent program element across its 13 YouthSource service sites. Perkins et al. (2016) found this program element significantly impacted youth performance on Workforce Innovation and Opportunity Act (WIOA) performance outcomes (US Department of Labor, 2015) They recommended examination of participants’ narrative evaluation statements to understand how the Assessment Experience impacts youth performance.

Background

The city of Los Angeles YouthSource program is a premier provider of service for in and out-of-school youth. The city serves some 17,000 youth in its annual and summer programs. The Los Angeles YouthSource program has been nationally acclaimed by the Department of Labor for its progressive commitment to serving out-of-school youth. In 2012, before the new WIOA service requirements, the City made a substantial commitment to serving this population when it required that 70% of the youth served at its 13 YouthSource sites be out-of-school youth. As a result, they have become the nation’s most successful program in addressing the needs of this underserved population.

YouthSource program leadership desired to offer a consistent program element to serve youth across its 13 sites. Most importantly they wanted a program element that is developmental for each participant while meeting WIOA expectations and requirements. Specifically, they were seeking a program that would accomplish the following:

• Actively engage participants in the development of their individual plan.
• Provide caseworkers with a foundation for development of individual plans grounded in participant’s assessment results.
• Connect individual plans and strategies to specific educational, training and occupational goals.
• Help caseworkers and clients see those skills needed to achieve occupational fit.
• Provide documentation of participant and case manager mutual engagement and results oriented planning.

They believe a program grounded in participant personal development that connects youth with high demand occupations using their preferences and interests will facilitate persistence in the attainment of educational, training and career goals. To accomplish this, the city issued an RFP and selected the InnerSight Assessment Experience to meet this unique program and developmental objective.

The InnerSight Assessment Experience

The InnerSight Experience™ gives participants a vocabulary, context, and framework for bringing their personal interests and preferences to bear on life’s choices. This experience puts the “Power of the Person” in the decision-making processes of life. Through the Experience, participants come to see they are the “Key” to their personal, educational, and career training choices and gain important information about themselves upon which to reflect when making those choices.

The InnerSight Assessment Experience consists of an online interest and preference inventory (The Strong Interest Inventory^®) resulting in a personalized InSight Guide booklet which is the foundation for a 3-hour, formative, in-person, group interpretative experience. This interactive experience cognitively and psychologically engages participants in individual and group exercises that emphasize and highlight the distinctiveness of their personal results. By self-validating their personal interests and preferences for working, learning, leading, risk-taking, and team orientation, participants are equipped to explore occupations of people who are satisfied in various occupations and share the participants’ interests.

When interests and preferences are aligned with an occupation (work tasks), satisfaction, happiness, and superior performance occur. Youth with cultural, familial, and economic barriers are no exception. When engaged in subjects of interest (music, dance, skateboarding, etc.), individuals succeed.

The InnerSight Experience is grounded in the prolific research and universally acclaimed Work Types Model of the eminent vocational psychologist Dr. John Holland. The Holland Model was adopted by the Department of Labor as the foundation for their online career exploration system O*NET. This system can be directly accessed from the electronic Insight Guide booklet participants receive after completing the InnerSight Experience by simply clicking on a preferred occupation.

For over 90 years, interests and preferences have been known as key factors in motivation. Using interests and preferences to help individuals make better life choices is a time-tested and valid approach for assuring continued effort/success in education, career, and personal life.

InnerSight uses this approach, but has changed the traditional career assessment/exploration model. Youth are accustomed to being processed; that is, being asked what they know (education), or being told what they need to know and do next. InnerSight begins the journey by first exploring and having participants validate what they may like or enjoy doing. Additional exercises guide the participants to discover the connection between what they enjoy doing and potential occupations in which people who share their interests are satisfied.

The InnerSight Assessment Experience continues when participants meet with their case managers to begin Next Steps planning in the InSight Guide booklet based on their self-validated results. Case managers who work with the youth complete the InnerSight Experience and training in “Facilitating the Journey.” “Facilitating the Journey” training provides case managers with the simple skills needed to engage the participant in a meaningful discussion of their results and use them as the foundation of the individual service plan.

Each InnerSight Experience session is evaluated by the participants. Their ratings provide continuous feedback on characteristics of the Experience from the InSight Guide booklet to facilities and the accuracy of their results. In addition, they are asked to respond to three open-ended questions regarding what they liked, what they would change, and any additional thoughts they might have. These personal responses provide a rich source of qualitative information for understanding, in participants’ words, how they were impacted by the Experience.

The InnerSight Assessment Experience is designed to help WIOA staff align youth and adults with education, training programs, internships, and actual job openings that are a best fit for them. This approach has been empirically demonstrated to increase youth performance on WIOA outcome measures. To better understand why it works, this study examines the youth’s perceptions of the Assessment Experience using their written evaluations.

Literature

Research on youth engagement, formative assessment and WIOA program impact offer a context for understanding and interpreting InnerSight Assessment Experience evaluations.

WIOA Program Impact

Moore and Gorman (2009) observed “there is little in depth analysis of the performance of the WIA system or of the likely drivers of the new common measures. Specifically, there has been little published on the relationship between individual participant characteristics and the program performance measures.” They found demographic characteristics such as age, ethnicity, education etc. account for considerable variance in performance outcomes. They observe “there is a need for researchers to undertake more nuanced studies—-of training most likely to lead to successful outcomes.” (p. 394), thus there is a need for providers to understand what program elements, assessments, or experiences have a positive impact on participants’ performance on the common WIA metrics.

Finding no other research identifying program element impact, Perkins, Reeves, Mancheno-Smoak, Furlong (2016) conducted an empirical examination of the impact of the InnerSight Assessment Experience on youth WIOA performance outcomes in the City of Los Angeles YouthSource program in program year 14-15. Performance of youth who had the InnerSight Assessment Experience was compared with those who did not have the Experience on WIOA outcomes.

InnerSight Assessment Experience Youth were found to significantly outperform those who did not have the Experience on Attainment and Literacy. There was an 11% improvement in success rate for all youth in the InnerSight group over the control group on the Attainment performance measure. The Attainment performance success rate for out-of-school youth was 14% higher for those who completed the InnerSight Assessment Experience. There was a 6% improvement for the InnerSight Assessment participants on Literacy over the control group. No significant difference on the Placement outcome measure was identified. This study is the first to link a program element to increases in youth outcome measures.

Perkins, Reeves, Furlong (2017) replicated the 2016 study looking at program year 15-16 outcomes. Youth Attainment success rate was 74% for InnerSight participants compared to 52% for non-participants, for a significant difference of 22%. In-school youth with InnerSight had a 94% success rate compared to 71% for those who did not, for a significant difference of 23%.  Out-of-school participants with InnerSight had a success rate of 60% compared to 43% for those without InnerSight, for a 17% significant difference, which was 3% higher than in the previous study.

The InnerSight Assessment Experience was also found to positively impact Placement in the 15-16 program year study. The InnerSight in-school group performance was 88%, while the control group was 74% for a 14% difference. The out-of-school InnerSight Experience group performance was 79%, while the control group achieved 70% on the Placement outcome.

This research, while demonstrating the relationship between a program element and success in achieving WIOA outcome measures, does not assist in understanding why or how the program works. Understanding how participants feel a program impacts them could provide valuable insight into effectively working with out-of-school youth.

Youth Engagement

Kazis (2016) in a research brief on Career Pathways observes “there is little rigorous research that assesses the impact of comprehensive career pathways programs that follows individuals from different starting points through a coherent set of educational experiences and “stackable” credentials to the labor market.” (p. 2). Kazis further suggests there is a paucity of research on how to positively engage the participant in his/her personal development of a successful career pathway.

Dawes and Larson (2011), in a qualitative study, examine how youth engagement develops based on 44 youth narrative accounts of their experience in youth programs. Drawing on theories of psychological engagement such as flow, interest and self-determination, the authors suggest youth engagement emerges from personal connection.

Flow theory (Csikszentmihalyi, 1975: Csikszentmihalyi, Rathunde, & Watson, 1993) suggests deep engagement occurs when people experience the challenges in an activity as matched to their skills (not too hard or too easy relative to skill level) The challenges, however, must also have some meaning for the participant.

Interest theory is like Flow in that the activity must be personally meaningful involving “focused attention, increased cognitive functioning, persistence and affective involvement.” (Hidi, 2000, p. 312). However, for interests to be sustained over time, a person needs to gain a base of knowledge about the activity and develop positive subjective feelings toward it. (Hidi & Renninger, 2006)

Psychological engagement occurs, according to Self-Determination Theory (SDT), when the activity requires the participant to be associated with more than just meaning or positive feelings; the activity must be integrated into self. In this theory, psychological engagement varies as a function of how much a person has internalized the goals of the activity. (Ryan & Deci, 2000) Increased motivation and engagement occurs on a continuum as a person identifies with, internalizes, and integrates the activity’s goals into the self-system. Strongest motivation occurs when participation in an activity is completely internally regulated (Ryan & Deci, 2000). Further research reveals that the processes of internalization are driven by three basic universal psychological needs of the self: competence, autonomy and relatedness. (Ryan & Deci, 2000)

Dawes and Larson (2011), building on these theories and the narrative research data in their study, developed the following operational definition of “forming a personal connection” to guide their research:

They found that “Youth described this personal connection as occurring through changes in both themselves (developing knowledge, skills, values, future goals) and in their perception of the activity (seeing new things in it, learning its relevance to goals). The process appeared to involve experiencing increased convergence between self and the activity.” (p. 263)

They discovered personal connections fit into three categories: learning for the future, developing a sense of competence, and pursuing purpose.

Learning for the future – The largest number of youth attributed the change in their psychological engagement to “a connection they discovered between the skills they were learning through participating in program activities and goals for their future” (p. 263) and “…engagement or motivation in the activity became stronger as they realized that they were gaining knowledge, exploring and developing skills that would be valuable to them later, often for a desired college major or career choice” (p. 264). In a separate analysis focused on helping youth think about career choices Rickman (2009) describes this process as that of “finding fit”.

Developing a sense of competence – For a second group of youths in the study “Doing well in program activities….and having that acknowledged by others …. provided meaningful self-affirmation. This experience of competence connected youth to program activities and fueled motivation to pursue new challenges in the program”. (p. 264)

Pursuing purpose – A third group reported increased psychological engagement when “forming personal connections to goals that transcended their own self-interest”. (p. 265)

Dawes and Larson (2011) concluded “For youth to benefit from many of the developmental opportunities provided by organized programs, they need to not only attend, but become psychologically engaged in program activities”. Their research reveals that “youth who experience a positive turning point in their motivation or engagement” do so through a change process that involves forming a personal connection. The authors observe, “that youth need not enter programs intrinsically engaged—motivation can be fostered—and that programs should be creative in helping youth explore ways to form authentic connections to program activities”. (p. 1)

WIOA encourages implementation of career pathway approaches that support post-secondary education and training for out-of-school youth that is related to the demand for qualified workers in local labor markets. However, research shows that many high-school-aged youth have little knowledge about or commitment to career pathways (Schneider & Stevenson, 1999) and as suggested by Meijers (1998), this can create anxiety and avoidance. Findings in Dawes and Larson (2011) tell us “that when youth do begin to connect to meaningful career paths, it can create a marked increase in their motivation and engagement”. (p. 266)

The Workforce Innovations and Opportunity Act expects the individualized pathway to be grounded in an assessment of skills and interests to inform participants’ career decision-making when identifying logical short-term next steps consistent with long-range goals. Formative assessment approaches have been shown to be valuable in accomplishing this important developmental goal.

Formative Assessment – A Context for Youth Engagement

Nichols and Dawson (2012) see assessment as a context for student engagement, observing “that summative testing systems tend to connect with traditional motivation processes such as goals and efficacy-related beliefs, whereas formative systems tend to connect with engagement-related processes such as self-regulated learning and self-determination.” They suggest formative assessment requires active participation of the learner.

Black and Wiliam (1998) state: “The core activity of formative assessment lies in the sequence of two actions. The first is the perception by the learner of a gap between a desired goal and his or her present state (of knowledge, and/or understanding, and/or skills). The second is the action taken by the learner to close that gap in order to attain the desired goal.”(p.11)

Teachers and facilitators play an equally important role with the learner in bringing about this sequence of events. For the first event to occur according to Black and Wiliam (1998),

“The prime responsibility for generating the information may lie with the student in self-assessment or with another person, notably the teacher, who discerns and interprets the gap and communicates a message about it to the student. Whatever the procedures by which the assessment message is generated, in relation to action taken by the learner it would be a mistake to regard the student as the passive recipient of a call to action. There are complex links between the way the message is received, the way in which that perception motivates a selection amongst different courses of action, and the learning activity which may or may not follow.”(as cited in Nichols and Dawson, 2012, p. 466)

Nichols and Dawson (2012) observe that formative assessment processes emphasizing engagement related approaches like self-reflection and self-determination are more effective than summative assessment practices in promoting persistence and academic achievement.

Others have shown that formative assessment practices provide students with greater opportunities to demonstrate autonomy and choice through feedback processes that are more informational than controlling which enhance engagement related actions and beliefs. (Deci & Ryan. 1995; Grolnick & Ryan, 1987)

Research shows formative assessment approaches to be a valuable context for the engagement of youth both cognitively and psychologically in self-regulation that leads to the exercise of student will or volition essential for “buy-in”. (Nichols & Dawson, 2012)

Manno, Yang, and Bangster (2015), in their work on engaging disconnected young people in education and work, state that “support from case managers and other adult staff seem to help promote youth engagement.”(p. iii) Dawes (2008) in her study of engaging adolescents in youth programs, found that leaders facilitate engagement by fostering a welcoming interpersonal climate, ensuring that serious activities are balanced with fun experiences and providing youth with verbal encouragement and strategic assistance on their projects. This research suggests that formative assessment can be a powerful connector, especially when placed in the hands of engaging case managers and program leaders.

WIOA research has focused on program performance with discussions of topics such as the importance of the employer network, career demand, issues of program integration, curriculum, and process. Until recently studies have focused on what is “done to” the participant and rarely on what is “done with them” or what “they thought or felt about” the program or their engagement. Essentially the focus has been on the process. Research rarely has examined if the process works or engages participants.

Method

While empirical impact studies provide, valuable results regarding program efficacy they offer little understanding of why a program works or how it achieves the desired result. This study builds on the qualitative research of Dawes & Larson (2011) who examined the narrative statements of 44 youth across 10 youth programs to develop a grounded theory of youth engagement. This study seeks to understand how and why the City of Los Angeles YouthSource Assessment Experience positively impacts WIOA youth performance rates.

Every participant in an InnerSight Assessment Experience is asked to complete an evaluation. The evaluation consists of 10 structured questions which participants may rate from strongly disagree to strongly agree. These questions are followed by three open ended questions that offer narrative content for analysis in this study.

What did you like most about the Experience?

What would you change about the Experience?

What additional comments do you have?

A copy of the evaluation form can be found in Appendix A.

Study Population: The study population consists of youth participating in the City of Los Angeles YouthSource program in 2014-15 at one of 13 provider sites across the City. A total of 899 individual evaluations were completed by youth participating in 68 InnerSight Assessment Experience sessions.

Information for gender and ethnicity of participants was not available in the data extract The City of Los Angeles provided total program gender and ethnicity information for program year 2014-15 reflecting a robust and diverse population of participating youth. For 2014-15 there were 1947 females (54.4%), 1630 males (45.6%). The reported ethnic makeup as expected is very diverse with the largest reporting group being Hispanic or Latino followed by White and African American/Black. The mix of Race or Ethnicity in the population is so large that participants are permitted to select more than one race or ethnicity, suggesting that the standard race identification categories are rapidly becoming ineffective demographic descriptors as study variables for this population.

Qualitative Analysis: Considering the significant impact the InnerSight Assessment Experience has been shown to have on WIOA outcomes, it would be helpful to understand how youth perceive the InnerSight Assessment Experience. To gain such an understanding, the narrative evaluation comments provided by youth in response to the three open ended questions on InnerSight Assessment Experience evaluations were examined to learn what they thought and felt about the Experience. A total of 1539 youth evaluation comments were obtained from 899 youth evaluations of the InnerSight Assessment Experience Sessions in the City of Los Angeles in program year 2014-15. These evaluative comments were collected from three open ended questions as noted above and in Appendix A.

Key word content analysis was used to organize responses to each question. Individual responses were grouped by their key word to identify meaningful content categories. Content categories for each question were then developed. The results for each category were examined and the number of responses for each recorded. Categories with 20 or more statements, or 5% or more of the responses, were considered major content or topical areas. Content areas with fewer than 20, or less than 5% of the responses, were collapsed under other observations.

Results of this content organization were examined by evaluation question to understand what participating youth felt about the Assessment Experience and why it worked for them.

Results

Results of the content analysis of participants’ responses to the three open-ended evaluation questions are presented by question below.

Most Like About the Experience

Responding to the question what they liked best about the Experience, the youth provided some 791 statements. These statements suggested eight major content categories of 20 or more statements. The remaining topics with 19 or fewer statements are reported as other observations and consist of some 142 statements in 13 topical areas. The number and percent of statements in each of the major content categories are identified in Table 1. Participants’ statements by question, content category and subcategory, where appropriate, can be found at: https://www.myinnersight.com.

Table #1
Summary of Participant Statements by Category
What Did You Like Most About the Experience?

Examining the statements provided by category offers insight into how the InnerSight Assessment Experience resonated with the youth and what was most important to them.

Myself – 182 (23.01%) of participants’ statements included the word “myself” and was the largest category expressing what they liked about the Experience. Simply stated they like that it focuses on them.

“What I liked most about this innersight was being able to learn more about myself and what I want to major in.”

” A great experience of discovering myself. I learned a lot of things about myself I didn’t know before.”

“What I liked the most was that I learned new things about myself that I would’ve never thought I had in me.”

“I enjoyed learning about myself.”

“I learned some reassuring things about myself.”

Career Options – 175 (22.12%) statements mentioned “career” and was the second largest category of expressions of what youth liked about the Experience. Participants said:

“It opened my eyes to all of my available job opportunities. I loved this.”

“I was able to become much more open minded to the career of interest. I found out a few careers that were never on my mindset. Interesting.”

“InnerSight was an amazing experience for me. It helped me figure out what career path to take and reassured me on what I like to do.”

“That it not only gave you careers but also college courses you could take and it gave you an idea of what you would be doing in the work fields.”

“I most liked, that I was given ideas about what to do with my future.”

These statements suggest participants gained insight regarding their future. They not only identified careers of interest, but found some they had never considered and now wanted to explore further. They began to see what they might need to study to pursue a career as well as what it might be like to work in the career field. They consistently enjoyed having new ideas regarding options that would be of interest in their future suggesting a developing intrinsic motivation.

The Presenters – 80 (10.11%) of the statements related to the presenters, or certified InnerSight Guides, who facilitate the Assessment Experience sessions. Participants shared the following:

“I loved the communication and even though it was a large group, the topics and session as a whole was very personalized. The presenters were very helpful, friendly and genuine.”

“I liked the fact that the speaker was very outspoken and listened to what we (students) had to say. Very polite! ?”

“Everyone got to share their dreams and goals, good examples and narrow down our requirements.”

“I liked how he interacted with us and got us to participate. Didn’t have a monotone and made it interesting.”

“It was easy to understand and personalized to meet my needs.”

“1 chill instructor.”

“The presenter was really nice.”

These reflections emphasize as noted in the literature the importance of connecting with the youth in both a personal and professional manner.

What I Learned – 73 statements or 9.23% focused on what youth felt they learned.

Youth said:

“I realized I need some work experience.”

“I learned different potentials that I have.”

“I’m very indecisive but need to expand horizon.”

“I liked most the learning experience and how well everything was explained.”

This again suggests that participants are building an intrinsic connection as they learn about who they are. These statements imply a discovery of the gap between where they are and where they might like to go.

Validation/Confirmation – Participants like receiving validation/confirmation as 43 (5.44%) of the statements touched on these ideas. For example, some participants said the following:

“I liked getting validation of my career choice. InnerSight also gave the many other choices that I would be interested in.”

“I liked seeing confirmation of my career and seeing all the other options that fit me.”

“Positive vibes, it got in touch with the inner me, and reassurance of my goal/path.”

Participants shared how satisfying it is to receive validation and confirmation of their personal goals and career paths. They also liked seeing other options that are a good fit with their already validated choices and interests.

The Information Provided – 37 responses (4.68%) relate to the Information received by the participants in the Assessment Experience sessions. Participants’ comments included the following:

“I liked that it was descriptive and full [of] information and gave me a lot of clarity.”

“I liked the useful information given by the presenters, it was very helpful.”

“All the various information will help me, help my community, friends and family. The money I make will not only be for me, but will be donated to give back.”

“I liked that it was very interesting and informative.”

“The amazing and helpful information given to me.”

“I liked that it was very interesting and informative.”

“Gave good information on what I wanna do.”

“The collected data/statistics.”

These observations suggest that the new information participants gain while learning about themselves is an important part of the Experience. They expressed amazement in finding fit between the information and their personal desires.

The Booklet – Some 37 responses (4.55%) pertained to the InSight Guide booklet provided to each participant. This is what several participants had to say

“I liked the booklet, it contains really useful information that can be used along our educational paths, job hunting, and to see our preferences and interests.”

“I loved how the booklet broke down what to do after you know what career path.”

“What I liked most was that we received a booklet of ourselves for ourselves.”

“I like the most about innersight was the book because it had the info that was so me.”

“The take home booklet plus the emailed file.”

Participants indicated that they like the InSight Guide booklet but, clearly it was because it focused on them and was all about them. Most importantly, after the Assessment Experience and work with their case managers, it contains their conclusions about their next steps or career pathways.

Helpful – Some 23 responses (2.91%) addressed the helpfulness of the Assessment Experience session. Participants found the Experience helpful. They wrote:

“Eye opener”

“It showed me what I could become and do”.

“It is very helpful.”

“Helped figure out more about me and careers.”

“They helped us learn more what we needed to learn.”

Participants indicated the Experience was helpful in seeing and realizing who they are, suggesting it was much like a personal epiphany.

Other Observations – There were 13 additional topics containing fewer than 20 statements each offering insight into what participants liked best about the Experience. These additional 142 statements make up 17.95% of the total observations of what participants like about the Experience. The 13 topics include: personalization, presentation, everything, participation aspects of the experience, easy to understand, specific things participants liked, accurate information, options/opportunities, interview skills, interests, participant interaction, fun and interactive, and activities.

Personalization of the Experience was noted in some 19 responses. This also speaks to the idea that it is all “about them”. Participants shared:

“I like how it all felt very personal and helped many different people.”

“That it involved my interests and not just common jobs that everyone likes.”

“I like the way that it was designed just for me with all my interests.”

“It was easy to understand and personalized to meet my needs.”

“I liked that they focused on everyone at the same time but also on everyone individually.”

“The individualized support.”

Some 18 statements touched on the presentation itself. Many participants liked the way the information in the Assessment Experience session was presented, as evidenced by the following statements:

“The fact that it was a key to open a door.”

“I love how the whole process is, from answering about 300 questions on that survey and then coming here to read about what that says about you is pretty awesome.”

“I liked the comforting and accepting atmosphere created.”

“The presentation was fun and engaging.”

These statements suggest that the formative process of presenting the information to participants in the Assessment Experience session is important. The interpretative approach appears to be effective and resonates with participants.

Participation in The Experience and that it was Easy to Understand each received 15 observations. Participants shared the following about their engagement and how easy it was to comprehend:

“I liked that we all got an opportunity to express our thoughts.”

“I liked how I felt comfortable and safe to say what’s on my mind”

“I enjoyed how everyone participated in the experience.”

“I liked how we all got to express our self in what we are.”

“The hands on experience.”

“That we shared our thoughts.”

“I liked that we talked about how it’s all up to me how we think.”

“What I like most about this experience was how easy it was to understand and how much information was provided.”

“Very clear.”

“How understandable it was.”

“Everything was clear and organized despite the fact that we jumped around in the booklet.”

“I like most about the Experience that it was clear and was really about me!”

“I clearly understood the booklet and results.”

“Easy to understand.”

These observations identify characteristics of the Assessment Session Experience. Participants appreciate the clarity, as well as the autonomy of choice and self-determination, in the Experience.

Thirteen observations touched on specific things participants liked about the Assessment Session Experience while 11 addressed the accuracy of the information. Participants shared that they liked things such as:

“The discussion”

“The paragraph”

“The communication.”

“I liked that it listed the skills I had.”

“It has statistics.”

“The power point”

“My results”

Participants also shared their thoughts on the accuracy of the information provided:

“The inventory results were very accurate, helped me learn more about my career and I’m 100% sure that physical therapy is what I want to do.”

“I like that it was well organized and it was very accurate on my interests.”

Ten statements touched on the options and opportunities participants learned about/ while six noted they gained new interview skills and six more felt their interests were clarified. These statements included observations such as:

“The information and the vast amount of opportunities for me.”

“I think I’m more prepared to talk to employers.”

“That the guides helped prepare me for interviews.”

“I learned what to say in an interview and financial support”

“It helps with my options.”

The final three topics in the other observation category included 6 statements about participant Interaction, 4 statements that the Assessment Experience was Fun and Interactive, and 2 regarding Activities.

“Hearing other’s opinions”

“That I got to conversate with different people and that we all shared something

about each other.”

“That we got to hear each other’s ideas.”

“It was really fun, I knew things I never heard about before!”

“It was entertaining and straight to the point.”

These statements suggest that the participants see the Assessment Experience as more than one dimensional. It appears to engage youth and allow them to use the information provided to self-validate their potential educational and career pathways.

What Would You Change

In commenting on what they would change about the Experience, the youth offered some 404 observations in six content areas. The content areas and associated percentage of responses in each are identified in Table 2. Content areas with less than 4% of the responses are reported under other observations. Some ten topics are covered under other observations.

Table #2
Summary of Participant Statements by Category to
What Would You Change About the Experience?

Content Category	Number of Statements	Percent of Total Statements
Nothing	247	61.14%
Time	43	10.64%
The Presentation	20	4.95%
My Personal Thinking	18	4.46%
Other Observations*	76	18.81%
Total	404	100%
*other observations include 10 topics

Nothing – Participants indicated in 247 statements (61.14%) that they would change nothing in the Assessment Experience indicating a high level of satisfaction with the Experience.

Time – The Assessment Experience Session requires three hours of personal engagement. Feedback on time represented 10.64% or 43 of the What [I] Would Change responses. Twenty-two of the participants wanted it shorter, 16 were unclear, 3 wanted it longer and 2 wanted it at a different time of day.

The Presentation – Twenty statements (4.95%) touched on the presentation suggesting changes that might be considered. Participants shared:

“Making the experience more individual in order to make it clearer.”

“It would be awesome if we could have a one-on-one meeting.”

My Personal Thinking – There were some 18 observations (4.46%) regarding how the Assessment Experience changed participants’ personal thinking. Participants said;

“I would recommend my friends at UCLA and bring them to the program.

“I would have loved to more about InnerSight. I felt I came in blindly and would have never expected this to truly help.”

“I would like to change that I thought I wouldn’t like it, but I did.”

“I would change my attitude about the experience.”

“The only thing I would change would be to try and participate more.”

“The only thing that I would change is my major.”

“I should have taken the survey more serious and spent more time on it.”

Other Observations – A total of 76 statements (18.81%) fell into 10 categories touching on topics such as games or interaction, boring, inventory and results, facilities, Assessment Experience environment, food, and breaks.

Several participants wanted more interaction or games as part of the presentation. A few felt that more information or having more people present would enhance the Experience.

“Maybe more activities; involve or incorporate some physical or active interactions.”

“More interactiveness with the other students.”

“Go even more in depth, with like education-wise.”

“Have more people attend.”

Only 4 commented on the session being Boring saying:

“More interaction, just sitting down was kind of boring.”

“I would add more excitement “.

Others stated the following about the Inventory and Results:

“What I would change is maybe the answers I gave to some questions.”

“I would want to take the quiz again and strongly agree on more topics.”

Some 29 statements focused on the facilities and environment for the Assessment Experience.

Room temperature and facilities received the most mention with statements such as:

“I would change the room temperature, it was a little cold.”

“Turn the AC on.”

Food and the break were also noted by participants:

“Add some snacks.”

“I would eat before coming. I was hungry.”

“That we could get two breaks.”

Additional Comments

When offered the opportunity to share additional comments, participants offered 344 observations in 5 major content areas. The content areas and associated percentage of responses in each are provided in Table 3. Content areas with less than 5% of total responses are combined in other observations and cover 6 topic areas.

Table #3
Summary of Participant Statements by Category to
What Additional Comments Do You Have?

Content Category	Number of Statements	Percent of Total Statements
The Experience	139	40.41%
Thank You	52	15.12%
The Presenters	47	13.66%
Useful or Helpful	40	11.63%
The Presentation	20	5.81%
Other Observations*	46	13.37%
Total	344	100%
*Other Observations includes 6 topics

The Experience – The Assessment Experience session was the focus of 40.41% or 139 of the additional comments. Participants said:

“Extremely happy about InnerSight and the way it makes me feel about understanding myself.”

“I loved it.”

“This class was an amazing experience and I would love to do it again.”

“This was a long experience, but it was worth it.”

“Love it (heart)”

“InnerSight is such a great program and is valuable to anyone who takes it.”

“The InnerSight is a good way to let people know who they really are. InnerSight has helped me reveal myself. Thank you InnerSight

“It was legendary”

These observations suggest that, InnerSight offers personal value for the participant and contributes to building an understanding of self.

Thank You – Fifty-two observations (15.12%) simply made the effort to say, “thank You:”

“Thank you for a wonderful Saturday morning? It was worth it.”

“Just a thank you! I really appreciate you all taking the time to guide others in important aspects of life.”

The Presenters – Forty-seven additional observations (13.66%) touched on the presenters. Participants said:

“I like how friendly everyone was and how they encouraged us all to participate.”

“All my questions were answered and I was comfortable all the time.”

“Great job to the presenter not boring.”

“My additional comment is that it was entertaining to be part of a professional way to conduct a workshop.”

These comments highlight the importance of the quality of the interaction by engaging youth in both a respectful and professional manner.

Useful or Helpful – observations made up 40 (11.63%) of participant’s additional comments. In expressing the helpful nature of the InnerSight Assessment Experience, participants made the following comments:

“I’m looking forward to sharing what I learned today with my parents who are very interested in my occupations.”

“This session was helpful to me and it did help me reconsider my major and think about how I would use my major to help others.”

“This experience was great and helpful. Now, I have a better sense of what person I am.”

These statements suggest the personal value of the Assessment Experience in charting a future plan as well as in communicating with parents.

The Presentation – Twenty (5.81%) of the additional comments focused on the presentation. Participants shared:

“Maybe a video of the people that have had InnerSight help them and they can give feedback, suggestions and comments to the new InnerSight people.”

“Ask more questions to the clients, make them volunteer, have them come out of their shell to have a more active session.”

“The preference presentation is amazing and unique.”

“Overall it was a great presentation”

Other Observations – Include 46 responses (13.37%) covering some 10 additional topics such as; recommend to others, my future, environment, and food were shared by participants.

Eleven participants recommended that others should take part in the Assessment Experience offering statements such as:

“InnerSight is a great experience I would highly recommend it for incoming college students to get an idea on what career to choose.”

“I strongly recommend this to anyone because it’s a nice experience and you learn new things about yourself.”

Six responses focused on the participants’ future with statements like:

“I really like how InnerSight helps people in finding their majors or jobs for the future.”

“Through InnerSight I was able to see what kind of future I could have.”

The remaining statements were general in nature with four focusing on the environment expressing equal emphasis on temperature, facilities, and food with responses like the statements about environment reported under desired changes.

Discussion

The City of Los Angeles YouthSource program has implemented a developmental program element to actively engage youth in the preparation of individual service strategies as expected by WIOA. YouthSource partnered with InnerSight LLC to provide the InnerSight Assessment Experience as a foundational and integrating element for their program.

YouthSource views assessment as more than a test. Like Nichols and Dawson (2012) they view assessment as a powerful foundation and tool for personal growth and development. The key for YouthSource is youth engagement, which occurs when the youth use their self-understanding in charting educational and career pathways.

The literature is replete with studies (Hossain, 2015; Dawes & Larson, 2011; Nichols & Dawson, 2012) speaking to the importance of truly engaging the youth. The City of Los Angeles YouthSource program accomplishes this with the InnerSight Assessment Experience that serves as a common foundational program element across all service providers. Thus, the effective use of assessment becomes a common element in the development of individual service strategies across all programs.

The City recognized the key to success was not the assessment tool but the youth’s effective use of the information. They adopted the InnerSight Assessment Experience to provide an inventory, but most importantly, a three-hour interpretative Experience which fully engages the youth in self-validation of their preferences, interests and the careers they most prefer. When followed up with encouragement and strategic assistance by case managers, as Dawes (2008) suggests, the youth have a personal development plan grounded in who they are and what they most prefer.

Youth participating in the InnerSight Assessment Experience have significantly higher success rates on WIOA outcome measures (Perkins et. al, 2016 & Perkins et. al 2017). This suggests a powerful impact when Assessment outcomes are effectively used.

While these studies have provided empirical evidence of program impact on WIOA outcomes they offer little insight regarding how youth feel about the Experience and perhaps what makes it work. Building on the qualitative theoretical work of Dawes & Larson (2011) on how youth become engaged, this study examined some 1539 narrative statements from 899 youth participating in the InnerSight Assessment Experience to understand how and why it works for them.

The Results show that youth really like that the Assessment Experience is all about them. This is perhaps a powerful message about what happens when we “Put the Person in the Process”. This is consistent with motivational theory suggesting the activity must develop positive subjective feelings toward it (Hidi,2000), become integrated into self (Ryan & Deci, 2000), build a personal connection (Dawes & Larson, 2011), and connect to a future goal in a process described by Rickman (2009) as “finding fit”. It seems too much program time can be spent in activities needed to process the person rather than activating and engaging them in the process as suggested in the literature on youth engagement.

Youth observations across the three evaluative questions in this study provided important insights on youth perceptions and feelings regarding the Assessment Experience. Their narrative observations fell into 46 content categories across the three evaluation questions. Seven content areas received 50 or more responses and accounted for 948 or 61% of the narrative observations. For these seven content categories, the associated evaluation question and number of responses in the category are provided in Table 4 as a quick summary.

Table #4
Content Categories with 50 or more Responses by Evaluation Question

Content Category	Number of Statements	Evaluation Question
Nothing	247	What I would Change
Myself	182	What I Liked
Career Options	175	What I Liked
The Experience	139	Additional Comments
The Presenters	80	What I Liked
What I learned	73	What I Liked
Thank You	52	Additional Comments

Connecting with youth participants is key to sustaining their engagement as noted by Manno (2015). Dawes & Larson (2011) state participants “need to not only attend but become psychologically engaged in program activities”. The number one content category, with 247 responses, was to ‘change nothing’ about the Assessment Experience suggesting a strong affinity for the Experience from today’s youth. But what is the most compelling or psychologically engaging for them?

The narrative statements suggest that when the Experience is all about them it becomes a powerful connector and “turning point” (Dawes & Larson, 2011). When it comes to career options it also was all about their options not just possibilities. These options connected the experience activity with participants’ potential future goals which is important in fostering engagement (Dawes & Larson, 2011).

In the additional comments, participants spoke of the Experience in terms of how it made them feel. The InnerSight session Guides were seen as positively contributing to the personal connection as well as how participants felt about the Experience. Learning about themselves and what they might need to consider in their life journey appears to also be a powerful connector as noted by Dawes and Larson (2011). While the Experience clearly focuses on the participants it connects in a manner that elicits participants’ appreciation.

So why does the InnerSight Assessment Experience achieve significant empirical gains in WIOA success rates? When we listen to the participating youth in this qualitative study it is because it connects. First as suggested by Nichols and Dawson (2012) the InnerSight Assessment Experience uses a formative approach that engages the participant. Second, as Dawes and Larson (2011) suggest for achieving engagement it connects both cognitively and psychologically by putting participants in the process of self-determination. Everything is about them, from the InSight Guide booklet to the professional interaction in the session, the cognitive content material is all about who they are and where they might go. The Experience is all about the participant and “the connect.”

When case managers support this connection using the InnerSight Assessment Experience material as the foundation for charting the youth’s pathway, it is an engaging and integrated program experience for the youth. As noted by Manno et. al (2015) this case manager support facilitates engagement. The youth is clearly in charge, working side by side with the case manager as a helpful facilitator.

The City of Los Angeles YouthSource program has focused on positive youth development by establishing a common program element across the City. This consistent program effort produces significant WIOA results. These program accomplishments have been realized by “Putting the Person in the Process”.

References

Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy and Practice, 5(1), 7–74.

Csikszentmihalyi, M. (1975). Beyond boredom and anxiety: The experience of play in work and games. San Francisco, CA: Jossey-Bass.

Csikszentmihalyi, M., Rathunde, K., & Whalen, S. (1993). Talented teenagers: The roots of success and failure. Cambridge, England: Cambridge University Press.

Dawes, N. P. (2008). Engaging adolescents in organized youth programs: An analysis of individual and contextual factors (Unpublished doctoral dissertation).University of Illinois at Urbana-Champaign.

Dawes, N.P., Larson, R. (2011), How Youth Get Engaged: Grounded-theory research on motivational development in organized youth programs. Developmental Psychology, Vol 47(1), Jan 2011, pp 259-269.

Deci, E. L., & Ryan, R. M. (1995). Human autonomy: The basis for true self-esteem. In M. Kernis (Ed.), Efficacy, agency, and self-esteem (pp. 31–49). New York: Plenum Publishing Co.

Grolnick, W.S. & Ryan, R.M. (1987) Autonomy in Children’s learning: An experimental and individual different investigation. Journal of Personality and Social Psychology, 52, 890-898.

Hidi, S. (2000). An interest researcher’s perspective: The effects of extrinsic and intrinsic factors on motivation. In C. Sansone & J. M. Harackiewicz (Eds.), Intrinsic and extrinsic motivation: The search for optimal motivation and performance (pp. 311–339). San Diego, CA: Academic Press.

Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational Psychologist, 41, 111–127.

Hossain, Farhana (2015). Serving Out-of-School Youth Under the Workforce Innovation and Opportunity Act (2014). Retrieved from http://www.mdrc.org/publication/serving-out-school-youth-under-workforce-innovation-and-opportunity-act-2014

Kazis, Richard (2016). MDRC Research on Career Pathways. Retrieved from http://www.mdrc.org/publication/mdrc-research-career-pathways

Manno, M.S., Yang, E., Bangster, M. (Oct. 2015) Engaging Disconnected Young People in Education and Work. Retrieved from http://www.mdrc.org/sites/default/files/2015_Engaging_Disconnected_Young_People_FR.pdf

Meijers, F. (1998). The development of a career identity. International Journal for the Advancement of Counseling, 20, 191–207.

Moore, R. W., Gorman, P. C. (2009). The Impact of Training and Demographics in WIA Program Performance: A Statistical Analysis. Human Resource Development Quarterly, Vol 20(4), Winter 2009 pp.381-396.

Nichols, S. L., Dawson, H.S. (2012). Assessment as a Context for Student Engagement In, Christenson, S.L. (Ed.), Handbook of Research on Student Engagement (pp. 457-477), New York, NY, Springer Science+Business Media, LLC 2012.

O*NET OnLine. Retrieved from: https://www.onetonline.org/

Perkins, M.L., Reeves, J.E., Mancheno-Smoak, L., Furlong, D.K. (2016), Assessment Program Impact on Successful WIOA Program Performance in the City of Los Angeles YouthSource Program. Retrieved from https://www.myinnersight.com/impact-study-v2/

Perkins, M.L., Reeves, J.E., Furlong, D.K. (2017), Assessment Program Impact Increases WIOA Performance in the City of Los Angeles YouthSource Program: A Replication. Retrieved from https://www.myinnersight.com/

Rickman, A. N. (2009). A challenge to the notion of youth passivity: Adolescents’ development of career direction through youth programs (Unpublished master’s equivalency paper). University of Illinois at Urbana–Champaign.

Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55, 68–78.

Schneider, B., & Stevenson, H. (1999). The ambitious generation: America’s teenagers, motivated but directionless. New Haven, CT: Yale University Press.

United States Department of Labor, Employment and Training Administration (2015). Retrieved from https://www.doleta.gov/performance/guidance/laws_regs.cfm

 

 

Appendix A

 

InnerSight Experience Evaluation

Date
Session Name and Time

Print Name:______________________________

For each statement below, please circle how strongly you agree or disagree with the statement.

1=Strongly Disagree(SD) 2=Disagree(D) 3=Neutral(N) 4=Agree(A) 5=Strongly Agree(SA)

	SD	D	N	A	SA
1. The information given was easy to understand.	1	2	3	4	5
2. The PowerPoint presentation was clear and enhanced the presentation.	1	2	3	4	5
3. The room temperature, lighting, and seating were comfortable.	1	2	3	4	5
4. I was able to hear and see both the presenters and the other participants.	1	2	3	4	5
5. Overall, this session was of value to me.	1	2	3	4	5
6. This session met or exceeded my expectations.	1	2	3	4	5
7. My inventory results made sense and sounded like me.	1	2	3	4	5
8. My InSight Guide Booklet is of high quality.	1	2	3	4	5
9. I felt the presenters listened to me and to the other participants.	1	2	3	4	5
10. I would recommend the InnerSight Experience to my family and friends.	1	2	3	4	5

What did you like most about the Experience?

What would you change about the Experience?

What additional comments do you have?

I give InnerSight permission to use my name, any part or all of my recorded or written statements, any or all of my recorded actions, and my likeness for training marketing and advertising, and commercial project(s) in whatever various media forms InnerSight may choose for the development and distribution of the project(s) and the works coming out of the project(s). I am not receiving any compensation for this permission and forever waive any rights to compensation. I am at least eighteen years of age.

Signature:_________________________ Date_____________________________