LTM Programs Evaluation

Introduction
The London Transport Museum (LTM) has launched its first employability programmes for 18 to 25-year-olds, including the Young Freelancer, the Young Volunteer Projects, and Route into Work.

To better understand young people's needs, LTM assembled an age-matched six-person research team of young individuals to assess and enhance these programmes' effectiveness.

Explored Participants' Needs Through Creative Methods

Problem Statement
The London Transport Museum (LTM) must design accurate questions and use appropriate methods to collect participants' feedback, ensuring that the programs meet the needs of young people and provide practical career support.

Uncovering User Needs: What do young people generally want from the program?

Through Manifesto panel discussions and creative workshops with program participants, the research team gained an in-depth understanding of young people's career challenges and desired program support.

Manifesto panel discussion

Manifesto panel discussion
We created a declaration that outlines the goals and core values young people hoped to achieve through the programme.

Manifestos

Creative workshop
We adopted the relaxed method of collaboratively creating artistic posters to guide young people in writing down their responses to the manifestos.

Young people’s response to the manifesto

Diving Deeper into Participants' Experiences Across Different Programmes

Build up persona
During this phase, we mapped out the different programs to understand the user journey and context, connecting participants' needs with the manifesto and theory of change to develop an evaluation plan.

Persona

Persona

Emotional journey map
We conducted a journey mapping exercise for young individuals to understand their emotional progress from the programme. We aim to quantify their progression from an emotional aspect, providing a motivational tool for reflection.

Emotional journey map

Who Can Drive Change and How It Will Shape the Future？

After the manifesto analysis, the team used the Theory of Change framework to bridge the existing activities and young people's expectations.

Theory of Change framework

We brainstormed through workshops to implement the manifestos into their respective domains and then anticipated the potential short-term and long-term impacts.

Brainstorming workshop

Manifesto+Theory of Change framework

Tailored Evaluation Plans for Each Programme to Collect Data Effectively

Design Evaluation Tool
Capturing Data from Multiple Angles with Precise Tools
Given our interviewees' backgrounds—young people from less privileged circumstances—we designed our approach to minimise dynamic pressure during manager-led interviews. We developed diverse research tools to gather both qualitative and quantitative data:

Entry and Exit Interviews

RAG (Red, Amber, Green) Ratings

Skills Wheel

Head, Heart, Handbag, Bin/Feet

Make Evaluation Plans for each program
After developing the theory of change, an evaluation plan is essential to implement. While the ToC outlines desired outcomes and assumptions, the evaluation plan defines how we will measure success, collect data, and assess progress on specific programmes separately.

Evaluation plan template

The evaluation tools focused on skills development, professional growth, and networking—critical areas identified as young freelancer’s priorities.

Questions are designed based on a low-pressure environment for creative exploration rather than direct career development, aligning with volunteers’ preference for creativity.

The apprentice evaluation plan focuses on assessing time investment, enhancing existing skills, improving networking, setting clear expectations, and tracking long-term impacts.

Data Collection

Data CollectionData collection was conducted internally by the London Transport Museum (LTM) team using the following methods.

Interview

Feedback Workshop

Findings

General Findings
Networking is essential and valued across programmes
More consistency around skills language and definitions is needed – e.g. applied skills, soft skills, employability skills, employability
Qualitative data the most useful for understanding impact, especially considering the sample sizes

Young Freelancers Findings
There is much more emphasis on improving existing skills rather than forming new ones, which speaks to young people's experience when they join the programme.

Communication skills look like they've reduced, but they were among the highest to start with, so this could be why.

Networking skills doubled, but there is a lot of appetite for more opportunities.

CV skills for future jobs are also an essential part of the programme for participants.

Young Volunteers Findings
Participants prioritised creative and soft skills over employability, valuing a low-pressure environment to explore new skills.

Evaluation data revealed limited emphasis on employability, questioning if it should be a programme focus.

The findings highlighted creativity as necessary, frequently mentioning meeting artists as a valuable networking opportunity.

The programme often establishes a lasting connection with LTM, which should be tracked for re-engagement.

The Route into Work Findings
Qualitative Data: More effective in capturing the programme's impact.

Confidence Levels: The lowest change observed in participants' job-seeking confidence, indicating a need for further investigation.

Overall Success: The programme meets or exceeds expectations, with a +38% average positive change.

Other Considerations: Engine team members need to gain direct experience with the programme; case study data is pending.

Young Freelancers Recommendations
Recommendation 1: Monitor overworking on briefs, as young people aren't paid enough time to develop the required skills to a high standard.

Recommendation 2: Expand networking opportunities by including more 'guests' in monthly training workshops, addressing the high demand for this skill.

Young Volunteers Recommendations
Recommendation 1: Ensure room comfort and good food options to maximise participant engagement.

Recommendation 2: Focus on building foundational relationships with LTM by offering opportunities for creativity, new interests, and life skills instead of career emphasis.

Recommendation 3: Build a community of young people with shared interests to extend engagement and strengthen peer networks through events like Lates.

Route into Work Recommendations
The Engine did not have many programme recommendations for Route into Work, but it would like to see representation from this programme in next year's Engine group to support it in the future.

Young Freelancers Recommendations

Recommendation 1 : Monitor overworking on briefs, as young people aren’t paid enough time to develop the required skills to a high standard.

Recommendation 2 : Expand networking opportunities by including more 'guests' in monthly training workshops, addressing the high demand for this skill.

Young Vlolunteers
Recommendations
Recommendation 1: Ensure room comfort and provide good food options to maximise participant engagement.

Recommendation 2: Focus on building foundational relationships with LTM by offering opportunities for creativity, new interests, and life skills instead of career emphasis.

Recommendation 3: Build a community of young people with shared interests to extend engagement, strengthening peer networks through events like Lates.

Route into Work
Recommendations
The Engine did not have many programme recommendations for Route into Work but would like to see representation from this programme in next year’s Engine group to support this for next time.

Reflection

This experience has taught me the importance of approaching the evaluation from both the organisation's and the user's perspectives. By understanding the needs and expectations of both sides, I learned how to bridge these gaps and assign responsibilities accordingly. This approach not only helps align expectations but also identifies critical opportunities for evaluation.

Next Steps
The project is ongoing, and we are currently gathering information to support the 2024 evaluation needs. We are also exploring the potential of designing an internal digital product for LTM to store and manage this information effectively.

Project Menu

The Future of TfL Go

Ohisama Sushi Digital Menu

FoodCLUB