By C. Aaron Price and Faith R. Kares
From Dimensions
July/August 2016
In the 11 years since the inception of the Science Minors and Achievers program, the Museum of Science and Industry (MSI), Chicago, has had plenty of anecdotal evidence that this out-of-school time program has a meaningful impact on its participants. Word of mouth was the strongest driver of recruitment, and teens were open with staff about other aspects of their lives. These and other characteristics suggested a deep connection between teens and the program. As the program grew in popularity and size, the museum wanted to see whether these informal observations survived more rigorous scrutiny and, if so, to look for reasons behind these connections.
A study we conducted in 2015, when the program celebrated its 10th anniversary, surveyed alumni to answer the questions, “Where are they now?” and “How did the program help them get there?” While the findings were helpful, they left us wanting to draw a more causal connection between the program and its ultimate impact on the life of its participants, including those who did not necessarily have a positive experience and were less likely to respond to a survey. Thus, we developed a five-year study of current and future participants funded by a (U.S.) National Science Foundation grant. The study begins in mid-2016 and aims to determine the impact of participation in the museum’s program on the participants’ educational/career choices and overall relationship with science later in life.
Science minors and achievers
Currently, about 140 youth ranging in age from 14 to 18 participate each year in the Science Minors and Achievers program, which aims to help teens discover new interests in science and technology, develop public speaking and leadership skills, and prepare for college and careers. The typical participant is active in the program for more than two years while in high school, beginning in the Science Minors program and then graduating to the Science Achievers program after completing a 10-week session focused on scientific content and communication, and contributing 50 service hours to the museum. The Science Achievers program also encompasses 10-week sessions that occur three times per year, with participants free to take part in as many as they want. Programming addresses youth leadership; college readiness; public communication; science, technology, engineering, and math (STEM) topics; and more. While encouraging and preparing teens for STEM careers is not the singular goal of the program, this motivator is a guiding force in fostering other skills.
A first look
For the 2015 study, 167 Science Minors and Achievers program alumni returned surveys, and 28 of those alumni participated in interviews. We reached out to 575 alumni for whom we had contact information—almost everyone who had been in the program up until that point.
The results demonstrated that the program has a remarkably different impact on career interests of male and female youth. (Note: When we used an open-ended question to ask for participants’ gender identification, participants listed male, female, or no clear answer. However, we also recognize that gender and social identities are not fixed, and not all youth identify with a gender binary.) One of the most striking findings was that female alumnae reported a much higher increase in STEM educational and career interest while they were in the program than their male peers. The program model has no gender-specific programming that could easily explain the difference. We also found a gender difference in another part of the survey in which alumni were asked to compare program staff with their high school teachers. With a rating of 10 signifying “a lot like teachers,” female respondents rated the staff at 5.7 while male respondents rated them at 5.1, a statistically significant difference. The survey data did not reveal many other gender-related differences in the results, including those related to attitudes toward science.
The semistructured and open-ended nature of the interview questions helped us interpret the survey results. As a group, female alumnae were more likely to describe museum staff as teachers and family, while male alumni regarded them more as mentors. Students commonly defined staff members as teachers by identifying them as sources of content and technical knowledge (e.g., “They were like the best school teachers . . . . I definitely looked to them as a knowledge base.”) They also compared staffers with family members (e.g., “They loved us like any good parent.”) Quite often the alumni reported that staff actually served in familial and/or teacher roles, forming relationships otherwise missing in their lives.
In addition to relationships with staff, according to an analysis of interview data, other program elements that were key to success included the extensive amount of time students were able to participate in the program, the diversity of the youth (not only in demographics but also in personality), and the role the museum plays as both a physical space and a community institution. Notably, alumni did not specifically focus on the program’s STEM content, instead referencing it only while describing other aspects of their experience. For example, they would eagerly discuss how the program had influenced their self-esteem, and then mention how that impact was related to their current relationship with science. Science was ever-present in the interview discussions but was never the focus—it was the lens through which other discussions took place. Therefore, we conducted our analysis from a youth-development perspective rather than within a science-education framework.
The study confirmed some intuitive feelings among museum staff while also revealing surprising findings. The importance of the significant length of the program, for example, was well known. And the descriptions of the relationships as familial were not surprising either. Indeed, such findings are consistent with those of a recent study of arts-based adolescent development programs, which found that 75% of alumni referred to their program experience as being more influential than family (Linzer & Munley, 2016). However, the influence of gender on both career interests and participants’ perceptions of staff was unexpected.
So where are they now? Less than half of alumni are currently in a STEM field, but at rates higher than the general population. They also still maintain positive attitudes toward science. One participant officially listed art as a current career, but in the interview still mentioned self-identifying as a scientist—thinking like a scientist and applying science to daily life. There is a significant drop-off in female alumnae interest in STEM careers while in college. This is a well-known problem in the field and one that we hope to learn more about with our next study.
While the results were revealing and informative, the study had some shortcomings and left some gaps in our knowledge. First, those with positive attitudes were more likely to participate in the survey and interviews, even though we offered substantial incentives to encourage more widespread and representative feedback. Second, without a comparison group we cannot attribute the overall findings to the program itself (although the gender-specific findings can be related to the program since male and female students participated equally).
The next phase
Partly to address the limitations of the initial study, the museum has launched the Developing YOUth! Project, a five-year study of current and future program participants. The study is longitudinal, meaning that it will follow participants over a long period of time. It is also semiexperimental, meaning that it will follow control groups. Researchers will thus be able to draw stronger links between the results and participation in the program by controlling for factors such as prior interest in STEM. The project will follow three cohorts of program participants from the time they leave the program—most commonly coinciding with their high school graduation. Developing YOUth! will use a wide variety of instruments and frameworks that have been designed and validated for use with populations underrepresented in STEM fields. For example, we ask youth to define what the word “science” means to them and then use that definition in our analysis of the rest of the survey. This is in response to previous research by others describing how different cultural groups view the word “science” differently. (See Zacharia & Calabrese Barton, 2004.)
We will collect quantitative data through surveys, tests, and program participation metrics. In addition, we will collect qualitative data through semistructured interviews and deep-immersion ethnographic research, which involves building long-term relationships with both the participants and their community (family, friends, etc.) Two comparison groups will be recruited among museum visitors and participants in other afterschool science clubs in the Chicago area.
The Science Minors and Achievers program is demographically diverse. The alumni we are studying self-reported as 43% African-American, 24% Hispanic/Latino, and 15% White (non-Hispanic); the remaining categories each represented less than 6% of the respondents. In contrast to a one-size-fits-all approach to research, each identified group will be studied individually. Our project draws upon, and will contribute to, research on Positive Youth Development (PYD) theory and practice, which holds that all youth regardless of backgrounds have potential to thrive. (See Lerner, Dowling, & Anderson, 2003.) PYD programs meet the cognitive, social, and emotional needs of youth by fostering positive relationships with adults, providing safe environments, building confidence through meaningful work, and encouraging leadership opportunities. (See Larson, 2000.)
Our study and subsequent analyses will draw on several key theoretical frameworks. The first of these is hybridity, defined in this context as how participants combine elements of home, school, and their social life to create a unique space for themselves within the museum. We will consider identity, meaning how participants see themselves. The study will take into account agency—the ability to control and influence one’s own life’s experience. By factoring in the culturally relevant and community-specific needs of the participants, and in keeping with emergent voices in the field that call for integrating race/ethnicity as context rather than mere variables, we will be able to independently investigate on a variety of scales from the very small to very large (Williams & Deutsch, 2016). This will require multiple assessment techniques, long-term relationships with participants, and the involvement of a community of advisers (from academic as well as lay and practitioner backgrounds) at every stage of the process. A recent study of informal STEM programs for girls found that consistent use of a variety of advisers (including participants themselves) provides a rare level of reflection and the multiple types of perspectives needed for studying such complex relationships (McCreedy & Dierking, 2013).
By combining research on current program alumni with a study of future alumni, we hope to more deeply understand how out-of-school time programs ultimately affect participants’ career decisions and relationships with science. In terms of broader impacts, rigorous, culturally sensitive investigations provide policy makers and program designers with the feedback they need to make key decisions about how best to support these programs.
C. Aaron Price is manager, evaluation and research, and Faith R. Kares is postdoctoral researcher/project manager at the Museum of Science and Industry, Chicago.
About the image: Science Achievers program participants lead demonstrations on the museum floor. Photo by JB Spector/Museum of Science and Industry, Chicago
REFERENCES
Larson, R.W. (2000). Toward a psychology of positive youth development. American Psychologist 55(1), 170–83.
Lerner, R.M., Dowling, E.M., & Anderson, P.M. (2003). Positive youth development: Thriving as the basis of personhood and civil society. Applied Developmental Science 7(3), 172–80.
Linzer, D., & Munley, M.E. (2016). Room to rise: The lasting impact of intensive teen programs in art museums. New York City: Whitney Museum of American Art.
McCreedy, D., & Dierking, L.D. (2013). Cascading influences: Long-term impacts of informal STEM programs for girls. Philadelphia: The Franklin Institute.
Williams, J.L., & Deutsch, N.L. (2016). Beyond between-group differences: Considering race, ethnicity, and culture in research on positive youth development programs. Applied Developmental Science 20(3), 1–11.
Zacharia, Z., & Calabrese Barton, A. (2004). Urban middle school students’ attitudes toward a defined science. Science Education, 88(2), 197–222.