Challenge

​

• This project involved two users:

  • ​Marginalized students (female students in STEM, racial minorities, and first-generation college students – students whose parents did not attend college) who tend to underperform in college in comparison to their peers

  • Universities and colleges that are attempting to curtail this issue using workshops and other resources

• We addressed two different user needs:​

  • For students:

    • identify factors contributing to lower academic performance​

  • For institutions:

    • lower the costs of helpful resources by suggesting targeted and effective resources

      • develop a method to identify students at most risk of lower academic performance

      • identify techniques that can be used to improve their performance

 

Goal

 

To identify factors that affect the academic performance of marginalized students and develop a survey to measure the effect of these factors on marginalized students. 

 

Why is this project important?

​

The academic underperformance of marginalized students has been an issue that research has been grappling with since the 90's. 

​

College education has been shown to lead to a significant improvement in a person's earning potential and thus we need to ensure that individuals from all social groups have an equal chance at this improvement. 

​

This project provides an effective solution to improving marginalized students' academic performance that is economical and thus saves resources for educational institutions.

​

Results

​

• We identified a theory that explains why marginalized students tend to have lower academic performance that takes into account past theories regarding this issue.

  • Marginalized students are more susceptible to experiencing a lack of fit in college, which leads to lower performance.

• The survey we developed successfully measured how much fit these students experienced.

• Educational institutions can use this survey to identify students in most need of help

  • The theory we identified can be used to structure workshops in order to increase fit and academic performance​

​

Team

 

My team consisted of:

1. My supervisor, Dr. Diane Mackie - Distinguished Professor in the Psychological and Brain Sciences (PBS) Department at the University of California, Santa Barbara (UCSB)

2. Dr. Andrew Maul – Associate Professor in the Department of Education at UCSB

3. Audrey Aday – graduate student at the University of British Columbia, Vancouver

4. And my Research Assistants who are all undergraduates in PBS at UCSB

   1. Jessamy Johnson

   2. Mario Velasco

   3. Shalmali Patil

   4. Adela Ochoa Peralta

   5. Katie Sabini

   6. Katie Peterson

   7. Sharon Lanre-Orepo

   8. Midory Ibanez

   9. Ruth Ashaolu

   10. Marcos Leos

​

Research process

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Stage 1: Understanding what work has already been done

 

I conducted a literature review to find all the factors that have been shown to negatively affect marginalized students’ academic performance.

 

The factors:

• Stereotype threat

  • The threat a person feels when they are asked to perform a task that their group is stereotyped as being bad at

• Belonging uncertainty

  • The uncertainty a person feels about whether members of their group belong to in a certain environment

• Cultural mismatch

  • The experience a person has when their cultural values clash with the cultural values of the environment that they are in

• The lack of ambient belonging

  • The feeling a person has when the items (e.g., decorations) in an environment communicate a culture different from the culture they belong to

 

I also discovered a new model that encompassed all the above factors to explain why marginalized students would underperform in college. The model consisted of three types of fit that marginalized students were less likely of feeling in college.

 

The three types of fit:

• Self-concept fit: the extent to which students felt that they could be their true selves in college

  • A lack of this results in à stereotype threat and/or belonging uncertainty

• Goal fit: the extent to which students felt that the goals they had to pursue in college were the same as their personal goals

  • A lack of this results in à cultural mismatch and/or a lack of ambient belonging

• Social fit: the extent to which students felt that they were similar to others in college and/or were accepted by others in college

  • A lack of this results in à belonging uncertainty

 

I held an in-person presentation, where I discussed these findings with my team and got approval to develop a survey to predict marginalized students’ academic performance by measuring the three types of fit.

​

Stage 2: Gathering information from marginalized students

 

I had conducted a literature review, but I also wanted to hear from the population that I was studying, marginalized students.

 

I hired research assistants who were of diverse backgrounds and could speak to the experiences of marginalized students in college. I conducted a form of ethnography with my team, asking them questions about their experiences and putting myself in their shoes to truly understand the viewpoint of marginalized students in college environments.

 

Stage 2.1: Creating the interview questions

 

As part of my ethnographic study, I created open-ended questions about marginalized students’ experiences in college and interviewed several of my research assistants using these questions. My research assistants talked about issues they’d experienced, which mapped on to the model of fit.

 

Using the information my research assistants provided as well as what I had learned from the literature review, I created a series of interview questions. I then held a focus group with my research assistants where we discussed each of the interview questions and whether it would be suitable and useful for understanding marginalized students’ experiences in college.

 

Stage 2.2: Fine-tuning and finalizing the interview

 

Based on their feedback, I added new questions, removed questions that could have induced anxiety in my participants, and added more detail to questions that were deemed confusing.

 

I also created an interview script for these questions, as I wanted my research assistants to administer these interview questions to the participants. My intention here was to have marginalized students interview similarly marginalized students, to affirm a sense of authenticity and trust in the interview process.

 

To test out the interview script, I performed a form of contextual analysis where I asked my Research Assistants a few questions about their thoughts on the interview script and then performed a moderated usability test where I observed them run through the test script using each other as participants. There were points where the interviewee mentioned something that I wished for the interviewer to probe further, as well as points where the interviewee mentioned something that the interview script was not prepared to respond to.

 

To make the interview script more sensitive to these issues, I created more branches of interview questions and special rules to follow given specific answers.

 

Stage 2.3: Administering the interview

 

My research assistants received proper training for conducting interviews and then proceeded to conduct interviews on ~20 marginalized students. I interviewed ~5 students myself. 

​

I analyzed the interviews using thematic analysis.

 

Stage 3: Developing the survey

 

The interviews with marginalized students resulted in added knowledge to the topic of marginalized students’ experiences in college.

 

Using this information, I engaged in survey design where I created questions about the three different types of fit that could be answered on a rating scale.

 

In order to understand whether my survey was going to measure the academic experience of marginalized students, I designed a lab study, where I asked my research assistants to take this survey and provide feedback on it.

 

I also conducted think aloud testing of this survey by asking my research assistants to talk through their process of answering the survey, giving me an understanding of how they were comprehending the survey items and the response scale.

 

The response scale was found to have too many points (it ran from 1 – Strongly disagree to 5 – Strongly agree with a neutral middle point), so it was reduced to a 4 point scale, where participants could strongly agree, agree, disagree, or strongly disagree with an item but could not indicate that they felt neutral about it.

 

A few of the survey questions were flagged as confusing or redundant, so these questions were either adjusted or removed.

​

Stage 4: Data collection and analysis

 

The data was administered to ~650 participants of different marginalized groups.

 

My first step was data manipulation, where I removed unnecessary columns, merged all the tables of data (for each separate marginalized group) into one large datafile, and flagged potentially untrustworthy entries. I used R for this step but have since then learned to carry out the exact same steps using SQL.

 

I had collected my data using both Amazon MTurk and Prolific, so I wished to know if the participants in the two platforms differed from each other in a distinct way. I conducted A/B testing to determine whether Amazon Mturk participants and Prolific participants differed in terms of the errors they made in the survey (on attention-check questions) or in their general response patterns. Results showed that there was no distinct difference.

 

I then conducted a form of benchmark testing on my data. Research had defined three dimensions for marginalized students’ academic experience (self-concept fit, goal fit, and social fit), so I conducted a multidimensional Rasch rating scale model analysis on the data to find out if my data fit this model. Our team’s expectation was that students would respond to questions measuring their self-concept fit in a way that differed from how they responded to questions measuring their goal fit, which in turn would differ from how they responded to questions measuring their social fit. This model analysis also tested whether students with similar levels of fit responded to the same questions in a similar manner (e.g. two students high on social fit both agreed with an item measuring social fit).

 

Results revealed that students responded very similarly to questions regarding their self-concept fit and social fit. This suggests that students may think of their social relationships as part of how they see their identity fitting into the environment. It does not affect the usability of the survey, but rather lends thought to the theory this survey was established on.

 

In terms of the usability of the survey, our results revealed that students with similar levels of fit responded to the same questions in a similar manner. This finding established that this survey could be used for future work measuring marginalized students’ academic experience.

 

Since completing this project, I have engaged in advocating my research to others. I walked through my analyses with my research assistants and have explained my findings to undergraduate students with very little background on factors affecting the academic performance of marginalized students. I have also presented my work to experts in the fields of psychology and diversity, and to groups invested in increasing the engagement of marginalized students.

 

The long-term goal is for this survey to be used by colleges to measure the extent to which marginalized students feel like they fit into college, and to use this information to connect marginalized students especially likely to feel a lack of fit to workshops and groups that can help them feel like they belong and protect them from underperforming in their classes.

​

Constraints

​

I had certain constraints on my study.

​

The first was time. This project was also the Honors Thesis project of my student and mentee, Jessamy Johnson. Thus, all aspects of the project from conceptualization to presentation of results needed to be completed within a year.

​

Although this curtailed the time I got to spend developing the survey before gathering quantitative data on it, the results of the model analysis showed that the usability of the survey was high enough.

​

My other constraint was that I could not confirm the relationship between fit and academic performance through an experimental test.

​

To remedy this issue, I am currently running an experiment, where I test this relationship. (Due to the ongoing nature of the study, I cannot provide further details as of now).

​