Testing meritocracy in university admissions
Top universities around the world are routinely criticised for encouraging social elitism in their undergraduate admissions. Yet the universities’ own attempts at increasing diversity are viewed by others as undermining academic merit to appease political demand. A different concern is that under-prepared students admitted via affirmative action cannot cope with the workload, which hurts their ultimate career-prospects (Sander 2004, Arcidiacono et al. 2016). These issues are frequently discussed by politicians and by news media, and have even resulted in several high-profile lawsuits, most notably in the US.1
In the UK, much of this debate concerns undergraduate admissions to Oxford and Cambridge, collectively known as ‘Oxbridge’. For example, a much-cited 2018 Sutton Trust report revealed that between 2015-17, Oxbridge “recruit(ed) more students from eight top schools than almost 3,000 other UK state-schools put together”2 – six of these eight being expensive, private schools which typically cater to wealthier households. On the other hand, the fraction of offers made to female applicants in both universities has been close to or higher than 50% in recent years. In their online admission statements, both universities claim to be looking solely for academic promise irrespective of social background.3 Given this context, an interesting question is whether and to what extent the aggregate admission statistics result from merit-based entry.
An economic characterisation of meritocratic admissions is that all applicants are held to the same bar of academic promise, with social/demographic background used only insofar as they predict intellectual merit. Had the admissions bar been higher for one group, the marginal (i.e. the weakest) entrants from that group would perform better academically in post-entry assessments than the marginal entrants from the favoured group. This notion of outcome-based fairness, going back to Becker (1957), has been used by researchers to test for fair decision-making in various settings including college admissions (e.g. Bertrand et al. 2010). However, detecting the marginal applicant is impossible when admissions are based on many separate indicators of academic merit, some of which are used by admissions officers but unobserved by researchers (e.g. the quality of confidential reference letters), leading to an ‘infra-marginality’ problem. Furthermore, the ad-hoc practice of predicting future academic performance for applicants by regressing the outcomes of admitted past students on their pre-entry characteristics and test scores suffers from classic sample selection bias. These problems apply to many other settings where discrimination is a concern, such as law-enforcement and legal sentencing (Arnold et al. 2018).
Arcidiacono et al (2015) offer an overview of the existing empirical research literature on race-based affirmative action in US college-admissions. For the UK, Bhattacharya et al. (2017) investigated merit-based admissions using a small dataset for a single subject at a different university. Their approach was not outcome-based, and addressed the infra-marginality problem by assuming positive association between observables and unobservables (Bhattacharya 2013). Educational sociologists have also documented lower application success rates at selective UK universities for ethnic minority and state school students with similar observable qualifications as their ethnically white and private school counterparts (Boliver 2013, Zimdars et al. 2009). These studies do not consider the role of unobservables, or any post-admission outcomes.
In a recent paper (Bhattacharya and Rabovic 2020), we propose a novel outcome-based test for meritocratic admissions using applicant-level, administrative micro-data from Cambridge, matched with post-admission performance of admitted students in Cambridge’s blindly marked internal exams, which we take to be our measure of merit. Our strategy to bypass the infra-marginality problem is to use an institutional feature of Cambridge-admissions whereby a subgroup of applicants is admitted in a second stage through the ‘pool’. Students placed in the pool are those perceived to be less meritorious than the first-round direct admits, but worth keeping in contention. The data can identify this subgroup. The applicants perceived to be the best among the pooled may eventually be admitted, depending on availability of spots after the first round. Therefore, comparing the post-entry exam performance of pooled admits of social group g (who are below the bar for direct entry of g-types) with that of the direct admits of group h (who are above the bar for direct entry of h-types) can reveal whether the criteria used to rank applicants in the first round are statistically consistent with a meritocratic goal. In particular, if the former perform systematically better in post-admission exams, then that suggests that g-types face a higher bar of academic merit for direct admissions than h-types, whether or not we observe all applicant characteristics observed by the admissions tutors.
We apply this idea to administrative data on nearly 8,000 students who entered Cambridge between 2013 and 2017 to study one of six undergraduate subjects with highly competitive entry (Economics, Engineering, Mathematics, Natural Sciences, Law and Medicine). Nearly 20% of these students were admitted from the pool. The aggregate admission success rates are nearly equal between men and women (18.89% and 18.04%, respectively), while students from independent (i.e. fee-charging) private schools had a considerably higher success rate (28.7%) relative to those from state-schools (20.65%).
Following entry, in each year of their study, students sit exams which are centrally set and marked blind. We use the aggregate scores for each student, expressed as percentages of the total and standardised by subject, as our measure of academic merit.
Applying our method to these data, first with the groups g and h denoting men and women, respectively, we obtain Figures 1 and 2, which plot the cumulative distribution function of first year exam-scores for four subgroups of admitted students – pooled men, non-pooled men, pooled women, and non-pooled women – for ESTEM (STEM plus Economics) and non-ESTEM (Medicine and Law), respectively. The height of each curve at x denotes the fraction of students in that category scoring below x, so a lower curve implies better performance.
In ESTEM, we see clear evidence that the performance distribution of directly admitted men first-order stochastically dominates (FOSD) the rest, followed by pooled men, non-pooled women and, finally, pooled women. The fact that pooled men have stochastically higher exam scores than non-pooled (i.e. directly admitted) women entrants throughout the distribution implies a higher admission bar for men. In contrast, pooled women are dominated by both the pooled and the non-pooled men. FOSD also implies that the gender-gap cannot be explained by risk considerations (such as male students being more ‘risky’), because any increasing utility function preserves the distributional dominance. On the other hand, in the non-ESTEM but highly competitive subjects of Law and Medicine, the performance distribution of pooled men and non-pooled women are seen to be very similar.4
This graphical evidence is corroborated by regression analysis, which shows that compared to non-pooled omen applicants, the pooled men in ESTEM score about 0.17 standard deviations higher, implying that the men face an admissions threshold that is at least 0.17 standard deviations higher than women. This finding is robust to inclusion of various controls, including various college-fixed effects,5 fraction of female students in the subject/college, year and subject fixed-effects.
Next, we investigate whether the gender gap in the first year, when all students sit the same examinations, persists into later years when students sort into different optional papers. When judged by third-year exam performance in ESTEM subjects, the gap shrinks considerably (0.17 versus -0.049 standard deviations). This narrowing could either reflect ‘catch-up’ by female students or could be a result of efficient sorting into optional papers.
Finally, we apply our methods using candidates from independent and overseas schools (called “others”) as group g,6 and compare them with group h consisting of candidates from the state-maintained, free UK schools. The corresponding CDFs, plotted in Figure 3, show that the non-pooled dominate the pooled within and across school types, and the CDF for pooled others and directly admitted maintained are almost identical, which is also corroborated by regression results accounting for various controls and fixed effects. The above findings suggest that while there is some suggestive evidence of lower admissions standards for state school applicants, the strength of this evidence is much weaker than the gender results.
To conclude, the overall empirical finding from our test is as follows: there is strong evidence of higher merit thresholds for male applicants in STEM and Economics where men are a significant majority; however, this evidence weakens if one uses performance in later years of study, when students sort into optional papers, as the measure of merit. In the non-STEM but competitive fields of law and medicine, with gender parity in enrolment, these gaps are non-existent in every year of study. On the other hand, the evidence of a higher admission bar for private school applicants is at best weak across subjects and years. These findings contrast sharply with the aggregate admission success rates, which are equal across gender but significantly higher for private school applicants.
Arcidiacono, P, E M Aucejo and V J Hotz (2016), “University differences in the graduation of minorities in STEM fields: Evidence from California”, American Economic Review 106(3): 525-62.
Arcidiacono, P, M Lovenheim and M Zhu (2015), “Affirmative action in undergraduate education”, Annual Review of Economics 7(1): 487-518.
Arnold, D, W Dobbie and C S Yang (2018), “Racial bias in bail decisions”, The Quarterly Journal of Economics 133(4): 1885-1932.
Becker, G (1957), The Economics of Discrimination, University of Chicago Press.
Bertrand, M, R Hanna and S Mullainathan (2010), “Affirmative action in education: Evidence from engineering college admissions in India”, Journal of Public Economics 94(1-2): 16-29.
Bhattacharya, D, S Kanaya and M Stevens (2017) “Are university admissions academically fair?”, Review of Economics and Statistics 99(3): 449-464.
Bhattacharya, D (2013), “Do elite universities admit the academically best students?”, VoxEU.org, 13 April.
Bhattacharya, D and R Rabovic (2020), “Do Elite Universities Practise Meritocratic Admissions: Evidence from Cambridge”, working paper.
Boliver, V (2013), “How fair is access to more prestigious UK universities?”, The British Journal of Sociology 64(2): 344-364.
Chan, J and E Eyster (2007), “Traditional measures to increase diversity in US universities penalise the poor”, VoxEU.org, 05 April.
Estevan F, T Gall, P Legros and A Newman (2017), “How university admissions can help integrate secondary schools”, VoxEU.org, 23 May.
Sander, R H (2004), “A systemic analysis of affirmative action in American law schools”, Stanford Law Review 57: 367-483.
Zimdars, A, A Sullivan and A Heath (2009), “Elite higher education admissions in the arts and sciences: Is cultural capital the key?”, Sociology 43(4): 648-666.
1 For example, Regents of University of California v. Bakke 1978, Grutter vs Bollinger (University of Michigan 2003), and Fisher v. University of Texas 2016.
4 The same pattern is also observed within each individual subject comprising ESTEM and non-ESTEM.
5 Cambridge admissions are conducted by the constituent colleges on behalf of the university. We check robustness to both application-college and admitting college fixed-effects; see Bhattacharya and Rabovic (2020) for more details.
6 The two groups are combined because both attract candidates from relatively more affluent backgrounds