Graduate medical education · Applied generative AI, document processing · Built by Ryan Murphy
Every application reviewed in full — and a better predictor of fit than the process it replaced.
Summary
Tulane's surgical residency program receives thousands of applications a cycle, and 800 were manually reviewed by ten surgeons and three program directors. Reading them consumes clinical and teaching hours, and the volume meant most of the field was cut by a single test score before anyone read an application in full. We built a system that scores every complete application — personal statements, letters of recommendation, the whole file — against a rubric drawn from the program's own values. Faculty then reviewed half as many applications more carefully, with two readers each, producing 80 interviews.
The problem
Every hour a surgeon spends reading applications is an hour not spent in the operating room and not spent teaching. Thirteen faculty against up thousands of applications is a volume problem with no good answer.
The practical answer most programs reach is a test score cutoff. It's fast, it's defensible, and it means the majority of applications are never read in full by anyone. Whatever is in a personal statement or a letter of recommendation from someone who has watched an applicant work simply doesn't enter the process for most candidates.
The applications that did get read had a second problem: the score depended significantly on which faculty member happened to read it. Some graded hard, some generously, and an applicant's outcome varied with an assignment that was essentially random. This was known. It's difficult to fix by asking busy people to be more consistent.
What we built
A scoring system that reads the complete application against a rubric built from the program's stated values — every application, not a filtered subset.
The output is not a decision. It's a consistent first read that gives faculty a differentiated starting point, so their attention goes where it matters most. Human reviewers made every actual determination, with two readers per application to limit individual grader effects.
Python, running locally on Tulane's own machines, calling an LLM API with a rubric for each document type — one for personal statements, one for letters of recommendation.
Ingestion had to handle whatever the application files actually were. Most parsed cleanly with PDF-to-text libraries; the rest were scanned or otherwise unextractable, and those went through machine vision to get usable text. Once a document was text, it was scored against its rubric.
Deliberately, there's no infrastructure here. Applicant files never left institutional systems, nothing was stored anywhere Tulane didn't control, and there was no pipeline to secure because there was no pipeline. For a program handling 800 people's personal statements and letters written in confidence, the right architecture was the smallest one that did the job.
What was hard
The rubric as a prompt, by a wide margin.
Getting it to reflect what the program actually valued was one problem. Getting it to produce enough differentiation was another — a rubric that scores every strong applicant identically is useless for the thing a program needs it to do. Most of the work was in that tuning.
Then we back-tested. We ran the rubric and prompts against prior years' applications and compared the output to how faculty had actually scored those same candidates, which is what told us whether the prompting and model selection were calibrated or just plausible-looking.
Results
Every application received a full review. Where the prior process cut the field on a single test score, all applications were read in their entirety — statements, letters, and all — which changed what the program was selecting on.
Better predictive accuracy than the process it replaced. The rubric scores correlated more closely with applicants' post-interview scores than either the prior test-score screen with the single-reviewer. In other words: the automated first pass was a better signal of how the program would ultimately assess a candidate than the entirely human approach.
Why this one matters
The assumption about this kind of work is that automating it means caring less about the people being evaluated. Here it went the other way: the manual process was cutting most of the field on one number because there was no other option at that volume. Reviewing everything, consistently, is something the program wanted to do and couldn't, and the new review process worked better than the manual one.
Related services: Agentic Automation · Application Development
Let's talk
Let us know if you've got something that needs shrinking.