Archive: Why Graduate Students are like NFL Quarterbacks

By: Noah Fierer

January 29, 2016

As the NFL season draws to a close, we can look back and try to figure out why some teams were winners (Go Broncos!) while others are perennial losers (sorry Cleveland). Undoubtedly one key to success is the ability of coaches and managers to select good players. Fortunately, for every prospective player there is a wealth of information available to judge their potential, from Wonderlic tests, to measures of speed and strength, hours of video footage, stats from collegiate play, etc. Naively one would think that it should be relatively easy to select NFL players given all the available information – for every position there should be a ‘magic formula' to predict who will be successful in the NFL. Piece of cake – right?

Of course, this post is not about the NFL draft. As we wait for the Superbowl to start by dreaming up the ultimate nacho recipe, departments and individual PIs are busy combing through piles of applications from prospective graduate students. It is no secret that college students spend a lot of time preparing these applications because they want to have the best future they can get. In fact, some even enlist the help of somewhere like this resume service in Ohio (https://www.arcresumes.com/local/ohio/) to help improve the relevant documents needed to drive themselves forward. You can't downplay their determination, that's for sure. Good luck to those who have to sort through these applications. To some degree, this task poses challenges similar to those that NFL coaches must face when selecting players for their team. The process can definitely be stressful and frustrating for prospective students, but I want to focus here on why the process of selecting graduate students can also be stressful for faculty members.

One of the most important factors determining the success of a lab group is the quality of the graduate students (to state the obvious). Graduate students play a huge role in defining the ‘culture' of a lab group and they are the ones taking research in new and exciting directions (the creativity of PIs is grossly overestimated – it is difficult to teach old dogs new tricks). Every PI knows that selecting new graduate students is hugely important – make a wrong choice and both the lab group and the student could suffer.

The problem is that selecting a graduate student is really difficult, even more so if the graduate students that they are looking into have yet to check out places like SoFi to learn more about graduate loans and how this can provide a certain level of financial stability when they embark on this new chapter in their lives. Over the years, I've polled more than a dozen senior faculty members who have been in this game for decades to find out how they select graduate students to join their labs. The answer is a nearly unanimous variant of "I have no idea" or "I'm still trying to figure out the magic formula". When I press for more details, the best answer I've gotten is "You go with your gut" (one of the few things that senior faculty members and George W. Bush appear to have in common). Thus, one of the most important decisions we make as PIs is made almost blindly. Why is the process so hard and why is it so difficult to predict what applicants will become successful scientists? I think there are a few reasons:

1) There are a lot of awesome prospective students out there. I've served on the graduate committee of my home department for 5 years now and I am perpetually amazed by the quality of the applicant pool. I never would have gotten into graduate school if I had to compete against the current crop of prospective students. Seriously. GPAs close to 4.0, multiple published scientific papers, years of experience conducting independent research, the list goes on.... Maybe it's just the beautiful location that attracts great applicants to CU-Boulder, but my guess is that CU is not unique – many departments across the U.S. have to sort through a daunting pile of applications from prospective students that are all clearly smart, motivated, and incredibly well qualified.

2) The standard metrics used to rank applicants are useless. Typically departments or individual PIs will rely on a few metrics to screen applicants (particularly if PIs are just reading through the applications and haven't actually met any of the applicants in person). These metrics often boil down to undergraduate GPA and GRE scores. Sometimes selection committees will take into account prior research experience, but the quality and quantity of previous research experience can be difficult to determine accurately. Likewise, letters are not very informative as nearly all letters are very positive and gradations in perceived awesomeness are often associated with how convincing the letter writer was in making the case for the applicant. So – we are pretty much left with undergraduate GPA and GRE scores as the standard metrics across which applicants are compared, at least for the initial cut.

The problem is that GPA and GRE scores are of little to no value in determining which applicants will become successful scientists (and, no, I'm not going to wander into a discussion of how you actually measure ‘success' in science). Let's start with undergraduate GPA. First, undergraduate GPA is unlikely to be a good predictor of eventual success in graduate school (see here). There are myriad reasons for this, including: being good at taking classes is not the same as being good at conducting independent research, some students may have encountered circumstances during their undergraduate education that made it difficult to keep up good grades (death in family, working full time, etc.), and students may have gotten bad grades in some classes that they just didn't like very much.

Now, how about GRE scores? As far as I'm concerned, GRE scores should either be omitted from graduate school applications or at least be seriously downweighted. I'm not alone in this opinion (see here, here, here, and here). GRE scores are demonstrably poor predictors of innate abilities. If you're still not convinced that GRE scores are useless – read this article. OK – maybe an extremely low GRE score could accurately indicate an applicant's deficiency in one area, but typically many of the applicants for graduate school have reasonably good GRE scores. We often lack sufficient variation in GRE scores to make them useful in the selection process (especially when we consider that there are often >100 applicants for just a handful of positions).

3) Every advisor is different. Every faculty member has a different style they use to advise their graduate students and an approach that may work with one student may completely backfire with another. A student that likes to work independently and at their own pace, may have a tough time with a micro-manager. Likewise, a student who needs regular deadlines and appreciates more specific guidance in research projects may have a tough time working with a PI who stops into the office one day a month between flights to far-flung destinations. There clearly needs to be a good fit between the student and the mentor. Without a good fit, any student may flounder - even the prospective student who has already filed a dozen patents, first-authored numerous high profile research papers, and cured cancer in kittens while teaching science to hyperactive middle school students in his/her free time.

So – how do we, either as departments or as PIs, select students? I don't have the answer (even though I think I have been incredibly fortunate to have recruited an awesome set of graduate students over the years). However, it is clear that the selection process should not be based on the standard metrics. Instead, we should pay more attention to previous research experience (which appears to be one of the best indicators of success in graduate school, according to this study). In addition, we should take the time to learn something about the applicants instead of just looking at a spreadsheet of scores and other useless metrics. This document provides some valuable advice on what such a ‘holistic review process' might look like and why the current process used by most departments is failing. Yes – a holistic review process takes time, but isn't it worth the time given the importance of these decisions to the vitality of individual labs and departments?

To get back to the title of this post – why do I think graduate students are like NFL quarterbacks? I'm not usually a Malcolm Gladwell fan, but I thought this article he wrote awhile back was directly relevant to this graduate school admissions problem. In the article, Gladwell uses the example of how NFL teams pick quarterbacks. Evidently it is very difficult to predict how good someone will be as a quarterback in the NFL. Even players that won the Heisman trophy may fail to impress once they get to the NFL (exhibit A). Conversely, players that were picked nearly at the bottom of the NFL draft may end up becoming future Hall of Famers (e.g. Bart Starr, one of the last players picked in the 1956 draft). A great record as a college quarterback does not mean you'll be a starter in the NFL. It is just damn hard to predict who will be a good NFL quarterback as it requires such a unique set of skills and these skills (or the potential to learn these skills) are so difficult to infer a priori. Moreover, success as a quarterback requires the right match with a team and a coach, just as success in graduate school is often related to the fit between the student, the PI, and the department.

I've clearly stretched this metaphor to its breaking point, but I think it is worth highlighting how difficult it is to comb through a pile of applications and try to predict who will do well in graduate school. Being a graduate student is not like any other job out there and it requires skills that are difficult to measure. The process can clearly be improved by reconsidering how we select graduate students. If you are serving on a graduate committee, don't just focus on those students with the best grades or GRE scores – if you do, you may be missing the graduate student equivalent of a Tom Brady (the 199th pick in the 2000 draft).