As someone who got a perfect score on the GRE Quant section, I generally agree with your criticism of that test.
However, I think you don't necessarily make the best argument about preferring GRE Quant vs SAT Math.
There are large differences in the groups taking the tests. First of all, way more kids take SAT than GRE. About 2mn kids take SAT per year, while about 300k or so take GRE. That means that if about 1% of kids get a perfect score on SAT Math, that's about 20k kids. If 9% of kids who take GRE get a perfect GRE Quant score, that's about 27k. Assuming just that these are the same groups of people (ignoring that they don't take it the same year), then it's really not all that different at identifying the best people per year at math in the full population. But you would also want to take into account that the people taking the GRE are most of the way through their college experience and thinking about grad school. Much different than the population taking the SATs.
The problem with the Putnam test is on the other end of things. It is too hard, especially if you gave it to everyone. If you gave it to everyone, then the median isn't 2, it's 0. So it would only be useful for the top programs trying to get the top people.
Interestingly enough, the GRE purposefully restricts the score at the top end because of strong Asian scorers. See Section 2.2:
I think the idea is to limit the advantage that comes from grinding test prep. So the trick would be coming up with a Quantitative test that is more correlated with IQ at the high levels and such that test prep wouldn't help you too much.
“Interestingly enough, the GRE purposefully restricts the score at the top end because of strong Asian scorers.”
I don’t think you are making the persuasive argument you think you are making.
Your point re: gaming the system is fair enough. But your identical point suggesting hard work and effort (“grinding”) being bad… is not.
I suspect I am not the only one who would say that selecting only for IQ as opposed to the union of IQ and diligence would be far from an optimal thing.
I was about to reply that although IQ is easy to test for and is a broadly useful advantage, high levels of diligence, while also broadly useful, are harder to test for unless they've already been invested in an unhealthily narrow field. Aiming all your diligence at a metric for years in order to make your diligence legible is quite doable - but not usually good for you.
However, then I realized that these tests don't exist to incentivize behaviors that are good for people. They don't even exist to identify the smartest and most diligent people. They exist to identify the people who will perform best in graduate programs and the careers that follow from them.
Are graduate programs and careers that require graduate degrees generally known for accommodating healthy well-roundedness? They are not.
So in a sense, perhaps it would be a good thing for these tests to identify the people who have a precedent of aiming all their diligence at a metric for years and not turning aside for something else. Because that's what they'll most likely be in for in the following decades as well.
I think IQ and conscientiousness are good qualities, but it's tricky when the test is measuring both, especially when some subgroups are incentivized particularly strongly to invest in test prep.
I took the GREs in 1969 at Occidental College, along with many Caltech undergrads, Occidental being the nearest place to take it. At the starting signal, we all opened the exam booklets, and five or ten seconds later a wave of chuckles filled the room -- relief.
Not disagreeing with the blog but I think it’s worth noting that a test with an mcat-like distribution of scores will be much more rewarding/informative for the average test taker compared to a Putnam style test aimed at differentiating amongst the very best. Counts for something.
I think you have to differentiate clearly between undergrad and graduate admissions. Logistically it is very different - professors control grad admissions and have the ability to request unusual things. Undergrad admissions have to operate at a larger scale.
The Putnam is too hard to be a useful standardized test. It's really 12 questions each worth 10 points and partial credit is just a tiebreaker. The difference between getting a 10 and a 0 is pretty flukey - it's just whether you got one question right, or zero questions right. It's only reliably meaningful for the top 100 or 200 students, most of which won't go into grad school at all, and the ones who do generally won't have any problem getting into a good one. (Grad schools, Wall Street and Silicon Valley all know very well that a high Putnam score is meaningful.)
But this is the right idea. I would just point in a slightly different direction. Better tests than the SAT are the AMC 12 and the AIME. Logistically, these are already standardized - they are machine-gradable and given widely to high school students across the country, the AMC 12 to about 50,000 people and the AIME to about 6,000. But a high score on these is much more meaningful than the easy-to-achieve perfect SAT math score.
So you professors can lobby your admissions departments right now - pay attention to AMC 12 and AIME scores!
School is an inefficient use of time for most people, I would estimate.
I hope that if we find ourselves in a future with increased GDP and a decreased population of children, we could at least treat those that do exist better, by funding public schools to have half the student-to-teacher ratio we currently do. Assuming that doubles learning speed, we could then halve the school day and give them their childhood back. But we'd need to revive the cultural status of stay-at-home parents so someone could watch them. Or at least have tons more recess and PE.
My memories of K-12 are that probably 90% of what they taught, I did not remember past taking tests. I love history and fiction, but school focused on a few worn-out classics and dates and names. Their forced art and music classes bored me to tears. I can think of very little that I used on jobs, even from college. STEM fields are different, but K-12 taught very little of that.
I think I would have done better with a library with an encyclopedia geared to children exploring what they want, and music and art teachers for those who want them as beginners. My grandfather had a baby grand piano, and I liked fiddling with it, but I despised music classes. Focus on Readin' Writin' and 'Rithmetic, and leave the exploration to the kids.
I do like the idea of leaving children to explore and play on their own half the day, but not by doubling down on the intensity of forced teaching. There is something wrong about forcing children to spend all day in classrooms until they are 18 or 22, stuck memorizing stuff they will forget after a weekly test, whose only purpose is a piece of paper, and which alienates them for years from stuff they might well like as adults.
Opinions like this have been expressed and argued for hundreds of times. Top colleges and top programs are not trying to select for the best test takers.
After my daughter did the GRE she realized that she should have taken it when she took the ACT in high school - she would have gotten a higher score. It was not asking her for material she had covered in college.
That was certainly my experience taking it more than 20 years after I graduated high school. I had scored in the 99th percentile on the SATs in both math and verbal back then, but while my verbal on the GRE remained high, my math score dropped precipitously. I still knew how to do the problems, but was just way slower than I had been. Very telling reminder of what age and lack of practice do to you.
I took the GRE's over 50 years ago and I certainly was not impressed by them - I had done a Physics degree with a lot of additional math and nothing I had studied was covered in the math section, and my college studies did not help with the verbal portion, I would have been better taking it with the SAT. 20 years later I did take the GMAT and placed in the 98/99 th percentile, I had a much higher opinion of the GMAT than the GRE, mind you I still did well in the GRE, but it definitely felt irrelevant.
The purpose of tests is to evaluate intelligence, so grinding is indeed bad. Particularly when the grinding doesn't increase knowledge, just provide a higher test score.
If one person can easily achieve a high score and another can achieve a high score with months and months of prep to acquire test specific domain knowledge that doesn't represent ability, the test is flawed. If you want to know who the grinders are, use another metric.
There is an old joke: What do you call someone who fails to get into Veterinarian School? Dr. because they then get into Medical School.
One of the factors that is usually omitted in these discussions is the memory factor. Most test are really about memorization. The more "photographic" your memory is the better you do on the tests. A spelling test is the obvious totally memory reliant. A great many others are, they just pretend they aren't. The key to success in many STEM tests is remembering formulas And yes, math too. The more you remember of what a professor said or wrote on the black\white board the greater your chances are of passing.
Neither very low-level nor very high-level spelling tests are reasonably described as "totally memory reliant" (although the intermediate range may come close). Low-level spelling tests, for young children or ESL students, are predominantly about understanding phonics principles. Very high-level spelling, like you see in elite competition, rely significantly upon etymological awareness, phonological deduction and strategic guessing.
"Strategic guessing"? "phonological deduction"? So if it's close then you get points? E=MC^2 is correct M=EC^2 is not correct. You can't guess. It's either right or wrong. The better you are at remembering formulas and equations the better you do on tests. Maybe not 100% but the more photographic your memory is the better you do. Other things help. Most fraternities keep records of prior tests that their members can review. A lot of college profs recycle test questions. If you can review a question from previous years (along with the answers it raises the probability you will get the correct answer. It also lessens the time you have to take in trying to solve the problem if you know the answer. The SAT's and others are time limited. Years ago I took different IQ tests as an experiment. One strictly adhering the rules, the other taking my time in answering the questions. I did much better on the second than the first. Even though I was watching TV at the same time and noshing. My college roommate was very smart. He also had a near photographic memory. Rarely studied. The night before a test he would read the book or the relevant chapters. Look at his notes and the notes of others (because he skipped a lot of classes) and usually he'd ace the test, quite easily. I'm not saying it was all due to his ability to memorize but the better you are at doing that the easier it makes taking tests.
You are correct. I didn't. But any test that does what you say is more of a guessing test than a knowledge test. You might call it a reasoning test, but the reality is it's still guessing.
Social critics clamor to break from the standardized tests or at least the Ivy League's hold on them, yet all suggested replacements are just more of the same. There was a time when there were no such tests. Are these tests just a reflection on the degree of conformity that is present because so many people want to go to college?
There are a law of good arguments related to Goodhart's law for standardized tests being unreliable. The problem is, you need some way to sort people, and the alternatives are *way* worse. The social critics tend to focus on the first bit, while forgetting the last.
The limitations of the GRE math test go back even farther. I took the GRE in 1982 to get into an engineering masters program, and got a perfect score on the math. It required nothing beyond what I had learned in high school algebra and geometry, and I went to a small high school (34 in my graduating class) with no AP courses.
My daughter missed a perfect score on the math GRE by one, so instead of an American PhD she went to Portugal for an Econ master’s. Thanks to the GRE, I get to go to Portugal every year!
I took the GRE in 1992, and things might have changed since then. But—
When I took it, there was the general test, with the verbal and math sections, taken by everyone. But there were also tests in specific subjects. I blew the doors off the general test, but only scored 85% on the math-subject test.
So while I'd agree with Dr. Caplan that the general GRE would be of little use to graduate programs trying to pick people who were exceptionally good at math, I would assume that such programs would insist that their applicants take the math-subject test as well as the general GRE. I'm surprised to see no mention of that by Caplan, since I've just looked at the ETS website, and it informs me that the math exam is still offered.
The high school Advanced Placement exams are scored 1 to 5 with about 20% scoring in each tier. But that means that there are radical differences between test-takers who score 5s. For example, I recall reading about 15 years ago that to score a 5 on AP Chemistry only required getting 56% of the maximum score.
Hence, Caltech doesn't give any credit for Advanced Placement results, even for 5s.
But, it wouldn't be hard to extend the maximum score from 5 to 7 or even to 9. If you take the test on a computer, you can be thrown tougher questions on the fly if you've gotten the early easy questions right. This info would be highly useful for admissions decisions to HYPS-MIT-Caltech. "Hey, this kid got an 8 on AP Chemistry!"
As someone who got a perfect score on the GRE Quant section, I generally agree with your criticism of that test.
However, I think you don't necessarily make the best argument about preferring GRE Quant vs SAT Math.
There are large differences in the groups taking the tests. First of all, way more kids take SAT than GRE. About 2mn kids take SAT per year, while about 300k or so take GRE. That means that if about 1% of kids get a perfect score on SAT Math, that's about 20k kids. If 9% of kids who take GRE get a perfect GRE Quant score, that's about 27k. Assuming just that these are the same groups of people (ignoring that they don't take it the same year), then it's really not all that different at identifying the best people per year at math in the full population. But you would also want to take into account that the people taking the GRE are most of the way through their college experience and thinking about grad school. Much different than the population taking the SATs.
The problem with the Putnam test is on the other end of things. It is too hard, especially if you gave it to everyone. If you gave it to everyone, then the median isn't 2, it's 0. So it would only be useful for the top programs trying to get the top people.
Interestingly enough, the GRE purposefully restricts the score at the top end because of strong Asian scorers. See Section 2.2:
https://www.ets.org/s/research/pdf/gre_compendium.pdf
I think the idea is to limit the advantage that comes from grinding test prep. So the trick would be coming up with a Quantitative test that is more correlated with IQ at the high levels and such that test prep wouldn't help you too much.
“Interestingly enough, the GRE purposefully restricts the score at the top end because of strong Asian scorers.”
I don’t think you are making the persuasive argument you think you are making.
Your point re: gaming the system is fair enough. But your identical point suggesting hard work and effort (“grinding”) being bad… is not.
I suspect I am not the only one who would say that selecting only for IQ as opposed to the union of IQ and diligence would be far from an optimal thing.
I was about to reply that although IQ is easy to test for and is a broadly useful advantage, high levels of diligence, while also broadly useful, are harder to test for unless they've already been invested in an unhealthily narrow field. Aiming all your diligence at a metric for years in order to make your diligence legible is quite doable - but not usually good for you.
However, then I realized that these tests don't exist to incentivize behaviors that are good for people. They don't even exist to identify the smartest and most diligent people. They exist to identify the people who will perform best in graduate programs and the careers that follow from them.
Are graduate programs and careers that require graduate degrees generally known for accommodating healthy well-roundedness? They are not.
So in a sense, perhaps it would be a good thing for these tests to identify the people who have a precedent of aiming all their diligence at a metric for years and not turning aside for something else. Because that's what they'll most likely be in for in the following decades as well.
I think IQ and conscientiousness are good qualities, but it's tricky when the test is measuring both, especially when some subgroups are incentivized particularly strongly to invest in test prep.
Well, with you logic, what test other than an IQ test is worthwhile??
I took the GREs in 1969 at Occidental College, along with many Caltech undergrads, Occidental being the nearest place to take it. At the starting signal, we all opened the exam booklets, and five or ten seconds later a wave of chuckles filled the room -- relief.
Oh yeah? When I went to Caltech, we had to walk five miles...in the snow...each way. And we liked it.
Ha! From my house on a dirt road, it is uphill, then downhill, to the paved county road, so it actually is uphill both ways.
When I thought about applying to Caltech in 1975, I couldn't even see the snow from the campus due to all the smog.
So I didn't apply.
Well, the smog and my SAT Math score.
Not disagreeing with the blog but I think it’s worth noting that a test with an mcat-like distribution of scores will be much more rewarding/informative for the average test taker compared to a Putnam style test aimed at differentiating amongst the very best. Counts for something.
I think you have to differentiate clearly between undergrad and graduate admissions. Logistically it is very different - professors control grad admissions and have the ability to request unusual things. Undergrad admissions have to operate at a larger scale.
The Putnam is too hard to be a useful standardized test. It's really 12 questions each worth 10 points and partial credit is just a tiebreaker. The difference between getting a 10 and a 0 is pretty flukey - it's just whether you got one question right, or zero questions right. It's only reliably meaningful for the top 100 or 200 students, most of which won't go into grad school at all, and the ones who do generally won't have any problem getting into a good one. (Grad schools, Wall Street and Silicon Valley all know very well that a high Putnam score is meaningful.)
But this is the right idea. I would just point in a slightly different direction. Better tests than the SAT are the AMC 12 and the AIME. Logistically, these are already standardized - they are machine-gradable and given widely to high school students across the country, the AMC 12 to about 50,000 people and the AIME to about 6,000. But a high score on these is much more meaningful than the easy-to-achieve perfect SAT math score.
So you professors can lobby your admissions departments right now - pay attention to AMC 12 and AIME scores!
in my understanding, math graduate programs are already good at taking putnam scores into account
The Olympiads in high school are very hard, have multiple levels and are very good to distinguish within the top 1% of SAT or ACT top scorers.
The CS one is taken from home online.
The others are math and physics, chemistry, biology.
You should do well in your intended major, in these.
You don't need to look any further.
Art of Problem solving trains kids in most of these Olympiads, as school is an inefficient use of time for the top 0.1% anyway.
School is an inefficient use of time for most people, I would estimate.
I hope that if we find ourselves in a future with increased GDP and a decreased population of children, we could at least treat those that do exist better, by funding public schools to have half the student-to-teacher ratio we currently do. Assuming that doubles learning speed, we could then halve the school day and give them their childhood back. But we'd need to revive the cultural status of stay-at-home parents so someone could watch them. Or at least have tons more recess and PE.
My memories of K-12 are that probably 90% of what they taught, I did not remember past taking tests. I love history and fiction, but school focused on a few worn-out classics and dates and names. Their forced art and music classes bored me to tears. I can think of very little that I used on jobs, even from college. STEM fields are different, but K-12 taught very little of that.
I think I would have done better with a library with an encyclopedia geared to children exploring what they want, and music and art teachers for those who want them as beginners. My grandfather had a baby grand piano, and I liked fiddling with it, but I despised music classes. Focus on Readin' Writin' and 'Rithmetic, and leave the exploration to the kids.
I do like the idea of leaving children to explore and play on their own half the day, but not by doubling down on the intensity of forced teaching. There is something wrong about forcing children to spend all day in classrooms until they are 18 or 22, stuck memorizing stuff they will forget after a weekly test, whose only purpose is a piece of paper, and which alienates them for years from stuff they might well like as adults.
Opinions like this have been expressed and argued for hundreds of times. Top colleges and top programs are not trying to select for the best test takers.
The lsat would be similar to the mcat.
180: 99.90%
179: 99.82%
178: 99.70%
177: 99.52%
176: 99.29%
175: 99.04%
174: 98.66%
173: 98.15%
172: 97.50%
171: 96.67%
170: 95.69%
169: 94.50%
168: 93.07%
167: 91.52%
166: 89.79%
165: 87.80%
164: 85.58%
163: 83.08%
162: 80.53%
161: 77.61%
160: 74.62%
159: 71.57%
158: 68.11%
157: 64.69%
156: 61.14%
155: 57.49%
154: 53.87%
153: 49.98%
152: 46.29%
151: 42.54%
150: 39.06%
149: 35.47%
After my daughter did the GRE she realized that she should have taken it when she took the ACT in high school - she would have gotten a higher score. It was not asking her for material she had covered in college.
That was certainly my experience taking it more than 20 years after I graduated high school. I had scored in the 99th percentile on the SATs in both math and verbal back then, but while my verbal on the GRE remained high, my math score dropped precipitously. I still knew how to do the problems, but was just way slower than I had been. Very telling reminder of what age and lack of practice do to you.
I took the GRE's over 50 years ago and I certainly was not impressed by them - I had done a Physics degree with a lot of additional math and nothing I had studied was covered in the math section, and my college studies did not help with the verbal portion, I would have been better taking it with the SAT. 20 years later I did take the GMAT and placed in the 98/99 th percentile, I had a much higher opinion of the GMAT than the GRE, mind you I still did well in the GRE, but it definitely felt irrelevant.
The purpose of tests is to evaluate intelligence, so grinding is indeed bad. Particularly when the grinding doesn't increase knowledge, just provide a higher test score.
If one person can easily achieve a high score and another can achieve a high score with months and months of prep to acquire test specific domain knowledge that doesn't represent ability, the test is flawed. If you want to know who the grinders are, use another metric.
There is an old joke: What do you call someone who fails to get into Veterinarian School? Dr. because they then get into Medical School.
One of the factors that is usually omitted in these discussions is the memory factor. Most test are really about memorization. The more "photographic" your memory is the better you do on the tests. A spelling test is the obvious totally memory reliant. A great many others are, they just pretend they aren't. The key to success in many STEM tests is remembering formulas And yes, math too. The more you remember of what a professor said or wrote on the black\white board the greater your chances are of passing.
Neither very low-level nor very high-level spelling tests are reasonably described as "totally memory reliant" (although the intermediate range may come close). Low-level spelling tests, for young children or ESL students, are predominantly about understanding phonics principles. Very high-level spelling, like you see in elite competition, rely significantly upon etymological awareness, phonological deduction and strategic guessing.
"Strategic guessing"? "phonological deduction"? So if it's close then you get points? E=MC^2 is correct M=EC^2 is not correct. You can't guess. It's either right or wrong. The better you are at remembering formulas and equations the better you do on tests. Maybe not 100% but the more photographic your memory is the better you do. Other things help. Most fraternities keep records of prior tests that their members can review. A lot of college profs recycle test questions. If you can review a question from previous years (along with the answers it raises the probability you will get the correct answer. It also lessens the time you have to take in trying to solve the problem if you know the answer. The SAT's and others are time limited. Years ago I took different IQ tests as an experiment. One strictly adhering the rules, the other taking my time in answering the questions. I did much better on the second than the first. Even though I was watching TV at the same time and noshing. My college roommate was very smart. He also had a near photographic memory. Rarely studied. The night before a test he would read the book or the relevant chapters. Look at his notes and the notes of others (because he skipped a lot of classes) and usually he'd ace the test, quite easily. I'm not saying it was all due to his ability to memorize but the better you are at doing that the easier it makes taking tests.
Uh... that's a lot of words to say you didn't understand that my response was specifically about spelling tests.
You are correct. I didn't. But any test that does what you say is more of a guessing test than a knowledge test. You might call it a reasoning test, but the reality is it's still guessing.
Social critics clamor to break from the standardized tests or at least the Ivy League's hold on them, yet all suggested replacements are just more of the same. There was a time when there were no such tests. Are these tests just a reflection on the degree of conformity that is present because so many people want to go to college?
There are a law of good arguments related to Goodhart's law for standardized tests being unreliable. The problem is, you need some way to sort people, and the alternatives are *way* worse. The social critics tend to focus on the first bit, while forgetting the last.
The limitations of the GRE math test go back even farther. I took the GRE in 1982 to get into an engineering masters program, and got a perfect score on the math. It required nothing beyond what I had learned in high school algebra and geometry, and I went to a small high school (34 in my graduating class) with no AP courses.
My daughter missed a perfect score on the math GRE by one, so instead of an American PhD she went to Portugal for an Econ master’s. Thanks to the GRE, I get to go to Portugal every year!
I took the GRE in 1992, and things might have changed since then. But—
When I took it, there was the general test, with the verbal and math sections, taken by everyone. But there were also tests in specific subjects. I blew the doors off the general test, but only scored 85% on the math-subject test.
So while I'd agree with Dr. Caplan that the general GRE would be of little use to graduate programs trying to pick people who were exceptionally good at math, I would assume that such programs would insist that their applicants take the math-subject test as well as the general GRE. I'm surprised to see no mention of that by Caplan, since I've just looked at the ETS website, and it informs me that the math exam is still offered.
The high school Advanced Placement exams are scored 1 to 5 with about 20% scoring in each tier. But that means that there are radical differences between test-takers who score 5s. For example, I recall reading about 15 years ago that to score a 5 on AP Chemistry only required getting 56% of the maximum score.
Hence, Caltech doesn't give any credit for Advanced Placement results, even for 5s.
But, it wouldn't be hard to extend the maximum score from 5 to 7 or even to 9. If you take the test on a computer, you can be thrown tougher questions on the fly if you've gotten the early easy questions right. This info would be highly useful for admissions decisions to HYPS-MIT-Caltech. "Hey, this kid got an 8 on AP Chemistry!"