Monthly Archives: January 2018

Curb Your Impact

Researchers are often advised to have an impact “beyond academia”. Indeed, Andrew Gunn and Michael Mintrum (2016) see the so-called “impact agenda” as focused on “university-based research that has non-academic impact, meaning it delivers some change or benefit outside the research community.” The basic idea is that the value of our universities lies in their contribution to society, for which the economy often serves as a proxy.

The EU’s 2020 target is to spend 3% of GDP on research and development, about a third of which is likely to be invested in university-based research (the rest is government and business sector R&D.) While 1% of GDP may not seem like much, it’s certainly a significant amount of money (about €150 billion) and it’s reasonable to expect a return. Indeed, Gunn and Mintrum detect “an impatience to see rapid and significant returns on the investments being made.”  This agenda can certainly be felt by researchers, whose performance is increasingly measured through indicators of societal impact. They are now acutely aware of their stakeholders outside the university and they are earnestly engaged in catching their attention. But perhaps the impatience that Gunn and Mintrum detect in all this activity should give us pause.

Consider another target in the Europe 2020 strategy: 40% of people aged 30–34 are to have completed higher education. Already today, about  25% of 20-29-year-olds are enrolled in university in most EU countries, so this target (which is already close to being reached) is entirely realistic. In any case, a few percentage points either way won’t affect the point I’m going to try to make. With such a high percentage of the population being either the product of higher education or in the process of getting such an education, it seems odd to get the people (researchers, faculty) who are spending a mere 1% of GDP to invest a great deal of their time (a portion of that economic activity) thinking about how their research makes a contribution “beyond academia”.  If the  researchers confined their efforts to their peers and students, their impact would by no means be insignificant.

With the passage of time this impact would spread throughout the population as the students graduate and begin to apply the skills, facts and theories they have learned. While this will be slower than the impact a piece of research might have by directly shaping government polices and corporate strategies, it will be deeper and more lasting. This is especially true when we consider that the 40% of thirty-somethings who have completed higher education will presumably have a disproportionate impact on GDP (i.e., as a group they will contribute more than 40% of their cohort’s GDP-contribution). Finally, by letting students round off their understanding of the theories in “the school of hard knocks” before they are implemented at the policy level some of the worst excesses of Ivory Tower thinking can be avoided.

Imagine for a moment what would happen if academics came to measure themselves only by the external performance indicators stipulated by the “impact agenda”. (Perhaps we could add to this an interest mainly in student evaluations as regards their teaching.) Suppose it was a matter of complete indifference to them what the students actually learned (so long as they were “satisfied” while at university) and what happened to the society and economy in the long run (so long as they were feted by bureaucrats and entrepreneurs.) In the short term, it is possible that high-quality research would inform well-crafted policy initiatives. But, after a time, researchers will learn (for they are not stupid) how to game the system to maximize their immediate, apparent impact. Mediocre but shiny ideas will crowd out the ideas that might actually work in practice. Many mistakes will be made. Meanwhile, since the minds of the students have been mostly neglected, the next generation of researchers won’t be up to the task of delivering what the best minds of the last generation were capable of. The race to the bottom will be on.

To avoid this, I would recommend evaluating academic research by measuring the value of higher education according to its long term impact on the economy and society, not the immediate measurable effects of discrete pieces of research on the policy process. I think the short-term indicators (student satisfaction, policy impact) are ultimately distractions, and we might find that academia is performing well by them but driving the culture into the ground. If researchers think the most important audiences are outside their own disciplines and classrooms, they will stop giving their best thinking to their students and colleagues, waiting instead for them to occupy some position of power wherefrom they can commission a policy paper. Education becomes merely a way of conditioning students to draw on “academic” expertise when they get out into the real world. “Life -long learning” becomes a kind of conditioned dependence on epistemic authority. And peer review becomes the mutual appreciation society it has long been accused of being.

In short, I propose we curb our socio-economic enthusiasm, our pursuit of “impact”. Let’s direct our energy back into academia and wait for it to diffuse organically throughout society. As T.S. Eliot once said, you don’t make flowers grow by pulling on them.

Update: See also Tina Basi and Mona Sloane’s post at the LSE Impact Blog.

Peer Grading and Peer Review

There is an important connection between my thoughts on peer grading and my thinking on peer review. Peer review in the familiar sense of having two or three qualified scholars read and comment on a paper before it is published in a journal was once necessary because publication was a costly process and there had to be some fairness about who got access to the scarce resource of printed pages. The process had to be blinded (in my opinion) to encourage frank assessment of work and discourage nepotism that would have wasted valuable space. Similar practical considerations justify the traditional approach to grading, in which students submit a single copy of their essay and the teacher gives it a grade that is none of anyone else’s business. For centuries it was simply impractical to demand that students produced 25 copies (or more for larger classes) of their essays. But information technology has made the cost of publishing and making copies negligible. Or, rather, once a platform exists, the cost of publishing an additional paper or making an additional copy is essentially zero.

This makes it possible to imagine an “open” peer review system in which papers are posted online along with the reviews as they come in. I’m not against such a system but I’m not sure it’s actually necessary. To me, it seems easier just to have people post their work to their own website (which may be personal or hosted by a university, but should in any case link to an institutional repository to provide a stable URL) and see what other people do with it. “Publication” just means uploading it to a server and sending an email announcing it to your peers. All review in this world would be “post-publication”. That is, our ideas would be evaluated as we would, ideally, evaluate the results in published papers. Today, unfortunately, the “peer-reviewed” stamp has a tendency to discourage serious criticism of published work. That’s an important reason we’re in the middle of a “replication and criticism crisis”.

Now think of students and their grades. They work alone (or in groups isolated from the rest of the class) on papers they submit to the teacher. After some time they get a grade that passes some final judgment on them. They do of course hear about the grades their peers received on the same assignment, but how many of them then sit down and read those papers to learn what a “better” or “worse” paper looks like? No, once the grade is received, closure has (in the vast majority of cases) been reached. Just as too many researchers put their published work behind. it’s time to focus on the next assignment, the next grant proposal.

But we have long had the technology to make a completely different approach possible. In addition to their assigned reading, students can be asked to read each other’s work. They can be expected to “publish” weekly assignments to a class website, where their peers can comment on them. Here, they can teach others what they’ve learned from their readings and discuss the issues that have come up in class. And they can do so by way of acquiring the writing skills that scholars have. The peer-feedback can be as mandatory as the assignments themselves. And it, too, can be graded. Indeed, even the students can grade each other, trying to apply the same criteria that their teachers apply. And the can in turn be graded on their ability to do this, their eye for quality work.

I have long suspected that the culture of peer review is an extension, or continuation, of the culture of examination. There are many who argue that the peer review system is broken, or at least seriously bent. Perhaps we can only change the way we engage with the published work of our peers by changing the way we engage with the work of students, and the way we ask them to engage with each other’s. After all, today’s students are tomorrow’s scholars.

Some Thoughts on Peer Grading

Suppose we listened to two pianists play the same sonata. Suppose one of them is a student who has been playing for a couple of years and the other is professional concert pianist. Or suppose we look at two drawings of the same hand, one by a high-school student of ordinary ability, the other by a professional illustrator. I’m going to presume we would quickly agree about which is the better performance. What I want to emphasize is that I can make this presumption without assuming anything about your abilities or claiming anything about mine. We can be less competent than all four of these “artists” and still judge them. We can say something about “how good” they are at playing the piano and drawing hands without being any particular good at it ourselves.

In the cases I’m imagining, our judgment–of relative competence–will often be quite accurate. Obviously, we’ll get into difficulties if we’re given twenty-five professional pianists to rank in order of mastery, but I don’t think we’d be as hard pressed if we’re given, say, a randomly selected group of twenty-five 18-year-old college students. Put them in front of a piano and ask them to play Bach, or give them a piece of paper and pencil and tell them to draw a hand, and we can pretty quickly sort them according to their relative skill at these things. In fact, if we’re asked to assign 3 As, 6 Bs, 8 Cs, 6 Ds, and 2 Fs, we could probably do it. That’s a rather marvelous fact to think about.

Now, if the students weren’t randomly selected but, instead, were taking a piano class or a drawing class, we’d want to have the competence of a teacher to confidently distribute grades like that. But what about the competence of the students? Could students in such a class distribute grades on the curve I’ve proposed? I believe they could, with a certain degree of accuracy, and I believe that the effort of doing so would be instructive–it would teach them something. Indeed, I believe that the ability to accurately discern the relative skill of two pianists or two artists (and therefore accurately predict the outcome of a competition between them) is itself a valuable skill that comes with mastering the relevant art. That is, as I suggested in my last post, it is in principle possible to grade students on their ability to grade each other. Developing their own eye for competence in the disciplines we teach should be part of the learning goals of any course. Even students who would give themselves failing grades in a course can make a qualified guess at what grades their fellow students will get by reading their essays. Just the effort to make that guess in 25 concrete instances will teach them something.

Let me say one last thing. I think it would be good for students to grade a class set of peer essays every once in while, even if just to develop an appreciation for the task that faces their teachers as a matter of course. Students often don’t take the problem of judging the competence of others seriously enough. By giving them this task, and by putting something at stake (i.e., grading them according to how well they predict the teacher’s grades), they will learn to value good, clear writing that is easily judged. After all, the purpose of academic writing is not impress the reader with your intelligence but to open your thinking to criticism from your peers. I think this idea needs to be tried.

I’m happy to hear from people who have.

Assignments

I’ve been focusing on students lately, not the craftsmanship of accomplished scholars. But it’s important to keep in mind that there is a close connection between what students are taught and what their teachers are able to do. When you give a student an assignment, even one that involves a significant amount of “independent research”, you are telling them to do something that you yourself could do. Like a conservatory teacher or an athletics coach, you need not presume that you could do it better or more easily than them. It’s reasonable for an aging master to be weaker in practice than the young apprentice. The important thing is that the teacher already knows what is possible, and is therefore able to evaluate the result, while students are only just beginning to discover what they are capable of. “The Great Learning,” said Confucius, “is rooted in watching with affection the way people grow.” Learning is always implicit in scholarship, and the student, therefore, is always implicated in the work of the scholar.

With this in mind, I want to suggest some assignments that can train and test the skills that scholars apply to their own learning. By means of these assignments, the skills can be passed on to their students and a culture of learning can thereby be conserved. These are high-minded ideals, but the assignments themselves are altogether practical and grounded.

 

Assignment #1

Give the students two short texts by writers on opposite sides of a debate that is, or was once, central to your discipline. Let them be around 3000 words long, which implies that they can be read–once–in about fifteen minutes. Make sure that they either are or represent “classic” positions. That is, make sure that the students are exposed to the ideas of two major figures in your field.

Give them a week to read the texts and, on that basis alone, make up their own minds about the issue. What is the debate about? Which arguments do they find most compelling? Who is right and who is wrong? Is there a middle ground that seems more reasonable to them than either of the two positions that are being defended?

Have them write a five-paragraph essay of no more than 1000 words to present their position on the question. Have them read and grade each other’s essays. (You can give each student a number of essays to grade. But in a smaller class of, say, 20 students, you can also tell them to spend 10 minutes on each paper and have them grade them all. That’s 200 minutes of, I hope you will agree, very instructive work. I would recommend having them distribute the grades on a normal curve.)

 

Assignment #2

Send the students to the library to inform themselves about the history of the issue you exposed them to in assignment #1. Have them locate the texts you assigned and the most interesting (to the student) sources they cited. Have them study the “impact” of the papers you assigned (or the authors who wrote them, or the classics they represented) on the discourse of your discipline by finding papers that cite them.

Give them a week to see what they can come up with. How important is this question today? What is the consensus among scholars? What sort of research has been done to test the relevant theories? What research is being done to extend and develop them? Encourage them to seek out work that disagrees with the position they took in assignment #1.

Have them write a five-paragraph essay of no more than 1000 words to present their results. Demand that they use a particular citation style. Again, have them read and grade each other’s essays.

 

Assignment #3

Have them reread their solution to assignment #1. In light of their work in the library, what do they think now? Have them rewrite essay #1 to take stock of this research, either defending their position in a more sophisticated way, or taking a new position in light of the better arguments they have encountered.

This time, you give them the grade.

 

Assignment #4

While you are grading assignment #3, have them grade each other too. This time, however, their job isn’t to evaluate it according to their own standards, but to try to predict the grade that you will give them. This is easiest to do if you grade on a curve, since the students will now have to distribute each other’s papers on a normal distribution. Their grade on this exercise will be based on how well they match the grades you give. If you don’t distribute the whole class set of essays to everyone (perhaps the class is too big), their job will simply be to rank-order them. You will then also have to rank each essay relative to the others.

 

I’ll let this stand without comment and offer some reflections on why I think this is a great way to spend a few weeks of your student’s time (and yours!) in subsequent posts. Do please offer your thoughts below in the meantime.

Paragraph 5

I have suggested that the key sentence of the first paragraph of a five-paragraph essay is, either implicitly or explicitly, self-referential. It does not assert the thesis statement of the essay; it asserts that this essay will demonstrate its truth. It refers, not to the world of fact that makes it true, but to the text before us that will show us how true it is. That gives it the task of justifying the writing of the essay by situating it in a practical and theoretical context, outlining the argument, and indicating the consequences for practice or theory or both. I will now suggest that the last paragraph will actually assert the thesis and present (not merely indicate) the consequences.

If the key sentence of paragraph 1 was

In this essay, I will argue that the Great Depression was caused by the monetary policy of the Federal Reserve,

then the key sentence of paragraph 5 will be simply

The Great Depression was caused by the monetary policy of the Federal Reserve.

They do not have to be this obviously similar, but the content should be. Notice that there is no reference to the essay in the latter, no “I have here argued…” or “I have tried to show…” Also, I should say that this is actually best understood as a way of mirroring paragraph 3 of a longer essay with paragraph 39 (out of 40). In a five paragraph essay, the first and last paragraph will do a bit more work than their key sentences make explicit. (You might want to plan this work so that paragraph 1 actually has three key sentences and paragraph five has two, but do remember to maintain a focus.)

Think of paragraph five as having the task of asserting both the thesis and its importance. To begin with, after having written the key sentence, you can just write a version of the key sentence of paragraphs 2-4. You now have four sentences written. Then add three sentences that suggest the consequences of realizing that the Fed’s monetary policy was to blame, whether for scholars or bankers or both (but be clear about who you’re talking about). Presumably, this gives us some guidance about how to avoid similar crises in the future. Tell us what the right policy would have been, perhaps. And what policy we should adopt today.

You have all the material in place. Now work on the paragraph until it feels like a conclusion of your argument. Remember it has three distinct parts or moments: (1) the thesis statement of your essay, (2) a summary of the argument, (3) an assessment of the consequences. It has to make these three moves as though they are one coherent gesture that asserts your overall meaning. One last thing: I’ve given you the steps of a dance or the chords to a song. Please grant that merely following these steps or playing these chords will not necessarily produce a good essay. You have to practice until you can do it well. And you have to have something to say.