A short while ago, I wrote about my adventures in test grading Why I Have A Scantron Gimme-Cap. In that bloated account of my life and hard times, I mentioned that I was introduced to machine-scored, scannable test answer sheets by a doctoral student in psychology at Texas Technique in the early 1970s. In my own studies on the South Plains of Texas in Lubbock, I majored in history and my minor field was literature in the English department. Now, I learn that Texas Technique's English department has online essays that are graded (or scored) online as well. The little frosh write essays and ship the files to the grad students (who teach these entry-level courses) and receive their grade (or score) online in an e-mail message. All of the transactions are untouched by human hands (or red pencils). The mad professor who is responsible for this "innovation" came to Texas Technique after I was long gone from the place. He has sold his software product to a major textbook publisher and he will end his days rich beyond the dreams of avarice. If this is (fair & balanced) bemusement, so be it.
[x The Chronicle of Higher Education]
A New Way to Grade: At Texas Tech, freshman composition has been revolutionized. Is the result too mechanical?
By Paula Wasley
Last semester Lindsay Hutton "taught" 1,940 students. She met only 70 of them in person. Those were the ones enrolled in the two weekly sections of English composition that she taught in an actual classroom. The hundreds and hundreds of others she knew only as anonymous numbered documents she read on her computer screen and then, with a click of a button, sent back out into the ether.
As one of 60 graduate students hired to teach freshman composition at Texas Tech University, Ms. Hutton had a weekly quota of grading.
Each week she was assigned to read, comment on, and grade 17 drafts of essays, offer a second grade on 18 more, and review about 25 peer critiques and 20 student self-evaluations to fulfill her 12-hour grading responsibility — allowing an average of 10 minutes per document. Ms. Hutton, a Ph.D. student in creative writing, is not paid per draft, but some graduate students who take on grading work during the holidays are. Last spring, for example, students were paid $2 for grading a preliminary draft, $4 for a final draft, and 50 cents for a peer or self-evaluation.
"Sometimes," Ms. Hutton says, "it feels like a factory."
Most colleges and universities require some kind of first-year writing course. Increasingly, the expensive task of getting masses of freshmen from widely varying educational backgrounds up to snuff falls to the cheapest labor: untenured professors, part-timers, and graduate students.
While institutions around the country are experimenting with technology to enhance or reinvigorate freshman comp, Texas Tech is using a computer system to entirely reinvent the experience. The university has cut class time in half, increased the amount of writing students do, and split the teachers into two groups: "classroom instructors" and "document instructors." The system allows faculty members to closely monitor the graders and collect piles of data about student writing and how graduate students evaluate it.
The system has divided the English department, pitting professors who say it not only saves time but prevents biased grading against those who find it dehumanizing and Orwellian. Most alarming to the critics is that the system's separation of instruction from grading threatens the traditional, and to some, sacrosanct, relationship between teacher and student.
Man and Machine
The man behind this divisive system is Fred O. Kemp, an associate professor of English and designer of Topic (Texas Tech Online-Print Integrated Curriculum), the Web-based computer application on which Texas Tech's first-year composition program, or ICON (for Interactive Composition Online), is based.
Mr. Kemp emphasizes that the system was designed to solve a set of problems particular to a large research institution like Texas Tech — namely how to use inexperienced graduate students to teach composition to 3,000 freshmen. Half of the graduate students entering Texas Tech have never taught before, says Mr. Kemp, or indeed even taken a basic composition course, having tested out of it in college. And with a 25- to 30-percent turnover rate, most who enter the system leave after three or four semesters.
"We have some folks who come in very talented, who like to teach and have a knack for it," says Susan M. Lang, an associate professor of English and a director of the composition program. "If you're a freshman and you luck out with the top 15 percent [of instructors], it's a phenomenal experience. But if you're in the bottom 15-20 percent, it's horrible."
Before ICON, says Mr. Kemp, the system for teaching freshman composition was rife with inconsistency. Or rather there was no system. Instructors drawn from creative writing, technical communication, rhetoric, and literature could not agree on either the content or criteria of good writing. Some instructors had students writing haiku and short stories, while others assigned lengthy research papers. At the beginning of each semester, says Mr. Kemp, the department dealt with wholesale movement between sections, while his office turned into a "complaint desk" for students carping about the program's inequities.
In the fall of 2001, when a graduate-student instructor was removed from the classroom for incompetence, Mr. Kemp, then director of the composition program, took over her two sections. Overburdened, he decided to divide the work of grading the 50 students' papers.
He tinkered with Texas Tech's homegrown database-driven software, Topic — which, like the commercial courseware WebCT and Blackboard, allows students to file and store papers online — so that other faculty members could read and grade the essays. The experiment was so successful that the new system was adopted programwide in 2002.
What is most radical about the system is the way it divides the labor of the traditional teacher into that of "classroom instructor" and "document instructor" or, in the local parlance, CI and DI.
Students meet once a week in a classroom with their classroom instructor to go over the finer points of grammar, style, and argumentation, and to discuss their weekly assignments, which are standardized across all 70-odd sections of the two required first-year composition courses. Each assignment cycle includes three drafts of an essay, reflective "writing reviews" commenting on students' own work, and two peer reviews of other students' work, all of which are submitted and stored online.
Document instructors, some of whom also work as classroom instructors, do the grading. Every piece of writing students produce is read by at least two anonymous graders from a pool of 60 to 70 graduate part-time instructors. The first reader reads, comments, and assigns a grade, from one to 100, to the document. When the second reader opens the file, he or she sees the essay, and the first DI's commentary, but not the grade. The second grader assigns a grade, and the computer averages the two. If the spread between the two grades is greater than eight points, the document goes to a third grader, and the two closest scores determine the composite grade. A student may appeal an assignment grade to his or her classroom instructor, who may choose to override it.
Mr. Kemp acknowledges his approach is controversial. The first semester the system was adopted, graduate students circulated a newsletter titled TOPIC Sucks, griping about the system's frequent technical glitches and industrial aspect. "It expressed a concern that they were being turned into pieceworkers," says Madonne M. Miner, an English professor and director of graduate studies in the department.
In the four years that the system has been in place, Mr. Kemp has been called upon by students, graduate students, and professors at Texas Tech and other institutions to defend and explain his creation. He is used to the attacks. Those who see the system as an insidious mechanical agent, he says, are often either Luddites with a visceral reaction to anything computerized or don't fully understand the system's operating principles.
"Simply to call it an assembly line and say, ipso facto, it's wrong, sounds like a 19th-century point of view," he says. "Henry Ford built an awful lot of automobiles, and he made them cheap so that an awful lot of people could buy cars that couldn't have bought cars without the assembly line. So the idea that efficiencies within a system are inherently bad and dehumanizing, I think, is wrong."
Students Required to Write More
Chief among the program's benefits, administrators say, is simply that it requires students to write more. "We make the assumption that students benefit more from writing and receiving commentary than by sitting in a classroom," says Mr. Kemp. Classroom time for first-year comp was cut in half, from 160 to 80 minutes a week, and the cap on class size was raised from 25 to 35 students in each section.
The changes were intended to give students more time to work on assignments, and graduate students more time to grade them. Students now turn in an average of 35 pieces of writing a semester, nearly three times as many as before. Although Mr. Kemp acknowledges that he has no data showing that the program has improved students' writing, there is also "no proof that what people were doing before ICON was any better," he says.
In addition, blind grading eliminates bias, says Mr. Kemp. His maxim: "We are not grading the writer, we are grading the writing."
Moreover, ICON removes what Mr. Kemp calls "the suck-up factor." No longer can a student earn good marks by buttering up the instructor. Teachers can't inflate the grade of a student who turns in consistently poor work just because he or she is deemed to be trying hard.
ICON is designed to inject objectivity into the subjective process of evaluating writing. Thanks to standardized assignments, standardized evaluation criteria, and shared grading, Mr. Kemp says, an A means something uniform.
"In the old days," he says, "if your students were all making A's, that could mean that they were a great group of students, or it could mean that you're a lousy teacher."
Faculty members can look in and determine whether some graders are giving higher than average marks and review their commentary. "From an administrative point of view, it's a dream child," says Ms. Miner. "You've got complete oversight over what your graduate part-time instructors are doing."
First-year comp students can also grade their graders. When retrieving their documents, students can enter a ranking of one to five, evaluating the helpfulness of the DI's commentary. Whenever a student enters a one — what some instructors refer to as a "nastygram" — the system immediately notifies the program's directors.
The program's Big Brother aspect may rankle his fellow instructors, but Michael Likhinin, a master's degree student in technical communication, says online teaching is the future of higher education, and training in ICON is ideal preparation for "this brave new world which is to come."
He cites the "flexibility factor" as one of the program's greatest attractions. As a document instructor, Mr. Likhinin can tailor his work hours to his schedule, grade more papers early to reach his quota if he knows he has a difficult week ahead, and log on anytime from anywhere to read and comment on papers. Once, on vacation and with little to do, he graded papers from his hotel in Mexico.
Grading online documents goes more quickly than if he were marking hard copies, he says. He can use search functions to find frequently made errors. And instead of writing out repetitive commentary by hand, he can insert links to helpful Web sites explaining grammar and style rules. "It reduces the amount of menial labor," he says.
Jonathan Markley, a sophomore at Texas Tech, liked all the commentary he received in his composition courses. And the software was certainly user-friendly: He always knew what his grade was, and he could turn in papers any time of day or night.
But there were times when he got the feeling that the graders were too overwhelmed to pay close attention. Like when an essay that had received an 85 or a 90 in the preliminary drafts — and which he then perfected by submitting it to his teacher and the university writing center for a read-through — came back to him on the final as a 72. But he could always go to his classroom instructor to appeal the grade, something he did half a dozen times each semester. All in all, the course was more helpful, and more challenging, than he had anticipated. The main drawback, he says, is that it was "kind of impersonal."
The Instructor as 'Middleman'
While Mr. Kemp contends that anonymous group grading leads to greater consistency, Lindsay Hutton sees too many cooks in the kitchen. "You don't know what the student has been told before," says the graduate instructor. And with so many "teachers" commenting on their work, the students received mixed messages and "didn't know how to differentiate which advice to take."
The tension, she says, affects the morale of both classroom instructor and student. "You've got 35 people who come in who know you have nothing to do with the grade they receive, aside from simply typing it into the computer," says Ms. Hutton. "People think you're just there as this kind of middleman, which you are. Once I started to feel that way, it became difficult for me to really put much into it."
As a document instructor, says Ms. Hutton, she felt even more disconnected from her students. Because of a system backlog, she was sometimes commenting on early drafts of assignments she knew that students had already written final versions of. And without the opportunity to sit down and explain her comments to the students, it often felt as if her feedback was escaping into a vacuum.
"You develop a certain amount of ambivalence," she says. "You're under a time crunch with that work and with your own, and you don't see these people, so your responsibility to them, even though it is not less, may feel like less."
Ms. Hutton, who has taught composition elsewhere under more traditional arrangements, says she prefers having a single teacher for each class, which allows her to "attach a face to every page."
"I felt more that I could help the students" in the traditional classroom, she says. "If they had questions they knew who to ask, and it wasn't someone else's comment I was trying to interpret for them. It was my own."
Who's in Charge?
Indeed, at the core of the debate over the merits of the Texas Tech system is the question of what the relationship between teacher and student should be.
For some, says Shirley K. Rose, president of the Council of Writing Program Administrators, separating the evaluator of student writing from the person teaching it is "going too far."
"It raises the question of whether we care about the texts or the student writing the texts," she says. Proponents of more-mainstream instruction models, she says, focus on "not what the papers of the student are, but what the writing is for the student," emphasizing context and process over product.
Some question whether the Texas Tech English department's reconfigured teacher-student model is really more fair or more productive. By casting a student's relationship with the teacher "as something questionable or surreptitious or evil," says Deborah H. Holdstein, chairwoman of the English department at Northern Illinois University, the model discounts a teacher's ability to inspire and reward students' intellectual development. "Why is the addition of technology making that a pedagogically sound thing to do?" asks Ms. Holdstein.
The most common complaint among grad students is that the system erodes their authority and autonomy as teachers.
Although her experiences were largely positive, Melonie R. McMichael, a Ph.D. student in technical communications at Texas Tech, says that during her first semester teaching she often felt "emasculated" in the classroom. "At times," she says, "I felt like I had no way to encourage my students to do anything in the class." Without control over their final grades, she says, it was often hard to gain students' attention.
She wished she could give extra credit to students who worked hard, she says, but realized that could thwart the objectivity of the system. Still, her frustration peaked when classroom instructors were told they could not round a final computer-generated grade of 89.9 up a tenth of a point, converting it from a B+ to an A.
Mr. Kemp has a term for this frustration. He calls it "the psychology of loss." Even instructors who acknowledge ICON's pragmatism feel that "by losing the power of the grade, they've lost something intrinsic to the student-teacher relationship," says Mr. Kemp. "But that's a myth. Why is it pedagogically necessary for the classroom instructor to be the one grading?"
Rather than focus on loss of authority, Mr. Kemp urges his instructors to reimagine their role as teachers. Now, he says, they are the coach, not the policeman. The theory is that students are more inclined to approach instructors who aren't doling out grades. And classroom instructors can become advocates for their students. "The coach doesn't determine who wins the football game every Saturday," he says. "The coach helps you win the football game."
External Review
Meanwhile, the discontent that produced the TOPIC Sucks newsletter has often divided faculty members and graduate students at Texas Tech along disciplinary lines. Technical-communications majors "get" ICON, it is said; literature majors and creative writers do not. And the tension between the two factions is palpable.
Some faculty members feel that composition administrators have become so invested in the computer system that they've lost sight of the program's initial aims, and technology has become the tail wagging the dog. "They've monkeyed with the system," says Ms. Miner, a professor of American literature, "but they've never been willing to throw out the system."
In August 2005, 19 faculty members petitioned the English department's chairman, Sam Dragga, to commission an external review of the Texas Tech composition program. In October, within a week of a departmental vote in favor of an external review, English professors received an e-mail message from Mr. Dragga announcing his proposal to move undergraduate and graduate writing programs (including composition, rhetoric, and technical communications) out of the English department entirely and into the department of communication studies.
His message cited the "growing similarities among specialists in oral and written communication and the growing differences with specialists in literature, language, and creative writing" as the inspiration for the move. But many professors, says Ms. Miner, see a "curious synchronicity" between the vote and the chairman's proposal, which is still pending.
Mr. Kemp and his composition colleagues welcome the external review, which is to be performed in April by the Council of Writing Program Administrators. He anticipates that a positive evaluation, combined with data produced by ICON, will vindicate the system against its critics.
Composed of nine linked databases, the program contains a veritable gold mine of data. According to a message Mr. Kemp posted on the council's e-mail list, the system produces "201 discrete searchable/sortable chunks of information" on students, teachers, and writing that could be analyzed according to criteria as useful or as obscure as "whether 8 a.m. classes turn in more late papers than 3 p.m. classes ... whether a class is generating a higher than average number of comma splices or semicolon errors than other classes ... whether women comment differently than men... ."
The glut of data the program has amassed has yet to be fully analyzed, and when it is, Mr. Kemp hopes that it will prove empirically what he believes to be true: that ICON "is the best deal for freshmen that I've ever seen."
The program may not fit a liberal-arts college or be a good replacement for a composition program taught by experienced professors, Mr. Kemp acknowledges, but it may suit other large research institutions that, like Texas Tech, rely almost entirely on graduate-student teachers.
Soon those universities will have an opportunity to see for themselves. With the aid of Mr. Kemp, Thomson Learning and Wadsworth are modifying their courseware, InSite, to include a distributed-grading function. The commercial version should be available this fall.
In addition to winning the Genauer Prize in Literary Criticism from the Columbia University Graduate School of Journalism, Paula Wasley was a McCloy Journalism Fellow in 2004. Presently, Wasley covers teaching issues for The Chronicle.
Copyright © 2006 by The Chronicle of Higher Education
Get an RSS (Really Simple Syndication) Reader at no cost from Google at Google Reader. Another free Reader is available at RSS Reader.
