The Heller Report: ETS brings E-Rater essay scoring technology online

ETS brings E-Rater essay scoring technology online

ETS Technologies, Inc., the for-profit subsidiary of Educational Testing Service (both of Princeton, NJ) is in the process of bringing an automated essay scoring technology onto the market. Richard Swartz, vp, says the technology is currently being offered to academic institutions in a pilot program, and he is in negotiations with several publishers and online content providers wishing to use the technology as a value-added service to their offering.

The technology, called E-Rater, was developed by researchers in natural language processing. It can score two types of essays–analysis of an issue and analysis of an argument. Both approaches work by generating data from analyzing large collections of essays on a topic. The system can evaluate features such as being on topic, using examples and supporting materials, how sentences proceed from one another, and the kinds of sentences that are used according to their variety and length.

Developing a topic requires evaluating a minimum of 470 representative essays. ETS Technologies has developed ten topics in three categories: college-level topics used for placement decisions or introductory composition courses, academic skills topics used for assessing general writing ability, and national standards topics used in 8th and 12th grade expository writing classes. Swartz says they will soon add upper division writing. E-Rater has scored more than half a million essays for the Graduate Management Admissions Test (GMAT), and those scores, says Swartz, agree with expert faculty readers over 97% of the time.

Swartz says that ETS Technologies has two channels to make the technology available. With Criterion, a direct model, the company provides E-Rater scoring services on a subscription basis to schools. Schools currently using the service are in a free, pilot program until December. Subsequent pricing has not been determined

ETS will also build E-Rater topics and host scoring for learning companies. Fees are likely to be based on a revenue sharing or per-submission basis.

Interest, says Swartz, is coming from online learning and assessment providers, especially those working in the K-12 markets with a focus on state-based standards. Textbook publishers wanting to provide supplementary online materials are also emerging as an especially interested market.

The first announced client is QuestionMark, a testing company that will integrate Criterion as a part of their product called Perception, a subscription service to schools. Swartz is in negotiations with about ten other companies. Many more are expressing interest.

Existing topics typically don’t meet the needs of these potential clients, so discussions include issues of building E-Rater scoring models, whether or not clients already have collected and scored essays, and whether the client wants to maintain ownership of those topics or allow them to be resold.

The original intent for developing the technology was to predict scores for high stakes test. As the company moves the technology forward, Swartz anticipates making it more useful instructionally. They will, for example, be adding enhancements to make feedback more diagnostic with commentary on issues such as grammar, syntax and the structure of the writing.

COPYRIGHT 2000 Nelson B. Heller & Associates

COPYRIGHT 2000 Gale Group