Advanced SearchSearch Tips
Automatic Detection of Off-topic Documents using ConceptNet and Essay Prompt in Automated English Essay Scoring
facebook(new window)  Pirnt(new window) E-mail(new window) Excel Download
  • Journal title : Journal of KIISE
  • Volume 42, Issue 12,  2015, pp.1522-1534
  • Publisher : Korean Institute of Information Scientists and Engineers
  • DOI : 10.5626/JOK.2015.42.12.1522
 Title & Authors
Automatic Detection of Off-topic Documents using ConceptNet and Essay Prompt in Automated English Essay Scoring
Lee, Kong Joo; Lee, Gyoung Ho;
This work presents a new method that can predict, without the use of training data, whether an input essay is written on a given topic. ConceptNet is a common-sense knowledge base that is generated automatically from sentences that are extracted from a variety of document types. An essay prompt is the topic that an essay should be written about. The method that is proposed in this paper uses ConceptNet and an essay prompt to decide whether or not an input essay is off-topic. We introduce a way to find the shortest path between two nodes on ConceptNet, as well as a way to calculate the semantic similarity between two nodes. Not only an essay prompt but also a student's essay can be represented by concept nodes in ConceptNet. The semantic similarity between the concepts that represent an essay prompt and the other concepts that represent a student's essay can be used for a calculation to rank "on-topicness" ; if a low ranking is derived, an essay is regarded as off-topic. We used eight different essay prompts and a student-essay collection for the performance evaluation, whereby our proposed method shows a performance that is better than those of the previous studies. As ConceptNet enables the conduction of a simple text inference, our new method looks very promising with respect to the design of an essay prompt for which a simple inference is required.
off-topic document;ConceptNet;automated English essay scoring;text inference;
 Cited by
Jill Burstein and Derrick Higgins, "Advanced Capabilities for Evaluating Student Writing: Detecting Off-Topic Essays Without Topic-Specific Training," Proc. of the International Conference on Artificial Intelligence in Education, Jul. 2005.

Robert Speer and Catherine Havasi, "Representing General Relational Knowledge in ConceptNet 5," LREC, pp. 3679-3686, 2012.

Higgins, D., Burstein, J., Attali, Y., "Identifying offtopic student essays without topic-specific training data," Natural Language Engineering, Vol. 12, No. 2, pp. 145-159, 2006. crossref(new window)

Annie Louis and Derrick Higgins, "Off-topic essay detection using short prompt texts," Proc. of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 92-95, 2010.

Spagnola, S., and Lagoze, C., "Edge dependent pathway scoring for calculating semantic similarity in ConceptNet," Proc. of the Ninth International Conference on Computational Semantics, pp. 385-389, 2011.

Peter Norvig, "Inference In Text Understanding," AAAI-87 Proceedings, 1987.

Sanda M. Harabagiu and Dan I. Moldovan, "A Parallel System for Text Inference Using Marker Propagations," IEEE Transactions on Parallel and Distributed Systems, Vol. 9, No. 8, Aug. 1998.

Gyoung Ho Lee, Kong Joo Lee, "Developing an Automated English Sentence Scoring System for Middle-school Level Writing Test by Using Machine Learning Techniques," Journal of KIISE, Vol. 41, No. 11, pp. 911-920, 2014. crossref(new window)