Abstract
The importance of open questions requiring argumentative answers to assess student's competence, along with the increasing number of people applying to colleges, have increased the demand to have systems which automatically score written essays. Developing such a system faces two main challenges. The first is, in many cases, scoring a free answer is largely subjective and does not have well-defined criteria. The second is scoring free answers requires deep language understanding. In this paper, we present an automatic scoring system for Arabic with these two challenges being considered. We only consider the essays of learners of Arabic as a second language in the beginning and intermediate levels. We omit essays of students at advanced levels as these essays might pose different challenges that require deep language understanding. The essays are scored by extracting specific features from the three linguistic levels, lexical, syntax and semantics. Syntactic level scoring is based on the sentence structure. Each level is scored independently and then the final score of the essay is a combination of these scores. We present different experiments with linear and non-linear combination methods on a real dataset. The results obtained from our experiments show that the trained models with respect to a human rater achieve accuracies and quadratic weighted kappa values similar to the agreement between two human raters. It is evident from our results that, with some realistic assumptions, a decision support Arabic scoring system can be achieved.