The problems with NICE 2 and the potential influences on the studies using NICE 2 data

The problems with NICE 2

So far, the problems with NICE 2 have been described on the Web pages since the NICE 2.2 was open to the public on November 25, 2013. The main points are summarized here.

“The finding of these problems is largely based on Professor Tsutomu Ohna’s critical review of the corpus. He deserves special thanks for his pointing out these issues.”

1. We have had explained that the data collection was controlled in the way saying “Write an English essay on one topic in one hour without looking up any references or dictionaries.” However, the following conditions were not controlled or consistent:

(1) In the case of learner

  • Whether we mentioned the rough indication of the length of the essay, which was about 500 words
  • Whether there was a proctor
  • Whether the selection of the topics was free of choice
  • Whether one person writes one essay or more
  • The use of grammar checker of the word processor

(2) In the case of native speakers

  • We mentioned the rough indication of the length of the essay, which was about 500 words, for the majority
  • In many cases there was no proctor
  • Whether the selection of the topics was free
  • Whether one person writes one essay or more

2. There are some essays written by someone who conducted studies with NICE.

3. There are some inconsistencies in the writers’ profile information.

Potential influences on the studies using NICE2:

1. Ishida, T., & Sugiura, M. (2015). [Corrected version] Linguistic features of recurrent word combinations by Japanese EFL learners: The utilization of discriminant analysis,” English Corpus Studies, 22, 21–34.

The data were obtained from a selection of files from learner data (JPN001 to JPN209) and native speaker data (NS001 to NS200). In the corrected paper, the number of tokens, types and the Guiraud index of repetitive word combinations were analyzed. When collecting the data, there was some ambiguity in the instructions. The agreement stated, “write as much as possible within an hour (or, approximately 500 words),” while the instruction sheet included expressions such as “main session (1 hour)” or “essay (please aim to write 500 words).” In addition, there were times when these instructions were provided, and other times when they were not. Due to this inconsistency, we cannot say that the writing conditions were exactly the same. In particular, it is assumed that when native speakers were given the direction to “write as much as possible within an hour (or, approximately 500 words),” they were likely to write “500-word essays”. If it were not for the yardstick of “500 words” in the directions during the data collection, native speakers could have written much longer essays; thus, there is a probability that the number of the tokens and types of repetitive word combinations could have been different.

This paper fails to consider this point, and it didn’t mention that the instruction included the yardstick of “500 words”, which may mislead the readers into believing that the essays were written under the condition of “one hour” only. We regret that we should have had more careful considerations.

2. Eguchi, A., & Sugiura, M. (2015). Reconsideration of the Group Score Method in Accuracy Order Studies of Grammatical Morphemes for Japanese EFL Learners. ARELE: Annual Review of English Language Education in Japan, 26, 157–172.

In this paper, we pointed out the potential problems of the scoring method (GSM) which has been applied widely in the acquisition order studies of grammatical morphemes of the second language, and proposed a new calculation method (GSMII) with which overuses are also included in the score. In this study, out of the learner data of NICE, we used the files (JPN001 to JPN201) to which the correction sentences by a native speaker were attached, and compared the scores calculated by GSM and GSMII concerning verbal morphemes among the three proficiency level groups. Based on the results, we discussed the possibilities that we could analyze the acquisition status of individual morpheme in more detail by taking into account of the proficiency and overuse.

However, the files used in this study were taken from NICE (JPN001 to JPN201) and during the data collection procedures of these files no attention was paid to the grammar checker function of the word processing software. To our knowledge, it has not been made clear whether and how much the grammar checker has influence on the grammatical accuracy in English essay writing. Therefore, we cannot totally deny the possibility that there is an effect of the grammar checker. For that we did not mention these possibilities in the paper, we regret that consideration was not enough.

3. Sugiura (ed.) (2008) Comprehensive and systematic analyses of the errors of English learners by applying the natural language processing technology. KAKENHI Grant-in-Aid for Exploratory Research (JP16652044) Report.

In this study, as a part of the analysis data, the NICE ver. 1 was used, which contains the above-mentioned problems of the NICE 2. Possibility that these problems affected the analysis cannot be denied. When reading this report, please read in light of the above-mentioned problems. When you would like to use this report as a reference, please contact Sugiura.

 

Should you have any questions when you are using the NICE 2 data or the studies using the NICE 2 data, please contact Sugiura. Thank you.