ОЦЕНИВАНИЕ КОМПЕТЕНЦИЙ ТЕСТИРУЕМОГО НА ОСНОВЕ ЛОГИКО-ЛИНГВИСТИЧЕСКОЙ МОДЕЛИ

ОЦЕНИВАНИЕ КОМПЕТЕНЦИЙ ТЕСТИРУЕМОГО НА ОСНОВЕ ЛОГИКО-ЛИНГВИСТИЧЕСКОЙ МОДЕЛИ

Научная статья

Домшенко Н.Г.¹, Морозова М.Н.^2, *,  Рубцова С.Ю.³, Спесивцев А.В.⁴

¹ ORCHID: 0000-0001-8887-9205;

² ORCHID: 0000-0002-8514-2803;

³ ORCHID: 0000-0003-2684-5872;

^{1, 2, 3} Санкт-Петербургский Государственный Университет, Санкт-Петербург, Россия;

⁴ Санкт-Петербургский институт информатики и автоматизации Российской академии наук, Санкт-Петербург, Россия

* Корреспондирующий автор (morozova.m[at]mail.ru)

Аннотация

Для повышения точности и объективности решения построена логико-лингвистическая модель количественного оценивания компетенций тестируемых по курсу английского языка по конвертированной семибалльной шкале оценок ECTS в 7-факторном пространстве. Модель отражает знания и опыт ведущих преподавателей, который можно использовать для мониторинга знаний студентов в процессе обучения, как базу знаний при построении экспертных систем, а также в прикладных науках и информационных технологиях.

Ключевые слова: оценивание компетенций, факторное пространство, логико-лингвистические модели, экспертные знания, экспертные системы, мониторинг успеваемости.

THE ASSESSMENT OF THE EXAMINEE’S COMPETENCES ON THE BASIS OF THE LOGICAL-LINGUISTIC MODEL

Research article

Domshenko N.G.¹, Morozova M.N.^2, *, Rubtsova S.Y.³, Spesivtsev A.V.⁴

¹ ORCHID: 0000-0001-8887-9205;

² ORCHID: 0000-0002-8514-2803;

³ ORCHID: 0000-0003-2684-5872;

^{1, 2, 3} St. Petersburg State University, St. Petersburg, Russia;

⁴ St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, St. Petersburg, Russia

* Corresponding author (morozova.m[at]mail.ru)

Abstract A logical-linguistic model of competencies quantitative assessment of the examinee taking  English language tests based on a converted -grade scale ECTS in a 7-factor space has been built to improve the accuracy and objectivity of the decision made by the examiner. The model reflects the expertise of teachers, which can be used to assess students' competence in the learning process. Moreover, it can be applicable as a knowledge base when building expert systems, as well as in technical and IT sciences. Keywords: assessment of competences, factor space, logico-linguistic models, expertise, expert systems, monitoring success. Introduction

The main challenges in the shared learning space are the realization and successful implementation of programmes reinforcing a whole number of compulsory competences. These include: axiological values; knowledge; the skills and abilities of students; the monitoring of performance; and the identification of comparative characteristics of various teaching packages. Therefore, the mechanisms for assessing the established skills are of primary importance [1], [2], [3].

The current trend for simplification and automation of learning progress control aims to increase the accuracy and objectivity of the decision made by the examiner [3], [4]. However, the differences in testing and assessment materials while implementing language competences raise questions over the flexibility, non-linear nature, and individualisation of assessment tools when ranging examinees.

There are works providing methodologies for the assessment of competences based on the expertise of teachers in the development of expert systems in this sphere [5], [6], [7]. There is a vagueness of the very concept of competence within the ambiguous semantics of specific terms, and a vagueness and inconsistency of specific requirements for students. As a result, these works are mostly of a declarative nature, even though they demonstrate the growing need for the development of expert systems.

Problem statement

This research is motivated by the pressing need to optimise the assessment of verbal linguistic competences by applying quantitative criteria. These criteria are characteristic of the logical-linguistic models [8] that are based on the expertise of highly-qualified specialists and their involvement as experts. An expert is thus viewed as an “intellectual measuring and diagnostic system” [9]. This definition applies completely to the profession of a teacher, as it is certain that they: possess intellect; carry out knowledge assessment in a specific area of a discipline; and diagnose the general extent of competence development in the examinee. This definition meets the description of expert’s work in any sphere and is increasingly applied in technical and IT sciences. The decision-making mechanism in the situation of uncertainty is described with the help of logical-linguistic models based on the knowledge of an expert and their professional experience [8]. The advantage of a logical-linguistic model is that it makes it possible to obtain a quantitative assessment of students’ knowledge using the entire diagnostic scale [3, 10] rather than just its opposite ends (where “OK” means that the competence is formed and “NOT OK” represents the undeveloped competence [7]). This allows for a more objective attitude to an individual student.

In order to build such models, it is essential to use a set of variables constituting the factor space within which the teacher decides whether the level of required competence was achieved.

The current research is limited to the language (linguistic) and speech competences which include: grammar and lexico-semantic competences; and discourse, phonetic and social interaction competences. These competences have been monitored in monologue and dialogue. The research was carried out with several groups of students studying English at B2 level (under the CEFR [10]) in situations of cross-cultural foreign language communication [4].

Methods

Building a logical-linguistic model [8] for each separate task comprises the following stages: identification and justification of the factor space where the expert makes the decision; preparation of a special examinational matrix to be filled in by the expert; and synthesis and professional analysis of the logical-linguistic model.

Within this research the factor space is described by seven input variables X₁-X₇.

The choice of competences corresponds to: the systemic approach to the examinee knowledge assessment procedure; and the overall index of course competence Y.

Let us consider in more detail the factor space in the task of building a logico-linguistic model for examinee competence assessment.

X₁ – discourse competence when evaluating: coherence, cohesion and consistency of the examinee’s answer; and the ability to produce reasoned statements and to indulge in critical thinking. There should be a particular emphasis on the ability of students to support the statement with relevant examples, statistical data and references to the latest research in a specific area.

Grammar competence implies two factors:

X₂ – communication in a natural manner, using proper and adequate grammar structures depending on the context of the utterance.

X₃ – usage of diverse grammatical structures of the English language in accordance with the indicated level, namely all: tense-aspect forms of the verb; degrees of comparison of adjectives; conditionals (types 1-3); words functioning as verbs; modal verbs in combination with infinitives; direct and indirect speech; active and passive voice; as well as specific constructions for future actions or habits in the past.

Lexico-semantic competence has also been considered in two aspects:

X₄ – skills to select vocabulary depending on the situation, diversity of vocabulary and, as a result, absence of repetition.

X₅ – adequacy of lexical elements. This category includes partly the compensatory competence; in which the examinee demonstrates his skills to cope with the shortage of vocabulary by using: international words; referring to the text; paraphrase; synonyms; substitution; and so forth.

X₆ – phonetic competence, which implies proper intonation, speech fluency, presence or absence of pronounced accent, pausing or mispronunciation of separate words.

X₇ – social interaction competence. This has been limited to the ability of the examinee: to understand direct or implied sense (meaning) from the speech of the interlocutor; as well as to use the language for specific purposes depending on the characteristics of social and professional interaction including the situation and status of the communicators and the addressee.

As a grading scale for the learning outcomes for separate competences and the subject as a whole a converted 7-grade scale ECTS (European Credit Transfer and Accumulation System) was adapted. This is shown in table 1.

The first five grades are sufficient for getting credits, the last two are not being used.

Table 1 – Examinee grading scale

Grade	Percentile %	Verbal definition	Corresponding score
A	90-100	Excellent	5
B	80-89	Very good	4+
C	70-79	Good	4
D	60-69	Satisfactory	3
E	50-59	Performance meets the minimum criteria	3-
FX		Unsatisfactory	2
F		Fail	1

 

Each of the competences was presented as a linguistic variable (see Fig. 1) and assessed on a specific scale. Each class was defined so that the learning outcome could be related to a specific class and then expressed numerically as in

Fig. 1 – Y as a linguistic variable

 

The linguistic variable (Fig. 1) is used for converting a verbal grade definition into quantitative information. It contains three scales on the x-axis: linguistic (the upper scale); numerical (to convert into natural scale); and standardized scale to be used in experimental design theory [«-1» – lowest grade Е; «+1» – highest grade А]. On the y-axis, there is the membership function scale of μ (у) grade. This means that the closer the grade is to the class mode, the higher is its accuracy.

For example, grade Y awarded for the whole subject could refer to class A if the examinee:

- produces well-structured syntactic constructions;

- has a good command of various language functions;

- demonstrates flexibility in using various linguistic forms;

- has a wide range of relevant expressions to perform the assigned task;

- demonstrates a high level of vocabulary, making mistakes which do not affect communication;

- has a consistently high level of grammatically connected speech;

- can support conversation on a particular topic.

The level of competences is graded as E class if the examinee:

- demonstrates a limited range of linguistic means;

- makes a lot of language and phonetic mistakes;

- is unable to conduct a logical and coherent conversation;

- depends on the help of the interlocutor;

- only partially performs the communicative task.

Then, in accordance with the methodological algorithm [8], [9], the expert/teacher fills in a special examinational matrix of linguistic variables (see Table 2), where each line stands for an implicative condition-action rule “if …, then …” (“situation” – “grade”). Thus, line 62 of Table 2 is the following condition – action rule: “If X₁ – discourse competence – is “high”, X₂ – communication in a natural manner – is “low”, X₃ – usage of diverse grammar structures – is “high”, X₄ – skills to select vocabulary depending on the situation – is “high”, X₅ – adequacy of lexical elements – is “high”, X₆ – phonetic competence – is “high” and X₇ – social interaction competence – is “low”, then Y – the overall index of competence – is in between classes B-A”.

Table 2 – Excerpt from the examinational table with the experts’ answers in linguistic form

Conversion of expert grades into numerical form using the scales of Figure 1 or Table 1 and processing these data using the methods of experimental design theory has resulted in the following analytical equation:

         (1)

where variables  are presented on a standardized (numerical) scale according to a formula:

        (2)

The adequacy of analytic equation (1) was verified by two criteria: correlation between the expert assessment and calculations based on the values in (1) (Fig. 2a); and the correlation between calculated values in (1) and grades awarded to a group of eight students by the teachers who were not familiar with this methodology (Fig. 2b).

Fig. 2 – Evaluating the adequacy of calculations based on (1):

a – knowledge and experience of experts; b – independent experiment data

 

Results and discussion

Following from the analysis of the correlation point field graph, theoretical regression lines are at 45 degrees. This means that: there are no systematic errors when comparing functions; and the high correlation co-efficient (more than 0.96) demonstrates that the results of these calculations correspond to independent grades in the target group of students. These two conclusions make it possible to say that analytical equation (1) is a mathematical model of assessment of a student’s competence in a specific subject.

Conclusion

Thus, the research demonstrates the effectiveness of the method of consolidating the expertise of leading experts by logical-linguistic models. They make it possible to monitor the learning outcomes of examinees in quantitative terms, and to receive comparable grades with the help of common methodology and draw on them at any moment in time. Also, it is possible to preserve and replicate the expertise of a qualified specialist for the practical use of junior teachers.

This method of building logical-linguistic models is universal. It can be applied to solving various tasks, drawing on the expert’s knowledge and experience. The availability of the equation allows for its use as a knowledge database to design systems of any complexity.

Конфликт интересов Не указан.

Conflict of Interest None declared.

Список литературы / References

1. Аскеров Ш. Философия оценки знания. / Ш. Аскеров http://www.courier.com.ru/ scores/askerov.htm(01.02.2007)
2. Ромашкина Г. Ф. Оценка качества образования: опыт эмпирического исследования. / Г. Ф. Ромашкина // Университетское управление: практика и анализ, 2005, No5, C. 83–88.
3. Сон И. С. Особенности оценивания знаний по иностранному языку (говорение, письмо) по балльно-рейтинговой системе [Текст] // Теория и практика образования в современном мире: материалы IIМеждунар. науч. конф. (г. Санкт-Петербург, ноябрь 2012 г.). — СПб.: Реноме, 2012. — С. 185-188. — URL https://moluch.ru/conf/ped/archive/64/2911/
4. Тихомирова А. В. Типология моделей коммуникаций в контексте инокультурной / интеркультурной коммуникативной компетенции учащихся неязыкового вуза / А. В. Тихомирова, А. А. Богатырёв // Международный научно-исследовательский журнал. – 2016. – № 1 (43) Часть 4. – С. 50—57.
5. Чванова М. С. Проблемы использования экспертных систем в образовании / М. С. Чванова, И. А. Киселева, А.А. Молчанов // Вестник Тамбовского университета. Серия: гуманитарные науки. 2013. №3 (119). –  С. 39-47.
6. Котова Е. Е. Имитация системы управления процессом обучения с Fazzy-алгоритмами принятия решений / Е.Е. Котова //Международная конференция по мягким вычислениям и измерениям. – Санкт-Петербург, 2014. Т.2 – С. 212-215.
7. Котова Е. Е. Постановка задачи оценивания компетенций выпускника методами мягких измерений / Е.Е. Котова, А. Г. Степанов // ХХI Международная конференция по мягким вычислениям и измерениям SCM 23-25  мая 2018. – Санкт-Петербург, 2018. Т.2 – С. 281-282.
8. Спесивцев А. В. Управление рисками чрезвычайных ситуаций на основе формализации экспертной информации. Под ред. проф. В.С. Артамонова / А. В. Спесивцев.– СПб: Изд-во Политехн. ун-та, 2004. – 238 с.
9. Спесивцев А. В. Эксперт как «интеллектуальная измерительно-диагностическая система». / А. В. Спесивцев, Н. Г. Домшенко //Сб. докладов. ХIIIМеждународная конференция по мягким вычислениям и измерениям SCM, 23-25  июля 2010, Санкт-Петербург, 2010, Т.2. – С. 28-34.
10. Дьяконов Г. Н. Размытые шкалы компетентно-ориентированных оценок качества подготовки бакалавров / Г. Н. Дьяконов // ХVМеждународная конференция по мягким вычислениям и измерениям SCM 25-27 июня 2012. – Санкт-Петербург, 2012. Т.2 – С. 125-128.
11. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Companion Volume with New Descriptors. Council of Europe, February, 2018.

Список литературы на английском языке / References in English

1. Askerov Sh. Filosofija ocenki znanija [Knowledge Assessment Philosophy]/ Sh. Askerov http://www.courier.com.ru/ scores/askerov.htm (accessed: 01.02.2007). [In Russian]
2. Romashkina G. F. Ocenka kachestva obrazovanija: opyt jempiricheskogo issledovanija. [Education Quality Assessment] / G. Romashkina // Universitetskoe upravlenie: praktika i analiz, [University management: practice and analysis] 2005, No 5, P. 83–88. [In Russian]
3. Son I. S. Osobennosti ocenivanija znanij po inostrannomu jazyku (govorenie, pis'mo) po ball'no-rejtingovoj sisteme [Peculiarities of Foreign Knowledge Assessment] // Teorija i praktika obrazovanija v sovremennom mire: materialy II Mezhdunar. nauch. konf. (g. Sankt-Peterburg, nojabr' 2012 g. [Theory and practice of education in the modern world: Proceedings of the II International. scientific conf.)]. — SPb.: Renome, 2012. — S. 185-188. — URL https://moluch.ru/conf/ped/archive/64/2911/[In Russian]
4. Tihomirova A. V. Tipologija modelej kommunikacij v kontekste inokul'turnoj / interkul'turnoj kommunikativnoj kompetencii uchashhihsja nejazykovogo vuza [Typology of the Communication Models in the Context of Foreign- and Inter-cultural Communicative Competence of Students from a Non-Linguistic Institution] / A. V. Tihomirova, A. A. Bogatyrjov // Mezhdunarodnyj nauchno-issledovatel'skij zhurnal. [International Research Journal].– 2016. – # 1 (43) Chast' 4. – P. 50—57. [In Russian]
5. M. S. Problemy ispolzovanija jekspertnyh system v obrazovanii [Problems of expert systems in education]/ M. S.Chvanova, I. A.Kiseleva, A. A.Molchanov//Vestnik Tambovskogo universiteta. Serija: gumanitarnye-nauki [Tambov University Review. Series Humanities].- 2013.- №3 (119). – P. 39-47 [In Russian]
6. Kotova E. E. Imitacija sistemy upravlenija processom obuchenija s Fazzy-algoritmami prinjatija reshenij [Simulation of learning process management based on a fuzzy inference system] // Mezhdunarodnaja-konferencija-po-mjagkim-vychislenijam-i-izmerenijam [International Conference on Soft Computing and Measurement] Sankt-Peterburg,-2014.-Vol. 2- P. 212-215. [In Russian]
7. Kotova E. E. Postanovka zadachi ocenivanija kompetencij vypusknika metodami mjagkih izmerenij [The problem of graduate student assessment using soft measurement methods] / E. E.-Kotova, A. G. Stepanov // ХХI –Mezhdunarodnaja konferencija po mjagkim vychislenijam I izmerenijam [ХХI International Conference on Soft Computing and Measurement] SCM 23-25 maja 2018. Sankt-Peterburg, 2018.-Vol. 2., P. 281-282. [In Russian]
8. Spesivcev A. V. Upravlenie riskami chrezvychajnyh situacij na osnove formalizacii jekspertnoj informacii. [Management of emergency risks based on formalized expert information] / A. Spesivcev . Spesivcev, V. S. Artamonova. - SPb:-Ed.-Politehn.-un-ta,-2004.- 238 p. [In Russian]
9. Spesivcev A. V. Ekspert kak «intellektualnaja izmeritelno-diagnosticheskaja sistema» [Expert as an “intellectual measurement and diagnostic system”]/ A. Spesivcev // Domshenko-N.G.-//Sb.-dokladov.- ХIII –Mezhdunarodnaja konferencija po mjagkim vychislenijam I izmerenijam [ХIII International Conference on Soft Computing and Measurement] SCM, 23-25 ijulja 2010,-Sankt-Peterburg,-2010,-Vol. 2. P.28-34 [In Russian]
10. D'jakonov G. N. Razmytye shkaly kompetentno-orientirovannyh ocenok kachestva podgotovki bakalavrov [Fuzzy-Scales of Competence-Oriented Assessments of the Quality of Bachelors’ Studies] / G. N. D'jakonov // HV Mezhdunarodnaja konferencija po mjagkim vychislenijam i izmerenijam SCM [International Conference on Soft Computing and Measurement SCM ]25-27 ijunja 2012. – Sankt-Peterburg, 2012. Vol. 2 – P. 125-128.
11. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Companion Volume with New Descriptors. Council of Europe, February, 2018.