---
abstract: |
    Interactive Educational Systems (IESs) have developed rapidly in recent years to address the issue of quality and affordability of education. Analogous to many other domains in Artificial Intelligence (AI), there are specific tasks of AI in Education (AIEd) for which labels are scarce and expensive. For instance, labels like exam score and grade are considered important in educational and social context. However, unlike interactive features automatically collected by IESs, obtaining the labels is costly as they require student actions taken outside the system. Likewise, while student events like course dropout and review correctness are automatically recorded by IESs, they are few in number as the events occur sporadically in practice. A common way of circumventing the label-scarcity problem is the pre-train/fine-tine method, where a model is trained in a relevant auxiliary task with a large amount of data before the main task. Accordingly, existing works pre-train a model to learn representations of contents in learning items (e.g. exercises). However, such methods fail to utilize the student interaction data available and model student learning behavior.

    To this end, we propose assessment modeling, fundamental pre-training tasks for general IESs. An assessment is a feature of student-system interactions which can act as pedagogical evaluation, such as the correctness of a student’s answer or the time taken for the student to answer. Assessment modeling is the prediction of assessments conditioned on the surrounding context of interactions. Although it is natural to pre-train interactive features available in large amount, narrowing down the prediction targets to assessments holds relevance to the label-scarce educational problems while reducing irrelevant noises.

    To the best of our knowledge, this is the first work investigating appropriate pre-training method of predicting educational features from student-system interactions. While the effectiveness of different combinations of assessments is open for exploration, we suggest assessment modeling as a guiding principle for selecting proper pre-training tasks for the label-scarce educational problems.
author:
- 'Youngduck Choi^1,2^, Youngnam Lee^1^, Junghyun Cho^1^, Jineon Baek^1,3^, Dongmin Shin^1^'
- 'Seewoo Lee^1,4^, Youngmin Cha^1^, Byungsoo Kim^1^, Jaewe Heo^1^'
title: |
    Assessment Modeling: Fundamental Pre-training Tasks\
    for Interactive Educational Systems
---

&lt;ccs2012&gt; &lt;concept&gt; &lt;concept\_id&gt;10003456.10003457.10003527.10003540&lt;/concept\_id&gt; &lt;concept\_desc&gt;Social and professional topics Student assessment&lt;/concept\_desc&gt; &lt;concept\_significance&gt;500&lt;/concept\_significance&gt; &lt;/concept&gt; &lt;concept&gt; &lt;concept\_id&gt;10003456.10003457.10003527.10003541&lt;/concept\_id&gt; &lt;concept\_desc&gt;Social and professional topics K-12 education&lt;/concept\_desc&gt; &lt;concept\_significance&gt;500&lt;/concept\_significance&gt; &lt;/concept&gt; &lt;/ccs2012&gt;

Introduction
============

![image](fig1.pdf){width="75.00000%"}

Interactive Educational System (IES) interacts with students to assess them and design individualized optimal learning paths. The observations of student behaviors collected automatically by IESs are large in scale, thus power data-driven approaches for many Artificial Intelligence in Education (AIEd) tasks and improve the quality of IESs further. However, there are important tasks with insufficient data that prevent relevant models from attaining the full potential. A possible case is when the task depends on labels external to IESs that is rare in number. For example, the labels for a grade prediction model in an in-class IES are collected at the end of class. In some cases, the data points are collected by IESs but simply lacks in number because the events occur sporadically in practice. For instance, the number of actions reviewing previously solved exercises may be outnumbered by number of actions solving unseen exercises, as it is likely for a student to invest more time to solve new exercises than to review.

To circumvent the lack of data, transfer learning has been studied in AIEd literature [@hunt2017transfer; @ding2019transfer]. However, the method works only when the available data and the task to complete have the same form. Instead, the pre-train/fine-tune paradigm can be considered. In this paradigm, a model is first pre-trained in an unsupervised auxiliary task with abundant data. Then, the model is modified slightly and trained in main tasks with possibly scarce data. This approach was successful in other AI fields including Natural Language Processing (NLP), computer vision, motion planning and so on [@devlin2018bert; @studer2019comprehensive; @schneider2019wav2vec]. In this line of thought, content-based pre-training methods have been studied [@huang2017question; @sung2019improving; @yin2019quesnet]. However, the pre-trained models fail to capture student learning behavior as student interactions are not considered.

In this paper, we propose assessment modeling, fundamental pre-training tasks for general IESs. Here, an assessment is any feature from student-system interactions which can act as pedagogical evaluation criterion. Examples of assessments are the time a student has spent on each exercise and the correctness of student response to a given exercise. While there is a wide range of interactive features available, narrowing down the prediction targets to assessments holds relevance to the label-scarce educational problems while reducing irrelevant noises. Also, most of assessments are available in large amount as they are automatically collected by IESs.

Inspired by the recent success of bidirectional representations in NLP domain [@devlin-etal-2019-bert], we develop an assessment model with a deep bidirectional Transformer encoder. In the pre-training phase, we randomly select a portion of entries in a sequence of interactions and mask the corresponding assessments. Then, we train a deep bidirectional Transformer encoder based assessment model to predict the masked assessments conditioned on surrounding interactions. After pre-training, we replace the last layer of the model with a layer corresponding to each downstream task, and all parameters of the model are then fine-tuned to the downstream tasks.

We empirically evaluate assessment modeling as pre-training tasks on *EdNet* [@choi2019ednet], a large-scale dataset collected by an active mobile education application, *Santa*, which has 1M users as well as 72M response data points gathered since 2016. The results show that assessment modeling provides a substantial performance gain in downstream AIEd tasks. In particular, we obtain an improvement of $14.16$ mean absolute error and $0.014$ accuracy from the previous state-of-the-art model for exam score and review correctness prediction respectively.

In summary, our contributions are follows:

-   We propose assessment modeling, fundamental pre-training tasks for general IESs. Assessment modeling as pre-training tasks allows the model to hold relevance to the label-scarce educational problems while reducing irrelevant noises.

-   We give formal definitions of knowledge tracing and assessment modeling in a form that is quantifiable and objective from a particular IES design.

-   Inspired by the recent success of bidirectional representation in NLP domain [@devlin-etal-2019-bert], we develop an assessment model with a deep bidirectional Transformer encoder.

-   We empirically verify that assessment modeling as pre-training tasks achieves an improvement of $14.16$ mean absolute error and $0.014$ accuracy from the previous state-of-the-art model for exam score and review correctness prediction respectively.

Related Works
=============

### Artificial Intelligence in Education

AIEd supports education through different AI technologies, including machine learning and deep learning, to “promote the development of adaptive learning environments and other AIEd tools that are flexible, inclusive, personalised, engaging, and effective" [@luckin2016intelligence]. A large body of work have committed to the development of AI models for AIEd tasks, including knowledge tracing [@corbett1994knowledge; @piech2015deep], question analysis [@yin2019quesnet; @liu2018finding], student score/grade prediction [@iqbal2017machine; @morsy2017cumulative; @almasri2019emt; @patil2017effective], educational content recommendation [@liu2019exploiting] and many more. With these models, an IES constantly updates the state of each student through constant feedback and provide appropriate learning experience to learners.

### Pre-training Methods in Education

Pre-training is the act of training a model to an unsupervised auxiliary task before the supervised main task [@erhan2010does]. Models using pre-training have been shown to outperform existing models in various fields including NLP [@devlin2018bert], Computer Vision [@studer2019comprehensive], Speech Recognition [@schneider2019wav2vec], Medical Science [@choi2016learning] and many more.

Pre-training techniques have been also applied to educational tasks with substantial performance improvements. For example, [@hunt2017transfer] predict whether a student will graduate or not based on students’ general academic information such as SAT/ACT scores or courses taken during college. They predict graduation of 465 engineering students by first pre-training the data of 6834 students in other departments with TrAdaBoost algorithm [@dai2007boosting]. [@ding2019transfer] suggests two transfer learning methods, each named Passive-AE transfer and Active-AE transfer, to predict student dropout in Massive Open Online Course (MOOC). Their experimental results show that both methods increase the prediction accuracy, with Passive-AE transfer more effective for transfer learning across the same subject and Active-AE transfer more effective for transfer learning across different subjects.

Most of the pre-training methods used in interactive educational system are NLP tasks on the content of learning materials. For example, the short answer grading model suggested in [@sung2019improving] use pre-trained BERT to ameliorate the limited amount of student-answer pair data. They used pre-trained, uncased BERT-base model and fine-tuned it with ScientsBank dataset and two psychology domain datasets. The resulting model outperformed existing grading models. Test-aware Attention-based Convolutional Neural Network (TACNN) [@huang2017question] is a model that utilize the semantic representations of text materials (document, question and options) to predict exam question difficulty (i.e. the percentage of examinees with wrong answer for a particular question). TACNN uses pre-trained word2vec embedding [@mikolov2013distributedWORD2VEC] to represent word tokens. By applying convolutional neural networks to the sequence of text tokens and attention mechanism to the series of sentences, the model quantifies the difficulty of the question.

QuesNet [@yin2019quesnet] is a question embedding model pre-trained with context information of question data. Since existing pre-training methods in NLP is not applicable to heterogeneous data in a question (such as images and metadata), the authors suggest Holed Language Model (HLM), a pre-training task in parallel to BERT’s masking language model. HLM differs to BERT’s task, however, because it predicts each input based on the values of other inputs aggregated in the Bi-LSTM layer of QuesNet, while BERT masks existing sequence randomly with certain probability. Also, QuesNet introduces another task called Domain-Oriented Objective (DOO), which is the prediction of the correctness of answer supplied with question, to capture high-level logical information. QuesNet adds two losses for HLM and DOO respectively for final loss in training. Compared to other baseline models, QuesNet shows the best performance in three downstream tasks: knowledge mapping, difficulty estimation, and score prediction.

Assessment Modeling
===================

Formal Definition of Assessment Modeling
----------------------------------------

![The relationship of features predicted in general AIEd tasks, knowledge tracing and assessment modeling. Assessment modeling predicts the distribution of assessments, the subset of interactive features which can act as pedagogical evaluation criteria. Note that predicting an exam score (`exam_score`), a grade (`grade`) and whether a student will pass to get a certificate (`certification`) are tasks outside knowledge tracing.[]{data-label="fig:am_def"}](assessment_def.pdf){width="50.00000%"}

Recall that knowledge tracing is the task of modeling a student’s knowledge state based on his/her learning activities over time. Although the task is considered as fundamental and studied extensively over time, it has not been defined formally and explicitly in existing literature. In this subsection, we first define knowledge tracing in a form that is quantifiable and objective from a particular IES design. Subsequently, we introduce the definition of assessment modeling that addresses the educational values of the label being predicted.

A learning session in IES consists of a series of interactions $[I_{1}, \dots, I_{T}]$ between a student and the system, where each interaction $I_{i} = \{f_{i}^{1}, \dots, f_{i}^{n}\}$ is represented as a set of features $f_{i}^{k}$ automatically collected by the system. The features represent diverse aspects of learning activities provided by the system, such as exercises or lectures being used, and the corresponding student actions. Under the given notations, the definitions of knowledge tracing and assessment modeling are as follows:

\[def:kt\] A *knowledge tracing* is a task of predicting a feature $f_{t}^{k}$ of the student in the $t$’th interaction $I_t$ given the sequence of interactions $[I_1, \dots, I_{t-1}, I_t, I_{t+1}, \dots, I_T]$. That is, the prediction of $$\label{eq:kt}
p(f_{t}^{k} | I_1, \dots, I_{t-1}, I_t \setminus \Gamma(f_{t}^{k}), I_{t+1}, \dots, I_T)$$ where $\Gamma(f)$ is the set of features masked when the feature $f$ is guessed. This is to mask input features not available at prediction time, so that a model does not ‘cheat’ guessing $f$. In particular, the feature $f$ itself is always masked so that $f \in \Gamma(f)$.

This definition follows the prior works on knowledge tracing models [@piech2015deep; @zhang_2017; @liu_2019; @pandey2019self]. Although a common set-up of knowledge tracing models is to predict a feature conditioned on only past interactions, we define knowledge tracing as a prediction task that is also conditioned on the future interactions to acknowledge the recent success of bi-directional architecture in knowledge tracing [@lee2019creating].

\[ex:kt\] Knowledge tracing is typically instantiated as response correctness prediction, where the interaction $I_i = \{e_i, r_i\}$ consists of a consumed exercise $e_i$ and a corresponding response correctness $r_i$ [@piech2015deep; @zhang_2017; @liu_2019; @pandey2019self; @lee2019creating]. In this setup, only the response correctness $r_T$ of the last interaction $I_T$ is predicted and features related to predicted student response $r_T$ are masked. Following our definition of knowledge tracing, the task can be extended further to predict diverse interactive features such as:

-   `offer_selection`: Whether a student accepts to study the offered learning items.

-   `start_time`: The time when a student starts to solve an exercise.

-   `inactive_time`: The duration a student being inactive in a learning session.

-   `platform`: Whether a student responds to each exercise on web browser or mobile app.

-   `payment`: Whether a student purchases charged items.

-   `event` : Whether a student participates in application events.

-   `longest_answer`: Whether a student selected the answer choice with longest description.

-   `response_correctness`: Whether a student responds correctly to a given exercise.

-   `timeliness`: Whether a student responds to each exercise under the time limit recommended by domain experts.

-   `course_dropout` : Whether a student drops out of the entire class.

-   `elapsed_time`: The total time a student took to solve a given exercise.

-   `lecture_complete`: Whether a student completes studying a video lecture offered to him/her.

-   `review_correctness` : Whether a student responds correctly to a previously solved exercise.

In the aforementioned example, features like `response_correctness` and `timeliness` directly evaluates the educational values of a student’s interaction, while it is somewhat debatable whether `platform` and `longest_answer` are also capable of addressing such quality. Accordingly, we define the concept of assessment and assessment modeling as the following.

\[def:am\] An *assessment* $a_{t}^{k}$ of the $t$’th interaction $I_t$ is a feature of $I_t$ which can act as a criterion for pedagogical evaluation. The collection $A_{t} = \{a_{t}^{1}, \dots, a_{t}^{m}\}$ of assessments is a subset of the available features $\{f_{i}^{1}, \dots, f_{i}^{n}\}$ of $I_t$. An *assessment modeling* is the prediction of assessment $a_{t}^{k}$ for some $k$ from the interactions $[I_1, \dots, I_{t-1}, I_t, I_{t+1}, \dots, I_T]$. That is, the prediction of $$\label{eq:am}
p(a_{t}^{k} | I_1, \dots, I_{t-1}, I_t \setminus \Gamma(a_{t}^{k}), I_{t+1}, \dots, I_T).$$

\[ex:am\] Among the interactive features listed in Example \[ex:kt\], we consider the `response_correctness`, `timeliness`, `course_dropout`, `elapsed_time`, `lecture_complete` and `review_correctness` as assessments. For example, `response_correctness` is the primary assessment as whether a student responded correctly for each exercise provides strong evidence for the mastery of concepts required to solve the exercise properly. Also, `timeliness` also serves as assessment since a student responding to each exercise within limited time is expected to have high proficiency in skills and knowledge necessary to solve the exercise. Figure \[fig:am\_def\] depicts the relationship between assessments and general knowledge tracing features.

Assessment Modeling as Pre-training Tasks
-----------------------------------------

![Not all interactive features are collected equally often. For example, `payment`, `event`, `course_dropout` and `review_correctness` are interactive features obtained more sporadically than other features. Among the sporadic interactive features, `course_dropout` and `review_correctness` are assessments.[]{data-label="fig:am_freq"}](assessment_freq.pdf){width="45.00000%"}

![Assessment modeling is highly appropriate pre-training method for the label-scarce educational problems. First, most of assessments are available in large amount as they are automatically collected from student-system interactions (Interactive). Also, since assessments share educational aspects, narrowing down the scope of prediction targets to assessments reduces noise irrelevant to the problems (Educational).[]{data-label="fig:am_alt"}](assessment_alt.pdf){width="50.00000%"}

![Possible pre-train/fine-tune scenarios. We may pre-train a model to predict `start_time`, `response_correctness`, `timeliness` and `review_correctness` for estimating `exam_score` (red). Likewise, a model pre-trained to predict `start_time` and `response_correctness` can be trained to predict `review_correctness` (green). However, pre-training a non-educational interactive feature like `start_time` is not effective for the label-scarce educational problems (dotted lines).[]{data-label="fig:am_pre_fine"}](assessment_pre_fine.pdf){width="50.00000%"}

In this subsection, we provide examples of important yet scarce educational labels and argue that assessment modeling enables effective prediction of such labels.

\[ex:non-interactive\] In many applications, an IES is often integrated as part of larger learning process. Accordingly, the ultimate evaluation of the learning process is mostly done independently from the IES. For example, academic abilities of students are measured by course grades or standardized exams, and the ability to perform a complicated job or task is assured by professional certificates. Such labels hold great importance due to the pedagogical and social needs for consistent evaluation of students’ ability. However, obtaining the labels are often challenging due to their scarcity compared to features automatically collected from student-system interactions. We give the following examples (see Figure \[fig:am\_def\]).

-   `exam_score` : A student’s standardized exam score.

-   `grade` : A student’s final grade in enrolled course.

-   `certification` : Professional certification obtained by completion of educational programs or examinations.

\[ex:sporadic\] All assessments are collected by IESs automatically, but some are few in number as the events occur rarely in practice, which introduces data scarcity problem. For example, it is natural for students to invest more time in learning new concepts than reviewing previously studied materials. `course_dropout` and `review_correctness` are examples of sporadic assessments (Figure \[fig:am\_freq\]).

To overcome the lack of aforementioned labels, we consider the pre-train/fine-tune paradigm that leverages the patterns of data available in large amount. In this paradigm, a model is first trained in an auxiliary task relevant to the tasks of interest with large-scale data. With pre-trained parameters as initial values, the model is slightly modified and finally trained in main tasks. This approach has been successful in AI fields like NLP, computer vision and speech recognition [@devlin2018bert; @studer2019comprehensive; @schneider2019wav2vec]. Accordingly, existing methods in AIEd pre-train the contents of learning materials like the word sequence of a passage, but such methods do not capture student behaviors and only utilize a single feature from the data.

Instead, one may pre-train different features automatically collected by IESs (see Figure \[fig:am\_alt\]). However, training every available features is computationally intractable and may introduce irrelevant noise. To this end, assessment modeling narrows down the prediction targets to assessments, the interactive features that are also educational. Since multiple assessments are available, a wide variety of pre-train/fine-tune pairs can be explored for effective assessment modeling (see Figure \[fig:am\_pre\_fine\]). This raises the open-ended questions of *which* assessments to pre-train for label-scarce educational problems and *how* to pre-train multiple assessments. In Section \[sec:experiments\], we explore these questions for exam score (a non-interactive educational feature) and review correctness (a sporadic assessment) respectively. Experimental results support that assessment modeling are effective pre-training tasks for the label-scarce educational problems.

Assessment Modeling with Deep Bidirectional Transformer Encoder
---------------------------------------------------------------

![image](figure-Bert.pdf){width="70.00000%"}

While there are several possible options for the architecture of the assessment model, we adopt the deep bidirectional Transformer encoder proposed in [@devlin-etal-2019-bert] for the following reasons. First, [@pandey2019self] showed that the self-attention mechanism in Transformer [@vaswani_2017] is effective for knowledge tracing task. The Transformer-based knowledge tracing model proposed in [@pandey2019self] achieved state-of-the-art performance on several datasets. Second, the deep bidirectional Transformer encoder model and pre-train/fine-tune method proposed in [@devlin-etal-2019-bert] achieved state-of-the-art results on several NLP tasks. While [@devlin-etal-2019-bert] conducted experimental studies on NLP domain, the method is also applicable to other domains with slight modifications.

Figure \[fig:am\_model\] depicts our proposed pre-train/fine-tune approach. In the pre-training phase, we train a deep bidirectional Transformer encoder based assessment model to predict the assessments conditioned on past and future interactions. After the pre-training phase, we replace the last layer of the assessment model with a layer appropriate for each downstream task and fine-tune whole parameters in the model to predict labels in the downstream tasks. We provide detailed descriptions of our proposed assessment model in the following subsections.

### Input Representation

The first layer in the assessment model maps each interaction to an embedding vector. Firstly, we embed the following attributes:

-   `exercise_id`: We assign a latent vector unique to each exercise.

-   `exercise_category`: Each exercise has its own category tag that represents the type of the exercise. We assign a latent vector to each tag.

-   `position`: The relative position $t$ of the interaction $I_t$ in the input sequence. We use the learned positional embedding from [@gehring_2017] instead of the sinusoidal positional encoding that used in [@vaswani_2017].

As shown in Example \[ex:kt\], IES collects diverse interactive features that can potentially be used for assessment modeling. However, using all possible interactive features for assessment modeling is computationally intractable and does not guarantee the best results when the assessment model is fine-tuned on downstream tasks. For experimental studies, we narrow down the scope of interactive features to the ones available from an exercise-response pair, the simplest but widely considered interaction in knowledge tracing task. In particular, we embed the following interactive features:

-   `start_time`: The month, day and hour of the absolute time when a student started to solve each exercise is recorded. We assign a latent vector to every possible combination of month, day and hour.

-   `response_correctness`: The value is 1 if a student response is correct and 0 otherwise. We assign a latent vector corresponding to each possible value 0 and 1.

-   `timeliness`: The value is 1 if a student responds within a specified time limit and 0 otherwise. We assign a latent vector corresponding to each possible value 0 and 1.

-   `longest_answer`: The value is 1 if a student selects an answer choice with the longest description. We assign a latent vector corresponding to each possible value 0 and 1.

Among the interactive features above, only `response_correctness` and `timeliness` serve as assessments. However, we study various combinations of the above interactive features for pre-training in Section \[sec:experiments\].

Let $e_t$ be the sum of the embedding vectors of `exercise_id`, `exercise_category` and `position`. Likewise, let $c_t$ and $t_t$ be the embedding vectors of `response_correctness` and `timeliness` respectively. Then, the representation of interaction $I_t$ is $e_t + c_t + t_t$.

### Masking

Inspired by the masked language model proposed in [@devlin-etal-2019-bert], we use the following method to mask the assessments in a student interaction sequence. Firstly, we mask $C$ fraction of the interaction in a sequence chosen at random. If the $t$-th interaction is chosen, we replace the corresponding input embedding with (1) $e_t + \texttt{mask}$ for $M$ fraction of the time and (2) $e_t + c_{{\mathrm{rand}}} + t_{{\mathrm{rand}}}$ for $(1-M)$ fraction of the time. Here $\texttt{mask}$ is a learned vector that represents masking, and $c_{{\mathrm{rand}}}$ and $t_{{\mathrm{rand}}}$ are embedding vectors for assessments uniformly chosen at random in the sequence. We determine $C$ and $M$ through ablation studies in Section \[sec:experiments\].

### Model Architecture

After the interactions are embedded and masked accordingly, they enter a series of Transformer encoder blocks, each consisting of a multi-head self-attention layer followed by position-wise feed-forward networks. All the inputs of layers have dimension $d_{\mathrm{model}}$. The first encoder block takes the sequence of interactions $I_1, \dots, I_T$ embedded in latent space and returns a series of vectors of the same length and dimension. For all $2 \leq i \leq N$, the $i$’th block takes the output of the $i-1$’th block as input and returns the series of vectors accordingly. We describe the architecture of each block as the following.

The multi-head self-attention layer takes a series of vectors, $X_1, \dots, X_T$. Each vector is projected to latent space by projection matrices $W^{Q}, W^{K}\in \mathbb{R}^{d_{\mathrm{model}} \times d_{K}}$ and $W^{V}\in \mathbb{R}^{d_{\mathrm{model}} \times d_{V}}$: $$\begin{aligned}
\label{eqn:attn-proj}
\begin{split}
Q = [q_1, \dots, q_T]^{T} & = X W^Q \\
K = [k_1, \dots, k_T]^{T} & = X W^K \\
V = [v_1, \dots, v_T]^{T} & = X W^V
\end{split}\end{aligned}$$ Here $X = [X_1, \dots, X_T]^{T}$ and each $q_i$, $k_i$ and $v_i$ are the query, key and value of $X_i$ respectively. The output of the self-attention is then obtained as weighted sum of values with coefficients determined by the dot products between queries and keys: $$\begin{aligned}
\label{eqn:attn-softmax}
\mathrm{Attention}(X) = \mathrm{Softmax}\left(\frac{QK^{T}}{\sqrt{d_k}}\right) V\end{aligned}$$ Models with self-attention layers often use multiple heads to jointly attend information from different representative subspaces. Following this, we apply attention $h$ times to the same query-key-value entries with different projection matrices for output: $$\begin{aligned}
\label{eqn:multihead}
\mathrm{Multihead}(X) = \mathrm{concat}(\mathrm{head}_{1},  \dots ,\mathrm{head}_{T}) W^O\end{aligned}$$ Here, each $\mathrm{head}_{i}$ is equal to the output of self-attention in Equation \[eqn:attn-softmax\] with corresponding projection matrices $W_i^Q$, $W_i^K$ and $W_i^V$ in Equation \[eqn:attn-proj\]. We use the linear map $W^O$ to aggregate each attention result.

After we compute the resulting value in Equation \[eqn:multihead\], we apply point-wise feed-forward networks to add non-linearity to the model. Also, we apply the skip connection [@he2016deep] and layer normalization [@ba2016layer] to the output of the feed-forward networks.

Assume that the last encoder block returns the series $H = [H_1,$ $ \dots, H_T]^T$ of vectors. For pre-training, the prediction to $i$’th interaction is made by applying a linear layer with softmax activation function to $H_i$. The final output is the estimated probability distribution of four possible combinations of assessment $A_t$: $(\texttt{response\_correctness}_t, \texttt{timeliness}_t) = (0, 0)$, $(0, 1)$, $(1, 0)$ or $(1, 1)$. The overall loss is defined as $$\mathcal{L} = \sum_{t=1}^{T} m_{t}\mathcal{L}_{t}$$ where $\mathcal{L}_{t}$ is the cross-entropy between estimated distribution of $A_t$ and actual one-hot distribution of $A_t$. The value $m_{t}$ is a flag that represents whether the $t$-th question is masked ($m_{t}=1$) or not ($m_{t}=0$). In other words, the total loss is the sum of cross-entropy losses for masked questions.

The input embedding layer and encoder blocks are shared over pre-training and fine-tuning. For fine-tuning, we replace the linear layers applied to each $H_i$ in pre-training by a single linear layer that combines all entries of $H$ to fit the output to respective downstream tasks.

Experiments {#sec:experiments}
===========

Label-Scarce Educational Problems
---------------------------------

We utilize assessment modeling to exam score (a non-interactive educational feature) and review correctness (a sporadic assessment) prediction. The tasks are described in detail as the following.

### Exam Score Prediction

Exam score prediction (ES) is the estimation of a student’s scores in standardized exams, such as TOEIC and SAT, based on the student’s interaction history in educational system. ES is one of the most important tasks of AIEd, as it is crucial for both students and educational system to assess the performance in a standardized measure. Because a substantial amount of human effort is required to develop or take the tests, the number of data points available for exam score is considerably less than that of student interactions automatically collected by educational systems. By developing a reliable ES model, a student’s score is estimated in a universally accepted score by interactive educational system with considerably less effort. ES differs from student response prediction (e.g. the prediction of assessment $\texttt{response\_correctness}_t$) because standardized tests are taken in a controlled environment with specific methods independent of interactive educational system.

### Review Correctness Prediction

Assume that a student incorrectly responds to an exercise $e_{\mathrm{rev}}$ and receives corresponding feedback. The goal of review correctness prediction (RC) is to predict whether a student will be able to respond to the exercise $e_{\mathrm{rev}}$ correctly or not if she encounters the exercise again. The significance of this AIEd task is that it can assess the educational effect of an exercise to a particular student in a specific situation. In particular, the correctness probability estimated by this task represents the student’s expected marginal gain in knowledge as she goes through future learning process. For example, if the correctness probability is high, it is likely that the student will obtain relevant knowledge in future even if his initial response was incorrect.

         Part            5    6    7         
  ------------------ -- ---- ---- ---- -- -- --
   Time limit (sec)      25   50   55        

  : Time Limits[]{data-label="tab:timelimit"}

Dataset
-------

We use the public *EdNet* dataset obtained from *Santa*, a mobile AI tutoring service for preparing TOEIC Listening and Reading Test [@choi2019ednet]. The test consists of two timed sections named Listening Comprehension (LC) and Reading Comprehension (RC) with a total of 100 exercises and 4 or 3 parts respectively. The final test score ranges from 0 to 990 in a gap of 5. Once a user solves each exercise, *Santa* provides educational feedback to his response such as explanations or commentaries on exercises. *EdNet* is the collection of user interactions of multiple-choice exercises collected over the last four years. The main features of the user-exercise interaction data consists of six columns: user id, exercise id, user response, exercise part, received time and time taken. We describe each column as the following. Firstly, the user (resp. exercise) ID identifies each unique user (resp. exercise). The user response is recorded as 1 if the user response is correct and 0 otherwise. The part the exercise belongs to is recorded as exercise part. Finally, the absolute time when the user received the exercise and the time taken by user to respond are recorded respectively. In the dataset, 627,347 users solved more than one problem. The size of the exercise set is 16,175. The total row count of the dataset is 72,907,005.

### Dataset for Pre-training

For pre-training, we first reconstruct the interaction timeline of each user by gathering the responses of a specific user in the increasing order of absolute time received. For each interaction $I_t$, the assessment value `response_`$\texttt{correctness}_t$ is the recorded user response and $\texttt{timeliness}_t$ is calculated as 1 if the user responded under the time limits recommended by TOEIC experts (Table \[tab:timelimit\]). We exclude the interactions of users involved in any of the downstream tasks for pre-training to avoid learning the downstream data beforehand. After processing, the data consists of 251,989 users with a total of 49,944,086 interactions.

### Dataset for Label-Scarce Educational Problems

For the exam score prediction, we aggregate the real TOEIC scores reported by users of *Santa*. The reports are scarce in number because a user have to register, take the exam and report the score at his own expense. *Santa* offered corresponding reward to the users who have reported their score. A total of 2,594 score reports have been obtained through 6 months, which is comparably less than the number of 72M exercise responses. For our experiment, the data is divided into training set (1,302 users, 1815 labels), validation set (244 users, 260 labels), and test set (466 users, 519 labels).

For the review correctness prediction, we look over each student’s timeline and find exercises that have been solved at least twice. That is, if an exercise $e$ appears more than once in a student interaction sequence $I_1 = (e_1, r_1), I_2 = (e_2, r_2), \dots, I_T = (e_T, r_T)$, we find the first two interactions $I_i$ and $I_j$ ($i < j$) with the exercise $e$. The sequence of interactions $I_1, I_2, \cdots, I_i$ until she encounters $e$ for the first time is taken as the input. The assessment $\texttt{response\_correctness}_j$ for the next encounter $I_j$ is taken as the label. The total of 1,084,413 labeled sequence are generated after pre-processing. For our experiment, the data is divided into training set (28,016 users, 747,222 labels), validation set (4,003 users, 112,715 labels), and test set (8,004 users, 224,476 labels).

Setup
-----

### Models

Our model consists of two encoder blocks ($N = 2$) with the latent space dimension of 256 ($d_{model} = 256$). The model takes 100 interactions as input. For comparison, the following pre-training methods are also applied to respective encoder networks with the same architecture. Since the existing pre-training methods embed the content of each exercise, we replace the embedding of $\texttt{exercise\_id}$ in our model with respective exercise embedding for fine-tuning.

-   *Word2Vec* [@mikolov2013distributedWORD2VEC] is one of the most well-known word embedding model which is used in [@liu_2019] for exercise embedding. The model is pre-trained on the Google News dataset containing around 100 billion words. The embedding of each exercise is the average of embedding vectors of all words appearing in the exercise description. Embedded vectors are not fine-tuned.

-   *BERT* [@devlin2018bert] is a state-of-the-art pre-training method featuring the Transformer architecture with masked language model. As above, we embed each exercise by averaging out the representation vectors of words in the description without fine-tuning.

-   For *QuesNet* [@yin2019quesnet], the image and meta-data embeddings are not used because the exercises used in our experiment consist of text only. The model follows the architecture (Bi-directional LSTM followed by self-attention) and pre-training tasks (Holed Language Model and Domain-Oriented Objective) suggested in the original paper.

### Metrics

We use the following evaluation metrics to evaluate performances of models on each downstream task. For exam score prediction, we compute the Mean Absolute Error (MAE), the average of differences between predicted exam scores and real values. For review correctness prediction, we use accuracy (ACC).

### Training Details

The strategy we use for finding the optimal model parameters is the following. For pre-training, the model weights are first initialized with Xavier uniform distribution and trained with $n_{pre} = 100$ epochs. After each $i$’th epoch, the model parameters are stored as $\mathcal{P}_i$. Then we fine-tune each $\mathcal{P}_i$ for downstream task with a specified number $n_{down} = 100$ of epochs. Likewise, the model after $j$ epochs of fine-tuning is stored as $\mathcal{D}_{i, j}$. Among all downstream task models $\mathcal{D}_{i, j}$ with $1 \leq i \leq n_{pre}$ and $1 \leq j \leq n_{down}$, the model with the best result on the validation set is chosen and evaluated with the test set. We use the Adam optimizer with hyperparameters $lr=0.001, \beta_1=0.9, \beta_2=0.999, epsilon=1-e8$. The batch size is 256 for pre-training and 128 for fine-tuning.

The labels available for exam score prediction are scarce. To alleviate this, we apply the following data augmentation to fine-tune our model. Given the original interaction sequence with score labeled, we select each element with 50% probability repeatedly to generate augmented sequences with the same score label.

                            ES          RC      
  --------------------- ----------- ----------- --
    Without pre-train      64.37       0.648    
        Word2Vec           77.93       0.649    
          BERT             70.19       0.646    
         QuesNet           64.00       0.642    
   Assessment Modeling   **49.84**   **0.656**  

  : Experimental results[]{data-label="tab:expres"}

Experimental Results
--------------------

### Model Performance

The experimental results of 5 different pre-trained models on the two aforementioned downstream tasks are shown in Table \[tab:expres\]. In all downstream tasks, our model with assessment modeling outperforms the other baseline models. Compared to the model without pre-training, MAE for exam score prediction is reduced to 14.53 and ACC for review correctness is improved by 0.9 points. Also, our model performs better than other pre-trained models. This demonstrates that the assessment modeling is more suitable for knowledge tracing tasks than content-based pre-training approaches. Interestingly, some of the pre-trained models even show worse performance than the model without pre-training. This shows the importance of choosing pre-training tasks relevant to the downstream tasks.

        Labels Pre-trained           ES          RC
  ------------------------------ ----------- -----------
          `correctness`             54.28     **0.657**
           `timeliness`             61.98       0.648
   `correctness` + `timeliness`   **49.84**     0.656

  : Assessment Modeling Tasks[]{data-label="tab:pretask"}

   Masking Positions      ES          RC
  ------------------- ----------- -----------
         Front           50.26       0.656
         Back            51.91     **0.657**
        Random         **49.84**     0.656

  : Masking Positions[]{data-label="tab:maskpos"}

### Effect of Assessment Modeling Labels

Recall that our model is pre-trained to predict whole assessment $$A_{t} = (\texttt{response\_correctness}_{t}, \texttt{timeliness}_{t}).$$ We demonstrate the importance of pre-training multiple assessments by comparing our model with two variations pre-trained to predict only one of `response_`$\texttt{correctness}_{t}$ and $\texttt{timeliness}_{t}$. The results are shown in Table \[tab:pretask\]. For exam score prediction, the results show that it is the best to pre-train the model to predict whole assessment $A_{t}$. This is because exam score depends on not only the student’s answer but also the time she spent on each exercise. However, for review correctness prediction, the model only predicts correctness of user’s response and does not care about timeliness. This explains why the model pre-trained to predict only correctness of user’s response performs slightly better than the model pre-trained to predict both components of assessments.

### Effect of Masking Positions

As we mentioned in Section 3, we used BERT-like masking method for pre-training to represent bi-directional information. We compared this with other possible masking approaches - masking the first 60% of exercises (Front) or the last 60% of exercises (Back) - and the results are shown in Table \[tab:maskpos\]. For exam score prediction, our approach with random mask is more effective than other masking methods with fixed positions. In case of review correctness prediction, there is no big difference among three models, although the model with last 60% of interactions masked performs slightly better than others.

          $C$   $M$   $N$   $d_{\mathrm{model}}$      ES          RC
  ------ ----- ----- ----- ---------------------- ----------- -----------
   Base   0.6   1.0    2            256            **49.84**     0.656
          0.2                                        53.87       0.655
          0.4                                        51.79       0.655
          0.8                                        52.01     **0.657**
                0.0                                  52.25       0.655
                0.5                                  52.51       0.655
                       1            128              50.79     **0.657**
                       4            512              51.85       0.653

  : Ablation Study. Rows in (A) represent the proportion $C$ of masked interactions. Rows in (B) represent the masking rate $M$. Rows in (C) denote the parameters for model architecture (number of encoder blocks $N$ and dimension $d_{model}$ of latent space)[]{data-label="tab:ablation"}

### Ablation Study {#sec:ablation}

We conduct several ablation experiments to understand how each property of our model affects the model’s performances. First, we observe the effect of the hyper-parameters for masking: the proportion of masked interaction $C$ and the masking rate $M$. Then, we varied the size of the model, the number of layers $N$, and the dimension of model $d_{\mathrm{model}}$. As a result, the model with $C=0.6$, $M=1.0$, $N=2$, and $d_{\mathrm{model}} = 256$ performs the best.

Conclusion
==========

We introduced assessment modeling, fundamental pre-training tasks for IESs. Our experiments show the effectiveness of assessment modeling as pre-training tasks for the label-scarce educational problems like exam score and review correctness prediction. Future works will include assessments beyond correctness and timeliness of the responses and investigate further label-scarce educational problems.
