A review of recent advances in learner and skill modeling in intelligent learning environments
---
paper_title: Mainstreaming AIED into Education

paper_content:

What will Education look like in 2010, and what will be its problems? Will AIED have anything substantial to offer, and what can we do to maximise its contributions? We argue that Education is, for understandable reasons, the discipline of classroom teaching and learning, with some attention paid to learning by individuals but very little indeed to teaching that is effective in a one-to-one situation. Further, it is not strong in formulating precise theories of learning and teaching. It is therefore not well-equipped to support the inevitable increase in learning activities that will be undertaken at computers. We argue that AIED has made great progress in adopting broader and more sophisticated conceptions of learning, and should thus at last be able to communicate well with Education. It can offer ways to develop and test precise theories, and also important concepts relevant to individualised learning that have largely been overlooked by Education, such as that of learner modelling by teachers. By 2010 AIED has the potential to be making a much broader contribution to Education. If this optimistic prediction can be achieved, and the educational mainstreaming of AIED does occur, both AIED and Education will benefit. (http://aied.inf.ed.ac.uk/members00/archive/vol_11/cumming/full.html)
---
paper_title: Advances in Intelligent Tutoring Systems

paper_content:

The idea for this book on Intelligent Tutoring Systems (ITS) was sparked by the success of the ITS08 international conference. The number of presentations and their quality bore witness to the vitality and maturity of the field, and the enthusiasm of the participants held out a promise of sustainability and innovative research. Long life to ITS research! The book is divided into five parts. The introductory chapters to these parts, which summarize foundations, developments, strengths and weaknesses in each of the areas covered, are addressed to all readers. For those who want more in-depth knowledge, we give the floor to researchers who present their work, their results, and their view of what the future holds. It is our hope that all readers will find the book informative and thought-provoking.
---
paper_title: Adaptive and Intelligent Web-based Educational Systems

paper_content:

Currently, Web-based educational systems form one of the fastest growing areas in educational technology research and development. Benefits of Web-based education are independence of teaching and learning with respect to time and space. Courseware installed and maintained in one place may be used by a huge number of users all over the world. A challenging research goal is the development of adaptive and intelligent Web-based educational systems (W-AIES) that offer some amount of adaptivity and intelligence.
---
paper_title: Addressing the testing challenge with a web-based e-assessment system that tutors as it assesses

paper_content:

Secondary teachers across the country are being asked to use formative assessment data to inform their classroom instruction. At the same time, critics of No Child Left Behind are calling the bill "No Child Left Untestedo emphasizing the negative side of assessment, in that every hour spent assessing students is an hour lost from instruction. Or does it have to be? What if we better integrated assessment into the classroom, and we allowed students to learn during the test? Maybe we could even provide tutoring on the steps of solving problems. Our hypothesis is that we can achieve more accurate assessment by not only using data on whether students get test items right or wrong, but by also using data on the effort required for students to learn how to solve a test item. We provide evidence for this hypothesis using data collected with our E-ASSISTment system by more than 600 students over the course of the 2004-2005 school year. We also show that we can track student knowledge over time using modern longitudinal data analysis techniques. In a separate paper [9], we report on the ASSISTment system's architecture and scalability, while this paper is focused on how we can reliably assess student learning.
---
paper_title: Fifteen years of constraint-based tutors: what we have achieved and where we are going

paper_content:

Fifteen years ago, research started on SQL-Tutor, the first constraint-based tutor. The initial efforts were focused on evaluating Constraint-Based Modeling (CBM), its effectiveness and applicability to various instructional domains. Since then, we extended CBM in a number of ways, and developed many constraint-based tutors. Our tutors teach both well- and ill-defined domains and tasks, and deal with domain- and meta-level skills. We have supported mainly individual learning, but also the acquisition of collaborative skills. Authoring support for constraint-based tutors is now available, as well as mature, well-tested deployment environments. Our current research focuses on building affect-sensitive and motivational tutors. Over the period of fifteen years, CBM has progressed from a theoretical idea to a mature, reliable and effective methodology for developing effective tutors.
---
paper_title: Knowledge tracing: Modeling the acquisition of procedural knowledge

paper_content:

This paper describes an effort to model students' changing knowledge state during skill acquisition. Students in this research are learning to write short programs with the ACT Programming Tutor (APT). APT is constructed around a production rule cognitive model of programming knowledge, called theideal student model. This model allows the tutor to solve exercises along with the student and provide assistance as necessary. As the student works, the tutor also maintains an estimate of the probability that the student has learned each of the rules in the ideal model, in a process calledknowledge tracing. The tutor presents an individualized sequence of exercises to the student based on these probability estimates until the student has ‘mastered’ each rule. The programming tutor, cognitive model and learning and performance assumptions are described. A series of studies is reviewed that examine the empirical validity of knowledge tracing and has led to modifications in the process. Currently the model is quite successful in predicting test performance. Further modifications in the modeling process are discussed that may improve performance levels.
---
paper_title: Extending the scaffolding metaphor

paper_content:

A brief overview overview is provided of how the scaffolding metaphor has been defined in educational contexts. This includes a discussion of what theories guide decision-making regarding what to scaffold as well as considerations as to whom or what does the scaffolding (human teachers, peers and tutors or computer tutors or support tools designed for learning environments). The scaffolding construct intersects instruction and assessment in that instructors assess learners to determine what type or level of scaffold is sufficient to help learners reach their potential. Such assessments are dynamic and ongoing and can occur through dialogue and social interactions with or without the use of technology. Hence scaffolds are provided when and where necessary but they are also removed when evidence of learning exists. This article describes how the contributors to this issue have extended the scaffolding metaphor to open-ended technology based environments. Empirical studies are reviewed with regard to how they extend the scaffolding metaphor in terms of the theories that guide the design of scaffolds, the metrics designed to assess how scaffolding affects learning and in terms of teaching scaffolding techniques to others.
---
paper_title: Optimising ITS Behaviour with Bayesian Networks and Decision Theory

paper_content:

We propose and demonstrate a methodology for building tractable normative intelligent tutoring systems (ITSs). A normative ITS uses a Bayesian network for long-term student modelling and decision theory to select the next tutorial action. Because normative theories are a general framework for rational behaviour, they can be used to both define and apply learning theories in a rational, and therefore optimal, way. This contrasts to the more traditional approach of using an ad-hoc scheme to implement the learning theory. A key step of the methodology is the induction and the continual adaptation of the Bayesian network student model from student performance data, a step that is distinct from other recent Bayesian net approaches in which the network structure and probabilities are either chosen beforehand by an expert, or by efficiency considerations. The methodology is demonstrated by a description and evaluation of CAPIT, a normative constraint-based tutor for English capitalisation and punctuation. Our evaluation results show that a class using the full normative version of CAPIT learned the domain rules at a faster rate than the class that used a non-normative version of the same system.
---
paper_title: Fifteen years of constraint-based tutors: what we have achieved and where we are going

paper_content:

Fifteen years ago, research started on SQL-Tutor, the first constraint-based tutor. The initial efforts were focused on evaluating Constraint-Based Modeling (CBM), its effectiveness and applicability to various instructional domains. Since then, we extended CBM in a number of ways, and developed many constraint-based tutors. Our tutors teach both well- and ill-defined domains and tasks, and deal with domain- and meta-level skills. We have supported mainly individual learning, but also the acquisition of collaborative skills. Authoring support for constraint-based tutors is now available, as well as mature, well-tested deployment environments. Our current research focuses on building affect-sensitive and motivational tutors. Over the period of fifteen years, CBM has progressed from a theoretical idea to a mature, reliable and effective methodology for developing effective tutors.
---
paper_title: Knowledge tracing: Modeling the acquisition of procedural knowledge

paper_content:

This paper describes an effort to model students' changing knowledge state during skill acquisition. Students in this research are learning to write short programs with the ACT Programming Tutor (APT). APT is constructed around a production rule cognitive model of programming knowledge, called theideal student model. This model allows the tutor to solve exercises along with the student and provide assistance as necessary. As the student works, the tutor also maintains an estimate of the probability that the student has learned each of the rules in the ideal model, in a process calledknowledge tracing. The tutor presents an individualized sequence of exercises to the student based on these probability estimates until the student has ‘mastered’ each rule. The programming tutor, cognitive model and learning and performance assumptions are described. A series of studies is reviewed that examine the empirical validity of knowledge tracing and has led to modifications in the process. Currently the model is quite successful in predicting test performance. Further modifications in the modeling process are discussed that may improve performance levels.
---
paper_title: Adaptive Hypermedia

paper_content:

Adaptive hypermedia is a relatively new direction of research on the crossroads of hypermedia and user modeling. Adaptive hypermedia systems build a model of the goals, preferences and knowledge of each individual user, and use this model throughout the interaction with the user, in order to adapt to the needs of that user. The goal of this paper is to present the state of the art in adaptive hypermedia at the eve of the year 2000, and to highlight some prospects for the future. This paper attempts to serve both the newcomers and the experts in the area of adaptive hypermedia by building on an earlier comprehensive review (Brusilovsky, 1996; Brusilovsky, 1998).
---
paper_title: Learned student models with item to item knowledge structures

paper_content:

Probabilistic and learned approaches to student modeling are attractive because of the uncertainty surrounding the student skills assessment and because of the need to automatize the process. Item to item structures readily lend themselves to probabilistic and fully learned models because they are solely composed of observable nodes, like answers to test questions. Their structure is also well grounded in the cognitive theory of knowledge spaces. We study the effectiveness of two Bayesian frameworks to learn item to item structures and to use the induced structures to predict item outcome from a subset of evidence. One approach, Partial Order Knowledge Structures (POKS), relies on a naive Bayes framework whereas the other is based on the Bayesian network (BN) learning and inference framework. Both approaches are assessed over their predictive ability and their computational efficiency in different experimental simulations. The results from simulations over three data sets show that they both can effectively perform accurate predictions, but POKS generally displays higher predictive power than the BN. Moreover, the simplicity of POKS translates to a time efficiency between one to three orders of magnitude greater than the BN runs. We further explore the use of the item to item approach for handling concepts mastery assessment. The approach investigated consist in augmenting an initial set of observations, based on inferences with the item to item structure, and feed the augmented set to a BN containing a number of concepts. The results show that augmented set can effectively improve predictive power of a BN for item outcome, but that improvement does not transfer to the concept assessment in this particular experiment. We discuss different explanations for the results and outline future research avenues.
---
paper_title: Adaptive and Intelligent Web-based Educational Systems

paper_content:

Currently, Web-based educational systems form one of the fastest growing areas in educational technology research and development. Benefits of Web-based education are independence of teaching and learning with respect to time and space. Courseware installed and maintained in one place may be used by a huge number of users all over the world. A challenging research goal is the development of adaptive and intelligent Web-based educational systems (W-AIES) that offer some amount of adaptivity and intelligence.
---
paper_title: Mind in Society: Development of Higher Psychological Processes

paper_content:

Introduction Michael Cole and Sylvia Scribner Biographical Note on L. S. Vygotsky Basic Theory and Data 1. Tool and Symbol in Child Development 2. The Development of Perception and Attention 3. Mastery of Memory and Thinking 4. Internalization of Higher Psychological Functions 5. Problems of Method Educational Implications 6. Interaction between Learning and Development 7. The Role of Play in Development 8. The Prehistory of Written Language Afterword Vera John-Steiner and Ellen Souberman Notes Vygotsky's Works Index
---
paper_title: Dynamic Courseware Generation: at the Cross Point of CAL, ITS and Authoring

paper_content:

This paper presents a further development of our architecture for dynamic courseware generation which allows explicit representation of teaching expertise. The instructional course is generated automatically for a given teaching goal and can be dynamically changed according to specified teaching rules to suit better the student's individual progress and preferences. The architecture of the system is based on explicit representation of the structure of the domain concepts and relationships (what has to be taught) and the instructional tasks and methods (how to teach). The separate representation of the actual teaching materials allows better flexibility and individualisation, as well as easier updating and re-use of ready CAL materials. In this way our approach provides an alternative to traditional CAL-authoring. An implementation of this approach in a simple engineering domain has been done and an attempt for evaluation of the advantages for authoring is presented.
---
paper_title: Competence-based knowledge structures for personalised learning

paper_content:

The ELeGI project focuses on integrating technology-enhanced learning methodologies into a pedagogy-driven and service-oriented architecture based on Grid technology. It aims at a system that is capable of realising personalised, adaptive, and experiential learning. This requires to have available a framework that, on the one hand, allows for representing existing domain knowledge, and, on the other hand, provides a representation of the learner's current state of knowledge. It is shown that a competence-based extension of Knowledge Space Theory provides a representation of the conceptual organization of a domain that not only allows for an adaptive assessment of an individual's knowledge, but also for generating personalised learning paths. The discussion of this framework emphasizes its application within an open distributed service model, and in the context of Virtual Scientific Experiments.
---
paper_title: Using planning techniques in intelligent tutoring systems

paper_content:

Abstract This paper proposes an architecture for building better Computer-Assisted Instruction (CAI) programs by applying and extending Artificial Intelligence (AI) techniques which were developed for planning and controlling the actions of robots. A detailed example shows how programs built according to this architecture are able to plan global teaching strategies using local information. Since the student's behavior can never be accurately predicted, the pre-planned teaching strategies may be foiled by sudden surprises and obstacles. In such cases, the planning component of the program is dynamically reinvoked to revise the unsuccessful strategy, often by recognizing student misconceptions and planning a means to correct them. This plan-based teaching strategy scheme makes use of global course knowledge in a flexible way that avoids the rigidity of earlier CAI systems. It also allows larger courses to be built than has been possible in most AI-based “intelligent tutoring systems” (ITSs), which seldom address the problem of global teaching strategies.
---
paper_title: Knowledge tracing: Modeling the acquisition of procedural knowledge

paper_content:

This paper describes an effort to model students' changing knowledge state during skill acquisition. Students in this research are learning to write short programs with the ACT Programming Tutor (APT). APT is constructed around a production rule cognitive model of programming knowledge, called theideal student model. This model allows the tutor to solve exercises along with the student and provide assistance as necessary. As the student works, the tutor also maintains an estimate of the probability that the student has learned each of the rules in the ideal model, in a process calledknowledge tracing. The tutor presents an individualized sequence of exercises to the student based on these probability estimates until the student has ‘mastered’ each rule. The programming tutor, cognitive model and learning and performance assumptions are described. A series of studies is reviewed that examine the empirical validity of knowledge tracing and has led to modifications in the process. Currently the model is quite successful in predicting test performance. Further modifications in the modeling process are discussed that may improve performance levels.
---
paper_title: More Accurate Student Modeling Through Contextual Estimation of Slip and Guess Probabilities in Bayesian Knowledge Tracing

paper_content:

Modeling students' knowledge is a fundamental part of intelligent tutoring systems. One of the most popular methods for estimating students' knowledge is Corbett and Anderson's [6] Bayesian Knowledge Tracing model. The model uses four parameters per skill, fit using student performance data, to relate performance to learning. Beck [1] showed that existing methods for determining these parameters are prone to the Identifiability Problem:the same performance data can be fit equally well by different parameters, with different implications on system behavior. Beck offered a solution based on Dirichlet Priors [1], but, we show this solution is vulnerable to a different problem, Model Degeneracy, where parameter values violate the model's conceptual meaning (such as a student being more likely to get a correct answer if he/she does not know a skill than if he/she does).We offer a new method for instantiating Bayesian Knowledge Tracing, using machine learning to make contextual estimations of the probability that a student has guessed or slipped. This method is no more prone to problems with Identifiability than Beck's solution, has less Model Degeneracy than competing approaches, and fits student performance data better than prior methods. Thus, it allows for more accurate and reliable student modeling in ITSs that use knowledge tracing.
---
paper_title: Inspectable Bayesian student modelling servers in multi-agent tutoring systems

paper_content:

User modelling shells and learner modelling servers have been proposed in order to provide reusable user/student model information over different domains, common inference mechanisms, and mechanisms to handle consistency of beliefs from different sources. Open and inspectable student models have been investigated by several authors as a means to promote student reflection, knowledge awareness, collaborative assessment, self-assessment, interactive diagnosis, to arrange groups of students, and to support the use of students' models by the teacher. ::: ::: This paper presents SModel, a Bayesian student modelling server used in distributed multi-agent environments. SModel server includes a student model database and a Bayesian student modelling component. SModel provides several services to a group of agents in a CORBA platform. Users can use ViSMod, a Bayesian student modelling visualization tool, and SMV, a student modelling database viewer, to visualize and inspect distributed Bayesian student models maintained by SModel server. SModel has been tested in a multi-agent tutoring system for teaching basic Java programming. In addition, SModel server has been used to maintain and share student models in a study focussed on exploring the existence of student reflection and analysing student model accuracy using inspectable Bayesian student models.
---
paper_title: Advances in Intelligent Tutoring Systems

paper_content:

The idea for this book on Intelligent Tutoring Systems (ITS) was sparked by the success of the ITS08 international conference. The number of presentations and their quality bore witness to the vitality and maturity of the field, and the enthusiasm of the participants held out a promise of sustainability and innovative research. Long life to ITS research! The book is divided into five parts. The introductory chapters to these parts, which summarize foundations, developments, strengths and weaknesses in each of the areas covered, are addressed to all readers. For those who want more in-depth knowledge, we give the floor to researchers who present their work, their results, and their view of what the future holds. It is our hope that all readers will find the book informative and thought-provoking.
---
paper_title: Building Intelligent Interactive Tutors: Student-Centered Strategies for Revolutionizing e-Learning

paper_content:

Computers have transformed every facet of our culture, most dramatically communication, transportation, finance, science, and the economy. Yet their impact has not been generally felt in education due to lack of hardware, teacher training, and sophisticated software. Another reason is that current instructional software is neither truly responsive to student needs nor flexible enough to emulate teaching. The more instructional software can reason about its own teaching process, know what it is teaching, and which method to use for teaching, the greater is its impact on education. ::: ::: Building Intelligent Interactive Tutors discusses educational systems that assess a student's knowledge and are adaptive to a student's learning needs. Dr. Woolf taps into 20 years of research on intelligent tutors to bring designers and developers a broad range of issues and methods that produce the best intelligent learning environments possible, whether for classroom or life-long learning. The book describes multidisciplinary approaches to using computers for teaching, reports on research, development, and real-world experiences, and discusses intelligent tutors, web-based learning systems, adaptive learning systems, intelligent agents and intelligent multimedia. ::: ::: *Combines both theory and practice to offer most in-depth and up-to-date treatment of intelligent tutoring systems available ::: *Presents powerful drivers of virtual teaching systems, including cognitive science, artificial intelligence, and the Internet ::: *Features algorithmic material that enables programmers and researchers to design building components and intelligent systems
---
paper_title: The Role of Probability-Based Inference in an Intelligent Tutoring System.

paper_content:

Probability-based inference in complex networks of interdependent variables is an active topic in statistical research, spurred by such diverse applications as forecasting, pedigree analysis, troubleshooting, and medical diagnosis. This paper concerns the role of Bayesian inference networks for updating student models in intelligent tutoring systems (ITSs). Basic concepts of the approach are briefly reviewed, but the emphasis is on the considerations that arise when one attempts to operationalize the abstract framework of probability-based reasoning in a practical ITS context. The discussion revolves around HYDRIVE, an ITS for learning to troubleshoot an aircraft hydraulics system. HYDRIVE supports generalized claims about aspects of student proficiency through probabilitybased combination of rule-based evaluations of specific actions. The paper highlights the interplay among inferential issues, the psychology of learning in the domain, and the instructional approach upon which the ITS is based.
---
paper_title: Modeling Diagnostic Assessments with Bayesian Networks

paper_content:

This paper defines Bayesian network models and examines their applications to IRT-based cognitive diagnostic modeling. These models are especially suited to building inference engines designed to be synchronous with the finer grained student models that arise in skills diagnostic assessment. Aspects of the theory and use of Bayesian network models are reviewed, as they affect applications to diagnostic assessment. The paper discusses how Bayesian network models are set up with expert information, improved and calibrated from data, and deployed as evidence-based inference engines. Aimed at a general educational measurement audience, the paper illustrates the flexibility and capabilities of Bayesian networks through a series of concrete examples, and without extensive technical detail. Examples are provided of proficiency spaces with direct dependencies among proficiency nodes, and of customized evidence models for complex tasks. This paper is intended to motivate educational measurement practitioners to learn more about Bayesian networks from the research literature, to acquire readily available Bayesian network software, to perform studies with real and simulated data sets, and to look for opportunities in educational settings that may benefit from diagnostic assessment fueled by Bayesian network modeling.
---
paper_title: Optimising ITS Behaviour with Bayesian Networks and Decision Theory

paper_content:

We propose and demonstrate a methodology for building tractable normative intelligent tutoring systems (ITSs). A normative ITS uses a Bayesian network for long-term student modelling and decision theory to select the next tutorial action. Because normative theories are a general framework for rational behaviour, they can be used to both define and apply learning theories in a rational, and therefore optimal, way. This contrasts to the more traditional approach of using an ad-hoc scheme to implement the learning theory. A key step of the methodology is the induction and the continual adaptation of the Bayesian network student model from student performance data, a step that is distinct from other recent Bayesian net approaches in which the network structure and probabilities are either chosen beforehand by an expert, or by efficiency considerations. The methodology is demonstrated by a description and evaluation of CAPIT, a normative constraint-based tutor for English capitalisation and punctuation. Our evaluation results show that a class using the full normative version of CAPIT learned the domain rules at a faster rate than the class that used a non-normative version of the same system.
---
paper_title: Learning Bayesian Networks

paper_content:

This chapter addresses the problem of learning the parameters from data. It also discusses score-based structure learning and constraint-based structure learning. The method for learning all parameters in a Bayesian network follows readily from the method for learning a single parameter. The chapter presents a method for learning the probability of a binomial variable and extends this method to multinomial variables. It also provides guidelines for articulating the prior beliefs concerning probabilities. The chapter illustrates the constraint-based approach by showing how to learn a directed acyclic graph (DAG) faithful to a probability distribution. Structure learning consists of learning the DAG in a Bayesian network from data. It is necessary to know which DAG satisfies the Markov condition with the probability distribution P that is generating the data. The process of learning such a DAG is called “model selection.” A DAG includes a probability distribution P if the DAG does not entail any conditional independencies that are not in P. In score-based structure learning, a score is assigned to each DAG based on the data such that in the limit. After scoring the DAGs, the score are used, possibly along with prior probabilities, to learn a DAG. The most straightforward score, the Bayesian score, is the probability of the data D given the DAG. Once a DAG is learnt from data, the parameters can be known. The result will be a Bayesian network that can be used to do inference. In the constraint-based approach, a DAG is found for which the Markov condition entails all and only those conditional independencies that are in the probability distribution P of the variables of interest. The chapter applies structure learning to inferring causal influences from data and presents learning packages. It presents examples of learning Bayesian networks and of causal learning.
---
paper_title: Using Bayesian Networks to Manage Uncertainty in Student Modeling

paper_content:

When a tutoring system aims to provide students with interactive help, it needs to know what knowledge the student has and what goals the student is currently trying to achieve. That is, it must do both assessment and plan recognition. These modeling tasks involve a high level of uncertainty when students are allowed to follow various lines of reasoning and are not required to show all their reasoning explicitly. We use Bayesian networks as a comprehensive, sound formalism to handle this uncertainty. Using Bayesian networks, we have devised the probabilistic student models for Andes, a tutoring system for Newtonian physics whose philosophy is to maximize student initiative and freedom during the pedagogical interaction. Andes' models provide long-term knowledge assessment, plan recognition, and prediction of students' actions during problem solving, as well as assessment of students' knowledge and understanding as students read and explain worked out examples. In this paper, we describe the basic mechanisms that allow Andes' student models to soundly perform assessment and plan recognition, as well as the Bayesian network solutions to issues that arose in scaling up the model to a full-scale, field evaluated application. We also summarize the results of several evaluations of Andes which provide evidence on the accuracy of its student models.
---
paper_title: Numerical uncertainty management in user and student modeling: An overview of systems and issues

paper_content:

A rapidly growing number of user and student modeling systems have employed numerical techniques for uncertainty management. The three major paradigms are those of Bayesian networks, the Dempster-Shafer theory of evidence, and fuzzy logic. In this overview, each of the first three main sections focuses on one of these paradigms. It first introduces the basic concepts by showing how they can be applied to a relatively simple user modeling problem. It then surveys systems that have applied techniques from the paradigm to user or student modeling, characterizing each system within a common framework. The final main section discusses several aspects of the usability of these techniques for user or student modeling, such as their knowledge engineering requirements, their need for computational resources, and the communicability of their results.
---
paper_title: A Computational Framework for Granularity and its Application to Educational Diagnosis

paper_content:

Many artificial intelligence systems implicitly use notions of granularity in reasoning, but there is very little research into granularity itself. An exception is the work of Hobbs [1985], which outlines several characteristics of granularity. In this paper we describe an approach to representing granularity which formalizes in computational terms most of Hobbs' notions, often refining and extending them. In particular two types of granularity have been delineated: aggregation and abstraction. Objects can be described at various grain sizes and connected together into a granularity hierarchy which allows focus shifts along either aggregation or abstraction dimensions. We briefly discuss how we have used granularity hierarchies in the recognition of novice LISP programming strategies and show how this enhances the recognition process and can lead toward planning appropriate feedback for the student.
---
paper_title: Bayesian Networks in Educational Testing

paper_content:

In this paper we discuss applications of Bayesian networks to educational testing. Namely, we deal with the diagnosis of person's skills. We show that when modeling dependence between skills we can get better diagnosis faster. We present results of experiments with basic operations that use fractions. The experiments suggest that the test design can benefit from a Bayesian network that models relations between skills, not only in the case of an adaptive test but also when designing a fixed (non-adaptive) test.
---
paper_title: Using Bayesian Networks to Manage Uncertainty in Student Modeling

paper_content:

When a tutoring system aims to provide students with interactive help, it needs to know what knowledge the student has and what goals the student is currently trying to achieve. That is, it must do both assessment and plan recognition. These modeling tasks involve a high level of uncertainty when students are allowed to follow various lines of reasoning and are not required to show all their reasoning explicitly. We use Bayesian networks as a comprehensive, sound formalism to handle this uncertainty. Using Bayesian networks, we have devised the probabilistic student models for Andes, a tutoring system for Newtonian physics whose philosophy is to maximize student initiative and freedom during the pedagogical interaction. Andes' models provide long-term knowledge assessment, plan recognition, and prediction of students' actions during problem solving, as well as assessment of students' knowledge and understanding as students read and explain worked out examples. In this paper, we describe the basic mechanisms that allow Andes' student models to soundly perform assessment and plan recognition, as well as the Bayesian network solutions to issues that arose in scaling up the model to a full-scale, field evaluated application. We also summarize the results of several evaluations of Andes which provide evidence on the accuracy of its student models.
---
paper_title: 2005, ‘Introducing Prerequisite Relations in a Multi-layered Bayesian Student Model

paper_content:

In this paper we present an extension of a previously developed generic student model based on Bayesian Networks. A new layer has been added to the model to include prerequisite relationships. The need of this new layer is motivated from different points of view: in practice, this kind of relationships are very common in any educational setting, but also their use allows for improving efficiency of both adaptation mechanisms and the inference process. The new prerequisite layer has been evaluated using two different experiments: the first experiment uses a small toy example to show how the BN can emulate human reasoning in this context, while the second experiment with simulated students suggests that prerequisite relationships can improve the efficiency of the diagnosis process by allowing increased accuracy or reductions in the test length.
---
paper_title: Bayes Nets in Educational Assessment: Where Do the Numbers Come From?

paper_content:

As observations and student models become complex, educational assessments that exploit advances in technology and cognitive psychology can outstrip familiar testing models and analytic methods. Within the Portal conceptual framework for assessment design, Bayesian inference networks (BINs) record beliefs about students' knowledge and skills, in light of what they say and do. Joining evidence model BIN fragments- which contain observable variables and pointers to student model variables - to the student model allows one to update belief about knowledge and skills as observations arrive. Markov Chain Monte Carlo (MCMC) techniques can estimate the required conditional probabilities from empirical data, supplemented by expert judgment or substantive theory. Details for the special cases of item response theory (IRT) and multivariate latent class modeling are given, with a numerical example of the latter.
---
paper_title: Using Bayesian Networks to Manage Uncertainty in Student Modeling

paper_content:

When a tutoring system aims to provide students with interactive help, it needs to know what knowledge the student has and what goals the student is currently trying to achieve. That is, it must do both assessment and plan recognition. These modeling tasks involve a high level of uncertainty when students are allowed to follow various lines of reasoning and are not required to show all their reasoning explicitly. We use Bayesian networks as a comprehensive, sound formalism to handle this uncertainty. Using Bayesian networks, we have devised the probabilistic student models for Andes, a tutoring system for Newtonian physics whose philosophy is to maximize student initiative and freedom during the pedagogical interaction. Andes' models provide long-term knowledge assessment, plan recognition, and prediction of students' actions during problem solving, as well as assessment of students' knowledge and understanding as students read and explain worked out examples. In this paper, we describe the basic mechanisms that allow Andes' student models to soundly perform assessment and plan recognition, as well as the Bayesian network solutions to issues that arose in scaling up the model to a full-scale, field evaluated application. We also summarize the results of several evaluations of Andes which provide evidence on the accuracy of its student models.
---
paper_title: 2005, ‘Introducing Prerequisite Relations in a Multi-layered Bayesian Student Model

paper_content:

In this paper we present an extension of a previously developed generic student model based on Bayesian Networks. A new layer has been added to the model to include prerequisite relationships. The need of this new layer is motivated from different points of view: in practice, this kind of relationships are very common in any educational setting, but also their use allows for improving efficiency of both adaptation mechanisms and the inference process. The new prerequisite layer has been evaluated using two different experiments: the first experiment uses a small toy example to show how the BN can emulate human reasoning in this context, while the second experiment with simulated students suggests that prerequisite relationships can improve the efficiency of the diagnosis process by allowing increased accuracy or reductions in the test length.
---
paper_title: Student modeling from conventional test data: A bayesian approach without priors

paper_content:

Although conventional tests are often used for determining a student's overall competence, they are seldom used for determining a finegrained model. However, this problem does arise occasionally, such as when a conventional test is used to initialize the student model of an ITS. Existing psychometric techniques for solving this problem are intractable. Straightforward Bayesian techniques are also inapplicable because they depend too strongly on the priors, which are often not available. Our solution is to base the assessment on the difference between the prior and posterior probabilities. If the test data raise the posterior probability of mastery of a piece of knowledge even slightly above its prior probability, then that is interpreted as evidence that the student has mastered that piece of knowledge. Evaluation of this technique with artificial students indicates that it can deliver highly accurate assessments.
---
paper_title: Optimising ITS Behaviour with Bayesian Networks and Decision Theory

paper_content:

We propose and demonstrate a methodology for building tractable normative intelligent tutoring systems (ITSs). A normative ITS uses a Bayesian network for long-term student modelling and decision theory to select the next tutorial action. Because normative theories are a general framework for rational behaviour, they can be used to both define and apply learning theories in a rational, and therefore optimal, way. This contrasts to the more traditional approach of using an ad-hoc scheme to implement the learning theory. A key step of the methodology is the induction and the continual adaptation of the Bayesian network student model from student performance data, a step that is distinct from other recent Bayesian net approaches in which the network structure and probabilities are either chosen beforehand by an expert, or by efficiency considerations. The methodology is demonstrated by a description and evaluation of CAPIT, a normative constraint-based tutor for English capitalisation and punctuation. Our evaluation results show that a class using the full normative version of CAPIT learned the domain rules at a faster rate than the class that used a non-normative version of the same system.
---
paper_title: An introduction to multidimensional measurement using Rasch models.

paper_content:

The act of constructing a measure requires a number of important assumptions. Principle among these assumptions is that the construct is unidimensional. In practice there are many instances when the assumption of unidimensionality does not hold, and where the application of a multidimensional measurement model is both technically appropriate and substantively advantageous. In this paper we illustrate the usefulness of a multidimensional approach to measurement with the Multidimensional Random Coefficient Multinomial Logit (MRCML) model, an extension of the unidimensional Rasch model. An empirical example is taken from a collection of embedded assessments administered to 541 students enrolled in middle school science classes with a hands-on science curriculum. Student achievement on these assessments are multidimensional in nature, but can also be treated as consecutive unidimensional estimates, or as is most common, as a composite unidimensional estimate. Structural parameters are estimated for each model using ConQuest, and model fit is compared. Student achievement in science is also compared across models. The multidimensional approach has the best fit to the data, and provides more reliable estimates of student achievement than under the consecutive unidimensional approach. Finally, at an interpretational level, the multidimensional approach may well provide richer information to the classroom teacher about the nature of student achievement.
---
paper_title: The Discriminating Power of Items That Measure More Than One Dimension

paper_content:

Determining a correct response to many test items frequently requires more than one ability. This paper describes the characteristics of items of this type by proposing generalizations of the item response theory concepts of discrimination and information. The conceptual framework for these statistics is presented, and the formulas for the statistics are derived for the multidimensional extension of the two-parameter logistic model. Use of the statistics is demonstrated for a form of the ACT Mathematics Usage Test.
---
paper_title: Adaptive testing for hierarchical student models

paper_content:

This paper presents an approach to student modeling in which knowledge is represented by means of probability distributions associated to a tree of concepts. A diagnosis procedure which uses adaptive testing is part of this approach. Adaptive tests provide well-founded and accurate diagnosis thanks to the underlying probabilistic theory, i.e., the Item Response Theory. Most adaptive testing proposals are based on dichotomous models, where he student answer can only be considered either correct or incorrect. In the work described here, a polytomous model has been used, i.e., answers can be given partial credits. Thus, models are more informative and diagnosis is more efficient. This paper also presents an algorithm for estimating question characteristic curves, which are necessary in order to apply the Item Response Theory to a given domain and hence must be inferred before testing begins. Most prior estimation procedures need huge sets of data. We have modified preexisting procedures in such a way that data requirements are significantly reduced. Finally, this paper presents the results of some controlled evaluations that have been carried out in order to analyze the feasibility and advantages of this approach.
---
paper_title: Using Restricted Latent Class Models to Map the Skill Structure of Achievement Items

paper_content:

This paper presents a new method for using certain restricted latent class models, referred to as binary skills models, to determine the skills required by a set o f test items. The method is applied to reading achievement data from a nationally representative sample o f fourth-grade students and offers useful perspectives on test structure and examinee ability, distinct from those provided by other methods o f analysis. Models fitted to small, overlapping sets o f items are integrated into a common skill map, and the nature o f each skill is then inferred from the characteristics o f the items for which it is required. The reading comprehension items examined conform closely to a unidimensional scale with six discrete skill levels that range from an inability to comprehend or match isolated words in a reading passage to the abilities required to integrate passage content with general knowledge and to recognize the main ideas o f the most difficult passages on the test.
---
paper_title: Skills Diagnosis Using IRT-Based Continuous Latent Trait Models

paper_content:

This article summarizes the continuous latent trait IRT approach to skills diagnosis as particularized by a representative variety of continuous latent trait models us ing item response functions (IRFs). First, several basic IRT-based continuous latent trait approaches are presented in some detail. Then a brief summary of estimation, model checking, and assessment scoring aspects are discussed. Finally, the Uni versity of California at Berkeley multidimensional Rasch-model-grounded SEPUP middle school science-focused embedded assessment project is briefly described as one significant illustrative application.
---
paper_title: Skills Diagnosis Using IRT‐Based Latent Class Models

paper_content:

This article describes a latent trait approach to skills diagnosis based on a particular variety of latent class models that employ item response functions (IRFs) as in typical item response theory (IRT) models. To enable and encourage comparisons with other approaches, this description is provided in terms of the main components of any psychometric approach: the ability model and the IRF structure; review of research on estimation, model checking, reliability, validity, equating, and scoring; and a brief review of real data applications. In this manner the article demonstrates that this approach to skills diagnosis has built a strong initial foundation of research and resources available to potential users. The outlook for future research and applications is discussed with special emphasis on a call for pilot studies and concomitant increased validity research.
---
paper_title: Learning Factors Transfer Analysis: Using Learning Curve Analysis to Automatically Generate Domain Models

paper_content:

This paper describes a novel method to create a quantitative model of an educational content domain of related practice item-types using learning curves. By using a pairwise test to search for the relationships between learning curves for these item-types, we show how the test results in a set of pairwise transfer relationships that can be expressed in a Q-matrix domain model. Creating these Q-matrices for various test criteria we show that the new domain model results in consistently better learning curve fits as shown by cross- validation. Further, the Q-matrices produced can be used by educators or curriculum designers to gain a richer, more integrated perspective on concepts in the domain. The model may also have implications for tracing student knowledge more effectively to sequence practice in tutoring/training software.
---
paper_title: RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY

paper_content:

Several authors from various disciplines such as cognitive psychology, artificial intelligence, and psychometrics have developed rule assessment methods for diagnosing students' misconceptions. Siegler's (1976, 1978) binary decision tree method was illustrated in the context of the problem of balance scales and Anderson's (1974, 1981) functional method was developed for diagnosing errors occurring during the operation of integrating functional rules. A group of artificial intelligence researchers (Brown & Burton, 1978; Brown & VanLehn, 1980; VanLehn, 1981) developed a computer program "DEBUGGY" that can diagnose a number of erroneous rules resulting from misconceptions ("bugs") in whole number subtraction problems. These investigations were motivated by an interest in the basic foundations of knowledge structure and development in human cognition. Tatsuoka and her associates (Tatsuoka, Baillie, & Yamamoto, 1982; Tatsuoka, Birenbaum, Tatsuoka, & Baillie, 1980) also developed a computer program that can diagnose a number of erroneous rules in signed-number addition and subtraction problems, but they were motivated primarily by the exploration of psychometric properties of bugs such as changes in error types or the stability of misconceptions committed by a student throughout a test (Birenbaum & Tatsuoka, 1980; Tatsuoka, 1981; Tatsuoka & Tatsuoka, 1983). Analysis of misconceptions can provide useful information in evaluating instruction or instructional materials as well as specific prescriptions for planning remediation for a student. For example, the source of many of the misconceptions committed by students is often the ambiguity of explanations or the lack of precise, accurate instructions in teaching material. It is useful to have such computer programs in educational practice. However, constructing a DEBUGGY-type system for domains more general than arithmetic is extremely difficult and time consuming. On the other hand, personal indices that are designed to summarize response patterns can be used for detecting anomalous patterns (Drasgow, 1982; Harnisch & Linn, 1981; Levine & Rubin, 1979; Sato, 1975; Tatsuoka & Tatsuoka, 1982; Wright & Stone, 1979) which may result from applications of some erroneous rules, but they have only limited power. For instance, these indices cannot diagnose sources of misconceptions or provide prescriptive information for remediating them. Nevertheless, the use of personal indices is economical and applicable to general areas because such indices can first be used to spot candidates among students who may possibly possess misconceptions and may hence need personal attention from teachers. This procedure seems to be widely used in Japan (Sato, 1982). This study develops a model to fill the gap between the rule-assessing methods and the personal-index approach. The number of different erroneous rules already discovered in
---
paper_title: Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory

paper_content:

Some usability and interpretability issues for single-strategy cognitive assessment models are considered. These models posit a stochastic conjunctive relationship between a set of cognitive attributes to be assessed and performance on particular items/tasks in the assessment. The models considered make few assumptions about the relationship between latent attributes and task performance beyond a simple conjunctive structure. An example shows that these models can be sensitive to cognitive attributes, even in data designed to well fit the Rasch model. Several stochastic ordering and monotonicity properties are considered that enhance the interpretability of the models. Simple data summaries are identified that inform about the presence or absence of cognitive attributes when the full computational power needed to estimate the models is not available.
---
paper_title: Improving Contextual Models of Guessing and Slipping with a Trucated Training Set

paper_content:

A recent innovation in student knowledge modeling is the replacement of static estimates of the probability that a student has guessed or slipped with more contextual estimation of these probabilities [2], significantly ::: improving prediction of future performance in one case. We extend this method by adjusting the training set used to develop the contextual models of guessing and slipping, removing training examples where the prior probability that the student knew the skill was very high or very low. We show that this adjustment ::: significantly improves prediction of future performance, relative to previous ::: methods, within data sets from three different Cognitive Tutors.
---
paper_title: K.R.: Performance Factors Analysis -- A New Alternative to Knowledge Tracing

paper_content:

Knowledge tracing (KT)[1] has been used in various forms for adaptive computerized instruction for more than 40 years. However, despite its long history of application, it is difficult to use in domain model search procedures, has not been used to capture learning where multiple skills are needed to perform a single action, and has not been used to compute latencies of actions. On the other hand, existing models used for educational data mining (e.g. Learning Factors Analysis (LFA)[2]) and model search do not tend to allow the creation of a “model overlay” that traces predictions for individual students with individual skills so as to allow the adaptive instruction to automatically remediate performance. Because these limitations make the transition from model search to model application in adaptive instruction more difficult, this paper describes our work to modify an existing data mining model so that it can also be used to select practice adaptively. We compare this new adaptive data mining model (PFA, Performance Factors Analysis) with two versions of LFA and then compare PFA with standard KT.
---
paper_title: 2011b) KT-IDEM: Introducing Item Difficulty to the Knowledge Tracing Model

paper_content:

Many models in computer education and assessment take into account difficulty. However, despite the positive results of models that take difficulty in to account, knowledge tracing is still used in its basic form due to its skill level diagnostic abilities that are very useful to teachers. This leads to the research question we address in this work: Can KT be effectively extended to capture item difficulty and improve prediction accuracy? There have been a variety of extensions to KT in recent years. One such extension was Baker's contextual guess and slip model. While this model has shown positive gains over KT in internal validation testing, it has not performed well relative to KT on unseen in-tutor data or post-test data, however, it has proven a valuable model to use alongside other models. The contextual guess and slip model increases the complexity of KT by adding regression steps and feature generation. The added complexity of feature generation across datasets may have hindered the performance of this model. Therefore, one of the aims of our work here is to make the most minimal of modifications to the KT model in order to add item difficulty and keep the modification limited to changing the topology of the model. We analyze datasets from two intelligent tutoring systems with KT and a model we have called KT-IDEM (Item Difficulty Effect Model) and show that substantial performance gains can be achieved with this minor modification that incorporates item difficulty.
---
paper_title: User modeling: a notoriously black art

paper_content:

This paper is intended as guidance for those who are familiar with user modeling field but are less fluent in statistical methods. It addresses potential problems with user model selection and evaluation, that are often clear to expert modelers, but are not obvious for others. These problems are frequently a result of a falsely straightforward application of statistics to user modeling (e.g. over-reliance on model fit metrics). In such cases, absolute trust in arguably shallow model accuracy measures could lead to selecting models that are hard-to-interpret, less meaningful, over-fit, and less generalizable. We offer a list of questions to consider in order to avoid these modeling pitfalls. Each of the listed questions is backed by an illustrative example based on the user modeling approach called Performance Factors Analysis (PFA) [9].
---
paper_title: Does help help? Introducing the Bayesian Evaluation and Assessment methodology

paper_content:

Most ITS have a means of providing assistance to the student, either on student request or when the tutor determines it would be effective. Presumably, such assistance is included by the ITS designers since they feel it benefits the students. However, whether--and how--help helps students has not been a well studied problem in the ITS community. In this paper we present three approaches for evaluating the efficacy of the Reading Tutor's help: creating experimental trials from data, learning decomposition, and Bayesian Evaluation and Assessment, an approach that uses dynamic Bayesian networks. We have found that experimental trials and learning decomposition both find a negative benefit for help---that is, help hurts! However, the Bayesian Evaluation and Assessment framework finds that help both promotes student long-term learning and provides additional scaffolding on the current problem. We discuss why these approaches give divergent results, and suggest that the Bayesian Evaluation and Assessment framework is the strongest of the three. In addition to introducing Bayesian Evaluation and Assessment, a method for simultaneously assessing students and evaluating tutorial interventions, this paper describes how help can both scaffold the current problem attempt as well as teach the student knowledge that will transfer to later problems.
---
paper_title: Using multiple Dirichlet distributions to improve parameter plausibility

paper_content:

Predictive accuracy and parameter plausibility are two major desired aspects for a student modeling approach. Knowledge tracing, the most commonly used approach, suffers from local maxima and multiple global maxima. Prior work has shown that using Dirichlet priors improves model parameter plausibility. However, the assumption that all knowledge components are from a single Dirichlet distribution is questionable. To address this problem, this paper presents an approach to integrate multiple distributions and Dirichlet priors. We show that modeling groups of students separately based on their distributional similarities produces model parameters that provide a more plausible picture of student knowledge, even though the proposed solution did not improve the model’s predictive accuracy. We also show Dirichlet priors might be hurt by outliers and models with trimming work better.
---
paper_title: Learning Factors Transfer Analysis: Using Learning Curve Analysis to Automatically Generate Domain Models

paper_content:

This paper describes a novel method to create a quantitative model of an educational content domain of related practice item-types using learning curves. By using a pairwise test to search for the relationships between learning curves for these item-types, we show how the test results in a set of pairwise transfer relationships that can be expressed in a Q-matrix domain model. Creating these Q-matrices for various test criteria we show that the new domain model results in consistently better learning curve fits as shown by cross- validation. Further, the Q-matrices produced can be used by educators or curriculum designers to gain a richer, more integrated perspective on concepts in the domain. The model may also have implications for tracing student knowledge more effectively to sequence practice in tutoring/training software.
---
paper_title: Learning Factors Analysis - A General Method for Cognitive Model Evaluation and Improvement

paper_content:

A cognitive model is a set of production rules or skills encoded in intelligent tutors to model how students solve problems. It is usually generated by brainstorming and iterative refinement between subject experts, cognitive scientists and programmers. In this paper we propose a semi-automated method for improving a cognitive model called Learning Factors Analysis that combines a statistical model, human expertise and a combinatorial search. We use this method to evaluate an existing cognitive model and to generate and evaluate alternative models. We present improved cognitive models and make suggestions for improving the intelligent tutor based on those models.
---
paper_title: Using Logistic Regression to Trace Multiple Sub-skills in a Dynamic Bayes Net

paper_content:

A challenge in estimating students’ changing knowledge from sequential observations of their performance arises when each observed step involves multiple subskills. To overcome this mismatch in grain size between modelled skills and observed actions, we use logistic regression over each step’s subskills in a dynamic Bayes net (LR-DBN) to model transition probabilities for the overall knowledge required by the step. Unlike previous methods, LR-DBN can trace knowledge of the individual subskills without assuming they are independent. We evaluate how well it fits children’s oral reading fluency data logged by Project LISTEN’s Reading Tutor, compared to other methods.
---
paper_title: Navigating the parameter space of Bayesian Knowledge Tracing models: Visualizations of the convergence of the Expectation Maximization algorithm

paper_content:

Bayesian Knowledge Tracing (KT) models are employed by the cognitive tutors in order to determine student knowledge based on four parameters: learn rate, prior, guess and slip. A commonly used algorithm for learning these parameter values from data is the Expectation Maximization (EM) algorithm. Past work, however, has suggested that with four free parameters the standard KT model is prone to converging to erroneous degenerate states depending on the initial values of these four parameters. In this work we simulate data from a model with known parameter values and then run a grid search over the parameter initialization space of KT to map out which initial values lead to erroneous learned parameters. Through analysis of convergence and error surface visualizations we found that the initial parameter values leading to a degenerate state are not scattered randomly throughput the parameter space but instead exist on a surface with predictable boundaries. A recently introduced extension to KT that individualizes the prior parameter is also explored and compared to standard KT with regard to parameter convergence. We found that the individualization model has unique properties which allow it to avoid the local maxima problem.
---
paper_title: Knowledge tracing: Modeling the acquisition of procedural knowledge

paper_content:

This paper describes an effort to model students' changing knowledge state during skill acquisition. Students in this research are learning to write short programs with the ACT Programming Tutor (APT). APT is constructed around a production rule cognitive model of programming knowledge, called theideal student model. This model allows the tutor to solve exercises along with the student and provide assistance as necessary. As the student works, the tutor also maintains an estimate of the probability that the student has learned each of the rules in the ideal model, in a process calledknowledge tracing. The tutor presents an individualized sequence of exercises to the student based on these probability estimates until the student has ‘mastered’ each rule. The programming tutor, cognitive model and learning and performance assumptions are described. A series of studies is reviewed that examine the empirical validity of knowledge tracing and has led to modifications in the process. Currently the model is quite successful in predicting test performance. Further modifications in the modeling process are discussed that may improve performance levels.
---
paper_title: Learned student models with item to item knowledge structures

paper_content:

Probabilistic and learned approaches to student modeling are attractive because of the uncertainty surrounding the student skills assessment and because of the need to automatize the process. Item to item structures readily lend themselves to probabilistic and fully learned models because they are solely composed of observable nodes, like answers to test questions. Their structure is also well grounded in the cognitive theory of knowledge spaces. We study the effectiveness of two Bayesian frameworks to learn item to item structures and to use the induced structures to predict item outcome from a subset of evidence. One approach, Partial Order Knowledge Structures (POKS), relies on a naive Bayes framework whereas the other is based on the Bayesian network (BN) learning and inference framework. Both approaches are assessed over their predictive ability and their computational efficiency in different experimental simulations. The results from simulations over three data sets show that they both can effectively perform accurate predictions, but POKS generally displays higher predictive power than the BN. Moreover, the simplicity of POKS translates to a time efficiency between one to three orders of magnitude greater than the BN runs. We further explore the use of the item to item approach for handling concepts mastery assessment. The approach investigated consist in augmenting an initial set of observations, based on inferences with the item to item structure, and feed the augmented set to a BN containing a number of concepts. The results show that augmented set can effectively improve predictive power of a BN for item outcome, but that improvement does not transfer to the concept assessment in this particular experiment. We discuss different explanations for the results and outline future research avenues.
---
paper_title: Skills and Knowledge Structures

paper_content:

Suppose that Q is a set of problems and S is a set of skills. A skill function assigns to each problem q i.e. to each element of Q — those sets of skills which are minimally sufficient to solve q; a problem function assigns to each set X of skills the set of problems which can be solved with these skills (a knowledge state). We explore the natural properties of such functions and show that these concepts are basically the same. Furthermore, we show that for every family K of subsets of Q which includes the empty set and Q, there are a set S of (abstract) skills and a problem function whose range is just K. We also give a bound for the number of skills needed to generate a specific set of knowledge states, and discuss various ways to supply a set of knowledge states with an underlying skill theory. Finally, a procedure is described to determine a skill function using coverings in partial orders which is applied to set A of the Coloured Progressive Matrices test (Raven, 1965).
---
paper_title: User-expertise modeling with empirically derived probabilistic implication networks

paper_content:

The application of user-expertise modeling for adaptive interfaces is confronted with a number of difficult challenges, namely, efficiency and reliability, the cost-benefit ratio, and the practical usability of user modeling techniques. We argue that many of these obstacles can be overcome by standard, automatic means of performing knowledge assessment. Within this perspective, we present the basis of a probabilistic user modeling approach, the POKS technique, which could serve as a standard user-expertise modeling tool.The POKS technique is based on the cognitive theory of knowledge structures: a formalism for the representation of the order in which we learn knowledge units (KU). The technique permits the induction of knowledge structures from a small number of empirical data cases. It uses an evidence propagation scheme within these structures to infer an individual's knowledge state from a sample of KU. The empirical induction technique is based, in part, on statistical hypothesis testing over conditional probabilities that are determined by the KUs' learning order.Experiments with this approach show that the technique is successful in partially inferring an individual's knowledge state, either through the monitoring of a user's behavior, or through a selective questioning process. However, the selective process, based on entropy minimization, is shown to be much more effective in reducing the standard error score of knowledge assessment than random sampling.
---
paper_title: The Assessment of Knowledge, in Theory and in Practice

paper_content:

Adaptations from a book and many scholarly articles are presented. It reviews the main ideas of a novel theory for the assessment of a student's knowledge in a topic and gives details on a practical implementation in the form of a software system available on the Internet. The basic concept of the theory is the 'knowledge state,' which is the complete set of problems that an individual is capable of solving in a particular topic, such as elementary algebra. The task of the assessor consists in uncovering the particular state of the student being assessed. Even though the number of knowledge states for a topic may exceed several hundred thousand, these large numbers are well within the capacity of current home or school computers. The result of an assessment consists in two short lists of problems, which may be labelled: 'what the student can do' and 'what the student is ready to learn'. In the most important applications of the theory, these two lists specify the exact knowledge state of the individual being assessed.
---
paper_title: A methodology for eliciting , modelling , and evaluating expert knowledge for an adaptive work-integrated learning system

paper_content:

We present a methodology for constructing and evaluating models for adaptive informal technology-enhanced workplace learning. It is designed for knowledge-intensive work domains which are not pre structured according to a fixed curriculum. We extend research on Competence-based Knowledge Space Theory which has been mainly applied in educational settings. Our approach employs systematic knowledge elicitation and practically feasible evaluation techniques performed as part of the modelling process for iterative refinement of the models. A case study was performed in the Requirements Engineering domain to apply and test the developed methodology. We discuss lessons learned and several implications for knowledge engineering for adaptive workplace learning.
---
paper_title: Competence-based knowledge structures for personalised learning

paper_content:

The ELeGI project focuses on integrating technology-enhanced learning methodologies into a pedagogy-driven and service-oriented architecture based on Grid technology. It aims at a system that is capable of realising personalised, adaptive, and experiential learning. This requires to have available a framework that, on the one hand, allows for representing existing domain knowledge, and, on the other hand, provides a representation of the learner's current state of knowledge. It is shown that a competence-based extension of Knowledge Space Theory provides a representation of the conceptual organization of a domain that not only allows for an adaptive assessment of an individual's knowledge, but also for generating personalised learning paths. The discussion of this framework emphasizes its application within an open distributed service model, and in the context of Virtual Scientific Experiments.
---
paper_title: Probabilistic Student Models: Bayesian Belief Networks and Knowledge Space Theory

paper_content:

The applicability of Knowledge Space Theory (Falmagne and Doignon) and Bayesian Belief Networks (Pearl) as probabilistic student models imbedded in an Intelligent Tutoring System is examined. Student modeling issues such as knowledge representation, adaptive assessment, curriculum advancement, and student feedback are addressed. Several factors contribute to uncertainty in student modeling such as careless errors and lucky guesses, learning and forgetting, and unanticipated student response patterns. However, a probabilistic student model can represent uncertainty regarding the estimate of the student's knowledge and can be tested using empirical student data and established statistical techniques.
---
paper_title: Learned student models with item to item knowledge structures

paper_content:

Probabilistic and learned approaches to student modeling are attractive because of the uncertainty surrounding the student skills assessment and because of the need to automatize the process. Item to item structures readily lend themselves to probabilistic and fully learned models because they are solely composed of observable nodes, like answers to test questions. Their structure is also well grounded in the cognitive theory of knowledge spaces. We study the effectiveness of two Bayesian frameworks to learn item to item structures and to use the induced structures to predict item outcome from a subset of evidence. One approach, Partial Order Knowledge Structures (POKS), relies on a naive Bayes framework whereas the other is based on the Bayesian network (BN) learning and inference framework. Both approaches are assessed over their predictive ability and their computational efficiency in different experimental simulations. The results from simulations over three data sets show that they both can effectively perform accurate predictions, but POKS generally displays higher predictive power than the BN. Moreover, the simplicity of POKS translates to a time efficiency between one to three orders of magnitude greater than the BN runs. We further explore the use of the item to item approach for handling concepts mastery assessment. The approach investigated consist in augmenting an initial set of observations, based on inferences with the item to item structure, and feed the augmented set to a BN containing a number of concepts. The results show that augmented set can effectively improve predictive power of a BN for item outcome, but that improvement does not transfer to the concept assessment in this particular experiment. We discuss different explanations for the results and outline future research avenues.
---
paper_title: The Assessment of Knowledge, in Theory and in Practice

paper_content:

Adaptations from a book and many scholarly articles are presented. It reviews the main ideas of a novel theory for the assessment of a student's knowledge in a topic and gives details on a practical implementation in the form of a software system available on the Internet. The basic concept of the theory is the 'knowledge state,' which is the complete set of problems that an individual is capable of solving in a particular topic, such as elementary algebra. The task of the assessor consists in uncovering the particular state of the student being assessed. Even though the number of knowledge states for a topic may exceed several hundred thousand, these large numbers are well within the capacity of current home or school computers. The result of an assessment consists in two short lists of problems, which may be labelled: 'what the student can do' and 'what the student is ready to learn'. In the most important applications of the theory, these two lists specify the exact knowledge state of the individual being assessed.
---
paper_title: Performance Comparison of Item-to-Item Skills Models with the IRT Single Latent Trait Model

paper_content:

Assessing a learner's mastery of a set of skills is a fundamental issue in intelligent learning environments. We compare the predictive performance of two approaches for training a learner model with domain data. One is based on the principle of building the model solely from observable data items, such as exercises or test items. Skills modelling is not part of the training phase, but instead dealt with at later stage. The other approach incorporates a single latent skill in the model. We compare the capacity of both approaches to accurately predict item outcome (binary success or failure) from a subset of item outcomes. Three types of item-to-item models based on standard Bayesian modeling algorithms are tested: (1) Naive Bayes, (2) Tree-Augmented Naive Bayes (TAN), and (3) a K2 Bayesian Classifier. Their performance is compared to the widely used IRT-2PL approach which incorporates a single latent skill. The results show that the item-to-item approaches perform as well, or better than the IRT-2PL approach over 4 widely different data sets, but the differences vary considerably among the data sets. We discuss the implications of these results and the issues relating to the practical use of item-to-item models.
---
paper_title: The Cognitive Structure of Emotions

paper_content:

1. Introduction The study of emotion Types of evidence for theories of emotion Some goals for a cognitive theory of emotion 2. Structure of the theory The organisation of emotion types Basic emotions Some implications of the emotions-as-valenced-reactions claim 3. The cognitive psychology of appraisal The appraisal structure Central intensity variables 4. The intensity of emotions Global variables Local variables Variable-values, variable-weights, and emotion thresholds 5. Reactions to events: I. The well-being emotions Loss emotions and fine-grained analyses The fortunes-of-others emotions Self-pity and related states 6. Reactions to events: II. The prospect-based emotions Shock and pleasant surprise Some interrelationships between prospect-based emotions Suspense, resignation, hopelessness, and other related states 7. Reactions to agents The attribution emotions Gratitude, anger, and some other compound emotions 8. Reactions to objects The attraction emotions Fine-grained analyses and emotion sequences 9. The boundaries of the theory Emotion words and cross-cultural issues Emotion experiences and unconscious emotions Coping and the function of emotions Computational tractability.
---
paper_title: Modeling and understanding students' off-task behavior in intelligent tutoring systems

paper_content:

We present a machine-learned model that can automatically detect when a student using an intelligent tutoring system is off-task, i.e., engaged in behavior which does not involve the system or a learning task. This model was developed using only log files of system usage (i.e. no screen capture or audio/video data). We show that this model can both accurately identify each student's prevalence of off-task behavior and can distinguish off-task behavior from when the student is talking to the teacher or another student about the subject matter. We use this model in combination with motivational and attitudinal instruments, developing a profile of the attitudes and motivations associated with off-task behavior, and compare this profile to the attitudes and motivations associated with other behaviors in intelligent tutoring systems. We discuss how the model of off-task behavior can be used within interactive learning environments which respond to when students are off-task.
---
paper_title: Adapting to When Students Game an Intelligent Tutoring System

paper_content:

It has been found in recent years that many students who use intelligent tutoring systems game the system, attempting to succeed in the educational environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. In this paper, we introduce a system which gives a gaming student supplementary exercises focused on exactly the material the student bypassed by gaming, and which also expresses negative emotion to gaming students through an animated agent. Students using this system engage in less gaming, and students who receive many supplemental exercises have considerably better learning than is associated with gaming in the control condition or prior studies.
---
paper_title: The Kappa Statistic: A Second Look

paper_content:

In recent years, the kappa coefficient of agreement has become the de facto standard for evaluating intercoder agreement for tagging tasks. In this squib, we highlight issues that affect κ and that the community has largely neglected. First, we discuss the assumptions underlying different computations of the expected agreement component of κ. Second, we discuss how prevalence and bias affect the κ measure.
---
paper_title: Exploring the relationship between learner EEG mental engagement and affect

paper_content:

This paper studies the influence of learner's affective states on the EEG-mental engagement index during a problem solving task The electrical activity of the human brain, known as electroencephalography or EEG was registered according to an acquisition protocol in a learning environment specifically constructed for emotional elicitation Data was gathered from 35 healthy subjects using 8 biosensors and two video cameras The effect of learners' emotional states on the engagement index was analyzed as well as their impact on response time variability.
---
paper_title: Motivating the learner: an empirical evaluation

paper_content:

The M-Ecolab was developed to provide motivational scaffolding via an on-screen character whose demeanour defended on modelling the learner's motivational state at interaction time. Motivational modelling was based on three variables: effort, independence and the confidence. A classroom evalu-ation was conducted to illustrate the effects of motivational scaffolding. Students had an eighty minute interaction with the M-Ecolab, divided into two sessions. The results suggested a positive effect of the motivational scaffolding, particularly for initially de-motivated students who demonstrated higher learning gains. We found that these students followed the suggestions of the on-screen character which delivered personalized feedback. They behaved in a way that was conducive to learning by being challenge-seekers and displaying an inclination to exert more effort. This paper gives a detailed account of the methodology and findings that resulted from the empirical evaluation.
---
paper_title: Repairing Disengagement With Non-Invasive Interventions

paper_content:

We evaluated the impact of a set of interventions to repair students' disengagement while solving geometry problems in a tutoring system. We present a deep analysis of how a tutor can remediate a student's disengagement and motivation with self-monitoring feedback. The analysis consists of a between-subjects analyses on students learning and on students' attitudes towards mathematics and perceptions of the software, and a deeper analysis on students' engagement within the tutor. Results show that the general trend over time is for students to become more disengaged while using a tutor, yet students can be coaxed into reengagement after viewing interventions that promote self-reflection and self-monitoring --a simple open-learner model accompanied by suggestions and encouragement.
---
paper_title: Improving Contextual Models of Guessing and Slipping with a Trucated Training Set

paper_content:

A recent innovation in student knowledge modeling is the replacement of static estimates of the probability that a student has guessed or slipped with more contextual estimation of these probabilities [2], significantly ::: improving prediction of future performance in one case. We extend this method by adjusting the training set used to develop the contextual models of guessing and slipping, removing training examples where the prior probability that the student knew the skill was very high or very low. We show that this adjustment ::: significantly improves prediction of future performance, relative to previous ::: methods, within data sets from three different Cognitive Tutors.
---
paper_title: Informing the Detection of the Students' Motivational State: An Empirical Study

paper_content:

The ability to detect the students' motivational state during an instructional interaction can bring many benefits to the performance of an Intelligent Tutoring System (ITS). In this paper we present an empirical study which provided us with a considerable amount of knowledge regarding motivation diagnosis. We show how this knowledge was formalised in order to create a set of motivation diagnosis rules that can be incorporated into a prototype tutoring system. We also briefly present how these motivation diagnosis rules were evaluated in another study.
---
paper_title: Learning to Identify Students' Off-Task Behavior in Intelligent Tutoring Systems

paper_content:

The paper proposes a machine learning model that can automatically identify off-task behaviors of students while using an intelligent tutoring system. Only log files that record students' actions with the system are used for the development of the model. The model utilizes a set of time features, performance features and mouse movement features and is compared to i) a model that only utilizes time features, ii) a model that uses time and performance features. In order to address data sparseness problem, a robust Ridge Regression algorithm is designed to estimate model parameters. An extensive set of experiment results demonstrate the power of using multiple types of evidence as well as the robust Ridge Regression algorithm.
---
paper_title: YALE: rapid prototyping for complex data mining tasks

paper_content:

KDD is a complex and demanding task. While a large number of methods has been established for numerous problems, many challenges remain to be solved. New tasks emerge requiring the development of new methods or processing schemes. Like in software development, the development of such solutions demands for careful analysis, specification, implementation, and testing. Rapid prototyping is an approach which allows crucial design decisions as early as possible. A rapid prototyping system should support maximal re-use and innovative combinations of existing methods, as well as simple and quick integration of new ones.This paper describes Y ale , a free open-source environment forKDD and machine learning. Y ale provides a rich variety of methods whichallows rapid prototyping for new applications and makes costlyre-implementations unnecessary. Additionally, Y ale offers extensive functionality for process evaluation and optimization which is a crucial property for any KDD rapid prototyping tool. Following the paradigm of visual programming eases the design of processing schemes. While the graphical user interface supports interactive design, the underlying XML representation enables automated applications after the prototyping phase.After a discussion of the key concepts of Y ale , we illustrate the advantages of rapid prototyping for KDD on case studies ranging from data pre-processing to result visualization. These case studies cover tasks like feature engineering, text mining, data stream mining and tracking drifting concepts, ensemble methods and distributed data mining. This variety of applications is also reflected in a broad user base, we counted more than 40,000 downloads during the last twelve months.
---
paper_title: A Response Time Model For Bottom-Out Hints as Worked Examples

paper_content:

Students can use an educational system’s help in unexpected ways. For example, they may bypass abstract hints in search of a concrete solution. This behavior has traditionally been labeled as a form of gaming or help abuse. We propose that some examples of this behavior are not abusive and that bottom-out hints can act as worked examples. We create a model for distinguishing good student use of bottom-out hints from bad student use of bottom-out hints by means of logged response times. We show that this model not only predicts learning, but captures behaviors related to self-explanation.
---
paper_title: Developing a generalizable detector of when students game the system

paper_content:

Some students, when working in interactive learning environments, attempt to "game the system", attempting to succeed in the environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. In this paper, we present a system that can accurately detect whether a student is gaming the system, within a Cognitive Tutor mathematics curricula. Our detector also distinguishes between two distinct types of gaming which are associated with different learning outcomes. We explore this detector's generalizability, and find that it transfers successfully to both new students and new tutor lessons.
---
paper_title: A Leisurely Look at the Bootstrap, the Jackknife, and Cross-Validation

paper_content:

Abstract This is an invited expository article for The American Statistician. It reviews the nonparametric estimation of statistical error, mainly the bias and standard error of an estimator, or the error rate of a prediction rule. The presentation is written at a relaxed mathematical level, omitting most proofs, regularity conditions, and technical details.
---
paper_title: The meaning and use of the area under a receiver operating characteristic (ROC) curve.

paper_content:

A representation and interpretation of the area under a receiver operating characteristic (ROC) curve obtained by the "rating" method, or by mathematical predictions based on patient characteristics, is presented. It is shown that in such a setting the area represents the probability that a randomly chosen diseased subject is (correctly) rated or ranked with greater suspicion than a randomly chosen non-diseased subject. Moreover, this probability of a correct ranking is the same quantity that is estimated by the already well-studied nonparametric Wilcoxon statistic. These two relationships are exploited to (a) provide rapid closed-form expressions for the approximate magnitude of the sampling variability, i.e., standard error that one uses to accompany the area under a smoothed ROC curve, (b) guide in determining the size of the sample required to provide a sufficiently reliable estimate of this area, and (c) determine how large sample sizes should be to ensure that one can statistically detect difference...
---
paper_title: Classifying Learner Engagement through Integration of Multiple Data Sources

paper_content:

Intelligent tutoring systems (ITS) can provide effective instruction, but learners do not always use such systems effectively. In the present study, high school students' action sequences with a mathematics ITS were machine-classified into five finite-state machines indicating guessing strategies, appropriate help use, and independent problem solving; over 90% of problem events were categorized. Students were grouped via cluster analyses based on self reports of motivation. Motivation grouping predicted ITS strategic approach better than prior math achievement (as rated by classroom teachers). Learners who reported being disengaged in math were most likely to exhibit appropriate help use while working with the ITS, relative to average and high motivation learners. The results indicate that learners can readily report their motivation state and that these data predict how learners interact with the ITS.
---
paper_title: A Dynamic Mixture Model to Detect Student Motivation and Proficiency

paper_content:

Unmotivated students do not reap the full rewards of using a computer-based intelligent tutoring system. Detection of improper behavior is thus an important component of an online student model. To meet this challenge, we present a dynamic mixture model based on Item Response Theory. This model, which simultaneously estimates a student's proficiency and changing motivation level, was tested with data of high school students using a geometry tutoring system. By accounting for student motivation, the dynamic mixture model can more accurately estimate proficiency and the probability of a correct response. The model's generality is an added benefit, making it applicable to many intelligent tutoring systems as well as other domains.
---
paper_title: Detection and Analysis of Off-Task Gaming Behavior in Intelligent Tutoring Systems

paper_content:

A major issue in Intelligent Tutoring Systems is off-task student behavior, especially performance-based gaming, where students systematically exploit tutor behavior in order to advance through a curriculum quickly and easily, with as little active thought directed at the educational content as possible. The goal of this research was to explore the phenomena of off-task gaming behavior within the Assistments system. Machine-learned gaming-detection models were developed to investigate underlying factors related to gaming, and an analysis of gaming within the Assistments system was conducted to compare some of the findings of prior studies.
---
paper_title: Case studies in the use of ROC curve analysis for sensor-based estimates in human computer interaction

paper_content:

Applications that use sensor-based estimates face a fundamental tradeoff between true positives and false positives when examining the reliability of these estimates, one that is inadequately described by the straightforward notion of accuracy. To address this tradeoff, this paper examines the use of Receiver Operating Characteristic (ROC) curve analysis, a method that has a long history but is under-appreciated in the human computer interaction research community. We present the fundamentals of ROC analysis, the use of the A' statistic to compute the area under an ROC curve, and the equivalence of A' to the Wilcoxon statistic. We then present several case studies, framed in the context of our work on human interruptibility, demonstrating how ROC analysis can yield better results than analyses based on accuracy. These case studies compare sensor-based estimates with human performance, optimize a feature selection process for the area under the ROC curve, and examine end-user selection of a desirable tradeoff.
---
paper_title: K.: An analysis of gaming behaviors in an intelligent tutoring system

paper_content:

We present results from an analysis of students' shallow behaviors, i.e., gaming, during their interaction with an Intelligent Tutoring System (ITS). The analysis is based on six college classes using the Andes ITS for homework and test preparation. Our findings show that student features are a better predictor of gaming than problem features, and that individual differences between students impact where and how students game.
---
paper_title: Engagement tracing: Using response times to model student disengagement

paper_content:

Time on task is an important predictor for how much students learn. However, students must be focused on their learning for the time invested to be productive. Unfortunately, students do not always try their hardest to solve problems presented by computer tutors. This paper explores student disengagement and proposes an approach, engagement tracing, for detecting whether a student is engaged in answering questions. This model is based on item response theory, and uses as input the difficulty of the question, how long the student took to respond, and whether the response was correct. From these data, the model determines the probability a student was actively engaged in trying to answer the question. The model has a reliability of 0.95, and its estimate of student engagement correlates at 0.25 with student gains on external tests. We demonstrate that simultaneously modeling student proficiency in the domain enables us to better model student engagement. Our model is sensitive enough to detect variations in student engagement within a single tutoring session. The novel aspect of this work is that it requires only data normally collected by a computer tutor, and the affective model is statistically validated against student performance on an external measure.
---
paper_title: Empirically building and evaluating a probabilistic model of user affect

paper_content:

We present a probabilistic model of user affect designed to allow an intelligent agent to recognise multiple user emotions during the interaction with an educational computer game. Our model is based on a probabilistic framework that deals with the high level of uncertainty involved in recognizing a variety of user emotions by combining in a Dynamic Bayesian Network information on both the causes and effects of emotional reactions. The part of the framework that reasons from causes to emotions (diagnostic model) implements a theoretical model of affect, the OCC model, which accounts for how emotions are caused by one's appraisal of the current context in terms of one's goals and preferences. The advantage of using the OCC model is that it provides an affective agent with explicit information not only on which emotions a user feels but also why, thus increasing the agent's capability to effectively respond to the users' emotions. The challenge is that building the model requires having mechanisms to assess user goals and how the environment fits them, a form of plan recognition. In this paper, we illustrate how we built the predictive part of the affective model by combining general theories with empirical studies to adapt the theories to our target application domain. We then present results on the model's accuracy, showing that the model achieves good accuracy on several of the target emotions. We also discuss the model's limitations, to open the ground for the next stage of the work, i.e., complementing the model with diagnostic information.
---
paper_title: Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor

paper_content:

The research reported in this paper focuses on the hypothesis that an intelligent tutoring system that provides guidance with respect to students' meta-cognitive abilities can help them to become better learners. Our strategy is to extend a Cognitive Tutor (Anderson, Corbett, Koedinger, & Pelletier, 1995) so that it not only helps students acquire domain-specific skills, but also develop better general help-seeking strategies. In developing the Help Tutor, we used the same Cognitive Tutor technology at the meta-cognitive level that has been proven to be very effective at the cognitive level. A key challenge is to develop a model of how students should use a Cognitive Tutor's help facilities. We created a preliminary model, implemented by 57 production rules that capture both effective and ineffective help-seeking behavior. As a first test of the model's efficacy, we used it off-line to evaluate students' help-seeking behavior in an existing data set of student-tutor interactions. We then refined the model based on the results of this analysis. Finally, we conducted a pilot study with the Help Tutor involving four students. During one session, we saw a statistically significant reduction in students' meta-cognitive error rate, as determined by the Help Tutor's model. These preliminary results inspire confidence as we gear up for a larger-scale controlled experiment to evaluate whether tutoring on help seeking has a positive effect on students' learning outcomes.
---
paper_title: IRT model fit from different perspectives

paper_content:

Item response theory (IRT) is a collection of statistical models that used for development, evaluation, and scoring of instruments. IRT models describe, in probabilistic terms, the relationship between a person's response to a test item and his or her standing on the construct being measured by the test. These measured constructs include any latent (i.e., unobservable) variable, such as depression, achievement, or attitude, that requires multiple test items to estimate a person's level on the construct. The properties of these models offer many well-known advantages in testing applications. However, the extent to which these properties are attained is dependent on the degree to which the IRT model itself is appropriate. This thesis is concerned with evaluation of model fit from two perspectives: the items and the respondents.
---
paper_title: Automated Posture Analysis for Detecting Learner's Interest Level

paper_content:

This paper presents a system for recognizing naturally occurring postures and associated affective states related to a child's interest level while performing a learning task on a computer. Postures are gathered using two matrices of pressure sensors mounted on the seat and back of a chair. Subsequently, posture features are extracted using a mixture of four gaussians, and input to a 3-layer feed-forward neural network. The neural network classifies nine postures in real time and achieves an overall accuracy of 87.6% when tested with postures coming from new subjects. A set of independent Hidden Markov Models (HMMs) is used to analyze temporal patterns among these posture sequences in order to determine three categories related to a child's level of interest, as rated by human observers. The system reaches an overall performance of 82.3% with posture sequences coming from known subjects and 76.5% with unknown subjects.
---
paper_title: Labeling Student Behavior Faster and More Precisely with Text Replays

paper_content:

We present text replays, a method for generating labels that can be used to train classifiers of student behavior. We use this method to label data as to whether students are gaming the system, within 20 intelligent tutor units on Algebra. Text replays are 2-6 times faster per label than previous methods for generating labels, such as quantitative field observations and screen replays; however, being able to generate classifiers on retrospective data at the coder’s convenience (rather than being dependent on visits to schools) makes this method about 40 times faster than quantitative field observations. Text replays also give precise predictions of student behavior at multiple grain-sizes, allowing the use of both hierarchical classifiers such as Latent Response Models (LRMs), and non-hierarchical classifiers such as Decision Trees. Training using text replay data appears to lead to better classifiers: LRMs trained using text replay data achieve higher correlation and A' than LRMs trained using quantitative field observations; Decision Trees are more precise than LRMs at identifying exactly when the behavior occurs.
---
paper_title: Emotion Sensors Go To School

paper_content:

This paper describes the use of sensors in intelligent tutors to detect students' affective states and to embed emotional support. Using four sensors in two classroom experiments the tutor dynamically collected data streams of physiological activity and students' self-reports of emotions. Evidence indicates that state-based fluctuating student emotions are related to larger, longer-term affective variables such as self-concept in mathematics. Students produced self-reports of emotions and models were created to automatically infer these emotions from physiological data from the sensors. Summaries of student physiological activity, in particular data streams from facial detection software, helped to predict more than 60% of the variance of students emotional states, which is much better than predicting emotions from other contextual variables from the tutor, when these sensors are absent. This research also provides evidence that by modifying the “context” of the tutoring system we may well be able to optimize students' emotion reports and in turn improve math attitudes.
---
paper_title: Knowledge tracing: Modeling the acquisition of procedural knowledge

paper_content:

This paper describes an effort to model students' changing knowledge state during skill acquisition. Students in this research are learning to write short programs with the ACT Programming Tutor (APT). APT is constructed around a production rule cognitive model of programming knowledge, called theideal student model. This model allows the tutor to solve exercises along with the student and provide assistance as necessary. As the student works, the tutor also maintains an estimate of the probability that the student has learned each of the rules in the ideal model, in a process calledknowledge tracing. The tutor presents an individualized sequence of exercises to the student based on these probability estimates until the student has ‘mastered’ each rule. The programming tutor, cognitive model and learning and performance assumptions are described. A series of studies is reviewed that examine the empirical validity of knowledge tracing and has led to modifications in the process. Currently the model is quite successful in predicting test performance. Further modifications in the modeling process are discussed that may improve performance levels.
---
paper_title: Student Preferences for Editing, Persuading, and Negotiating the Open Learner Model

paper_content:

This paper describes a study where students were able to view an open learner model in seven formats. They were provided with tools to edit this model directly, to attempt to persuade the system to change it, or to enter into a negotiation about the model contents. Results indicate that many students are less comfortable having direct control over the content of their learner model than in situations where the system has the final say over proposed changes.
---
paper_title: StyLE-OLM: Interactive Open Learner Modelling

paper_content:

There is a strong argument in Artificial Intelligence in Education which advocates that computer-based learning systems need to adapt to the needs of learners if they are to provide for effective personalised instruction (Self, 1999a). Diagnosing a learner's cognitive capacity is a crucial issue in building adaptive systems. We have explored an interactive open learner modelling (IOLM) approach which conceives diagnosis as an interactive process involving both a computer system and a learner that discuss and together construct the learner model. This paper outlines the architecture of an interactive open learner modelling system and illustrates the method in a terminological domain. We discuss an evaluative study of an IOLM demonstrator - a system called STyLE-OLM. The results from the study demonstrate potential benefits of the method for improving the quality of the learner model and providing a means for fostering reflective thinking. We argue that IOLM is a fruitful approach which may be employed in intelligent learning environments both for obtaining a better model of a learner's cognitive state and engaging learners in reflective activities.
---
paper_title: Student Models that Invite the Learner In: The SMILI� Open Learner Modelling Framework

paper_content:

In recent years, the learner models of some adaptive learning environments have been opened to the learners they represent. However, as yet there is no standard way of describing and analysing these 'open learner models'. This is, in part, due to the variety of issues that can be important or relevant in any particular learner model. The lack of a framework to discuss open learner models poses several difficulties: there is no systematic way to analyse and describe the open learner models of any one system; there is no systematic way to compare the features of open learner models in different systems; and the designers of each new adaptive learning system must repeatedly tread the same path of studying the many diverse uses and approaches of open learner modelling so that they might determine how to make use of open learner modelling in their system. We believe this is a serious barrier to the effective use of open learner models. This paper presents such a framework, and gives examples of its use to describe and compare adaptive educational systems.
---
paper_title: Open Learner Models

paper_content:

An Open Learner Model makes a machines’ representation of the learner available as an important means of support for learning. This means that a suitable interface is created for use by learners, and in some cases for others who aid their learning, including peers, parents and teachers. The chapter describes the range of purposes that Open Learner Models can serve, illustrating these with diverse examples of the ways that they have been made available in several research systems. We then discuss the closely related issues of openness and learner control and the ways that have been explored to support learning by making the learner model available to people other than the learner. This chapter provides a foundation for understanding the range of ways that Open Learner Models have already been used to support learning as well as directions yet to be explored.
---
paper_title: Using problem-solving context to assess help quality in computer-mediated peer tutoring

paper_content:

Collaborative activities, like peer tutoring, can be beneficial for student learning, but only when students are supported in interacting effectively. Constructing intelligent tutors for collaborating students may be an improvement over fixed forms of support that do not adapt to student behaviors. We have developed an intelligent tutor to improve the help that peer tutors give to peer tutees by encouraging them to explain tutee errors and to provide more conceptual help. The intelligent tutor must be able to classify the type of peer tutor utterance (is it next step help, error feedback, both, or neither?) and the quality (does it contain conceptual content?). We use two techniques to improve automated classification of student utterances: incorporating domain context, and incorporating students' self-classifications of their chat actions. The domain context and self-classifications together significantly improve classification of student dialogue over a baseline classifier for help type. Using domain features alone significantly improves classification over baseline for conceptual content.
---
paper_title: Clustering and Sequential Pattern Mining of Online Collaborative Learning Data

paper_content:

Group work is widespread in education. The growing use of online tools supporting group work generates huge amounts of data. We aim to exploit this data to support mirroring: presenting useful high-level views of information about the group, together with desired patterns characterizing the behavior of strong groups. The goal is to enable the groups and their facilitators to see relevant aspects of the group's operation and provide feedback if these are more likely to be associated with positive or negative outcomes and indicate where the problems are. We explore how useful mirror information can be extracted via a theory-driven approach and a range of clustering and sequential pattern mining. The context is a senior software development project where students use the collaboration tool TRAC. We extract patterns distinguishing the better from the weaker groups and get insights in the success factors. The results point to the importance of leadership and group interaction, and give promising indications if they are occurring. Patterns indicating good individual practices were also identified. We found that some key measures can be mined from early data. The results are promising for advising groups at the start and early identification of effective and poor practices, in time for remediation.
---
paper_title: Tutorial Dialogue as Adaptive Collaborative Learning Support

paper_content:

In this paper we investigate the role of reflection in simulation based learning by manipulating two independent factors that each separately lead to significant learning effects, namely whether students worked alone or in pairs, and what type of support students were provided with. Our finding is that in our simulation based learning task, students learned significantly more when they worked in pairs than when they worked alone. Furthermore, dynamic support implemented with tutorial dialogue agents lead to significantly more learning than no support, while static support was not statistically distinguishable from either of the other two conditions. The largest effect size in comparison with the control condition of individuals working alone with no support was Pairs+Dynamic support, with an effect size of 1.24 standard deviations.
---
paper_title: The Big Five and Visualisations of Team Work Activity

paper_content:

We have created a set of novel visualisations of group activity: they mirror activity of individuals and their interactions, based upon readily available authentic data. We evaluated these visualisations in the context of a semester long software development project course. We give a theoretical analysis of the design of our visualizations using the framework from the “Big 5” theory of team work as well as a qualitative study of the visualisations and the students' reflective reports. We conclude that these visualisations provide a powerful and valuable mirroring role with potential, when well used, to help groups learn to improve their effectiveness.
---
paper_title: Detecting and Understanding the Impact of Cognitive and Interpersonal Conflict in Computer Supported Collaborative Learning Environments

paper_content:

This paper presents a model which can automatically detect a variety o f student speech acts as students collaborate within a computer supported collaborative learning environment. In addition, an analysis is presented which gives substantial insight as to how students' learning is associated with students' speech acts, knowledge that will significantly influence how this model is utilized by running learning software. Within Piagetian theory, the cognitive conflict of ideas between students is seen as beneficial for learning. Which sorts of interpersonal behaviors lead to most effective learning, however, is open to debate, with some researchers arguing that cooperation is most effective and others arguing that interpersonal conflict is a natural part of collaborative learning. We find that, in fact, interpersonal conflict is associated with positive learning, a finding that must be taken into account, in designing interventions that rely upon detectors of students' speech acts in CSCL environments.
---
paper_title: The Ecological Approach to the Design of E-Learning Environments: Purpose-based Capture and Use of Information About Learners

paper_content:

Abstract: The semantic web movement has grown around the need to add semantics to the web in order to make it more usable ::: by people and by information systems. In this paper I argue that even more important than semantics is pragmatics; that is, to really enhance web usability it is critical to capture and react to aspects of the end use context. Most centrally, to make the web truly responsive to human needs, we need to understand the "users" of the web and their purposes for using it. In this paper I elaborate this argument in the context of of e-learning systems. I propose an approach to the design of e-learning systems that I call the ecological approach. Moving from the open web to repositories of learning objects, I show how the ecological approach shows promise not only to allow information about learners actual interactions with learning objects to be naturally captured but also to allow it to be used in a multitude of ways to support learners and teachers in achieving their goals. In a phrase, the approach involves attaching models of learners to the learning objects they interact with, and then mining these models for patterns that are useful for various purposes. The ecological approach turns out to be highly suited to e-learning applications. It also has interesting implications for e-learning research, and perhaps even for research directions for semantic web research. Invited Commentary: Ramondt, L., Smith, T and Bradshaw, P. (2004) Commentary on: McCalla, G. (2004). The Ecological Approach to the Design of E-Learning Environments: Purpose-based Capture and Use of Information About Learners [PDF] Editors: Terry Anderson and Denise Whitelock.
---
paper_title: Interoperability in Personalized Adaptive Learning

paper_content:

Personalized adaptive learning requires semantic-based and context-aware systems to manage the Web knowledge efficiently as well as to achieve semantic interoperability between heterogeneous information resources and services. The technological and conceptual differences can be bridged either by means of standards or via approaches based on the Semantic Web. This article deals with the issue of semantic interoperability of educational contents on the Web by considering the integration of learning standards, Semantic Web, and adaptive technologies to meet the requirements of learners. Discussion is m ade on the state of the art and the main challenges in this field, including metadata access and design issues relating to adaptive learning. Additionally, a way how to integrate several original approaches is proposed.
---
paper_title: A design model for lifelong learning networks

paper_content:

Please refer to the original article: ::: Koper, R., Giesbers, B., Van Rosmalen, P., Sloep, P., Van Bruggen, J., Tattersall, C., Vogten, H., & Brouns, F. (2005). A Design Model for Lifelong Learning Networks. Interactive Learning Environments, 13(1-2), 71-92. Accepted Jan.2004
---
paper_title: T.: Modeling Individualization in a Bayesian Networks Implementation of Knowledge Tracing

paper_content:

The field of intelligent tutoring systems has been using the well known knowledge tracing model, popularized by Corbett and Anderson (1995), to track student knowledge for over a decade Surprisingly, models currently in use do not allow for individual learning rates nor individualized estimates of student initial knowledge Corbett and Anderson, in their original articles, were interested in trying to add individualization to their model which they accomplished but with mixed results Since their original work, the field has not made significant progress towards individualization of knowledge tracing models in fitting data In this work, we introduce an elegant way of formulating the individualization problem entirely within a Bayesian networks framework that fits individualized as well as skill specific parameters simultaneously, in a single step With this new individualization technique we are able to show a reliable improvement in prediction of real world data by individualizing the initial knowledge parameter We explore three difference strategies for setting the initial individualized knowledge parameters and report that the best strategy is one in which information from multiple skills is used to inform each student's prior Using this strategy we achieved lower prediction error in 33 of the 42 problem sets evaluated The implication of this work is the ability to enhance existing intelligent tutoring systems to more accurately estimate when a student has reached mastery of a skill Adaptation of instruction based on individualized knowledge and learning speed is discussed as well as open research questions facing those that wish to exploit student and skill information in their user models.
---
paper_title: Ontology-based Framework for User Model Interoperability in Distributed Learning Environments

paper_content:

This paper presents an update of our work on developing a distributed architecture for adaptive E-Learning. We introduce ADAPT – an extension of KnowledgeTree architecture that was presented at one of the past E-Learn conferences. To accommodate a larger variety of adaptive components, ADAPT employs a higher-level mechanism for ontology-based interoperability of self-contained adaptive Web-based systems within our architecture. We provide an overview of ADAPT, argue for the importance of ontology-based interoperability, and describe our implementation of this functionality in ADAPT.
---
paper_title: The interoperability of learning object repositories and services: standards, implementations and lessons learned

paper_content:

Interoperability is one of the main issues in creating a networked system of repositories. The eduSource project in its holisticapproach to building a network of learning object repositories in Canada is implementing an open network for learning services. Itsopenness is supported by a communication protocol called theeduSource Communications Layer (ECL) which closely implements the IMS Digital Repository Interoperability (DRI)specification and architecture. The ECL in conjunction withconnection middleware enables any service providers to join thenetwork. EduSource is open to external initiatives as it explicitlysupports an extensible bridging mechanism between eduSource and other major initiatives. This paper discusses interoperability in general and then focuses on the design of ECL as animplementation of IMS DRI with supporting infrastructure andmiddleware. The eduSource implementation is in the mature stateof its development as being deployed in different settings withdifferent partners. Two applications used in evaluating ourapproach are described: a gateway for connecting betweeneduSource and the NSDL initiative, and a federated searchconnecting eduSource, EdNA and SMETE.
---
paper_title: A Framework for Browsing, Manipulating and Maintaining Interoperable Learner Profiles

paper_content:

Learners are assessed by several systems during their life-long learning. Those systems can maintain fragments of information about a learner derived from his learning performance and/or assessment in that particular system. Customization services would perform better if they would be able to exchange as many relevant fragments of information about the learner as possible. This paper presents the conceptualization and implementation of a framework which provides a common base for the exchange of learner profiles between several sources. The exchange representation of learner profiles is based on standards. An API is designed and implemented to create/export and manipulate such learner profiles. The API is implemented for two cases, as a Java API and as web services with synchronized model exchange between multiple sources. Application cases of the API are discussed shortly as well.
---
paper_title: Learner profile management for collaborating adaptive eLearning applications

paper_content:

Adaptive Learning Systems would perform better if they would be able to exchange as many relevant fragments of information about the learner as possible. The use of Web Services standards is recently gaining the attention of many researches as a promising solution for the problem of interfacing adaptive hypermedia systems. Existing Web Service standards, however, only provide very basic features and leave out many important issues like transactional management. In this paper we propose a mechanism for enabling consistency maintenance of Learner Profiles shared between collaborating adaptive learning systems.
---

Title: A Review of Recent Advances in Learner and Skill Modeling in Intelligent Learning Environments

Section 1: Learner Models Basics
Description 1: Write about the fundamental concepts in learner modeling, including how tutors diagnose and assess the learner’s state.

Section 2: Learner Models and Learning Environments Success Stories
Description 2: Discuss the success stories of Intelligent Tutoring Systems (ITS) and the learner modeling approaches they used.

Section 3: Tutors for Problem Solving and Solution Analysis
Description 3: Review problem solving and solution analysis tutors, focusing on Cognitive Tutors (CT) and Constraint-Based Modeling (CBM).

Section 4: Cognitive Tutors and Constraint Based Modeling
Description 4: Explain the aspects and differences between Cognitive Tutors and Constraint-Based Modeling.

Section 5: The Role of Student Models in CT and CBM
Description 5: Explore how CT and CBM use student models for feedback and scaffolding and discuss Knowledge Tracing.

Section 6: Content Sequencing Tutors
Description 6: Describe tutors that guide students through learning material, focusing on systems like ALEKS and their curriculum sequencing approaches.

Section 7: Learning Models Revisited
Description 7: Discuss the sources of success in intelligent learning environments, focusing on fine-grained skill diagnosis and transfer models.

Section 8: Models of Skills
Description 8: Review probabilistic approaches to skill modeling and the importance of uncertainty in assessing skills.

Section 9: Bayesian Networks and Graphical Models
Description 9: Explain how Bayesian Networks are used for modeling learner skills and the associated challenges.

Section 10: Issues and Challenges
Description 10: Discuss challenges in using Bayesian Networks, including the difficulty of deriving conditional probability tables.

Section 11: Item Response Theory (IRT) and Latent Trait Models
Description 11: Describe IRT and its application in personalized assessment environments.

Section 12: Latent Cousins DINA, NIDA, DINO, NIDO
Description 12: Explain the class of latent models derived from IRT, including the DINA, NIDA, DINO, and NIDO models.

Section 13: Bayesian Knowledge-Tracing
Description 13: Discuss Bayesian Knowledge-Tracing (BKT), its parameters, and how it models learning over time.

Section 14: Models Without Hidden Nodes
Description 14: Describe non-hidden node models such as Knowledge Spaces and their application in learner modeling.

Section 15: Probabilistic Assessment Algorithms
Description 15: Review algorithms used for assessing the student's knowledge state within the Knowledge Spaces approach.

Section 16: Assessing Learner Models
Description 16: Examine methods for validating learner models, including dynamic student models and the challenges of predicting student performance.

Section 17: Affect, Motivation, and Disengagement
Description 17: Explore the extension of learner modeling to include students' meta-cognition, motivation, and affect.

Section 18: Meta-Cognition and Self-Regulated Learning
Description 18: Review models that focus on meta-cognition and self-regulated learning within intelligent learning environments.

Section 19: Open Learner Modeling
Description 19: Discuss the concept and benefits of open learner modeling, including learner control and negotiation.

Section 20: Group and Collaborative Learner Modeling
Description 20: Explain the models used in group learning settings and their application in collaborative learning.

Section 21: Long-Term Learner Modeling
Description 21: Describe efforts and challenges in creating long-term learner models that are shared across learning environments.

Section 22: Conclusion
Description 22: Summarize the advances in intelligent learning environments and learner modeling, and discuss the future potential and remaining challenges.