Investigating Performance: Design and Outcomes with xAPI

By Janet Laane Effron, Data Scientist HT2 Labs


The promise for both big picture and individual benefits was compelling to me. Despite my interest and research around how people learn, I am a data person primarily and keenly aware of the potential impacts of having the right data readily available. From work in competitive intelligence, huge information disconnects exist within businesses, and both individuals and entire organizations are hurt by the lack of easily accessed and analyzed information. Interviewing people across five roles in a company can end up feeling like one is speaking to five different businesses, not one. This has tremendous costs in terms of employee growth and satisfaction as well as organizational success. I saw xAPI as a way for organizations to gain deeper internal understanding – by seeing what learning actually impacts performance, there is the potential to understand what the real learning needs are to support success for employee and employer alike.

This ties into the more obvious potential for xAPI. We can cite cognitive science until we are blue in the face, talking about how to best support meaningful learning. At the end of the day there is the issue of accountability: we need to provide some form of assessment. It’s nearly impossible to implement new approaches to learning design if we are stuck with the same old metrics of test scores and completions. We can’t offer a better learning experience until we have better means of assessment. xAPI was the key that could open the door to more effective learning.

To build better assessment and to increase internal organizational insight, I thought a lot about how to design courses, data streams, and analysis methods to get meaningful data. At the same time, my co-author, Sean, was looking at xAPI as a starting point to do data-driven course design — the path to a powerful feedback loop. I looked at how to design for data; Sean was looking at how to design from data.

We realized that we were working on the same problems from different directions and, for both of us, xAPI was the starting point to support better learning opportunities.

During that weekend, conversation went beyond the promise of xAPI and moved into explorations of the potential obstacles and difficulties involved in getting started. Like any technological innovation, the mechanics of implementation are the easy part; it’s the strategic plan and subsequent activities which determine success or failure. From those conversations, the outline of Investigating Performance emerged. The initial concepts and content evolved over the next two years as we gained experience working with xAPI. In that time, we’ve had valuable discussions with other early adopters about what works, what doesn’t, and what we wished we’d known when we got started.

While the technical aspects of an xAPI implementation are not unduly complicated, and L&D professionals do not need developer level skills, understanding the fundamentals of the spec matter. Sean does an excellent job of translating the spec into plain English so that non-developers can use it effective. He looks into the important topic of profiles (the rules and documentation needed to implement xAPI for one’s given use case), and the associated vocabulary needed to assure that the data you collect meets your analysis needs.

The book covers design: designing for data, and designing from data. Designing for data starts with the big picture: evaluating business goals which underlie instructional design goals; looking at the needs of one’s learning customers and data customers to understand how to measure success. Designing for data needs one to be forward-thinking, concerned with what actions and decisions the data will inform, not just concerned with documenting what has already happened.

Designing from data is the other side of that coin. xAPI data can provide a wealth of information that supports nuanced evaluation of not just the users, but of the course itself. What elements of a course resonate with users? Are there design flaws or technical issues that impede user success? What elements of the course can be tied to improved performance? The data interoperability provided with xAPI plays a key role in being able to explore how our learning interventions impact the real world.

Given that data analysis is not a topic that is traditionally in the wheelhouse for L&D professionals, the latter part of the book walks through the fundamentals needed by practitioners to get started. Beginning with the basics of quantitative and qualitative data, we explore the foundation issues including content strategy, and the design practices including establishing and testing hypotheses. There is an exploration of content strategy; finding the intersection between the data you need and the data that’s available. We also look at the basics of analysis including data exploration, correlations, and common pitfalls. Data stewardship, having a plan for managing data collection, ownership, and quality, are all critical to a successful long-term data program. This is outlined in the final chapter.

Writing Investigating Performance turned out to be a powerful learning experience. It pushed us to explore questions more deeply than we would have otherwise; and really dig into the challenges and benefits of xAPI; not in a theoretical way, but from wrestling with data in messy real-world situations, and finding unexpected insights. Those insights came not just from our data but from the work or others, as is shared in case studies included in the book. Getting started with learning data is a lot easier when it is not a solo effort. We benefitted greatly from conversations with colleagues about real world concerns regarding data stewardship, and the value of data for improved course design.

Writing the book was a constant reminder of the intersection of the big picture and the personal. Using data to understand both learner performance and course performance fits into a larger ecosystem that is bigger than any learner or any course; and yet it is the awareness of that big picture that allows us to better serve each individual, and improve each course.

At the end of the day, the point of the book is not so much to be a technical manual, as it is a starting point to explore the strategic opportunities which xAPI offers to L&D professionals.

Janet Laane Effron works as a Data Scientist with HT2 Labs. She has a penchant for applying diverse fields such as cognitive science and competitive intelligence to develop best practices in learning design & developing meaningful metrics for performance analysis.   Current research includes the application of machine learning to improve course design and provide customized learner experiences.  Janet is a frequent speaker on topics including analysis practices for learning data, data driven design improvements,  and iterative learning design.

Speak Your Mind