Please be sure to say hello in the chat session. Tell us where you're from and maybe give us some of your impressions of the first day of the conference. That would be great. And if you haven't already, please follow us on Twitter using the Twitter hashtag. I've seen a couple Eden 20 and Eden 2020. If you have questions, please enter them in the Q and A area. We will not be following questions that are posted in chat. So please be sure to post them there. You can also follow the conference from the Eden website using YouTube. So good morning, everyone. We're just waiting for everyone to file into the room this morning. And before we start, we're gonna wait a couple more minutes. If you haven't already, please say hello in the chat box and tell us a little bit about where you're from. Maybe your impressions of the conference from the first day. Welcome to day two of the Eden annual conference in Tima-Sawara or virtually in Tima-Sawara. And please enter your questions in the Q and A area of the Zoom conference. This is where we'll be reviewing the questions and then feeding them to our speakers. Once they're finished, our speakers will be giving their presentations and then we'll have a couple of questions and if there's time at the end, we'll answer more questions. So please say hello. And if you want to know the program of what's coming up for today, please be sure to check out the Eden website as well as you can also access the live streaming link from there. It's going to be a big day today. There's a lot of presentations, a lot of workshops. Today is also the micro-HE day where people will be learning about the results of the EU project, looking at micro-credentials. So, and we're very excited to have our speakers with us today. So, all right. I think I'm gonna go ahead and start. We have a really great lineup this morning of speakers to talk to us about the topic of the conference, artificial intelligence. Our first speaker today is Professor Demetria Samson, who will be followed by Denise Whitlock and then Anthony Camilleri will be speaking. So, each speaker will have about half an hour to speak and then we'll have a little time for questions and answers and then we should be wrapping up around 1030. So, let me start out with Professor Demetria Samson who is engaged in teaching and research in the field of learning technologies and digital learning since 1996. He is the co-author of 350 articles in scientific books, journals and conferences and the editor of 15 books, 35 special issues in academic journals and 40 international conference proceedings with more than 6,000 citations. Very impressive. He's been director, principal investigator and or research consultant in 70 research and innovation projects. Professor Samson supervised 155 honors and post-graduate students to successful completion. Currently he leads an international university industry consortium called Learn to Analyze that promotes professional development and educational data literacy for online education and training professionals and higher education students. He is also the recipient of the IEEE Computer Society Distinguished Service Award in July, 2012 and named a Golden Core member of IEEE Computer Society and the Golden Nikola Tesla Chain Award of the International Society for Engineering Pedagogy. So, the topic of his keynote today is educational data analytics combining human and machine intelligence and online teaching and learning. And Demetria, the floor is now yours. Thank you very much, Ibiza. It was a really nice introduction. I will indeed talk about educational data analytics, but I've decided to adjust a little bit the topic to the current situation that we are all facing all around the world with the health emergency situation. So, basically I will talk a little bit, I will focus a little bit more on the educational data literacy, which is something that more practical use nowadays for most of the people who are engaged in teaching and learning either at high education or at school education. So, my presentation will be around educational data literacy, the challenges and the opportunities, especially under the current circumstances and the follow-up situations that we all expect in the different parts of the work, the changes that we anticipate that would happen in teaching and learning because of the experience of the coronavirus. I'll provide some definitions and then I'll show some examples from a project that we are running now in Europe related with educational data literacy. So, I'll start with the challenges and the opportunities of educational data literacy. Now, I think that even if there were some doubts in the past, today all over the world, everybody thinks that, everybody accepts that digital teaching and learning is indeed a key innovation for both teaching and learning at university, but also for a professional development and vocational training. And it's actually what has kept education alive and alive during this health crisis. So, to this end, blended and online courses are nowadays widely used to meet the needs of both a higher education vocational training students and in-service professionals, but nowadays also for a high school students as well. At the core of these courses, obviously are the course or the learning management systems. And yesterday, you had a presentation by Martin DiYamas. He presented the case of Moodle that most of the people are actually familiar with and it's actually one of the most widely used platforms in terms of course management systems worldwide, especially because it's open access. So, within this context, the design, the delivery, the evaluation and the redesign of high quality online and blended learning courses and learning experiences are nowadays at the core of discussion about education and training in really in most parts of the world. I receive invitations to both consult but also provide lectures and training in many parts of the world nowadays in this context. The two important professional roles that have actually become a little bit more visible and they emerge, of course in the US they were already visible and emerged for a long time but now in other parts of the world including Europe, they have these two professional roles that become quite important. It's the structural designers, the ones who design and develop online and blended learning courses and the trainers and tutors who can support and deliver these online blended courses. These are two new professional roles, not new really, but two professional roles that they have received lots of attention during this crisis and many people around the world they actually realize now that schools of education need to be able to prepare the students for these two roles. They're all of the structural designer and they're all of the professional trainers and tutors who can support online courses. And indeed, within this context new professional competencies are required for these two professional roles especially taking into consideration the current developments, the mass use of online and blended teaching and learning not only in higher education professional development but also in school education. A recent advancement in online blended learning has been the educational data analytics and you are quite familiar. Yesterday there were a couple of presentations on data analytics. Basically, it's actually the use of educational data which are generated during teaching and learning including also the different methods of assessment that we use for better supporting individual learners in online but also in blended courses. This becomes very popular nowadays especially with so such a large scale implementation of online learning experiences in school and higher education. I can tell here that I had the experience from working in Australia for the past almost five years. First full time now part time as Australian universities are fully online and we offer all our courses online for many years now and therefore there is lots of experience there and all these issues that we discussed today in some parts of the world like Australia and higher education are already a daily practice that we use. Coming back to the educational data analytics most of the course management systems including Moodle they incorporate educational data analytics tools. However these tools are not widely used in a recent survey that I did with teachers in different countries not only in Europe but in different continents as well in China as well and in Asia and in Australia. Not that many teachers in school education but also teachers at tutors at higher education are actually using the educational data analytics tools that are incorporated in most course management systems either Moodle or Blackboard or the others. And one of the reasons that they are not actually used is because the professionals who are supposed to use them they report having very low educational data literacy competencies through their professional development or the initial studies. So it is a fact that professionals either start from designers and trainers or even K-12 teachers and higher education tutors who use and adopt online or play the learning strategies they are actually lacking educational data literacy competencies in most parts of the world. What is educational data literacy is actually core competence for all educational professionals including school teachers instructional designers, tutors even educational institution leaders it's basically the capability of collecting and analyzing data which are relevant either at micro level for analyzing teaching and learning in your classroom activities or even in a metro and macro level related with taking decisions at a higher level. Nevertheless, most of the existing professional competence framework for educators they pay very little attention on educational data literacy they are missing out the potential of using these methods and tools in online and blended learning teaching. Therefore, there is a need and this is one of the pieces I bring in this presentation there is a need for extending the existing professional competence framework for educators with new competence that they can accommodate the emerging field of educational data literacy. There are also very limited professional development courses for cultivating educational data literacy competencies therefore there is also a need for developing professional development programs to develop but also assess educational data literacy competencies. So, I'll give some definition about educational data literacy before I move to what we have already done in this field. So, basically it's about collecting and organizing data that are either I call them profiling data profiling data are data that can help us provide either the students or even the learning environment in some cases even the teachers or the course and interaction data are the data are created by the interaction all and between the three different entities that they play a role in teaching and learning and the learning environment usually a digital learning environment and these data are both qualitative but also they are derived both with qualitative which is the most quantitative method quantitative methods are probably the most commonly used qualitative methods are a little bit more difficult but I think they are also very important for the educational data collection and analysis. So, data literacy another definition for education is the ability to understand and use data effectively so as to inform educational and pedagogical decisions. Education I use the different words educational for the macro and meso level and pedagogical for the micro level the classroom based decision. So, it's basically a set of competencies that help educators to locate, to collect, to analyze upon educational data from different sources so that they can support the improvement of teaching, learning and the assessment process. This is an overview of the data literacy for an educator. Yes? I'm so sorry I just wanted to drag your attention that you haven't started your slides. You keep talking only you I'm so sorry for this. But I thought that I share my screen. No, you didn't. I'm sorry I kept texting you but this is why I have to...  Yeah, because I knew that you had a presentation. So, maybe if you can quickly just go over the images. Yes, you can go now on the slideshow and just go briefly over them but don't... Yeah, yeah. Don't drag me before. So, this is the slides anyway but it's basically text. So, now this is a figure probably that people can see. Is that okay now Diana? Yes, that's perfect. You look very good. Thank you so much Dimitris. Sorry. Thank you. No worries. Because we tested before and I was sharing my screen and I had to redo that. So, educational data technology. There are different type of educational data technologies. They are teaching analytics which are methods and tools that can help you visualize and analyze and assess a teaching practice. They are learning analytics which most of the people are actually familiar with which are the methods and tools for collecting and analyzing reports student related educational data and mainly towards monitoring the learning process and the combination of the two which is the teaching and learning analytics which in my view is the most interesting part. The reason why is... The reason I'm saying that it's the most interesting part is because the only learning analytics part the analysis of the learning process without analyzing the teaching practice that is the context where the learning is taking place is probably giving us a very limited view of what is actually happening in the teaching and learning process. So teaching and learning analytics are the methods and tools that can support the process of reflected practice facilitating the teacher to actually go back and reflect on their teaching design using evidence from the actual delivery and the feedback that they can receive and the analysis of the actions of their students. So these are the three different categories of educational data analytics and tools as far as I see them today. I know that most of the people are talking about learning analytics. We have introduced the term of teaching analytics and the combination of the two. The reason why teaching analytics are useful is not only... and I will insist a little bit on that because I know that you have lots of input from the learning analytics side is not only for the self-reflection improvement that is their tools that can help you visualize the elements of a lesson plan to visualize and align a lesson plan with educational objectives and standards. There are also tools that they have some kind of evidence in here of the artificial intelligence part that can help to validate, for example, a lesson plan with potential inconsistencies in its design, especially when it comes to applying strategies like... or principles like for example alignment, things like that. But it's not only for self-reflection improvement and teaching analytics can help us especially in cases of mass large-scale adoption of online courses to share with peers your work, your lesson plans for example your activities in the classroom and your experience of these lessons with your students. So share them both with peers but in case of non-experience not so experienced teachers and that was the case actually nowadays in the past few months with the coronavirus crisis because many people they were quite... even experienced teachers they weren't very experienced in actually moving their activities online therefore the phenomenon was that we had many younger teachers with less experience in teaching and learning but more experience in using digital tools to arch as mentors to the more experienced teachers but not very familiar with digital technologies as a result the way that things have developed especially in Greece but in other parts of the world as well as I have observed is that communities of teachers were actually creating and therefore the teaching analytics to support each other and therefore tools like teaching analytics can help these communities to share what they're doing share their experiences but also to start working as peers in analyzing and annotating common teaching designs and have some kind of core reflection and share experiences in a more let's say transparent way so therefore I insist that I highlight teaching analytics probably something that can be useful in large scale implementation of online education especially in free education I'm not going to go into details on the learning analytics you have seen probably quite a few presentations and you will see many, many more I can highlight though that mainly I see learning analytics as a tool for building an update in individual student profiles keeping track of what an individual student can do and there's different type of learner data that we are interested in it's not only the engagement and the performance but also the emotional data that I feel are very relevant especially in the situation that we face in the coronavirus crisis emotional data became extremely important like stress or anxiety or boredomness within the context of online courses and I suggested to many people that this is probably a good opportunity for doing a little bit more research on this particular topic the use of the collection the use of emotional data of course overcoming all the issues ethical restrictions privacy so I still think that this is an interesting topic if there are people who are interested in doing more research I would love to get in contact with them on this particular topic so the different type of technologies that are used in educational data analytics are the script analytics where we can say what happened in the past based on analyzing the past data this is useful but not very useful the predictive analytics so we can predict future trends based on identifying patterns things like that from the analysis of past data but what is more interesting to be honest is actually the perspective analytics what we could do or should do should be a little bit strong what we could do so basically a recommender system that's based on analytics what kind of recommendations for actions can be provided based on the analysis of these data this is where the artificial intelligence algorithms are coming and that's why it's a very relevant topic for machine intelligence systems now within this context because I need to properly rush a little bit up within this context we have we developed a European project called Learn to Analyze it's about promoting educational data literacy it's an academia and industry knowledge alliance program that provides, brings together both academic partners and industrial partners the scope of the Learn to Analyze project was to enhance existing professional development framework for structural designers and tutors to include educational data literacy competencies in a response to the challenge that I described before and also to develop and evaluate a professional development for MOOC for promoting these competencies in school teachers and start from designers and heat trainers therefore the main outcomes of this project have been complete educational data literacy competence profile also a US snapshot of that and a MOOC with a certificate of achievement of educational data literacy under the European Commission and of course to generate a professional community around this topic in Europe and in the world this is the snapshot of course I'll leave the slides with the organizers so we can have access to the slides that will be as a file download in my presentation this is a snapshot of educational data literacy competence profile that we have provided with I'm not going to go in details here but for each dimension of competencies there are statements competence statements related to this dimension you can see that it's not only about collection of data management analysis but also about the comprehension of interpretation in terms of education the application, the use of this in order to take decisions in educational and educational decisions but also an important topic is actually the data ethics related to privacy and all the other issues that are relevant in when we use data especially in sensitive areas as education so within this context we have developed a MOOC we run this MOOC last year for 8 weeks we had more than 2,000 enrollments eventually and the interesting part was that almost 50% which is actually quite high of the people who enrolled actually completely got the certificate creating a community of around 1,000 people mainly from Europe but also in other parts of the world that they have this certificate and now we are in the process of redesigning this MOOC based on the evaluation experience that we had with the first version and running again but it will be after September because of the issues related to the coronavirus I know many people had other priorities so we was born to be the next version I keep receiving lots of mails even the next version is going to be from September so basically this MOOC had two basic parties that it has two different types of modules one is the theory a theory related to the educational data educational data literacy and the other is actually the practice and the practical side was for one week there was actually a practice in both information but also practice with the 3 learning platforms one was Moodle which is an open access and most of the group were familiar with that and two other preparatory commercial products from European industry the exact street the IMC's learning street of course these are commercial products unlike Moodle which is an open learning management system and there was also a method for evaluating the competencies of the educational data literacy and leading to a certification of that system these are the topics as you can see there were eight modules, eight weeks first week was the orientation the last week was the conclusion and then there were three theory modules one online learning supported by educational data one on learning analytics and one on teaching analytics and then three different modules on applying educational data using specific tools and I must say from the feedback that we received from these almost 2,000 people who participated in this book the combination of theory and practice was actually the strong point of this book that kept them active into the process and gave them the motive to complete and get the certificate we are now working actually on redesigning this and we are working with different Ministry of Education to probably include this competence profile and also move in some of the programs the professional development programs that they propose to research teachers or the professional development teachers so we have some discussions also outside Europe with organizations Ministry of Education translating being localised in the local languages in order to promote and use it in the professional development programs in South America and another one in Asia this is the team who work on this project you can see people from the University of Peraios my team from an origin University of Science and Technology from the University of Manheim Professor Dirk and then the three companies that I've mentioned before it's LATAS you're learning and I have seen from Europe innovation which is very important in Europe so this is where you can actually get more information about the Learn to Analyze project and I think that I have now 30 minutes I managed to use the 30 minutes and I will now stop my presentation is that okay I had 30 minutes I understand yes that's great Demetrius thank you very much you did a fabulous job and thank you for bringing the slides up and these will be shared on the Eden website so please no worries and we've recorded the session as well there's a few questions that have come in in the question and answer chat one of them you've answered somewhat in talking about your MOOC but maybe you could expand upon it a little bit further this comes from Jacques Lecavalier what can you say about the digital competency education competency framework I can't pronounce that right which does include some data analytic competencies is it not detailed enough not the right competencies and do you think there have been enough attempts to integrate your work with their project to avoid multiplicity that's a great question I think that it's coming to competence framework which is a much more general framework is not detailed enough that's that's that and our intention is not to create a new competence framework but to create a sub set of competencies related with focus on educational data literacy and then plug this E to different existing educational data frameworks like BG.com and therefore with Moodle who has used the DG.com and do we are discussing now to include this framework as part of their program for the assessment and the development of Moodle users framework and also we are discussing with people in the United States on the competence related with instructional designers because they have more specific it's only in the US that they have more specific competence frameworks to actually consider including what we have developed and learned to analyze there is a sort of set that extends the the instructions and competencies that's a very good question the other one is about educational data literacy part of the teacher's training which is the right place for this kind of training there are two dimensions and this is where we actually used it so far one is to influence the curriculum of schools of education that they prepare students new graduates on as pre-service teachers the idea there was that it would be great to have at least one or even more courses that are relevant to educational data literacy because this is something that it's a competence that needs to be included to the competence profile of the graduates of pre-service teachers so this is relevant to higher education and in Australia for example we are discussions with the teacher association because they want to emphasize on this topic the topic of educational data literacy but it's also relevant to teachers professional competencies especially for example I know that in US it's actually part of the requirements for getting your license to demonstrate educational data literacy competence abilities in your exam so that was a good question too and I think that this is the reply for that the other one is in online learning and LMS the learning analytics data are collected automatically by LMS the question is whether are least bit analytics what are the emotional data so indeed many data are collected but I don't think that these the systems they do collect lots of data I don't think that they have developed so far tools that can analyze this data and to the extent that it's actually needed by the different teachers in different contexts therefore there are lots of interesting dashboards for example that you can do related with the engagement mainly of the students probably also the performance but what we need here in the LMS is actually tools that can be customized by the individual teachers in a simple way so each teacher can actually customize his dashboard and collect the data and analyze the data which is relevant to its own teaching and at the same time for the leaders however in order to be able to therefore what I'm actually saying is that we have reached a kind of plateau in terms of the software development efforts and the LMS vendors like Moodle for example they want to receive feedback from people who are using it on what to do next however in order to be able to receive meaningful feedback it is expected that people have educational data literacy and that's why I'm saying that this is something that it's a step stone we need to actually do that in order to inform the users more about educational data literacy instead of train them through the tools and the functionalities of the tools train them a little bit beyond these tools so that they can come back and use the tools and provide ideas on what else do they need to be able to do from these tools this is what is actually expected from the vendors and that's why they have invested in being part of the project that I've mentioned because they want to learn more on the use of these tools and the potential future so there is a question about the MOOC finished 15 of April as indeed we extended until 15 of April we had a period of time the question is if there is a possibility to follow a self-paced there was a period of time for two months until 15 of April to have a self-paced mode we stopped it in order to create a new version so we will probably reopen the self-paced we haven't decided yet but there will be a new version from September anyway with the opportunity to get a certificate actually there was quite an interest in several parts of the world about the certificate the other question was about what emotional data are emotional data are the data that we can actually collect through effective computing by having for example a camera that can collect face expression through face recognition there is a lot of software from industrial partners that they do that there are not that many projects who are actually using this in education that's why I'm saying that it's an interesting topic for further research especially under the given circumstances called the large scale implementation of online education so there is another question about oops there are two open questions I see could you post a website address for the MOOC? yes okay it's actually on the learn to analyze website and there is a direct link there so you can get this from the slides and what type of certification is offered for the completion it's not actually micro-credential it's actually certificate from the project under the umbrella of the European Commission for completing this course we are in discussion with a couple of universities in order to actually have some kind of credentials from the universities but that's probably a discussion that will meet continually a little bit longer because of other priorities that they had through the crisis so I think that I answered so this is we are not yet for the first version of the MOOC on the micro-credential side but we are under discussions for that on the future so I think that I have replied to most of the questions thank you very much Demetrius for answering the questions and thank you for a wonderful keynote it was very eye-opening lots of really valuable information there so thank you everyone for checking in for this first keynote so early in the morning we are now moving on to our next keynote who is Denise Whitelock from the Open University UK Professor Denise Whitelock is the interim director for the Institute of Educational Technology she is a professor of technology enhanced learning and has over 25 years experience in artificial intelligence for designing, researching and evaluating online and computer based learning in higher education she recently led the UK's contribution to the adaptive trust e-assessment system for learning Tesla and she is currently the editor of open learning the Journal of Open Distance and e-learning and international recognition through holding visiting chairs at the Atanoma University Barcelona and the British University in Dubai her presentation today is called digital assessment and artificial intelligence practice and promise so very much looking forward to this presentation Denise and the floor is now yours please be sure to share your presentation just to be sure for a certain reason we lost Denise about five minutes ago I tried to email her so apologies for this just to see that it's me around here not somebody else speaking so I don't know either she lost internet connection or I have no idea what's happening but I couldn't find her in the attendees anywhere I'll try to email her because I don't have anything else maybe you can carry on with Dimitris keep speaking as he's still around Dimitris please find Denise where she is absolutely if there are more questions or anything else you would like me to elaborate I'll be happy to answer more questions are there any more questions would anyone like to address a question to Dimitris one of the questions coming from Paul are there any data that you think that we should not collect well that's a very good question I think that lots of data that we need to probably it's not about not collecting data it's about giving access to the data to certain people for example the ownership of the data by the individuals and their ability to decide which data we become visible to whom is actually a very important element Paul as far as I'm concerned I think that all data from that line of thinking all data should not be collected so or alternatively they need to they can be collected but it's only the individual user who is the one who creates this data who should have the right to release this data and to whom this data will be released and for which use will be used so this is a very important issue and I think that this actually touches the issue of privacy and ethics I think this is a very important topic and I know that many people in our field because they are coming from the technical side they are actually not paying too much attention on that but this is probably the most hot topic and I know that Torre has done some very excellent work in this field so the other is about assuming students can opt out for emotional data yes absolutely yes it's the question by Brita absolutely yes it's the same question actually the same thing the data are owned by the individual and what will be released and how it will be used it has to be under the control of the individual otherwise it's completely unequal one of the issues of course which I would like to comment because I had this discussion here with the Minister of Education is the role of the partners especially when we are actually discussing issues related with recording for primary school and for very young children so it's a very complicated topic and there are no guidelines or works there and that's why I think that it's another interesting topic for doing extra work issues related with the sensitivity of the data especially when it comes to such young ages the other is what can you say about models and lead screen particularly about predicting which well I'm not in favor of any of the tools I mean I'm not very happy with any of the tools that I have seen in all course management systems I think that they have very simplistic prediction algorithms most of the times and that's why I'm saying that this is probably the start of that they have done the LMSs but probably more work can be done only users are engaged in the process and they start defining their needs in a more accurate way so I think that the vendors have done what they can do and the software developers and the researchers and everybody but at the end of the day now it's a time for people to use it in a large scale and learn from that Dimitris I think this is also something that Martin brought up in his keynote just the real need for more social media and more interaction amongst the educators in sharing their best practices so Moodle is trying to do what they can do but as you pointed out it's not something that is really in their domain moving on to the next question can you show some examples of visualization of useful data? Before that I would probably like to comment on what you said Lisa that's what insisted on the teaching analytics part because it's actually quite useful for people to understand that there is a design behind teaching and learning and there is effect from the delivery of our design and therefore in order to analyze our teaching and reflect on our practice is of course it's common sense for all of us here but it's probably not very common sense for the typology, for the mass of the teachers who are working in the field and therefore the teaching analytics tools are probably useful to have that's why I highlighted this part of the education analytics tools but I think we have our next speaker yes we have unfortunately she lost internet connection and she's now from her phone so welcome Denise okay so I'll stop here and give the floor to the next speaker thank you so much thank you good morning everyone it's a pleasure to be with you can you see me yes we can see you Denise I want to share my screen there's a button at the very bottom yeah can you click on that just click on the arrow pointing up I just want to get this let me get this up can you see my screen now not yet I've clicked on it and I need to you can't seem to see my screen now there is another share blue button in the right corner of the screen which comes up after you select what you want to that's what thank you good morning everyone it's a great pleasure to be with you this morning I'm so sorry I'm late my internet went down we're all suffering this in this new COVID-19 situation now underneath my look I'm the director at the moment Institute of Educational Technology at the Open University in the UK and we have 170,000 students and this is my building and IT the Institute of Educational Technology was one of the first departments in this new way of working this new university I've been working in AI for a long time now and I want to talk to you today about some of my own work and others work and about in digital assessment and how I want to take the student's point of view the teacher's point of view researchers, awarding bodies software developers and disruptors so this is how I'm going to run the presentation this morning I think it's very important when we're talking about assessment to think about the students and this is some work from Chris McKillop, a doctoral student who wanted students to write narrative stories about their feelings about assessment she didn't get very far but when she asked them to draw this is what she found and here's another picture from her collection assessment is very stressful for students and we need to bear that in mind and of course what happens to us in our teaching I would say if I asked you and I was in the room I wish I was with you all I asked you what's your way of working and your teaching you'd probably say you're all constructivists so we're moving pedagogy forward but also we work within institutions that are awarding bodies from the institution all this AI work learning analytics work that we heard from our colleague earlier I think the grand challenge for all of us is representing in some way an analysis of learning that can be readily understood by ourselves the teachers and also the students now here's an example of Ali Fahler's work an ESC program this is early AI stuff and what she was doing is there was a call from the teachers who were teaching Spanish in an English university and they had many thousands of students and they couldn't have enough tutors to respond to the students quickly enough and you know I'm speaking to you I'm a native speaker in English I'm sure a lot of you are not native speakers and you've learned English and it's very difficult if you make a mistake or you start making mistakes and they're not corrected early then you get this fossilization so what she did is she set up this program where students could start translating and they had two attempts if you can see on the slide attempt one they got 57% some feedback and then they tried again and so this is an early AI program but what's important there are lessons important here for us even now so if you want to give some feedback during a test that it is best if there are two attempts to unequally wait the marks and to give more weight to the first attempt otherwise the students won't try hard enough and they found that and so the final mark really depends on how much help and supports being given and so therefore you know it opens the agenda to people that you can give feedback during a test and change the marks accordingly and so I mentioned earlier students have a lot of anxiety about testing and how do we maintain our empathy with the learner and to give feedback that I've been writing about which I call advice for action and goodness look somebody's tried to write all their answers on their picture on their hand because they're so nervous and they have a passion of anxiety so how can we help students with their feedback reduce their anxiety and motivate them and I was given a task at the university to try and write some a program and I did this with Stuart Watt so that students in the arts faculty didn't want just multiple choice questions but they wanted to write something in some tax these were early days and get some feedback and before I started doing this work I was very impressed by the work of Carol Dweck and her work I'd like to share that with you where she realized in her research that praise often means to a student that they're clever and negative feedback often means that I haven't got any ability I can't change and I can't do very well and Dweck did this particular work in the United States with Muller they looked at thousands and thousands worked with thousands and thousands of school children in the US and gave them Raven's tests these are non-verbal reasoning tests and they set up three tests with the students and it was a randomized controlled child so some of the students were praised for effort some for ability they took a second test and then the third test and what was important here is that the students who were praised for effort scored one point more than those for ability so what does this work tell us and what is Dweck telling us she's saying your intelligence is something very basic with people and people when they receive feedback can believe that their intelligence is very fixed and it doesn't matter what they do and if you have a gross mindset you actually believe that you can try a lot of work is trial and error and that you can change and you can get better and what does a fixed mindset look like in people that you're working with often they're very super sensitive about being wrong and they're always trying to prove themselves whereas with a growth mindset people are more willing to stretch themselves confront obstacles that become challenges and there's a lack of tension when learning so how could we promote this growth mindset and start praising for effort and this is what we did in a program called open comment and we worked with the history department and here you see an example of a question in history we often ask we're always usually teaching about the first world war and here I put in they asked a question I put in the answer no idea and then I got some feedback but what I want to share with you now is what's underneath the program and how I tried to organize it because what I had to do is I had to work as a history student and work with the professor and try and understand his feedback and to build the system so what we did is the first phase the program looks at your input and tries to detect errors but more importantly we took on board Dweck's work and you can see underneath here there will be something response to you as a student you've done well to start answering this question but perhaps you must you misunderstood it instead of thinking about X consider Y and the next part of the program it starts to look at omissions things that you've left out we might have left something out and we'll give you feedback about that but always praising what's correct to point out what's missing showing the effort and then in the final stages of the analysis we'd look for clarification inferencing and often when you mark the work yourself you know students present the facts but they don't tell you why what is the inference what reasoning message should you take from what they told you now you know our systems aren't that clever and we had to check we only had questions that included causality because we could manage the programming in that way and the final stage we asked we looked to see where the causal factors weighted now you might think that's a bit of a strange thing especially if you're a scientist what would be happening in a history essay and what's was surprising to me was that when I started to answer the questions in the essays the students did myself I could only get a C grade and then I found out how I could get an A grade and that was because I was talking about the causes of the first world war and I wasn't waiting the factors I wasn't saying which cause was more important than another and once the students understand that they can do very well so we said about and I showed you at the beginning those diagrams that the students have written and drawn to show their emotional difficulties with assessment and it's very important to think about how can I support the students with the written feedback so what we did was we looked at the feedback student tutors had given in the open university on the assignments and I thought how can I code those comments it's very difficult to set up a coding system it takes years and I found Bayles was a psychologist in the 50s and he built this system to look at group interaction and what he found is that in any group that works very well you need a cognitive leader and you also need someone to provide sociomotive support and that could be in the same person and if you look here there are four groups categories A being at the top of the screen you'll see positive reactions at the bottom D there are the negative reactions B are the attempted answers which if you look at them closely are things that we do when we're teaching and there's also a set of questioning what I did was analyse thousands and thousands of scripts with the feedback and this is one particular open university if you look at this chart you'll see on the left hand side the A B, C and D categories that we saw before representing those that are in the Bayles group and then you'll see pass 1, 2, 3 and 4 pass 1 is the highest pass given to that assignment now let's look at just the bars for the A group, the praise you'll see there's more praise for passes 1 and 2 the highest marks given and less praise for the lower marks that is not unreasonable if you look at the teaching comments you'll find that there are more teaching comments given by the tutors in the lower grades the lower passes 3 and 4 less in the higher passes and again this is very reasonable if you think about it because if the student has a lower mark the tutor has to give them more feedback to help them get better let's move to the group C I believed that the students who got the pass 1 the best marks should have received more questions from the tutors encouraging them asking them pushing them to do better and to explore different areas in fact I was completely wrong there and that was due to the English language what we do and what tutors tend to say to their students if they haven't included something have you thought about including X in other words being very polite instead of saying this is something you should have included and so that was the change the unexpected result here but more importantly I think look at the D group and we train our tutors obviously we don't see our students they don't see us only in certain tutorials we don't see them day by day and there is less negativity which we were very pleased about we would be very worried because we do train our tutors to give feedback so with bearing that in mind we built this system called Open Mentor it strips out all the tutor comments that are typed in to the assignment and it classifies them and you can see examples here but more importantly from that last analysis here we built an algorithm so that for the mark awarded we could have an algorithm that would tell you what the number of comments you should see from the tutor and so here this is an assignment that's been put in assignment for Sandy Smith of course this is a made up name this isn't the student's real name but they were given 61 marks mark assigned 61 and if you look at the bar chart below you'll see for group A that's the praise the comments of praise that the tutor gave a certain number that are in blue, that's a blue block and the ideal number from the algorithm so they were less, they gave less praise than you'd expect for this mark they gave less teaching comments i.e. category B and less questioning and more negative comments so you could see how they were marking and how they were giving feedback we were not there to give feedback about the marking but about the feedback itself to help the student do better so this is a program that was developed and then obtained GIST money and has been used by the OU for training our tutors but also in Southampton University and Kings College, London I'd like to move on to some other work which I think is much more important now to think about in the times of Covid where this was work funded in the UK also with Oxford University and the problem that I tried to solve here was that as you know, in distance universities we have an open university you don't have to have qualifications to come students might not have written essays for years and they don't get support writing that first essay, their first assessment and I started to work and I wanted to work with Stephen Pullman from Oxford because he was an expert at summarisation and the reason I wanted to work with someone with that capacity is because if I understand what you've told me I can actually say it back to you and that was a theoretical stance I wanted to take as from the work of Gordon Pasc conversation theory which I studied and I was a doctoral student so we built a system that summarised the students essays and it actually looked at the structure the key words and key phrases and key sentences we were using natural language processing and here we can see how the feedback is given to the student, there are hints it splits up the essay into introduction evidence and conclusions it does different types of analysis and gives you different types of graphics one of the most important and valuable feedback part of this that the students found valuable was this keyword dispersion graph and here you can see the words that are used in the introduction of an essay and in the conclusion and that we know with a very good essay, an essay is a story with evidence so it tells you in the introduction what it's going to talk about and then it explains what all that meant in the conclusion and if the introduction and conclusions don't match to some extent then the essay is not working well so this was used we used this with our MAODE students or master's students in open and distance education they had to use the open essayist for essay one and essay two we found there was significant differences to their marks from using this system and the mean grade for the overall module was higher with students who'd used this system the year before with significant differences but I'd now like to show you something that we didn't try with this particular group what we built a bit later and if you look here I've shown you some text just a very small amount of text you can see that there are some sentences in colour and I'm going to show you how that text moves itself into a sentence graph and it shows you with the different colours the different parts of the little piece of text how connected those sentences are together and from this we built the rainbow diagrams the red nodes are the conclusions the violet nodes are the introductions let me show you if you put in ten identical paragraphs you get a pattern that looks like this now if you have 50 identical sentences you've got more of a globe representation but then we put in from Stanford University one of their prize boost essays now look now you can see that the whole of the phrases and sentences are very well connected the introduction the violet nodes and the red nodes are very close together and the rest of the essay which is the evidence of that essay are very well connected so we put in this essay from one of our students with a high grade obviously not as good as the Stanford Booth essay but we can see the connectedness and we can see that there are parts of the story not well connected they are on the periphery and here's a low grade where we can see less connectedness what we did then was we actually put 45 students to write two essays and we put them through the rainbow diagrams and we found that we could by eye look at these but we found that the differences that essays rated as high by the rainbow diagrams would be expected to receive 8.56 percentage points more than essays rated as medium and 17.2 percentage points higher than essays rated from rainbow diagrams as low a very important finding and we've been using this with our students in helping them to write essays giving them feedback formative feedback to improve the connectedness of their arguments I want to talk about now just one final example of work just recently a European project I was involved with again very important in this COVID time because it was about trust adaptive trust assessment and it was about securing reliable online assessment using a range of technologies voice and face recognition keystroke pattern detection as you know we all have our own way of typing that can be detected plagiarism and forensic analysis in this raft of technologies all put into one system and there were many partners in this project but these are the partners who actually tested the systems and we've looked at thousands and thousands of students testing these systems in Europe and the subject areas we looked at good range of subject areas in the arts and sciences social legal sciences health sciences engineering and architecture and we ran three pilots and you can see the types of assessments that we put through from these different areas and the language of the courses it wasn't just in English we had Turkish Finnish ourselves at the OU in the UK Dutch partners, Spanish partners Bulgarians as well trialling the system and we used a range of activities online essay writing short answer quizzes planning there was coursework reading, reviewing all sorts of different assessments on the whole there was a very positive experience but there were real challenges about people giving away their facial recognition their pictures of themselves and their audio but we were GDPR compliant the way the system was but giving away personal data is a difficult thing to do we're not always ready to do that but the most important finding I think was that the students recognised that we would be moving although this is pre-COVID just before last year that we would be moving more and more into e-assessments and final exams being carried out in this way and that they felt that students should not be able to plagiarise other students because that would make a difference to their results and that their institution needed to be trusted by employers that if they were undertaking this type of assessment that everything was proctored and invigilated correctly finally I'd like to talk about disruptors in our area of course there are commercial companies who are bringing AI into the market there are private online learning institutions so there will be different ways bringing new ways of working and government policies governments want shorter cheaper courses we're talking about micro-credentials and of course COVID-19 will be a major disruptor when people have to move online when they're not used to moving online and of course in this picture here you can see now he's a robot we've got in our lab at the computer science department which I've been working with and we've already got chatbot recommender systems and they are becoming better we are linking some of our feedback with Alexa so that you can query our systems our courses and we can direct you to a point in the text that you want to learn about you want to know about and how about language rehearsal with your own robot who won't be tired who will listen continuously maybe we're not at this phase next but I can take questions but it won't be 3 million in 5 minutes thank you for your attention and I've left you with some references and I'm sure my slides will be distributed thank you for your attention thank you Denise we really appreciated you coming in and also with the technical challenges that you faced that we were able to overcome there are a couple of questions in the chat box that perhaps I will direct to you one of them is many schools in Europe still have graduation exams when school students are tested after 10 to 12 years of school learning and the exams which are very stressful decide their future learning and careers we have a situation like that in Germany sometimes they cannot have the second chance even after a decade school graduation is engraved in their life determined by their diploma how would you comment on these practices and what would you recommend for the policy makers in education in these countries let's think about what the policy makers want we want a well educated workforce that is confident and people don't always do well at school and you heard me talk about Dweck's work and they don't feel they don't feel they are good learners and they can learn and improve so we have to look at ourselves and how we teach how we give feedback the sorts of tasks we give our students we need to the Tesla project helped us think about new ways of testing new types of tests where people can shine show their ability because some people are very nervous under terrible these test conditions I myself was immensely nervous I was with you for 24 minutes my internet went down I had to ring someone who looks after our internet in our house and they said we can't fix it we turn off the routers and nothing you must go to your internet provider so it was absolutely terrible but I had to overcome it and I couldn't even see your faces when I was talking so was that a good test well I managed to pass it I think somehow but for somebody else that might have really shaken them especially they've been a younger researcher so why don't we think about the range of tests that we want to give our students they shouldn't all be the same and they should be given at different times and to show and to try and bring students to the end of their studies where they know it is not just ability that if you do a thesis you know it's 10% inspiration 90% perspiration and it's the people who have who continue and keep going that come out in the end Thank you very good point about the growth mindset it's really important to start teaching our policymakers a little bit more about the importance of the growth mindset within our curriculum I have another question from Raza Greenspaugh and that is hello Denise I've been analyzing the concepts of assessment for learning and formative assessment and I found that some scholars think that they are synonymous while other researchers argue that and list certain criteria in which these assessment strategies differ what view do you support what is formative assessment different from assessment for learning Thank you For me formative assessment isn't any different for assessment for learning that's what it should be and that's why the type of feedback is so important and I showed you Open Mentor where we were trying to help tutors to give very good feedback and for me I've been writing about what I call advice for action so that feedback must help you move forward help you change and practice maybe what I'm really concerned about is that students don't have time to practice too much something they've just learned we wouldn't go and take a driving test after a couple of lessons we would make sure and our instructor would make sure that we were safe on the road and I've had students say to me sometimes if only I knew the questions were going to ask me before even in a class setting I would look cleverer I'd look better and I think assessment for learning is formative assessment so you can practice and you should be able to practice and like that essay Open Essayist you could practice and what we found was that when they had that tool students didn't give in their last minute attempt they did practice using the tool now you could say the tool isn't that smart well it isn't maybe it was the practice but does it matter because you need to practice you need feedback and you need confidence to be able to ask someone to give you who you trust to give you some feedback I have a question and this is based on my experience working with educators who have really large numbers of students in their classrooms how can we promote that growth mindset within classrooms that have these super high student enrollments for example University of South Africa this is where I think AI's got a big role and I showed you some of them as I've been working in AI over a number of years because what we need to be moving to is personalized feedback and we can do that with AI techniques it's not perfect we're getting there our previous speaker talked about the learning analytics predictive analytics we use this ourselves at the OU but then what are the measures the pedagogical interventions that can take place but that's a move but also what we can do more simply is for example we've got a lot of people here today with a lot of chat going on right and in MOOCs and in future learn our micro-credentials how can a tutor with thousands of these chats respond so the open essayist can actually analyze all that feedback and summarize so they can summarize where we are in the chat and how you can respond to that so the main issue that the students are having trouble with is this but ideally we all want a personalized tutor yes the next question I have is from Katrina and her question is are the systems you developed are they available for other institutions to build in their feedback and assessment and are any of the assessment systems yeah this is the same question the open essayist is in Github it's open source so you can go to Github open mentor you can contact me and I can see if I can give you a time to use it and Tesla that system you go to the website we have a package there where you can have part of the system and you can use it so and I think all I know in the UK if you have any funding now it must be open source okay thank you very much Denise are there any additional questions we have a couple of minutes if anyone would like to ask an additional question if not we'll move on to our next speaker okay well thank you again Denise it was a very informative presentation and it's great to see the research that you're doing I'm very happy you were able to overcome the technological challenges this morning thank you Lisa Marie so our third speaker today is Anthony Camilleri who is the director of the Knowledge Innovation Center in Malta Anthony's expertise is in quality assurance processes and knowledge transfer of research he's given training on techniques for peer review to quality assurance agencies across Europe for knowledge transfer he has worked with organizations to better describe their research outcomes through their communication channels he's also developed a methodology for improving impact measurement of dissemination and exploitation activities within EU projects Anthony was previously engaged as quality services manager for the European Foundation for Quality in e-learning FQL responsible for coordination of projects linked to OER including OER test and VM pass and he's going to be talking to us today about the introduction to the MicroHE project final conference micro credentials in the future European policy landscape so Anthony the floor is now yours thank you I just do a quick audio test make sure you can hear me fine I see nothing so thank you for the introduction so being that I am introducing a session I thought it would be useful to contextualize the discussion we're having on micro credentials all of a sudden this seems to be one of those crazes in education that is on everybody's lips and if we're going to talk about contextualizing any European policy initiative at the moment it is important to recognize the context in which we are operating namely and simply this spring the world changed I don't think I have had a conversation in the past four months that didn't have a reference to Covid or to corona suddenly all of us have become data analytics experts looking at the latest numbers and higher education in particular has been called upon to go above and beyond doing everything from providing distance learning to masses of students overnight to research arms being asked to produce vaccines to all kinds of innovations and things which we thought wouldn't even be possible two or three months ago and if you were to characterize this as an expression you might say that higher education is operating in a world changed utterly and hopefully those of you who are more perceptive will have noticed that there's a small problem with this article and that is that it's dated 2010 because as a matter of fact the last time we were saying higher education has to face new challenges and be more flexible and rise to the challenge and be more employable and you're all familiar with this set of trends was the last ten financial crisis hit in 2008 and part of the response at that time was the famous or infamous year of the MOOC which was famously declared by the New York Times in 2012 and if you brought in to all the hype that was around that MOOCs were supposed to give a student choice, they were supposed to allow for increased employability they were supposed to allow for reskilling and the more you read into it it was supposed to be part of the solution to the dark times we were living in and to the reduced opportunity of people in that crisis and when we're talking about this it is good to keep in mind the perspective of what we might call millennials and simply enough if you're from my generation which I just barely qualify as a millennial we are now facing the second once in a lifetime economic crash in the period of ten years and if we look at the results the last economic crisis you find that people are actually less will-off than earlier generations they have lower earnings fewer assets, less wealth fewer children, fewer marriages it has had a real concrete effect and that was the result of the one in 2008 now we're coming up to the second one and these people are going to get hit a second time so it would be worth asking do we as an educational establishment have a good answer a good value proposition for them and if we look around at the things that are being posted we find that MOOCs again are being touted as part of the answer and that people are showing increased numbers in MOOCs so these are just some class central rankings for MOOCs and basically what you're seeing is that in March and April the numbers of sessions of all the large MOOC platforms increased dramatically everything from 50% all the way up to 400% in terms of increased hits so my question over here would be are we quoting Peter Pan saying that all this has happened before and will happen again are we just repackaging solutions from 2010 for the modern crisis out of a dearth of better ideas or are micro-credentials something fundamentally different and something fundamentally new that will have a difference on education systems and before I jump into this why I would like to change this one thing which I would ask you to consider is one of the MOOCs that was launched in the past six months which was a MOOC on COVID-19 offered by Imperial College London the first version of this had 70,000 users it's being re-offered and at the moment it has something in the region of 70,000 people subscribed it is a 30-hour MOOC and I ask you to consider where else in the history of higher education have we been able to progress in a matter of weeks deploy it globally and have 200,000 people listen directly from the experts in the field in response to a public health emergency this was not even technologically possible 10 years ago and the fact that now let's say we can produce this with this kind of speed and this kind of accuracy say something about how the MOOC and the short learning program industry have grown up over the years so my question is are micro-credentials just rebranded MOOCs or are micro-credentials the crooks of 30 years of educational innovation and to consider that we need to jump into what we consider micro-credentials to be and one of the fun things about micro-credentials is that nobody really agrees on the definition so depending on who you ask micro-credentials can be anything from open badges to very, very formalized education units of about 5 to 10 years to private learning programs so there are a lot of definitions out there and if we're going to talk about a European landscape of micro-credentials we really need to settle on one of these but the one thing everybody seems to agree on is that a core element of micro-credentials is that it's made up of a system of interoperable building blocks and this concept of interoperability and of combining different micro-credentials with each other is the core difference between micro-credentials and what came before to some extent you might say that micro-credentials represent a more mature vision of education than books or you might rephrase that and say they represent a less libertarian vision of education than books and what we mean by this is just putting content out there hoping that people will find it and that people will put together personalized and that those personalized learning pathways will miraculously become recognized without much intervention is very, very idealistic but not necessarily the way the world works on the other hand taking that same that same set of open education modules that same set of opportunities and creating pathways in which they can be combined and creating structures that allow for that combination can become much, much more powerful and be a bridge between let's say the very, very open ideals of open education and the much, much more formalized ivory tower arguments you are used to hearing so in a way micro-credentials are trying to occupy this middle ground between each other and linked to that micro-credentials can be said to have five features the five features generally are considered to be modularity which means that micro-credentials can be grouped into small units stackability which means that they can be combined and the smaller credentials can be combined to make up larger credentials portability which means that you should be able to take micro-credentials earned at one institution and use them in another institution digital and here we mean that the credentials themselves are digital and universality which means that those micro-credentials should be available in different educational contexts and different education the other unique thing about micro-credentials is that if we think about an educational life cycle curriculum course specifications are set by educational institutions and they are the input which goes into a course courses and programs are taught by the educational institutions and the combinations within courses and programs are also controlled by institutions credentials are the output that comes after you have finished your learning and credentials are held by the student and are controlled by the student and when we talk within the sphere of micro-credentials of stacking and combining we think of the student being the one who is the final say and the final control about how to combine these input portfolios so the focus of learners is also one of the key features when we talk about it and so what I would ask you to consider is how are micro-credentials like electric cars and if you think how are micro-credentials like electric cars I would argue that if my PowerPoint will work the concept has been proven so MOOCs and short learning programs and also non-technology non-learning short programs such as summer schools have been proven as a concept they are known to work the technology behind them and the technology that powers them has also been proven we are able to easily broadcast this to tens of thousands of students without too much trouble there is a clear social imperative for flexibilizing education there is public demand for these courses as shown by the statistics I was showing you before of increasing load numbers we've also proven that the micro-credentials are in the ambit of new exotic providers only we've proven that legacy providers are able to provide micro-credentials and they are relatively easy to get you can go online and find a whole ecosystem of micro-credentials and finally they enable new types of educational mobility so how also micro-credentials like electric cars they also share some of the same problems in that they're not accepted everywhere they're not fully trusted they're not always interoperable and they face issues around scaling also they compete with legacy systems of let's say more traditional forms of degrees and if we are to actually help micro-credentials reach their potential we have to figure out how to address these challenges at our policy level and to to do this we first need to understand the prevalence of micro-credentials and if this is something that institutions are actually interested in and the answer to this a quick news a quick scan on news sources this is just a few selections the European MOOC consortium launching a common micro-credential framework made up of some of the largest MOOC providers this one is from yesterday the Australian Government building a 4.3 million dollar online micro-credentials marketplace the European Commission getting in on the act and putting a together a consultation group on micro-credentials in higher education so we see that there are a lot of actors which are putting their heads together and trying to address some of the issues around it so a quick and the reason this is a hard problem is because of the scale so just playing around with some numbers let's assume that micro-credentials become widespread tomorrow and let's assume that we have a thousand universities or educational institutions offering 50 courses per institution that would mean that we would have a European market of 50,000 micro-credentials and if we assume that we are going to combine those into five credential packages that gives us two six two impossible combinations that is an order of magnitude of complexity above what we are used to in education and the essential question is how do we move from a higher education system that is used to dealing with a limited number of known quantities in the forms of degrees and qualifications how does that system start dealing with these kinds of numbers around micro-credentials to bake the problem down a little more it brings adoption challenges for students because in that sea of credentials how do I as a student find the quality resources that will help me and help my career from a higher education institution perspective how do I recognize the credentials from other institutions and systems that might not be exactly like my own and from an employer's perspective how on earth do I assess the CVs of students coming in with all these micro-credentials each of them with a personalized learning profile how do I find the time to actually go through these and understand what each student has learned so micro-credentials offer some very very real challenges which are not necessarily straightforward to address helping us though we do have a whole bunch of drivers of adoption first of all open source technologies open source standards on how to describe education on how to secure credentials on how to share information between student information management systems all of these are things that help us deal with large volumes of data and education if we want to get fancy there are also emerging technologies such as AIs such as blockchain that can help make these things even easier we're seeing the emergence of common European standards everything from the Europas standards to the ELMO standards to IEEE standards to ISO standards who are all looking into how can we standardize the data that is in educational credentials we're also seeing the beginning of national strategies for qualifications national strategies for micro-credentials being offered in some countries and we're seeing the beginning of some noise around micro-credentials in the Bologna process and some indications that the topic might be taken up there so we do have a lot of technology and education policy activities coming together to try and address this field the question I would ask you then is what is the to-do list for European policy in this area and I would propose that there are some straightforward items that need to be dealt with if micro-credentials are to reach their true potential the first and simplest is that we need a common definition of micro-credentials it's high up on the agenda of most of the groups that are looking around this but at the moment micro-credentials are widely defined as anything less than a degree and the current definition is not necessarily useful a common definition is important because a common definition allows you to describe a unit which can be traded a unit that can be described and once we have that common definition the second thing we need to do is standardize module structure and descriptions what we mean by this is that we need some kind of credit system some kind of trading system that allows us to actually put these micro-credentials together that allows us to carry them for some that allows us to measure learning a person has actually done in what areas and since education isn't something that can be measured like currency we also need standardized descriptions to go along with those module structures the important of standardization here again I remind you about that slide with the sextibion on it without standardized module structures and standardized descriptions there is too much is too much information for us to be able to rationally process for us to be able to understand step three is integrating them into qualification frameworks so in every country in Europe or nearly every country in Europe we have qualification frameworks that define essentially what can be learned by students at the moment in most countries there are certain exceptions such as Ireland that already have micro-credentials but in most countries the menu of what can be learned stops at things which are at least a year long so if we want to promote micro-credentials we need some recognition in the legislative and policy frameworks that there are smaller units of learning that might be useful to learners and step four we need to define quality standards that cover these within higher education this is an easier problem we already have a quality assurance system that can be easily extended to cover micro-credentials if we think about a definition of micro-credentials that incorporates different forms of open learning that incorporates continuing professional education that incorporates various forms of adult learning that even might consist of courses offered by NGOs we need a way to understand quality across all of these different perspectives and we need a way for the quality systems that apply to these different areas of education to be able to communicate and to understand each other and step five once we've defined all these standards we need to define information and data models and the reason for this is that the only way we are ever going to be able to process the levels of information that will be generated by micro-credentials whether it be a student searching for a course or an employer processing CVs will be with the assistance of machines we will need a computer aided the technology enhanced approach and the basis of any technological approach is information and data models and finally we need to incentivize institutional networks for micro-credential mobility and what we mean by this is that once we have all the policy infrastructures in place no amount of descriptions and recognition policies are going to actually make institutions accept micro-credentials institutions will accept micro-credentials between them when they are working as networks when they are signing cross-recognition agreements and when they are using the entire infrastructure of recognition that exists in Europe for trusting the quality and examining the equivalence of credits that come from other institutions alongside of this we can also examine and test new technologies and we hear a lot about the potentials of blockchain and the potentials of AI they do all have a lot of potential but probably none of them can be considered a panacea as of yet without testing so if we go through these seven steps we have an idea of what needs to happen on micro-credentials but can possibly stop us from actually achieving what seems like seven quite straightforward activities to be done in the policy environment and the first tension is what we might call ivory towers and walled gardens ivory towers are usually associated with higher education and usually associated with higher education not being open to the wider world so one of the big mistakes we can do is say that micro-credentials are a creature of higher education that micro-credentials are broadly equivalent to ECTS and that micro-credential policy is basically about institutions sharing ECTS more freely amongst themselves while that definitely should be part of a micro-credential policy it leaves out all the rest of the education ecosystem and it leaves out most of the more interesting combinations of why students might want to follow these creatures the second part of this is and walled gardens are essentially private providers or sometimes public providers but mainly private providers saying we are going to create a micro-credential ecosystem but this is only available to on our platform through our subscription and essentially when you lose access to our platform you also lose access to all that learning and possibly even to your achievements so walled gardens or ivory towers go against the idea of openness and choice that micro-credentials are supposed to represent second tension and barrier is the eternal discussion of harmonization versus standardization it's very very easy for us to say standardization is hard so let's harmonize instead and I do understand the reasons you want to harmonize but simply enough the more you opt for harmonization versus standardization the more complexity you introduce into the system because we're talking about a system that's already vastly complex and has vast variety in it the more harmonization we increase the harder we make for all of this to be transferred so in the harmonization versus standardization debate it's dangerous to go for the easy answer of harmonization unless it's absolutely necessary third challenge is what I call AI optimism I hear it a lot in meetings these days where they say okay there's no problem how do we help students choose the right problem we'll throw AI at that problem how do we figure out which CVs are most appropriate if we have a vast CVs we'll throw AI at that problem how do we assure the quality of credentials we'll throw AI at that problem and no doubt we can do a lot with AI we can do a lot with natural language processing but AI in its infancy makes a lot of mistakes and BAI sets and if you think about how many high quality data sets there are on student choice and micro credentials there aren't really any yet maybe we can start thinking about it and start thinking how we can build data sets that will allow us to process micro credentials in the future and AI will definitely be part of as strategy on a longer let's say 10 year perspective but at the moment I would argue that AI isn't there we need slightly more traditional approaches and not just saying AI will solve this eventually last part of this is incumbent momentum and just like I made I said that listen there's a lot similar to electric cars and the challenges that are faced in micro credentials in the challenges we also have the challenges of incumbent momentum in the sense that full higher education full degrees, traditional education not only is it not dead not only will it not die it has strong ongoing momentum which will continue and one of the largest mistakes we could make is turn this into a fight between micro credentials and incumbents and waste time arguing which one of these is most relevant to education the answer is that they're both relevant and we need every tool into toolbox possible to give perspectives to people who want to learn and there's going to be plenty of role for traditional degrees and there's going to be plenty of role for micro credentials and incumbents have momentum it's a good thing it's not something we should spend time arguing about so linking this introduction to what you'll be hearing about in the rest of the micro-HE sessions within this overall context micro-HE has defined a number of components of the project the first is understanding prevalence understanding how to use micro credentials already the second part of the project was technology standards with technology standards actually defining how we should code data how we should translate data how we should transfer data the third part of our project was a technology demonstrator on a credentialing platform and the fourth part of it was forecasting in terms of looking to see how this will affect institutions more practically you will hear more about all of these in the next session the last thing I would like to say is that the stakes are high it's important to get this right if we get this right what micro credentials promise us is a future with more flexible and personalized offerings resolve the opportunity to maybe better resolve skill mismatches the opportunity to give better educational access to the disadvantaged and the ability for employers to find granular competence in a sea of graduates these are the things we are paying for in this policy field thank you for your attention I'd be happy to answer your questions thank you Anthony wonderful presentation quite the to-do list I think it will be a terrific basis for the ongoing discussions that happen today during the micro-HE conference unfortunately we don't have any time for questions I would encourage those who posted questions to attend some of the micro-HE sessions today where you'll have an opportunity to ask those questions directly on the team members from micro-HE I'd like to hand this over to you because you had some follow-up comments that you wanted to make yes thank you so much for this Lisa and thank you Anthony thank you Dimitris thank you so much Denise who managed to overcome quite a lot of stress over her lost the internet and to do a very good presentation so I will briefly just share my screen to do some technical things and to invite you to the other sessions so you again to remind you that on the Eden website which is Eden online at slash 2020 underscore Timishwara you find all the time the program in the CESTA in the Central European Standard Time Summertime, sorry programs hours so please have a look there if you are not very sure at what time is what session and then once you have logged in into the conference.upt.org which is the event structure where you find papers, other information and the connection there you can select your time zone and you will be able to see directly in the time zone. I need to say this again because quite a lot of people are asking us also on Twitter and also via email when they are supposed to show up for different sessions. This is where you can find exactly the time. We will have now a virtual networking coffee break in a different Zoom room my colleagues are going to be there they are going to show you some videos about Romania and allow you to talk and to speak around and to have some networking time then we have some parallel sessions where you have all the links into the conference.upt.org and you will be able to join them. I need to attract attention that besides the standard theory student need and paper presentation of full papers we have a workshop which is presenting the experience of EDM with the webinars the Monday webinars as we call them the online together webinars and then obviously we start the PhD symposium where we have three sessions in the PhD symp. Anthony just introduced the MicroHE workshop where you have other sessions coming up there are three of them we will have a virtual lunch break which I need to remind you again where our therapist Elena is going to come again live and you will be able to do some stretching exercises and also to be you will be able to have the possibility to chat around and to discuss things and I want to invite you for tonight we had quite a lovely virtual tour last night the recording is already on YouTube of the virtual tour because I got a message from several people asking when the recording are going to be available and that was an event which we supported it's also on the eLearning YouTube channel beside of the ones you can also see it there and we will do the same with tonight but please join us our partners from ambasada have gave us made available two very nice and very simple recipes one is a Grishkulap and the other one is Sarmalen so you will need to go here to see the recipes and also the videos and to try to do the cooking if you still have time to do that until tonight and then join us but if not please bring any of your food and a glass of wine or beer or whatever you have at home or nearby better some national drinks and we will have a lovely chat about community involvement and the cultural festival of Timisoara will have some jazz music with a very famous band from Romania, Begablus Band and we will be joining somehow virtually from a cultural community space which is called Barber so please also join us tonight for the conference dinner in a virtual space so I just want to tell you about I told you about the online together workshop and the PhD track and the Micro AG which will follow you find all the links live into the conference tool. Thank you very much to all. Let's go for coffee.