Hi everyone, welcome to the e-learning and evaluation accents and healthcare presentation as friends already mentioned My name is math yankars. I'm the IT and operations manager for the Sousa Institute First of all, I want to say thank you to everyone that is a part of Moodle Thank you to the creators the developers the community Everyone that is involved in making Moodle what it is We'll vouch you we would not have been able to accomplish nearly as much as we have We would have been encumbered by expensive licensing fees and proprietary code that wouldn't have been able to adapt to what we need So thank you to Moodle So give you a brief summary of who we are and what we do This will give you a little bit of context and back on before I explain how we use Moodle as mentioned in the description of this talk the increasing incidence of cancer and advances made to screen and treat the disease Demand health professionals with a very specialized set of skills and knowledge These health professionals require expert critical thinking Prom solving and decision-making skills to be able to provide quality cancer care for patients and their families Universities right now turn out mostly generalists They do not have the skills and knowledge necessary to provide the best care for cancer patients straight out of school That's where we come in We are fully not for profit originally created in 2008. So we're only seven years old Created by the government of Ontario to support nurses that care for cancer patients and families So far we have provided educational support to thousands of nurses We are dedicated to a high level of standards and accents in healthcare and always our end goal is to improve patient care Recently in 2013 we've expanded our support to not just nurses, but other healthcare professionals and to learners outside of Ontario We had Just two learners from New Zealand sent up for some of our courses to the past week We are a small group so far. We're just under 20 people as a whole and I am very lucky to lead the IT team Basically we run the platforms that this was answered runs on including Moodle and Had the instructional design and multimedia development team that designs and develops the courses that we offer In the last five years we've educated a little bit over 6,000 nurses Created more than 30 courses each 10 hours longer longer is our more or less university style distance education courses But really focused on the theory and and applicability of cancer care Topics range from things like radiation therapy to administration of chemotherapy to psychosocial topics like patient teaching and education and Managing grief and loss when people die So what makes an adult health care professional and not standard learner? These are nurses. They are doctors physiotherapists social workers even spiritual care professionals like chaplains They tend to be in their 30s to 50s And have lots of demands on their time It is very difficult to get health care professionals committed to a set class time You're not going to be able to get them to sit together in the classroom And if they are sitting in a classroom, they need somebody else to take care of the patients while they're off the unit So to meet this need we are essentially almost 90% online All of our courses have an asynchronous learning component and participation Our learners are able to log in When they have half an hour here or an hour there in between taking care of patients or at Starbucks after their shift Taking all this into account We want to provide health professionals a skin skills and confidence to be able to provide the best patient care One other thing we've noticed is even though they are self-directed and motivated and they know what they want They still need some structure In continuing education, you don't often get any structure There's no degree for you to see as the end goal So we created something like a degree. We created the the the Susan nurse designation To achieve this designation a person has to do our core curriculum Write a specialty certification exam of the Canadian Nursing Association and Complete a clinical fellowship minimum of 75 hours so they can actually apply these skills on the unit Only once they complete all of these requirements can they actually call themselves the Susan nurse Our director likes to say we want to change health care practice with a single course But if a pathway and promoting lifelong learning, we may be able to change the system One other thing I should mention is the four domains of practice We don't want a person that just knows how to treat a certain set of disease or type of cancer To achieve the designation they have to take a course from each of the four domains They have to know how to talk to families how to support Somebody that is crying because their husband just died They have to know how to lead the team and develop the professional practice And they have to know how to teach and coach how to teach patients how to take a medicine regimen All in all for a full-time working professional. It may take about three to five years to finish all these requirements So with all of these constraints, what is our approach to e-learning? How do we get learners interested in our courses and are coming back? When we started out, we had a little bit of leeway and since we were just starting up we could research the latest adult learning principles adult learning theory and Develop the courses in a way that they can be clinically oriented and applicable for the busy clinician This guy up on the screen has a name we call him think about guy We have to make sure to get them to think about what they are learning while going through the material How do they apply it on their day-to-day job? So in addition to think about guy, we have a range of activities. I'll demo some of these in a little bit and Engaging interactions and practice What do I mean by an interaction again? It's practice It's somehow applying their learning at least as much as can be done without practicing on an actual patient So these are things such as case studies and knowledge checks All of our courses include social interactions. We put in Forced discussion forums in some ways forced since it's an assignment to discuss something to get feedback from peers Many of our learners are specialists in their field. They may be the only oncology specialist in their city with Joining in a course with other specialists They can actually talk to other people get feedback on what are they doing and what are the best practices and connect the network They can connect across the entire country to get perspective on things and know that they are not alone and Of course assessment when when the things you're teaching can potentially kill people You have to be very careful that when you say somebody knows something that you're actually sure that they know it So a sample activity. I'll pull this up in just a minute, but This is essentially a way of combining I believe an 18 page PDF into something. That's much more digestible In this case, these are a variety of chemotherapy drugs and their treatments and side effects in this case We used visual anchors to help learning each chemotherapy drug is connected to the plant that was derived from and And developing something like this takes a combination of understanding of both the subject matter But the fields of user interface design user experience design and of course instructional design and learning itself Discussion forums as I mentioned before are meaningful interactions. They're not just you must post twice You can't just reply to person. I agree or say good post. These are actual Thinking required discussions often these turn into Case studies of situations that people have run into names changed of course But again, it's sharing learning being able to learn from what others have run into in Developing these activities. It's a fairly standard the learning production process I'd say instructional design is key here because they are the glue that glues the educator together with the multimedia developer that Can create the interactive interesting graphics We have the educator write some content at this point is essentially more or less like a textbook style We storyboard it produce it with the multimedia developers and an often run a pilot to Verify that what we are trying to achieve is actually being achieved Our content keeps changing there is no Set textbook that isn't out of date within a year or two in the healthcare field new treatments come out new guidelines are set out by government organizations and we help work with them to translate those into activities that Transite that learning and make it understandable We do use flash very heavily in our courses Unfortunately as though you may have heard flashes a little bit on its way out there have been a lot of complaints about the security on it and The big problem of flash not working on tablets like iPad We've slowly been transitioning to HTML5 development, but the tools there aren't quite good yet And for those of you thinking of if you do develop in flash and thinking of moving away from it carefully consider the timeline and What your learners are using we can't support the iPad with in HTML5 activities without not supporting Internet Explorer 8 which a lot of hospitals and a lot of corporate organizations still heavily use Sort of we have one foot in the past and one foot in the future and we're trying to do that big leap forward to catch up and Talking of catching up There's regular course updates again because the evidence keeps changing We're on a little bit of a treadmill. We have to keep up with changing the evidence changing the academic references We linked a lot of external resources in our courses and almost every single time that we run a course We find some broken links It's just link rod. It's amazing how fast things change on the internet That are pieces of course the user feedback from mandatory evaluations We get a lot of feedback in our courses and we do use it and I'll talk about evaluations a little bit So now let me actually try and pull up our e-learning Moodle install and see if I can show you some activities live If the gods of the demo will cooperate with me All right, there we go so This is essentially the Moodle clean theme customized a little bit just with our colors Shifted things around a little bit the check marks on the left as opposed to on the right and The course structure we use a format called collapsed topics which avoids the Scroll of death when you have a lot of resources in your course the majority of our content is Again an e-learning module since there are no textbooks that have the current evidence guidelines and Our pre and post evaluations that I'll go into in a little bit So let's see if I can pull up an activity here There we go So I'll pull up the PDF of what this is taken from We still provide the PDF since this that's something that they need to take away as you can see It's quite large with lots of red text and important little bits in it that cannot be forgotten So we took that and We turned it into This flash-based activity Splitting the drugs into their classes and doing a little bits of animation and again Chunking out the information to digestible formats is that when a person opens this they don't know there's an 18-page PDF in there They're not gonna get scared at least not right away Information also specific to things like community nurses is hidden away as needed administration tips And again things like red flags like fatal have given in a certain way And these aren't all that complicated as they're essentially click to show more's but it's makes a real difference in Getting people engaged since they have to lean forward and actually click something move their mouse They're not just passively reading another activity. This was quite inspired This is around personal protective equipment that they're supposed to put on when they're administering chemotherapy drugs to the make sure that They don't get poisoned over time from these drugs so it's fairly dry topic, but If we dressed the nurse we found out why it's important to put on a gown and the specific recommendations around what kind of gown it is or A mask when to use a mask. I'm just gonna skip through a few other ones just to give you an idea of the format of courses This is another activity a multi-step nine-step Oral chemotherapy patient pathway specifically for when patients are given oral chemotherapy pills to take it home It steps through what the healthcare professional has to do But again, it's made a little bit interactive. They have to click to find out more. There's illustrations to help them Put themselves in the situation Where they're actually gonna be doing that and if they need to jump back They can easily jump back in some cases we have had to hire actual medical illustrators to illustrate some of these This one's a work in progress by the way, so don't mind the yellow text Here we go. This is again another fairly complex topic around the mono amine Mono. Yeah, I think it's mono amine hypothesis of depression essentially when you don't get certain What's the term? Certain types of neural receptors not going to the right place, I believe Again a way of illustrating very complex information That lets the learner better understand and better contextualize it and the last activity that I'll show If I can find it here we go No, sorry 3 and 17. Here we go. So Things like airway obstruction. We need actual medical illustrators to draw the actual proper physiology They say a picture says a thousand words Presentation here No, I forgot to mention the case studies So we do both Moodle quizzes for actual formative assessment where they are marked and assessed and Then there's case studies interpersed throughout the content. So we provide a situation in this case a person that has a certain medical history and This is in line with the content We give them some additional information Fake clipboard of their assessment scores and then we asked them To answer a few questions and these are low stakes not Not actually recorded scores this lets them practice figure out what they are learning figure out Why they didn't answer correctly and to go back and review now. Let's quickly talk about evaluation. I think I have about 10 minutes left, okay So evaluation we have to be accountable to funders and learners is Our program actually achieving its intended objectives. We are spending publicly funded money So everything we do is very closely scrutinized This gives us a tool to say this is how you're spending your money This is the effect you are having One thing we do that's a bit unique. I believe is doing an evaluation at the start We establish a baseline confidence level and skill level for each learner before they begin the course before they allow to teach any Before they allow to touch any content We use Moodle's completion tracking it to lock everything until they fill out the baseline evaluation And at the end we don't give them their certificate order grade until they finish the final evaluation And again end up final evaluation. We ask the same confidence levels and we measure what difference the reaction make in their skills What gaps still exist what do they still need to learn? For our evaluations we use the Moodle questionnaire plug-in and thank you to Joseph Rizal and Mike Churchward for building and maintaining this plug-in So you can see in this case is a radiation therapy course These are some of the confidence ratings that we ask like the confidence in describing the principles of radiation therapy or Listing the common treatment modalities And then we take that and we Run reports on it and compare in this case. This is the Measuring cancer care quality improvement specifically with our chemotherapy course about 2,000 nurses have completed the course across Ontario There's also an annual maintenance course that they have to repeat each year to make sure they are still up to par on this skills required for chemotherapy administration and we took their confidence levels and ran them through an exam before the course and After the course and you can see there's a marked improvement in their exam score in this case This is also a special case of blended learning. We try and do as much as we can online, but for certain things like Doing giving an injection of chemotherapy. You can't really teach that online. You can teach this theory You can teach what they should do But then we actually have a distributed model of 30 facilitators at various hospitals across the province To actually teach them the specific skills of hands-on skills and also to add whatever context is relevant to their organization Since we're not attached to any one specific hospital we rely on these facilitators from each organization to fill in the gaps of What is specific to the organization? What do they specifically do that is unique to them? And these are just some other Samples of the reports that we generate. These are pre and post confidence levels on assessment tools used and Here ratings of our e-learning and tools. We explicitly asked them were the instructions clear How easy was it to navigate the course? How was our technical support and finally text comments When you get a hundred feedback comments from students, it's hard to parse all of it together So with a research background, we have a research analyst that uses a program called and vivo to group these together So for example to zoom in. These are all the comments around the content We know that it was useful content. It was informative thorough Yes, etc. And we do the same thing for weaknesses and figure out what do we need to change about our courses This is great for a single course now, unfortunately the questionnaire plug-in doesn't have an easy way to pull many evaluations from many courses In the past we've had to export a CSV from each single question here in each single course and Then try and combine them together in Excel, which was a nightmare since Some courses in the past would have had slightly different questions others in the future would have more or less evaluations There is a data tools from essentially used in the big data field. I use the tool called Penta hole It's a what's known as an extract transform load tool and You can use it to chain together a whole bunch of queries together So essentially with this we are able to pull out Every evaluation from every course we have ever run and group it together We plan to do use this to do things like for example Giving a learner feedback on their confidence levels So let's say in 2009 they took a course with us and they rated their confidence levels as such and such and such In 2010 they took another course and they had site improvement And now they're taking another course and we could potentially pull up that information that they gave us before and show them These are your gaps. This is where you need to focus your learning That's something that I'd like to do in the future, but it's Gonna be quite challenging and again, there's gonna be an issue of privacy If a learner gave us their confidence levels when they had to take a course from their employer three years ago Are we still allowed to keep that information and they even want to know that information still And we do get returning learners that come back after four years They've forgotten their password, but they still have history with us So we try and pull up their account make sure that they have all of their information preserved All right, and that's about it Thank you for your time. If you have any questions or comments Yep So while we force them to if they want to pass the course It's part of depending some courses also are self-directed Where they can essentially get away without talking to anyone else those tend to be the introductory courses other ones are in a weekly format Like the patient navigation course, which is all about talking with patients each week There is a forum discussion that they have to participate in meaningfully and if they don't participate in and they don't pass the course Yes It's hard to estimate as it really depends on the complexity of the activity What we try and budget is roughly for a half hour module It's about three to four weeks of production time in terms of instructional designer and a multimedia developer Like that plant alkaloids activity that I showed off that was probably Maybe two weeks of work from a multimedia developer They're not working on it a hundred percent time because there is a lot of back and forth They'll come up with a drawing of a prototype that they'll run by the educator Then they'll go back come up for working prototype again Go back make sure it's still as expected and then there's a QA pass as with with some of these things You can get into situations where you can't back out or they've made a button that shows up underneath something else So there's a QA component to this. It is it is a big time investment It is something that you want to make sure you get at least a few hundred to a few thousand learners through to really justified investment Yes So traditional e-learning does use voiceover audio and almost everything our challenge is that a lot of our learners are using health health organization or hospital computers to learn they don't have audio or they don't have the The the ability to turn on the audio because they are in a busy unit with potentially a lot of noise in the background So we don't use any audio voiceovers The other challenge of using voiceovers in our applications because we have to keep refining things every year or possibly more often We would have to keep rerecording audio, which is a great pain So part of it is from feedback from the evaluations if a link breaks we get feedback in the comments The teachers definitely hear about it and they let us know In terms of content educators attend their center the conferences We have partnership of cancer car Ontario who is actually issuing the health care administration guidelines So they very well let us know when something is coming down the pike as they often expect us to create a course for it to translate their recommended guidelines and to Some that can be actually taught that's typically these guidelines come out there like 20 pages of a dense PDF of Information that's not easily digested. Thank you everyone