For instance, like I said, it is very easy for students to set slightly different names for the functions or objects that we ask for. So what we do is the AutoHARP provides a shiny app for students to run their code before they actually submit it. So this is a kind of a pre-check, it looks something like this. So we run it on a server and let's say a student comes to the server and they want to check whether their solution is correct or not. So they come here and then they upload. So they look for their solution, they upload it and some lints appear. Now the lints are from the linter package, which is a really excellent package. And if we have time later, I can tell you why we didn't just use the linter object. It just has too much detail for us for what I needed. Okay, so lints appear and they can correct that. When they go to the next tab, they can see the output so that they can verify that I will get what they get when they run it. And then the correctness check tells them some basic checks. Okay, you were supposed to create these objects with these column names, but you're missing these. So this student needs to go back and verify why that doesn't happen. Maybe another student will come and then they get all true and they can happily submit this to the learning management system. We can add details here, like explaining exactly what this server is checking in this run. Because a common question I get is that if it passed all the checks, why am I not getting the full marks? We kind of run more extensive checking when we actually do it. Okay, the other functionality is that this is a class on visualization. So one of the things that the package provides is it generates thumbnails of all the visualizations that students create. And I can bring this back to the class.  When I click on this link, it brings me to the student's code. And I can discuss it with the students and you get their feedback. Why do you think this is good? Why do you think this is bad? And I can offer suggestions. It's anonymized, so no one feels too bad about it. Okay, so why is it a semi-automatic radar? Because the output it creates is something like this. It creates a data frame that I still need to go through. I spend about a couple of hours each week looking through this. So one row is for each student. And I can see these are the functions they used. Did they use the union function or intersect? Or did they write a loop on their own to do it? And then usually when I go to the end, I write some comments. And almost always the comment is use less for loops. Okay, so there's some other functionality that I'm hoping to develop, which is things like this. In a situation like this where the X output becomes the input in the next line, then the Y output becomes a Y input. I want to pick this up quickly and then identify who here's a place you could use piping. Or if they duplicate code, they write some snippet and then they cut and paste it many times and reuse it. I want to be able to identify that. And then encourage them to write a function so that their code is reusable. Now is the package the right one for you? Maybe not. So what I've developed is something very specific to the way I do things. So if it's not how you work, then maybe it's not appropriate, but there are other options out there. There is mark my assignment, homework, and from ours to do this to learn R together with the grade this package. If you want to know how more about how it works, the full manuals can be found at this URL. The package itself ships with some examples, like how some examples of solution templates, examples of student scripts, and question sheets. And one of these question sheets is the one that I just highlighted to you during this talk about the PDF for XQ. Yeah, if you need any help or something, or you have suggestions on how it can be improved, please feel free to email me at this email address. Thank you for listening. I hope you all stay safe and healthy. If you're wondering about the name, it's the title of a song by Belgian band called Cooverphonic. Claudio is listening, I think. He's a good friend, and he's gonna introduce me to the band. So thank you very much. Thank you, Rick, for such an interesting talk and interesting package. So here's one question for you. So does static code analysis works when students use tidyverse functions like per instead of a for loop? Yes, you can call, yes, it looks for the function. It doesn't grab for function. It looks for functions with a particular name. So you give it a regular expression. So it looks for function calls with that precise name. So it picks up things. It looks in the package name space and matches it there. Sometimes there are different packages with the same name. So you have to be careful and aware of that, that's all. So I have a question because I also write my own automatic grading workflow for my R class as well last semester. And so just wondering how do you check the students' answers correctness? Is it compared to your own saved object? Yes, so at the end of the requirements, they are required to generate a vector or a scalar. And then in my solution template, I will pick it up and check with the correct version. Yes. You also specify the variable name as well, right? Yes, that's right. That is a bit manual. I'm sure that I can't find a simpler way to do this. I have to specify. And that's why we came up with a Shiny app to make sure that simple mistakes get overcome. Yeah, I can't blame them for those. Yeah, I'm thinking like mine is very harsh because even the types the variable name run and I just penalize them. OK, yeah, I should do that, too. And the second question from me is, so because this class is about data analysis class, right? And when I design my questions, I often find my variable, if they do something wrong with question one, and it will also affect the rest of the questions. But I feel like they shouldn't be really penalized for that. But I don't think there's a good way to avoid it. So what's your approach about this one? Do you like, for each of your questions, when you design, you design them like isolatedly or like also like a natural data analysis flow? No, yeah, you're right. We should have that flow, right? But we can't completely penalize them if they get a small mistake and that causes the second part to be wrong. So I try to insert checks in between. So I check the first part and then if there's something wrong there, I go back and check it. That's why I still have to do a lot of manual stuff. I look at the data frame and I see where they went wrong. And then from that thumbnail, I can just click into the HTML file and look at the script more quickly. I can't avoid that, yeah. I wanted to, like you say, we want to avoid like the, there is a place for the data camp style or the learn R style where the environment is provided to them and we just check really specific correctness. But when we ask them to submit a script, I'm hoping that they get into the habit of writing it from start to finish, writing empathetic code, right? That someone else can run. And if they trip up, then we can't penalize them. We've got to spend the time looking at it. Yeah, I totally agree with you. Cool. So there's one more question from the audience. So have you noticed any change in the marks from using this automated process? I'm just assuming that full manual checking can be open to more subjectiveness and bias. I didn't do that comparison, but I do realize that. I am more consistent in the grading because once I have the data frame for everybody, I can see what everybody did. If later on I decide to change something slightly, I can just quickly go back and make it consistent throughout. This has helped me be more consistent, but that's a good idea. I should perhaps go and look at the comparison of the grades. Thank you. Just wondering how long have you run your automatic grading process? How many semesters have you run so far? So far, three. So it's quite mature to be like everybody can use that? Yeah, I think it can be used. Cool, I should try that. Sorry, I haven't been aware of that in your package. And now I just simply just write my own workflow. Yeah, I don't know. It only went up on Cran very recently. But if you need any help, just let me know, please. Thank you very much. Thank you. So I think, can I just move on to the next presentation? Because I think there's two more minutes to start. OK, so yeah, Rian is like naughty. I think I'm ready to introduce you. So now we have the second presentation from the session. So Rian, a data scientist from Jumping Rivers and a co-organiser of Our Lady's Lancaster. And she's talking about another automating online teaching with R. Yeah, it's obviously the fun topic at the moment. I wonder why. Thanks, Eero. Yeah, so it's a slightly different spin here. So my name's Rian. I'm a data scientist at Jumping Rivers. And I want to tell you about how we handle our infrastructure around delivering training. So I work at a company called Jumping Rivers. We're based in the United Kingdom. We do statistical consultancy. We build a lot of infrastructure. We're our studio partners. So we do a lot of setting up of their tools. But a big thing of what we do is we provide training. So we teach around, we have about 30 different courses at the moment, all one day courses. And that might be Intro to R, Machine Learning with Python. We do Docker courses, Git courses, et cetera. So quite a wide variety of courses that we deliver online. And like I said, about 100 courses a year. We've got quite a small training. We've probably got about five or six trainers delivering about 100 courses to around 2,000 attendees per year. So it's a large part of what we do as a company. And obviously, most of our courses pre-March 2020 were on site. So I'd be traveling around the UK delivering training courses. And then March happened. And suddenly we had all of these bookings. And we had the same scramble. I'm sure all of you are well aware of trying to move things online. What we actually did was we took our standard course. So lots of slides and talking and maybe half hour practicals. And we ran it online. So we said, if we're going to do this, let's get one of our trainers. So Jamie delivered our Intro to R course to the entire Jumping Rivers staff. And it was really, really dull. It did not lend itself. So we had this mad panic of like, OK, we need to make this more interactive. We need to make this engaging. We need to keep people engaged. So we changed from one day to sort of two half days. And we made things a lot more interactive. So we basically threw out all of our slides brought in live coding. So when I teach, say, spatial data analysis with our, it will mostly be a lot of me live coding and chatting with participants as we go through. And instead of longer practicals, we have lots of short exercises, plus lots of quizzes just to check in on formative assessment. And that's great. We've had some really nice feedback. When we do go back on site, we're going to stick with this format because we think it makes better training. But it has made a lot more work for me as a trainer. And there's so much more infrastructure now. I need to think about as a trainer in terms of running that session. And we don't do any sort of go away assessment in terms of the stuff that was talking about, although that looks amazing, in terms of assessment. It's more on the day chatting back and forth. So I want to tell you about the process we use. But to tell you about the process, our process always starts with this lovely lady. This is Deborah. She's our administrator at Jumping Rivers. And when you book a course with us, she's the person that talks to the client and finds out what course do you want, when are you running it, and information about the client. So everything starts with Deborah. And what Deborah does is she puts all of this information into Asana, which is our task manager that we use. So all of our project management stuff we do in the Asana tool. So the sorts of things that Deborah would take, she would know obviously the client name. So quite often when we teach, it might be NHS or government, but we also sort of go into clients and we do public courses. So there's lots of different clients that we work with. So we need to know who the client is, who's delivering the trainer. That's me here. And what the course is. Like I said, 30 courses across multiple languages. You wanna make sure you teach the right one on the day, right? And she would also set up the Zoom call information. And that's kind of everything really. And the idea is that from this one task on our project management system, we should be able to create all of the infrastructure required for training. Now the infrastructure that's required for training, we're teaching virtually. So we need somewhere for people to actually do our programming. And we know that particularly with intro courses, we don't want to get people setting up and installing our studio packages in advance. So we want to provide a virtual training environment where attendees can turn up, do the course, and have everything set up for them. So we need our studio. We need all of the right system dependencies and our package dependencies. We also want our notes there, our scripts. We need a Google doc for quizzes. We need links to Zoom. So there's a lot of stuff that we need to spin up every time we run a course. And obviously this is happening multiple times a week. So not something we could do manually. So we've effectively developed a series of our tools which are all internal at the moment, predominantly because, again, like Vic was saying, you sort of tweak things to work to your exact personal workflow, but most of the tools that we use are available as alternatives, which I'll highlight that are out there. And also very, very happy. I can't go into technical detail right now just because of the amount of time. But if anyone wants to know more about any of the little sort of connections here, please just get into contacts. I'm very happy to share some code and help you get set up with similar things. So everything comes from this R package. We've developed a central R package called JR Droplet. What JR Droplet does is effectively, I should be able to, as a trainer, run one line of R code, create Droplet from this task, give it the URL to this task, and it should create all of the infrastructure I need. So spin up a virtual machine with the right name on a URL with the client's name built in. And we're gonna talk you through all of the bits that JR Droplet does, because from creating Droplet to Droplet finished, although it only takes about five to eight minutes to run, it touches a lot of different technology. So digital ocean provides virtual machines. So it's what we use to create a virtual machine that we use for training. So a day before the training course, we will spin up this virtual machine and that's the environment that the students will use for the entirety of the training. They're given a username and a password so they can log in and out of the space during the training and a little bit after the training as well. So we pay to create virtual machines via digital ocean. And there is an R package that wraps around the digital ocean API. It's called Analog C and we use that a lot. So what that will allow us to do is create virtual machines and delete virtual machines from R. Now we did go into digital ocean and we set up, I guess, a sort of standard image. So what's the minimum that's required for all of our courses? So we would want something like Ubuntu 1804, some minimal system dependencies and R and Python installed because there are two big, most common courses. What JR Droplet can then do is it can take that base image and then we can say, well, look in Asana, what course are we teaching? If we're teaching intro to R, we're gonna want Dplyr GGplot. If we're teaching visualization with Python, we need the Seaborn library. So what happens is effectively, we get the base image and that's developed and then it looks for the course specific package dependencies and installs that as well. We use the RStudio workbench, which is the new name for the RStudio product, the server pro. So that's the system we use as a sort of training environment. But we do need to actually install the package dependencies as well to make sure we're all set up. So we've got a virtual machine with all the packages required for people to learn. We actually need the notes now. So we use, like I said, quite a lot of live coding and we also have PDF notes and slides. These are all stored in our GitLab repositories, all of our course materials. As you know, teaching programming can be, you have to keep on your toes. So we sort of update our courses monthly, effectively, either because new packages are coming out and we want to share the latest information or I've taught a course and I've spotted a typo or I've seen an exercise that I maybe want to make a little bit harder or a little bit easier. So we constantly update our courses. So version control is obviously really important there, but also we want to run a bunch of checks. So anytime I update the course material, we've got quite a long GitLab continuous integration process, which checks, do the notes run? Are they limited correctly? Are there any spelling mistakes? Simple stuff like that. At the very end of that process is something called making an artifact. In GitLab, what you can do is anything that's created through the continuous integration process, you can kind of have available somewhere where you can actually grab that down called an artifact. So what happens is anytime we change our notes, the notes will basically be built and be available on GitLab. So again, what JR Droplet can do is look in a sauna and say, okay, we're teaching intro to R. Let's pull down the introduction notes and put them on the virtual machine. Let's pull down the scripts required for the learners and put them on the virtual machine. So we've now got a virtual machine with all the right packages, all the right system dependencies, notes and packages. Now, if your head's getting a bit full, so is mine too, and so are the clients, right? We've already got quite a lot of tech going around, quite a lot of links. And that's a lot for the learner to think about. And when I'm teaching someone, I don't want them thinking about how do I log into RStudio workbench or how do I find the quizzes? We want them to just have a place for all of their stuff. So what we have is a welcome page. And what this welcome page does is effectively provide a central location for everything they need for the course. And again, this is just a one-day course. It's a lot of stuff that we have to share with them. So we want to keep it as simple as possible. So this is actually a Shiny app. It's hosted at a URL, which is memorable for the client. So we use a custom domain. And the Shiny app aspect is just so they can self-serve themselves a username and a password. And the reason is that's the sort of thing that can take quite a bit of time off the trainer and actually being able to go and grab your own username and password for the RStudio workbench is really helpful. So that's been a big benefit for us. And I mean, this is just a simple idea, right? This is just a placeholder, which is telling you, okay, this is my video room down in the bottom left. So that's where Zoom is. This is our quiz document, which I'll talk to you about in a minute. And this is how we can get to RStudio. So we found that having one central location for, okay, you're gonna start your course tomorrow. This is home. This is where everything is, has been really helpful. Yeah, and that points to quite a few things. So like I said, it points to Zoom, it points to our Google Docs, and it also points to the RStudio workbench where the actual learning happens. I'll show you the sort of thing that we do in the live coding. So we do provide PDF notes as well built in our markdown, but I don't want students reading those during the course. I'd much rather than be engaging in the teaching and have it as a kind of takeaway pamphlet for use afterwards. So we make sure it's on the virtual machine in case they want to read ahead or look back over something. But the main way we deliver things is through a sort of filling the gaps, but free text. So there's a couple of scripts for each chapter that we use. We have exercises, which they go away and have a go at and the solutions just so they can double check. There's nothing wrong with having a nosey at the solutions, you can always learn something. And we also give them a demo script. And the idea here is as I'm live coding, I'm going to be typing lots of stuff. And the last thing I want is for them to be worrying about writing all of that down. So the demo is effectively a good guess of what I'm going to code. I do go off script, I do like going off on tangents, but it gives them the confidence that they can sit back and relax and just watch me code and engage and ask questions. So this is kind of what the actual environment looks like in terms of when people are working. And that's their own personal space. You can see we're logged in here as user 45. What that means is at the end of the course, they can just click one button and everything will be downloaded onto their computer. So if they've created another script, just to have a play around with something that's not going to be lost, which is really important. So yeah, there's a lot of different technology going on here. And then the very last thing, oh, the Google Doc, that's it. So we use Google Docs for our quizzes. This is something I borrowed from Greg Wilson as part of the studio education program. Using Google Docs for quizzes has been really useful for us. So what we do is we create a single Google document for our clients after their course and it will have these sorts of ballot boxes where people can just go in and mark their answer. The reason I like this over sort of standard polls or something, other solutions is it's very free for us as trainers. So I can put any questions in there I like. I'm not restricted to just using texts. So I can put a code snippet in and say, what would X be? Yeah, sorry, what would Z be if I run this code? I can put plots in. So when I teach sort of spatial mapping, I can say, here's four maps. Here's a piece of code. Which of these plots would be generated by the code, right? So lots of freedom. It's also really nice because if you've got 12 people in a Google document, you think that'll be mad. Actually seeing people typing, starting typing, not quite sure gives me that feedback that I've really missed in doing online training. And people can and quite often do not only just fill in the poll, but they're quite often right in R. Is it plus or is it, which one do I operate first? I really don't have a clue or can you explain to me again? So using the Google docs has been great. Again, this was something that when we first started doing online training, took a lot of time as a trainer. I would have to go into our Google Drive, create a Google document, copy it across from a template. What we've actually moved to now is we use our Markdown. So we have a standard index file. We render that our Markdown as a Word document. And then we use the Google Drive package to upload it to the right place in our Google Drive. And the big thing for me is I would always forget to change the sharing settings. So by default, Google Drive is quite private. If you want anyone with the link to be able to edit, you have to go file anyone on the web can edit. And that's the sort of little thing that I could so easily just forget to do before I deliver a training course. And then it causes a bit of fluster and a bit of break in the training. So Google Drive, the R package has a function where you can specify the privacy of the doc. So you can say I want to upload this Google doc to this place and I want it to be editable by anyone. So it's all these kind of little steps in terms of changing the process and just putting in little checks that's really helped make sure that our courses run smoothly. They're only little things by themselves, but for me, I want to spend my time focusing on the learner and their understanding rather than worrying about the tech. So this whole process from, I guess, someone starting to run JL Droplet to having the machine up with all the notes, the Google doc creation and the welcome page takes between five to 10 minutes, depending on how many system dependencies there are. And when that's done, it pings our slack. It says, Rian, you're teaching tomorrow. Here's the welcome page. This is the client. Here's all the information you need. So I know, again, I don't have to worry about it. I get that message a day before and I know everything's set up and I can just go in and just be happy that the environment is there and that the learners are going to have a great day. Now I must confess, I work with a lot of techy geeky people who like making and building things and we do sometimes have a tendency to build tools for fun rather than really needing them from a business perspective. And I was thinking, we've built this really quite complex stack. We're touching GitLab, Asana and Slack. By the way, they all have R packages out there already for wrapping around the API. We've written our own because we're doing something quite bespoke, but there is Asana, R, Slack, R, et cetera. But yeah, we've built this complex stack and it doesn't just look after itself, right? APIs change, we have to go in, we have to maintain things. But we reckon as a team, we've saved around six hours plus per course and we're a very small team. So in terms of, you know, that's just the setup. That's a lot of work. We also have a similar command for cleaning up afterwards. So it checks who logged into the virtual machine and uploads a register to Asana so we can discuss with the client things like that. It sets up a nice R Markdown format ready for our questionnaires. And it also points to a Shiny app which generates certificates and emails, certificates direct to the attendees after. So there's quite a lot sort of in the backend as well afterwards. So yes, it's definitely worth it. It is a lot to maintain. The key things are me having everything in one central location. So I guess that's two points. The Asana listing everything that you need to run the course but also for the client knowing they can go to that welcome page and everything is there for them. It means I can focus on teaching rather than stressing about the admin which is the bit I love, right? But it has taken our full jumping rivers team. I've not built this stack, right? This is a collaborative effort because when you're touching all these different technologies it does take a wide range of skills but it's been very useful for us. Very happy to give people more detail. If you've gone, oh, I really want to know about Google Drive or how do you do certificates in Shiny? We're sponsoring the conference so you can come grab us in our Slack channel and there's our Twitter and our website. Thanks very much. Thank you, all right. We've got a couple of minutes left for one question, Honina. So there's a question. Are the virtual machines provided by digital options similar to cloud infrastructures such as Amazon Cloud 9? How did you decide for this infrastructure in particular for your setup? We went with DigitalOcean because the pricing model works well for us. We're costed per minute. That works great. But most of our infrastructure, DigitalOcean plus the studio workbench on top. But yeah, it came down to just looking at the flexibility of the different offerings and which model suited us, yeah. And so this is one more question. Impressive sense, a quick question. Have you used our studio clouds for delivering online training? And if so, why have you not carried on with using it? That's a really good question, yeah. So we are our studio partners so we work quite a lot with some of the paid models as well. And like I said, our studio workbench just gives us a little bit more in terms of our flexibility in the service we can provide. That's what used to be referred to as our server pro until a couple of months ago. It just gives us a few extra bits of functionality and gives us complete control over what we want to spin up and where. And we can host it ourselves, which means we can tweak everything we need. Thank you so much for the talk. Thanks, Eru. So now we have the next speaker, Christof, our software engineer and our studio working with Eva Scheer in the R Markdown team. We'll talk about the extending the functionality of your R Markdown documents. Hi, everyone. Thank you for joining. In the next 15 minutes, we'll talk about how to extend your R Markdown documents. First, a few words about me. I recently joined R Studio as a software engineer and I'm working the R Markdown team. You may find me on Twitter and GitHub and you find my handle on the slide. Now let's begin. In this talk, we will talk about the R Markdown package. We will focus on some recipes to extend your R Markdown document. This is inspired and based on the R Markdown code book. It is made with book done and you can read it online. The first thing to understand before extending efficiently your R Markdown document is what happens when document renders. Often the feedback we have is that it's like magic and a lot of without free. You may already know this illustration by Alison Hort and the little results that cook and mix some ingredients, so text code in order to produce beautiful documents. But this is just first is magic. At the end, there is some tool involved under the hood and it's important to understand how these works because it will help you know what to look for and what's part exactly to tweak when you want to extend your R Markdown document. Rendering the document is mainly a three-step process. First, you will write a RMD file that will be converted to a more classical .MD file, so it's a Markdown file. This is done by Nitara. The second step will be this Markdown to be the input of Pandoc. Pandoc is a universal converter. Pandoc is in charge of converting your Markdown file to any output format you ask for. So usually it's HTML, documents, PDF, or even PowerPoint presentation. The third and last step that can happen is when you render to PDF. When you render to these formats, Pandoc will just convert to a .Tech file. And after that, it will be Lattec that will convert to PDF. RMD will also run Lattec for you the same way it runs Pandoc for you. These are the three main steps of the process. Each one has its own features, its own set of configuration, and what R Markdown does for you is just making it easy with one function, R Markdown Render, to convert the RMD file to any output formats. In the following, I will show you a few features that you can use to tweak the default behavior and extend the reports. To do that, I will use a simple report as a base to add features on. So in the animated deep below, you can see this RMD source file. I do not expect you to read it from this slide. You can go into the R2D Cloud project or just download the RMD file directly following the link below the GIF. This report contains a quick analysis of the data from the Palmer-Pinguin's package. So this analysis is paid mainly on the content of the great package of the website. And it's just an example that we will build on to show the different features you can add to customize the report. This report is using HTML document format and it's really the default behavior. So there is some text, some graphics, some tables, some illustration. And you can see this in the GIF or you can also see the results in the R2D Cloud project or download the HTML file. So the aim is to extend and improve this RMD document to make a report that you can publish. But creating the report to publish, the first thing we want to do, unless it's for demo or example, is to hide the source code from documents. For that, you will need to set the eco chunk option, knitar chunk option to false. You can set that globally using the ob chunk object in knitar. And this will apply the eco false chunk option to any of the following chunks. You don't have to set that on a pure chunk basis. This configuration can be made in a setup chunk. In this chunk, the configuration eco false won't apply. So it's a good idea to use another chunk option called include that you can set to false. Include will evaluate the code, but don't show the source code or even don't include the outputs. And so it's a great use for the setup chunk. Another important thing to do for a published report is to think accessibility. And so for that, you will need to add alternative text on any of your figures or plot or images that you include in the report. In the recent knitar version, you can do that on a chunk basis by using the fig.alt chunk option. This fig.alt chunk option can be used with any external image you want to include. And for that, you can use the knitar function include underscore graphics to include an external image and put that into a chunk. In this way, you will be able to set the fig.alt attribute on this chunk. Obviously, you can also do that for our graphics. And what I show in this slide is that any chunk option can be set by the variable because it's evaluated during the knitting process. So here I'm setting the active text into a variable in another chunk for which I will set include false because I don't want to include this in my document just for it to be evaluated. And then I can use the value of the variable into the fig.alt chunk option for my chunk containing the R plots. It would also be nice if your published report would look a bit different from the default document and maybe all the other document that other R markdown user can produce. So what I want you to know is to style my document. For example, I would like to apply a color on the header. This color could be from some guidelines of my organization. And for that, I can use CSS, so cascading type sheets. It's a way to style any HTML output and you can use that with R markdown too. One interesting thing is that you can do that from inside the R markdown document and it's not mandatory to use an external CSS file. In Netor, there is a CSS chunk that you can use and in this chunk, you can put any valid CSS code. I will set Ecoforce on this chunk because I don't want the source of the CSS to be included and seen into the output of my documents. And what happens when it renders, the CSS will apply on the document directly. So in this chunk, I'm using this CSS engine and accepting the header 11, 12 and three to change the color and take it to a blue color. This is useful, for example, for quick iteration on a style you want to make for an HTML before setting that to an external file. But it's also interesting for demoing or for teaching purposes. But obviously, if you have an external file, you can also set that directly in your R markdown document. You can use the CSS argument from the HTML underscore document function. And this workflow is better suited when you have an external file that you want to share across document or with your colleagues. By doing any of the two previous methods, I will now have some blue headers inside my document. Next, it would be great if I can highlight some of the results of my analysis. With R markdown, you can do that with something that we call custom block. Custom block are based on the Pandoc features called FenceDiv. This is a special markdown syntax that Pandocs reports that you can create these blocks by using three columns following by brackets in which you can set some classes beginning with a dot or some ID beginning with a dash or any attributes with a name equal than a value. So in this custom block, I'm creating a special environment of class highlight box for which I want to apply a style that I provide as attributes. The content of the block will be usual markdown. So here I'm setting a value based on the previous results using meter inline code and I'm putting this value in bold to emphasize it. This will be rendered in HTML in the following way. You will have a div tag with the class that you set up in your custom blocks or your highlight box and the attributes will be set on this div tag. And so here my style attributes will be set. The content of the custom block will be inserted as the content of the div and will be converted as usual markdown. So my bold text is now between a strong tag and my previous variable have been evaluated by meter and replaced by its value. How to improve this highlighted result then? What I want to do, for example, is I would like to have a box. A box with a border, a border with the same color as I put for my headers. And I would like any strong tag that I put inside this box to be the same color also. So by just thinking about what I want to do, I see that I want to apply color on three elements. So there will be probably some duplication into my style sheets. And so can I use variable or can I simplify the writing of the style sheets? So obviously for those who know CSS, you can use variable in a CSS document. But what we'll see next is another way to improve the creation of a style sheet for your documents. This new way is using a tool called SAS. SAS is an extension language for CSS that will allow you to write some rules and select errors in a more much flexible way that you can do with plain CSS. But don't worry, I'm not saying that you need to use another external tool. You will be able to use that directly from R because there is now an R package called SAS, S-E-S-S. So it's the same name as the tool. This R package will allow you to use this tool to produce any CSS. But with R-Markdown, you won't even need to use this R package because the support for SAS has been built into R-Markdown thanks to this R package. So let's see now how to use that in your R-Markdown document. So here I'm showing the CSS that I will need to apply to have my box with the blue border and the strong text. So you can see that I need to set my color in three different place. It would be really interesting if I could simplify that, mainly because if I want to change the color, for example, I would like to change it in any place I use. This is where the SAS syntax comes into play. There is two syntax that you can use. The first is the SCSS syntax and the second one is the SAS syntax. How does it look like? Here, you can see that I'm using a special NITAR engine called SCSS. So it works the same way as the CSS NITAR engine, but it's specifically meant to be used with the SAS package. So any code you would put into this chunk will then be processed by the SAS package to produce a CSS-included document. Let's look at this SCSS syntax. You can see that it's very close to a usual CSS syntax with the brackets and the semicolon, but you can do more things. For example, here I'm setting the blue color inside a variable. The variable begins with the dollar. I can then use this variable in the different places that I want to apply on. You can also see that I can use a nested organization for my element. So for example, here I have a div that I want to style, but I want to style the div or flag a lightbox and put the solid border with the padding and the color. But I also want for div of this class to have their strong element with the same color. And so using this nested structure is kind of easier to read in a way and easier to write than if I had to build the CSS. This is exactly the same with the SCSS syntax. The SCSS syntax is the old one. It's much less closer to usual CSS because it only works with indentation. There is no brackets and there is no semicolon. So here in this slide, I'm just showing the way to write the previous code using this syntax. Obviously, you can write all this in external file with dot SCSS or dot SSS. And you can provide this file into the same CSS argument that you can do with the CSS file. By inserting the previous code into my RNG document, I will now have a custom block that will be styled and will render in HTML as a blue solid box with text inside and any bold text will be also set in blue. If we add all that into our document, we have no source code, no more source code in the document. We have our headers colorized and we have our reasons highlighted in special blue box. You can see this in the animated diff, but you can also see the results in the R3Doclub project or download the modified RMD or HTML with the link below the animated diff. How to go further if you want to extend even more our document? You can find a lot more information and adventuring into the Armageddon cookbook. I hope you learned something in these 15 minutes. Thank you for being listened to me and have a good day. Thank you, Christophe. Apparently, we have some technical fun in the beginning. And so I'm sure there are lots of questions for your own Slack channel, so if you're happy, just post your questions on Slack. And now we're going to move to the final speaker, Patricia Franciak, talking about psychometrics. And now we're sure about those pre-recorded videos we are playing well because now I've become the expert. Hello, and welcome to this talk. My name is Patricia Marcinkova. And I am from Prague in the Czech Republic, where I did the Computational Psychometrics School at the Institute of Computer Science of the Czech Academy of Sciences. I conduct research in psychometric methods. I also teach graduate courses in psychometrics at Charles University. And with my team, I have developed the Shining Item Analysis R package to bring psychometric methods to a wider audience. During this talk, I would like to share my insight on teaching computational aspects of psychometrics with R and Shining. Here is an outline of the talk. I will first introduce the field of psychometrics and the courses I teach. I will then focus on the Shining Item Analysis package, its newest features, and how it was integrated into the course. I will then describe how understanding important psychometric concepts was supported by collection of real and simulated data sets, which are part of the package. Last but not least, I want to discuss a book related to this talk, its contents and its aims. The field of psychometrics deals with the advancement of quantitative measurement practices and analysis of these measurements in behavioral sciences. Researchers and practitioners involved in psychometric research are organized in a psychometric society. You may check the web page and learn about an upcoming virtual conference taking place later this month. Psychometrics covers a number of statistical methods that are useful in the behavioral and social sciences. These include the estimation of reliability in order to deal with the only presence of a measurement error or detailed modeling with so-called IRT models and many other topics. There are a number of existing R packages developed specifically to psychometric models and methods. You might check the psychometrics crime task view for a list of existing packages. Let me now share my experience teaching psychometrics as an interdisciplinary course to a very heterogeneous audience. As a Fulbright alumna at the University of Washington in Seattle, I got the opportunity to teach psychometrics to graduate students at the DERA College of Education. Since my return back to class, I have built two interdisciplinary courses at the Faculty of Mathematics and Physics at Charles University, a graduate course devoted to the computational aspect of psychometrics, covering main psychometric topics and a seminar open to recent research and guests. The courses are attended by students from many different fields. We also conduct pre-conference workshops and workshops to test developers and other practitioners. We have students or participants at various levels of our proficiency, from beginners to very experienced users, and also students with various levels of statistical focus and proficiency. With this wide heterogeneous audience, the hardest task is not to allow anyone to lag behind while at the same time maintaining an interesting and challenging course for the more advanced students. The main goal of this course is to explain the psychometric models and methods in a wider context of statistics and data science, providing terminology links and interpretations, because the same concepts oftentimes are named differently in statistics and psychometrics. To support this understanding, my goal is to illustrate important computational aspects, such as why we sometimes update zero estimates or when can one method provide a deeper understanding than the other method. To answer these questions, I have collected suitable real data and also created simulated data sets, which I will describe in this presentation. My goal is also to provide a toolbox of existing arpeggios available and irrelevant functions. These functions will sometimes return different estimates. In this course, we will focus on explaining these differences. Finally, going back to the main and hardest task, my goal is to make the computational tools and our functions better available and understandable to those who are beginners in R. And for this purpose, we developed the shiny item analysis or package and interactive shiny application, which I will now describe in more detail. The shiny item analysis is an R package for psychometric analysis of multi-item measurements and ratings from multiple readers. Versio 1.37 is available on Chrome. The newest version is that from GitHub. It contains a number of functions now and it also contains an interactive shiny application, which is acceptable locally using the start shiny item analysis function. And for those who are new to R, it is also available online on the Shiny server from the Institute of Computer Sciences and on the Shiny XIO. The features of an earlier version of the package were presented at the USARC 2017 conference in Brussels and it was later described in an article published in the R journal. Since then, there were a number of improvements, which I will now describe. We are happy that the shiny item analysis package is being widely used and that we now have a total of more than $40,000 downloads only from RStudio at Chrome River and over 1,000 downloads from Chrome from GitHub. The online app is used in over 100 countries and we are glad to get emails by people from all over the world who are using it for their teaching in their research or in practice. Let me now focus on the interactive shiny item. What is displayed here is the intro page. The available methods are organized in separate steps, which correspond to the topics covered by the graduate course of Psychometrics, which I teach and which also correspond to individual chapters in the book. The newest developments for the shiny item analysis interactive application include new methods, which are made available within each of these steps, including factor analysis, reliability of restricted samples or models for polyethanol item data. A number of new data sets, toy data sets are now available and also new data types are allowed for individual uploads. These include continuous or ordinal items, covariates, matching variables, et cetera. Interactive training sections have been also extended. These allow the user to explore how the item characteristic curve and item information curves change when changing item parameters. And training sections have also been extended by interactive quizzes, allowing the users to check their understanding of the model interpretations. The last step of the interactive shiny item analysis application allows one to automatically generate reports for data uploaded by the user. We had a poster presented at the user 2017 conference in Brussels providing more details on how report generation is performed. In short, shiny provides the user interface and calculations are marked down as used for our code evaluation, take root generation and conversion to either PDF or HTML format. And under the hood, nature, pandoc and PDF are utilized. Let me go through this section by discussing the newest functionalities of the package. A very important new functionality is that the start shiny item analysis function when used in our studio now starts the interactive shiny application as a background process. This is very important because it makes the console available for trying the sample our code provided in the shiny application. The students can just copy and paste provided sample our code and learn our while also using the interactive application. Here is a practical demonstration. We start the application running the start shiny item analysis function here. Oh, sorry, we have to hold the library first. We then use the start shiny item analysis. And this already starts the interactive shiny app. And it is running here in the jobs. Going back to the interactive shiny app, we can select methods here, for example, on the disk. We can go over the tabs, read, select different methods, but also play with the graphs which are all plotted in plotly. It's possible to download the figures, provide situations, tables of parameters. And very down here, there is a sample our code which the student can copy and paste into the RStudio session. The console, or up here, line by line then possibly also make some changes. Let me go back to the presentation now. We are now also working on modularizing the app with the systems of the golden package. This is related to one issue we deal with, high number of dependencies. On one hand, we want to offer a wide range of approaches. On the other hand, a large number of dependencies makes the package vulnerable to any of them becoming unavailable. And it also returns a note in RStudio. Let me now move to the next section of my talk and discuss collection of data sets provided by the shiny item analysis package and further discuss in the book which are used to illustrate important concepts of psychometric methods. The first data set I will present here is the data from a grant proposal peer review. It was used in a paper published in the Journal of the Royal Statistical Society series A earlier this year to demonstrate why zero-integer reliability estimates are possible under a restricted range. An interactive illustration is offered in the shiny item analysis app with the ABS data set. Animation to illustrate this issue was created with the GGAnimate package. Here, we can see that when the proposal range is restricted only to the top samples, the between proposal variance of the peer review scores decreases causing the decrease in the inter-rater reliability estimates eventually reaching the zero. The second data set or rather collections of data sets illustrates the importance of the detection of so-called differential item functioning or in short data. As we have illustrated with a simulated GMA data set in a paper published in the CBA Life Sciences, this may be present even in cases where there is the two groups of interest have exactly the same distributions of the total scores. The data may signalize an unfair item, therefore the data analysis should be done routinely in development of assessments and these items should be checked for working by content experts. In longitudinal designs, the differential item functioning in change which we have proposed in a paper published in 2020 in the Learning and Instructional Journal can provide proofs of instructional sensitivity even in cases where differences in change are not visible in the total scores. In figures displayed here, we can see that two students from two different academic tracks and academic and the basic score, but otherwise with exactly the same characteristics and the exactly same baseline score in grade six have different probabilities of answering items 6A, 6B, and 6D in grade nine. The diff and diff C analysis is available interactively in the shiny item analysis app where the data is discussed above implemented within the diff NLR package for details data paper published in the arsenal in 2020. Further computational aspects of parameter estimation in group specific models which are implemented in the diff NLR package including a simulation study comparing several existing and novel approaches alongside with some research and reliability will be presented at the international meeting of the Psychometric Society later this month. Last but not least, let me introduce a book still in preparation which is planned for publication in 2022. The title of the book is Computational Aspects of Psychometric Methods, so it are. In this book, we hope to bring deeper understanding to psychometric methods, models and the book is aimed for a wide audience including graduate students with statistics like ecology, education, health and other fields as well as researchers and practitioners. The content is based upon the topics covered by the Psychometrics graduate course I teach and I described. In each chapter, we provide our code and a number of practical examples including those presented here today. Each chapter also includes a section demonstrating the methods in the interactive shiny item analysis app. I hope this can help attract those readers who are new to R. To conclude, I have presented ideas on teaching computational aspects with R and with the shiny item analysis package. The interactive shiny app provided in this package was developed to help attract those who are new to R. I want to highlight the fact that although available online and interactively the entire complex shiny app is built within R which demonstrates the power of R. I also want to highlight the importance of the sample code provided within the shiny app making the processes behind the app more transparent and helping users to start using R. I have also illustrated and want to highlight the importance of the collection of relevant simulated and real data examples to help explain important computational aspects. And finally, stay tuned for the upcoming book on psychometric methods. This concludes my presentation. Thank you very much for your attention. Thank you Patricia for the wonderful talks. I have a question for Patricia. So how do you set up your interactive quiz? Are you using the R or like you made something special about the shiny app? No, so it's inside the shiny app and there is one thing we are actually considering is to also generate these data sets. So right now there is a fixed data set and the student can answer to this data set but what we are thinking about in implementing is including the possibility to automatically generate a data set so that different students get different data set but this is not implemented. And also I got a lot of motivation here at the conference how to automatize the process of grading. So right now the students would send us a screenshot but if there is more students this might need some further step. So here's a couple of questions for our questions about our markdown. So is there a blog post to show a more step by state process on how to style the output using SAS? Currently there is no blog post specific to our markdown but as I said it's powered by the SAS package which is an R package developed by RStudio and in there there is a nice article in the package done website that you can find which is an intro to the SAS which is an external tool and also with a specific part on how to use it in shiny apps and in our markdown. So it would be a great place to start and after that there is also the DSLIP package. I didn't mention that into the talk but there was a talk by Carson Sever that RStudio talked last year about that so there is now support for Bootstrap and it's also using SAS and so you get a nice article into that on how to use SAS with our markdown because our markdown HTML document is using Bootstrap. So is there a way to make the images center align in our markdown? Yes, there is an option, you need our option for that which is FIG.align, I posted the link into the Slack for the other question too so you can find that error but for this specific question the answer is also in the Armageddon cookbook because there is a specific like the same question. Exactly, so when you have a question like that about how can I do that into Armageddon the Armageddon cookbook is the place for us to put this kind of answer that are not related to a specific more global topics so there is an option for that which should make it work for different type of outputs because CSS is only for HTML output if you want to style in and the presentation was about that but if you want to style for example PDF output you need to use other tools but some integrated features like FIG.align will make it work for several type of outputs. And the final question is can we change the font in the YAMA for our markdown? It depends on the output formats so you can change the font a font is quite specific because you need to have the font installed in your system and things like that but with HTML document you can use web fonts and so the BSLEAP package or some tools there is some specific package to handle fonts that you can find on Cran I will post the link in the Slack and so you can change that as a art chunk or inside the YAML if it's supported but you can also change the font I think there is something about that in the cookbook too or in the other Armageddon definitive guide if you want to look into that. So, well, I'm just going to wrap up now so thank you all the speakers for impressive talks about Shinyab and Armageddon and automatic gradings in online teaching so the next decision is our social event mixer and that's about it thank you so much for joining me for the session