All right. So now we're going to be having heard about how to embed or base practices around about creating questions. We're going to show four projects that have used H5P in their courses. So first we're going to start off with Kaylee, who's going to talk through how she has used H5P within her online textbook. Then we're going to be talking about how I've used it within my psychology course before going to Brenna Clark Gray, who's educational technologies at Thompson Rivers University, and how Brenna has used it within her course. And then Stephen Barnes, who is an associate professor of teaching at UBC, and how he's used H5P in a very inventive way. So to get us started. Project showcase, we're going to be talking first and then Brenna and then. So the problem just to give a little bit of a scope, each of these we're going to talk about why did we start using H5P and then give some definitions as to some examples as to how we did it. So the problem for Kaylee and I is that we know that university is expensive. Over the last 30 years, we see that prices have increased over 400% for universities. And we can look often domestic students tend to pay less than international students substantially less so for international students this burden is often significantly higher. And we know within the USA is the student debt crisis of 1.52 trillion. And we know from other resources that minoritized students tend to be hit harder with student debt. And quite often it's not only the cost of university, but it's the hidden cost that comes along with buying textbooks. And so the open education movements is really posed a nice challenge to textbooks in offering free open online resources. The textbooks have kind of hit back by having these homework systems where that offer the book and then formative assessment opportunities and test banks and whatnot, which is a little bit more difficult for open resources to reply to. And so we can see that these homework systems is often what now shifts these hidden costs because one program area and one medium sized British Columbian institution for one term of just under 1800 students of about $78 for access to this online homework system makes about 140,000 Canadian dollars. So we see that there's still these hidden costs and how are we able to lighten the burden for students. And then just as we've heard, H5P and these open sources offer the opportunity for best practices, right. It provides flexibility in how we can embed we can make the textbook more interactive we know active learning has better effects than passive reading and from what we've just looked at the keynote I feel a little bit silly for showing the slide after listening to Professor McDaniels talk. But we know practice tests improve overall performance. We know that multiple tests are better. And this goes nicely with what Professor McDaniel was saying is that it's not necessarily the number of questions you get right that correlates with learning but it's actually the number of questions that you attempt that correlates with learning and getting questions long leads to better learning as we've seen Professor McDaniel saying this meta memory, especially if this tends to be followed by feedback. And we see frequent formative assessment leads to lower anxiety and also greater motivation. So we hear H5P offers this beautiful opportunity to interface the problem of student burden with the opportunity for providing interactive formative assessment opportunities. Now I'm going to hand it over to Kaylee, who for me is one of the first people to develop an interactive textbook, a fully interactive textbook so I'd like to invite you to talk about your chemistry course please. Absolutely. So my I teach first year chemistry at UBC as one of the courses that I teach, and our department approached a colleague at Professor Glenn Samus and I in 2014 and asked us to write a custom textbook for the course that we were teaching. The reason that we were asked to do this was partially that it was getting really expensive for students because they would have to purchase two different textbooks that were amalgamated together to cover the material in the course. And then there was also a lot of superfluous material in the course that they were having to buy the textbook for but it wasn't covered. So they wanted this custom book that would relate to our course specifically as organic chemist our students often need to be doing lots of 3D visualization to understand how molecules interact. So we thought we proposed to the department that we would make an online textbook that would let us integrate different interactive components that would allow us to help students understand and develop this 3D visualization. When we were talking through different possibilities we wanted also to make interactive videos that would be able to branch depending on how a student was doing to be able to adapt to where the student was at. One of our colleagues at CTLT Richard Tape suggested looking into H5P at the time and so we found that H5P actually offered all of these different opportunities for being able to create interactive formative assessment opportunities within the textbook. And so we've written the textbook and integrated H5P questions into every page and we offer this for free to students so they don't need to buy a textbook at all. So I've got some screenshots for you to show you what this looks like in our textbook. Simon's going to show you some of these H5P elements actually in situ in this textbook but I'm going to save a little bit of time to tell you about how students find these questions and whether they're useful. So here is an example of what a page looks like in our textbook at the top there's a key point section that summarizes what they will see in the content section you can see here that there's some text but right in mixed with the text. Right here is an H5P question so immediately after learning something students are asked to try it out so that they can assess whether they understood what they just learned again going on that meta memory that Professor McDaniel spoke about. So once they click this is has a hotspot style H5P question when they click to identify they will immediately get feedback underneath about how they did. Then at the bottom of each page there is a practice section with a whole bunch of different questions. They're such as interactive videos fill in the blanks hotspots multiple choice etc. So another couple of examples of types of content from H5P that we've used in the course are things like the essay question. So here for example they're asked why do you think elementary reactions with three or more reactants are rare and they could give an answer and H5P is able to look for keywords within here and suggest any parts of the answer that they might have forgotten about. So, for example, not every reaction not every collision goes to a successful reaction what else might be required and they're able to either show a full solution, or to retry themselves to add in another component to their answer. And I've had students describe these as magic when they're made well that they ask me how the heck did the computer know what I was talking about, and able to mark an essay question on its own. Here's another example of a question so this is using a drag and drop H5P so students are able to take these different chemistry molecules which are inserted as images, and they can drag them into any three of these parts. This is a chemical mechanism that students really struggle with and find really complex so it's a way that before they are. I don't think I think I looked at the registration I don't think there's many chemistry so this is a way for them to be trying this out before they're having to get to the point of writing it out by themselves on by hand. So they drag each of the components and click check at the bottom and they'll get a little plus one or minus one to show them what they got right and wrong and they're able to retry it again. And then there's also more standard questions so this is using a question set or quiz content type from H5P. You can see the little dots down here showing there's four different questions, and this is more of us standard multiple choice questions we have lots of those incorporated as well that's one of our most used question types. So I wanted to talk a little bit about are these questions useful because H5P doesn't easily offer a lot of analytics so I do have it integrated. We have our own instance of H5P so we have a little bit more analytics than than normal, so I wanted to show you a little bit about do students like these things. We asked 1100 students taking first year chemistry to take this online textbook and rate how helpful they found each component of the book so over here on the right is 100 meaning that they found it very helpful zero was very unhelpful and 50 was neutral. So you can see first that they're finding the book helpful in general, but these things that I've highlighted in pink are the things that can be made use using H5P the interactive challenge questions we made with H5P. The standard interactive questions as well as interactive videos and you can see these are the things that are coming out on top as the most helpful components of the online book, so they're really valuing these. So in addition to valuing them we want to know well are they using them. So we have some analytics using XAPI statements for our instance of H5P that is able to show us how often students are using these features and interacting with them. So there's around 1600 students registered in this course, and you can see in blue on the left here is number of students so we have over 1000 students on at a high by accessing the chirp per day, and that we can see also that there are up to 90,000 interactions with the book per student in a single day. You can probably guess when our midterm and our final exam were in this course. So students are particularly finding these H5P questions helpful to help them prepare for exams but there's also a baseline of around 10,000 interactions per day from students in the course. So on this now I'd like to talk a little bit about how I've used H5P in my course and I'll show you some of the examples of where it is embedded in my question in my textbook. So this was done in collaboration with BC campus. I received a grant with with Kaylian and Dr. Stephen Barnes, and we said we'd make 150 formative assessment opportunities into the OpenStack Psychology textbook. And so we actually ended up doing over 500 elements with over 1000 questions. So in this project we aim to provide regular opportunity for students to practice what they have learned. So over here if anyone is interested in looking at the textbook you're welcome to see the project over here. So providing regular feedback. Over here students would read on operant conditioning. They'd read all the text and operant conditioning, and then over here they'd get a number of questions, test your understanding on it. And then they would go into primary and secondary reinforcers. They'd read the next section and at the end of the next section there'd be another set of questions. And the questions wherever possible were created on bird watching and actual processes that you can find in real life. Then you read the next little bit and once again at the end you get some more questions. So we wanted to provide regular opportunities for students to practice what they've just learned. We wanted to provide feedback on incorrect options so students could improve their understanding. So for instance students would go to this little course over here which we've been looking at correlations now that you've just learned about correlations let's test your understanding. How would you describe the most likely correlation between weight and height? Well as weight goes up, height goes up. Cool, correct, I got that correct. What direction do you think the correlation between hours of sleep and tiredness is? Now let's say that someone says the more you sleep the more tired you are so that's a positive correlation and they get it wrong. This would then branch to something saying oh you got it wrong. Here's a little bit of an explanation as to why it's wrong. Here are some other examples of correlations and now let's try this again and then you try and it would be a negative correlation and you'd go correct. So wherever possible we would provide feedback to increase their understanding. We also wanted to give students a sense of control and mastery and making learning a little bit more entertaining. So we created a whole bunch of branching interactive videos where students were often seen as the expert. So over here we have a guy called Preston that we created and Preston is a 16 year old intern who's a bit of a not the smartest tool in the tool shed. And he's put in all these types of situations where he comes up with problems. He asks a question and then you as the watcher who is put in the person of a position of power have to answer his questions. So I'll play a quick little example of Preston in a sleep lab. Hey doc your EEG machine thingy is beeping. What does this mean? So having just read it it paused to ask for a question. Students would hopefully know that this is what is it not a K complex but it's a sleep spindle which you found in stage two. Let's say they get this wrong and they go stage one. When we're awake and alert our brains exhibit beta waves. In stage one of sleep we would expect to see alpha waves which are lower in frequency and higher in amplitude than beta waves. As the patient continued in stage one we would also expect to see theta waves which are even lower in frequency and somewhat higher in amplitude. In the EEG that Preston showed the patient does seem to exhibit theta waves but there's also a burst of high frequency high amplitude activity that is not consistent with stage one sleep. Which stage of sleep involves theta waves as well as brief bursts of electrical activity. So over here you can see that then it pauses and asks for a question. The question again after the explanation a student can replay the explanation or they can try. They get it correct and they move on it. And they get to the next question. Over here it can also be used to demonstrate course content. One thing that I try and find difficult to explain over the internet is over here you can see from point one to point two the yellow dot gets brighter. Now over here the yellow dots increases in brightness by the same amount but because the context is different you don't witness it. And this is known as the just noticeable difference and it's something that's quite difficult to explain. And it also tries to practice recall wherever possible. So over here not only do we have these question sets test your understanding at the end of a section but over here we can also use something like the glossary. We can help put an accordion over here. Students would have to remember what agoraphobia is we call it then they can click and test. So it kind of creates the glossary almost like a flash card. And now great this is free to students and it's free to instructors and it can be embedded anywhere. So talking about these interactive videos I'm going to hand over to Katie to talk briefly about your experiences with interactive videos. Sure so the interactive video that Simon demonstrated there with Preston the intern is one of many interactive videos that we've worked on and so I have done some studies into the effectiveness of these interactive videos. So interactive videos are ones where it's guiding a student through a problem step by step pausing to ask for input and then branching based on what that input is and it allows students to control the pace of the video. Our goal is to keep students engaged when watching videos from home which can inherently be quite a passive activity and to ensure students are understanding one step before they're proceeding on to the next step of a solution or a concept. So our goal was to create videos to support learning in our context chemistry and psychology to ensure students are actively engaged rather than passive consumers. We and then we wanted to investigate to are these videos more effective than traditional videos. So we had students randomly assigned to watch either an interactive video or the identical video that was non except it was non interactive so it was as a worked example. And we asked them then about their experience watching the video so in green here you can see our students who watch the interactive version and in blue is non interactive. Anywhere where you see a start the top we saw statistically significant differences for students who watched interactive versus non interactive and keep in mind they have no idea that a different version of the video exists. So what they found what we found was that students who watched the interactive video felt significantly more able to answer questions. They found it more enjoyable. They found that they had better control over the pace. They thought the pace of the video was more appropriate. They better felt that they were able to master concepts before moving on and they thought it was a more effective method. In a further study we replicated these results actually in several further studies. We also added an additional question about engagement and students watching interactive version of the video reported feeling much more engaged. We then revealed to students that there were two different versions of the video available and we asked them what would you prefer over on the far right is a strong preference for the interactive version over on the strong left. On the left is a strong preference for the traditional video and you can see that regardless whether students watch the non interactive or interactive video to begin with so either green or blue. They had a strong preference for this interactive video thinking that it would be more enjoyable their preference more engaging and a more effective way to learn. So what we found overall is that all the videos were effective at promoting learning but students had a strong preference for interactive videos and we are currently working to further study to parse out those long term academic effects of watching an interactive video versus non interactive video versus text but I think we don't have time for that. We don't have time to that we've been doing a study with the sleep video that I showed you but I'm happy to talk to anyone offline I would actually like to give Brena Clark Gray some opportunity just a couple of things that people that have helped in both of our courses in both of our projects projects. Brena are you here we would love to hear about how you've been using h5p in your course. Sure am hi everybody let me just get my screen going here. Okay, thanks so much for having me I'm always excited to talk about each by feet I'm always excited to chat with Simon so this is a, this is a good fit I hope everybody's having a good time at this symposium. I'm going to link a chat going to put a link in the chat here. So this is a resource that I built for a talk on this project at only global that was a bit longer so it goes a little bit more into detail. If you're curious about any of these but the nice thing is that this site also has a little package where you can download all these objects so if you want to use them as a starting place for your own work they're there and ready to grab. So, my name is Brena Clark Gray I'm coordinator of educational technologies at Thompson Rivers University joining you from beautiful sweat McAulay this morning the sun shining it's glorious here. It's also extremely cold but I'm inside so I just see the sunshine. I am talking to you today about a project I built for a BC campus grant actually the same grant that Simon was talking about previously I think. And what we did was we wanted to really look at how we could make use of h5p in developing a composition course textbook or in improving a composition course textbook. There are tons of examples out there about using h5p for like closed ended questions or like sort of facts, but how you make use of h5p in a more open ended kind of context can be a bit, a bit more complicated or tricky to think through so my background before I became coordinator of educational technologies. Here is that I was a composition instructor well English instructor but half of my load was composition at Douglas College in the West for nine years. And so I've always been really interested in how we can provide more formative feedback to students in a course that is extremely demanding on the instructor. So there is a lot of feedback in a first year writing course and finding ways to offer additional feedback opportunities for students that don't necessarily will result in more instructor workload is something I'm really curious about. So for this project we developed 168 activities for the writing for success first year textbook. It is still going through its peer review process at BC campus so I can't link you to the book itself. But if you take a look at this link that I've dropped in the chat you'll be able to see all these exercises that I'm going to talk about today anyway there. And if you're curious about any others and more than happy to send along more examples you can reach out to me by email I'll pop that in the chat in a second to a real key for us was involving students in the development of these resources. So one nice thing about having access to undergraduate research students research assistants is that they've all been put through first year composition. Every student takes first year composition and they may be coming at it from a range of perspectives so one thing we did we had the BC campus grant enabled us to hire two student research assistants. And their first job was to sit down and read the textbook and flag the places where they thought interactive interactive activities would be more useful. And then we set them about developing those activities so it was really kind of ground up from the student experience of reading the book and where they saw utility and adding additional interactive. Anyway, all this to say I'm going to focus on three activity types today really briefly and as I said please do go check out the website for more information about all of them. In sort of in order of more open endedness so something that we spend a lot of time on in composition classes, often is paragraphing structure and different choices around paragraphing structure, organizing ideas chronologically or by order of importance for example so I find that the summary tool in h5p works really well for helping students to think through the order of ideas. So in this case they're being asked to put the statements in chronological order. And so there's lots of signaling language here right so I'm going to start with every morning I make my coffee the same way for maximum flavor there's my topic sentence right. First, I do this. Next, I do this. Finally, I do this. And then, you know, so he gives students feedback right away if they're understanding the concept of chronology likewise we do the same thing with ordered importance. But that's still pretty close and they compared to what we want them to be able to do in an English class by the end right. I also make a lot of use of the essay tool the essay tool is a little bit funny because it's not for writing essays at all it's really for writing some. I mean, ideally, the summary tool I use for writing paragraphs and the essay tool I use for writing summaries. Most composition classes include an assignment around a summary or appraise the writing. And what's nice about the essay is that you can flag the kind of key terms or key ideas that you want to raise. And you can make it so that students have to address them so if we check this. That's because I didn't put anything in the box here we go this is a good example. So they've been asked to paraphrase this right using the summary tool. And what I can do is I can tag key ideas that I want them to touch on in their summary, and give them a little bit of a key or an idea as to why and then I can also give them a sample solution. One thing that I spend a lot of time doing and developing these activities is thinking about ways to communicate that there's not just simply one right answer that this is an example. You'll see that the feedback sort of over and over again articulates this is just one way to address it. And I just saw a question in the chat are you able to save student work. So these are for practice they're not going to save or submit these for assessment. But with the documentation tool which I use a lot you can save that student work students are responsible for saving their own work. Yes you can so the documentation tool was actually the first one that I came to you for academic writing classes and it's because I wanted to make virtual an activity that I do in every single class which is the thesis development exercise where we start with the question that the students are responding to and we work through a series of steps together in class, and we come to a thesis statement. I wanted to try to formalize that as something that I could share with other instructors who might need use of it but also something that I could share with students learning at a distance right or particularly in a pandemic context. And so the documentation tool good question if it's the only one I Andrew I'm not actually 100% sure it's the only one of the three I'm showing you where we can export the work at the end, but I don't want to say that definitively. So you can see that students are asked to work through there also with this exercise, working through the kinds of steps you need to do to come up with a good thesis making sure you're answering the question making sure you're hitting all the key points, those kinds of things. But they're also asked to review the thesis statement rules. Right, so what do I need to know from a thesis statement what does it have to do. And what we can do with the documentation tool is set it up as these criteria, against which they will then reflect on their own thesis statement so in the exercise below they're asked to make three points about how did you address all the components. How did you make an assertion or a claim, and are there multiple perspectives on your topic so they're asked to address those three things using the criteria tool. And then, after they draft the thesis they're actually brought back to those original criteria that they set up and asked to reflect on whether or not they've achieved their goal in that regard. It's nice because you can build in some formative self checks and formative self reflection that's really concrete. One of the things I love about using documentation tool for this kind of free writing, or pre writing work is that I really do think we communicate the importance of what we're teaching students by giving it real estate. So, you know, I everybody has probably had the experience of the difference in sort of seriousness between having a prepared handout for students and asking them to just pull out a piece of paper right we signal importance or formality in all kinds of non verbal ways. And I think that pre writing and free writing exercises are often things that students skip right on through when they're working through a textbook. These tools allow us to give space concrete explicit space saying this is important I'm giving some textbook real estate to it and I want you to take the time to do it. Obviously, I didn't do the exercise for you but you do get a document export option at the end it just saves it as a text file and then you could have students submit that to you or you could use it for group work. In the critique pre writing exercise that I built, they actually share that document back and forth in order to give each other some feedback on the tool. And I have actually been increasingly playing with the idea of using this for the assignment preparation itself so that the document that they come out with at the end is the finished document now there's all kinds of formatting limitations obviously to a text file. So they do think that it has quite a lot of promise for short types of assignments, again where we're trying to sort of streamline the process for students and give them like a clear, a clear space in which to do the work within their textbook that's one of the real strengths I think about HPP is not asking students to go elsewhere to do the exercise but having it embedded right there while they're learning the content is I think very very helpful. Okay, I see that I am just about at the end of my time so I just want to give you one of my little bonus exercise which is, I've been using the multiple choice tool set to have multiple answers which you can do you can have multiple right answers. I set it up so that all the answers are right, and then I use it as a checklist for students to do before they submit their work so lots of instructors do like a final assignment checklist, this allows you to build that right into your textbook resource in HPP. Anyway, that's the end of my time I'll pop my email in the chat please don't hesitate to check out that resource and you can download all those activities as a starting point if you want to try them out in your own classes. Thanks. Thank you very much Brenna. A real pleasure to have you and thank you for making time. We invited Brenna to be on our panel but Brenna is giving a keynote tomorrow so we're going to hide him on so thank you very much for making the time to come talk to us. Very rarely happens that there's two things. And especially with you with your essay work it's phenomenal so thank you very much for sharing with us today. So now I'd like to invite Stephen Barnes and associate professor of teaching here at UBC Stevens sort of taken the open source part of h5p and adapted it to create a non linear remixable way of presenting work to students so we're going to do a lot better job of explaining there so Stephen, the spotlight is over to you. Thanks Simon, just as a preface here, actually I'm going to share a link to the project in the chat menu that you can section of link that you can you can follow it. So this tool, although I'm using it right now for a presentation wasn't built for a presentation. So I'm using it so I can show you the tool at the same time I give the presentation that's the only reason and I'll explain why it was built momentarily. So this is the tapestry tool. This is actually what we call a tapestry and tapestry uses h5p so every node in this tapestry that I'm going to show you is actually in this case an h5p widget. It could be an h5p widget it could be a video it could be an image it could be a website, all of that can be embedded. And we've used this people have used this now in a wide variety of context it doesn't have to be just at UBC or just in a post education post secondary educational context. A lot of, we've gotten a lot of uptake with this tool in healthcare settings actually specifically professional development, and also dealing with patient groups as well. It's hard because there's so much interest right now in something called learning health systems which depends on feedback from patients from caregivers and from doctors to inform how health research progresses. And that is what is important in the tapestry tool that is the notion that people can contribute to content and they can reuse content as well. Just like any h5p widget person can reuse content. I'm sure you've seen this little button at the bottom right hand corner and if you can see it right now it's pretty small. Basically that allows you to download the widget and reuse it. If you're not familiar. So this tool was actually developed with the help of many, many students as faculty members and staff. So we currently are in an alpha version that probably is calling it a bit less than it is. We're and I'll explain where we're going with it in a bit. So we, we had over 1000 students user test this, about 500 students in lab and about 500 students in class now in lab actually means in many cases zoom since we've been doing the research during the pandemic. So at UBC, this tool has been used for a variety of context. It's been used as a collaborative mind mapping tool by people. It's been used for remote field trips using the h5p video 360 video widget. And also for individual student projects and group projects, because it's a collaborative tool that allows multiple users to contribute to what we call a tapestry network of nodes. This means that people can co design content. And specifically what we're interested in or what we were hoping to encourage is in student instructor co design of content. So, basically an instructor could set up a framework, a basic tapestry and then allow students to add nodes, and these nodes could be formative assessments or summative assessments, sorry, summative assessments that a student creates that could be anything that they could desire other than VR at this point. And that that would grow the tapestry. So the notion here is that the students would you reuse that tap the instructor would reuse that tapestry carry that student content into the next offering of the course obviously with student permission. And outside of UBC, it's being used quite heavily. So right now it's being used by the tide project, which is an educational program designed to help youth with autism spectrum disorder transition from high school into post secondary education and or the workplace. And this shows you just how customizable the interface can be so this is actually the tapestry tool, and this is a pop up widget from the tapestry tool. And then in terms of professional development where we're getting a lot about uptake right now. It's going to be used by a space BC, which is the academic communication equity Association of British Columbia to educate post secondary ed employees about how to best manage accommodations for disabled students. Now part of that process is going to be us. Doing some major improvements on the tool in terms of accessibility to support this that is we don't want to have a tool that is inaccessible in any way to students who might be might have a disability and so we're working on tweaking it to make it as accessible as possible and I'll talk about that a bit. That's sort of an excellent. I failed to show you a slide. Okay, so the other PD effort that's been done is being used by the BC support unit which is funded by the Michael Smith Foundation to educate researchers about patient engaged research. This is another example. So this shows you basically the tapestry tool being used for that for that exact purpose. So what you can see here is something that is important in the in the tapestry tool that is this user has submitted a node, and in fact they submitted two nodes. So this is a node that no one can see except for the, the actual creator of the node. And what it, what it is is it's submitted to the to the administrator and then the administrator can peruse that node and decide whether it's appropriate or not. They can approve it. That is, in which case it'll show up in green, or they can reject it in which case it shows up in red. And what another feature here that you'll see is the ability to lock questions. So you can basically create a tapestry in such a way that certain nodes are not are locked until either a certain time, or based on their completion of previous nodes or previous assessments. So in this case this is a locked question. And then there's an interactive video and a virtual tour that is also locked. And if I go in and finish it. It opens up these other nodes. So then it basically is conditional release. And this, I don't know if this is working. I didn't create this. Yeah, this is a 360 virtual tour. As I said, people use this for virtual field trips. It's just a standard h5p widget. Okay, so and you can make it full screen, of course, if you want to. And just a few last words. Actually, I'm pretty good for time. So in terms of the next stages of development, we have a generous donation and no strings attached donation every every time someone sees a corporation attached to the development of tool they wonder whether it's going to stay open source or whether you're selling it. Our goal is to maintain this is open source. And this is part of our ability to make this a freely available tool to anyone who wants to use it. Right now it's a WordPress plugin. So the alpha version is a WordPress plugin with the beta version which we had a generous donation from the Microsoft accessibility team to to develop. So we are taking it a step further in many ways. We are partnering partnering with a PC, the organization I talked to you about before to improve accessibility to the point that we reach WCAG 2.0 double a plus guidelines. So we want to also build a tool that is faster. So it's faster on any machine so we want aspect of accessibility of course is making sure that students have low bandwidth connections can still access the content and so we're implementing a graph database to boost application We're creating a standalone platform so that you can still use all that content H5P widgets on a standalone platform making it easier to author so it's not locked to WordPress specifically. We'll keep on we'll keep on updating the WordPress plugin but we're going to go to a standalone platform. We're also going to develop a zoomable interface and the reason for that is that we've we've noticed that obviously when you get over about 100 nodes and we have tapestries now that are in the thousands of nodes. It gets overwhelming to students under that hundred market seems to be manageable by most students at least in terms of the user testing we've done. But when you get in some massive tapestries it also takes a performance hit so we're creating a zoomable interface which we have mockups for now we're actually in the process of testing with students. And then we're also enabling synchronous collaboration right now if students collaborate they see a node that's locked and they can't edit it. While another student's editing it what we want to have is that the tool should be synchronous in the sense that you should be able to see the other students who are collaborating on the tool on the tapestry where their cursors are that is exactly what they're working on. Just as you would in a tool like Miro if you're familiar with that or Google Google documents where you can see that the cursor of other people you're collaborating with. And the other thing we want to do is add node based video conferencing and what that is is basically you could click on a node and open up a chat room a video chat room. So that you could basically create a spatial layout that would, for example replicate a real space and then have like a hybrid environment where students can move from room to room and interact with that content with other people. So again, we're adding a node type that in this in this case this node type is video conferencing. And this is just the project timeline for the beta version, we're aiming to launch the beta version in September, doing some real world testing of it, and then developing an onboarding system as well. If you want to contact us if you want to use the tool we're willing to support anyone who's interested. The best of our ability if you want customization we might need to recoup that expense but it's not it's nothing that nothing to worry about. And so that's our contact information info at tapestry tool calm and I shared the website link to you as well. Thank you. Brilliant thank you very much. Thank you for that Steven. I'd like to open up to the audience for any questions that you have I see we've been answering questions that they've been going along. We'll have questions for about another five or so minutes and then we'll take another five to 10 minute break before getting going with getting started with h5p. I see that there's a. Okay, so Novak's replying to a question that I put out is currently WordPress plugin seems to going stand but seems as if it's going standalone. Is that right Steven. Yeah it's going standalone, but we're going to maintain the WordPress plugin as well. So really both will be available. Fantastic. Do we have any other questions is canvas able to integrate the tapestry tools, the tapestries I mean you can you can embed a tapestry in any website so if you actually go to our homepage you'll see that there's tapestries there. So it can be embedded in any HTML page as an as an iframe. So it could in theory be embedded in canvas we haven't done that but I'm sure it could be perfect. I'd like to ask you a question, a question that was asked earlier about the branching scenarios is that how time consuming is it and so quite often I find that a lot of planning helps reduce the amount of time. I imagine these essays because they're so foreign to me and please Kaylee feel jump in as well if you've got something to comment on this. How time consuming are these essays questions to make. So, um, it's about thinking through process so like anything once you've taught it a zillion times you probably know the process you work through with students like the back of your hand right so I don't find them that that time consuming to develop. They do tend to take iteration so you know oftentimes I'll find that working through with actual students I've missed the staff or I've assumed something is obvious that isn't which you know is something we have to battle all the time. And so I don't find them particularly time consuming to develop certainly not anything like a branching scenario, but similar to a branching scenario you want to have a plan in place of where you're trying to get to so you know the goal is to have a thesis statement that a student has spent time has spent time revising against a set of criteria so you need to have those pieces in place but no I mean, I find branches there is really hard so compared to that they're walking the park. I see that essays is very hard so I'll happily do a branching scenario. Um, Kaylee do you have any do you have any comments because I know that not only have you done these essay questions but you've also looked into branching interactive videos and they can also be quite time consuming. Do you have any golden tips for planning or comments on how time intensive they are. I think the interactive videos can certainly be time consuming I've found that they, if I if I include everything so again planning is key if I include everything it does take me probably up to 10 hours to make an interactive video but that can be like a half an hour interactive video with like interactions every minute so that's a lot of interactivity to be building in and that's right from that 10 hours includes planning scripting recording a video and adding all interactivity so that's kind of what I find my all in time is but that's for a resource then that you know a couple thousand students use every year and have been using for eight years so it's not too bad of a return on the investment but it is definitely an initial investment. In terms of the things like branching scenarios and essays and stuff I find the first few that I make have taken me a long time and I've gotten faster and faster because you start to figure out what types of questions. It works well for and what types of questions it doesn't work well for and you just start to get efficient knowing like when you think of the thing of the concept of the question you think ah, yes that's going to work well or no it's not and so you end up getting quite efficient. As you do more of them. It's fantastic I see someone asked about base practices regarding copyright. And so I see will has popped in saying, when creating an h5p object, you can set a creative comments license for that object that allows other people to reuse or adapt the question. That makes h5p a fantastic VR tool. There are several different types of copyright that you can use as well and we have links to those on the h5p site. And also when using images, do you have space to put in metadata. Does the images have any copyright attributions. So we're will be talking through in the next session, how to do all of that as well. These use to create conditional logic quiz. It can be used to create a conditional logic quiz. But once again, I think, in some instances that needs to be a little bit more of a, how to say a little bit more of a hack, I suppose thinking about how are you going to, how are you going to direct students down different paths and so we find interactive videos using a crossroads function is really good for that, and the branching scenarios pretty good for that too. But that seems to be possible. So the yes, I absolutely can send a link for the HSP website that there was a request for references from Professor McDaniels talk which are now on the website as well. You can find the website over here for the symposium. With that it's 11 o'clock we're supposed to start our first session getting to know. And so what I think we'll do is that we'll take a 10 minute break, and we will come back in 10 minutes to start talking about introducing you to H5P and start looking at grading our own questions so Stephen and Brenna and Kaley thank you very much for taking time out to come talk to us today giving us some ideas for what we can use over the next day and a half to create questions for our own courses in context it's been really fascinating and we appreciate your time. Thanks so much. I hope everybody has a great time. We'll see you in 10 minutes everyone.


Everybody forced here up in 2022 top scrim like baseball hobby 12 box full case number 37 Random teams. Let's find a big hit in here. Let's find a few big hits in here. Good luck everybody You've eaten the whole pitch Counter has a more effect on the picture of the batter Try the pitcher. I'll say pitcher. Yeah.  Maybe that'd be cool God Lou Bob within all two parallels. You got orange rookie Vidal Bruhan for Tampa Bay. It's a 25 20 of 25 and a refractor of indoor Mets that one's number 12 of 199 First auto goes to the Mariners Edgar Martinez. Nice. You go Edgar Martinez Mariners bomb the camera out of my way camera I Maybe Maybe maybe I'll look back over and I'll get annoyed because it's crooked set beer gold Bobby witt Royals rookie 41 of 50 Love to see that Royal Walter there you go Walter H and a Don Mattingly Yankees based on a Donny baseball That's me Donny baseball. He knows all about the baseball Yeah Yeah, nobody scored any points in this game that yet Donnie Refractor Christian yellows Brewer 64 of 199 and green refractor rookie out of Brian de la Cruz 29 of 99 Marlins I Want to pull a super I've never pulled a super out of any top-scrumb like baseball This year the year before the year or the further inaugural yours 2020 never I don't I don't which is really weird like Belanger blackman purple mini diamond Turner Dodgers 54 of 150 I hope today's the day and base auto rookie pirates or ones that can trance Good picture hit there Perez Hoy Park rookie green atomic Bruhan rays to 99 white Sox based auto Elohimenez Mike, I think we are just top heavy in those breaks today We got unlucky on some of the other teams. There are a lot of other names. We hadn't seen we saw a lot of duplicates. It was weird Happens every now and then though six to go NT basketball student everybody Also, that's six box NBA mixer. That's supposed to be Don Russ prism draft hobby and choice There's only one dot. We only had one Don Russ left our account was off So he listed an extra one on the accident in that break So to make up for it. We have two elites instead So gain and value on that still the same price the price isn't raised the same the same. That's our mistake You already had spots sold in it. So Yeah extra autos in there Purple mini diamond Pache number to 150 for the A's and rookie auto Colton Welker Rockies refractor to 150 do that probably so cool Vaughan. Oh man. I Can't wait to check out the dynasty checklist when I get home. What's that? What if you pull the one Beltra, there probably is only one Beltra Red Sox. He was only there for one season like 2010 Kaurali new bar rookie refractor Julio Rodriguez You already know yeah, this guy knows 148 of 199 Seattle good parallels in this good color so far Look good for some big autos, but we are getting good color. So still good break thus far Mariners, I don't know who has him Eric P Grats Eric and base auto Kyle Tucker Astros Let's go now her bone release We're doing that SGC break the day before so that'd be cool Breaking live at SGC Yeah, there's day before Next SGC set it up. We're gonna like all the autos that are pulled out of it are gonna get graded as like then in there Yeah, Torkelson say a Suzuki and a blue refractor you ought an Alvarez to 75 for the Astros We'll have more info on that coming up rookie out of it Al Bruhan rays. I mean yeah Whatever rich posted about or announced or whatever that's that's much as I know really so finalizing details on all that But it'll be just like any other group breaks sold on our site The last I heard we were thinking Three jumbo case random player style That's probably what we were what we were going to do Carlson, there's a damaged cabrion haze Yeah, just ding the top left corner here. I'll send a box ever to the pirate spot Purple mini diamond trout 31 of 150 Yeah, I think it's kind of like a two-part thing Rory for the most part I don't think like Like I'm not sure specifically the entire logistics of it But I think we do the break sort of thing first and have like a talk and like a conversation It's sort of like in such interview type thing and then it will transition into like the grading process and that whole aspect of it So I think that's how it's gonna go probably makes the most sense to do it that way Orange super futures auto Andrew Vaughn to 25 White Sox you go White Sox nice card But we'll see we're still got to iron out some of the details It'll be fun. It'll be fun random Marsh rookie O'Neill cruise variation rookie Pirates and green rookie Josh Lowe raised to 99 and Based out of Jorge Posada Yankees hip hip Jorge That's what they did this wasn't me. That's what they did Yeah, there's gonna be a whole like walkthrough I'm sure they're gonna want to get in depth That's you see as you see would probably want that like that quite a bit honestly It'll be a crazy day, but it'll be it'll be very cool. I'm excited Ron's you can chair us Malina green refractor Tatis Dude is crushing the miners. He's just it's almost unfair for those kids down there 10 of 99 and Gold refractor auto A's mark McGuire won a 50 Nice big Mac color match to with the A's First time we've anyone's ever done a break at a grading service and the cards from the break it graded like immediately First time there in general just in general wander rookie It's industry first Rookie green atomic price and start Philly's 94 of 99 It also makes it a lot very easy because they're literally in Boca. It's like a three-hour drive for us. Yeah, I know that close So it's so much easier than doing something else and gold mini diamond auto Barry Larkin at 36 of 50 reds Nice finish of the case there. Thanks again everybody for joining moving on


So, my name is Natalie Holia. And I'm Joe McLean. And we work at ThoughtWorks New York. We're product and experience design consultants. ThoughtWorks is a custom software consultancy. And we help our clients with ambitious missions to make great products. For the last year and a half, we've been working with a company called Amplify Access. Amplify Access is an education technology startup in New York City. And they're using lean product development to build a tablet for K to 12 education. During today's presentation, we'll spend the first half sharing some of our experiences and some of the things we've learned. And in the second half, we'll take your questions and have a discussion. So before we start, what do we mean by lean product design? Lean principles came out of Toyota manufacturing as a way to eliminate waste. And it's an idea that's become really popular in a lot of areas. So when we're talking about developing a product, it means moving away from upfront design to continuously changing and evolving the product as it's being developed. We do this through a process of prototyping, testing and feedback and take somewhat of a scientific approach. We come up with a hypothesis of what might provide value to the customer. Then we do user research and testing to validate if it actually does or not. And only if something's valuable do we continue to spend time and effort in developing those features. So it's a really flexible approach. It allows you to pivot your product quickly based on learnings and arrive at a more valuable product faster. Lean is a really exciting way to build a product, but it definitely comes with its challenges. One of the things that we've seen is that the challenges to lean product development change depending on your team's size and the maturity of your product. When you're starting out, if you have a small team or a relatively undeveloped product, your challenges are largely around finding the right starting point, finding ways to move really quickly and have an efficient process and make decisions effectively in an environment where there's a lot of uncertainty and a lot of unknowns. You may not be totally sure what the endpoint of your product looks like. However, as your team becomes larger and you have more of an investment in your product, there's more pressure to plan, especially if you have a large development team that you have to resist. There's a lot of potential features and a lot of different directions you could branch into and so you really have to be careful about where you invest. Because you have an investment in your existing product, it can also make changing more difficult. It becomes harder to pivot. And also you have many more decision makers. So making decisions really quickly and efficiently can become more challenging. In our time at Amplify, we have seen them grow from a very small organization to a much larger organization. And so we've experienced a lot of these challenges firsthand. We'd like to talk about some of the different tools and processes that we used at different points to address these challenges. So Amplify's first approach thought works with an ambitious mission to change education in schools across America. They had a huge vision for their product, this tablet for education. And we knew that we couldn't build it all at once. It was going to take quite some time. However, Amplify was really committed to building a product that real teachers and real students would use in real classrooms. So in order to do that, they wanted to get the tablet out early into classrooms and pilot it and get feedback from exactly those people to ensure that it would be successful. So our challenge was we needed to find a starting point and move fast to get the tablets into classrooms early well before commercial launch so we could start getting feedback and learning and improving. So when we were a small team and we were just getting started, we really had to focus on finding the right starting point. So we looked across all the things that we hoped the product would do, all of the areas that we hoped it would provide value and tried to identify a solid foundation for moving forward. So we really decided at the beginning to focus on just facilitating the communication between teachers and students. We thought by focusing on this value we would give ourselves a foundation both from a product perspective and from a technical perspective that we could build forward from and expand into the other areas that we hoped our product would provide value. So once we had our starting point, it really became about finding ways to move as quickly and efficiently as possible to provide value. So a lot of our practices that we enabled were about getting that speed and efficiency. We did a lot of sketch to code and to design dev pairing. So rather than building wire frames we just sketched things out and in many cases had the designers sitting right next to the developers as we built functionality. This accomplished a few things. First of all it helped us resist the temptation to plan too much up front and do a lot of design. But it also meant that when we encountered things that were maybe more difficult to develop than we anticipated we could make changes on the fly and not have developers wasting time trying to implement something that was maybe harder than we realized when we designed it to actually implement. A big part of that was really tight collaboration between all the different roles on our team. So this is a picture of a developer talking with a QA and a product person and a BA and the designers all working together to try to figure out how we were going to approach building a particular functionality. This really helped us avoid waiting for decisions to come down the pipe from different groups. We did a lot of this stuff in flow as we went. Finally we had a design wall so we took all the things that were going on in the design team all of our different wire frames our sketches and even our visual design and printed everything out and put it up on the wall. This was really helpful not just for facilitating design critique but also for just showing the entire team what was going on in the design process. This was really important especially early on because things were changing really rapidly. So having that visibility for the entire team was really helpful and it allowed us to see kind of the entire scope of what we were working on. So all of those things were designed to work together and they really helped us move really quickly and kind of the net result of all that stuff was being able to get a product out to market in less than six months. So we had a product in schools and we were getting feedback and learning. So Amplify began as a startup with this very small single team that were able to work closely together. But obviously as we grew we reached a certain point where that was no longer possible. We began to have all these different moving parts which made communication and staying in sync really hard. However our product was still young. We still wanted to make a lot of changes. We were getting a lot of feedback. We were pivoting and so to keep the product consistent and coherent it became a worthwhile investment in our time to make some of our tools more efficient to allow us to continue to make changes quickly. So one of the things that we did straight away was change to a flat UI. Instead of having designers slicing assets so we could have rounded corners and shaded backgrounds we changed our visual design so that it looked similar to this so that developers could change the size, the color, the shape, all the UI elements they could make them directly in the code. Flat UIs become really popular recently and this is exactly why. It's not just a fad it actually makes designing and developing products much faster. The other thing that we did was use style tiles of our flat UI. So instead of creating a visual comp for every single wireframe for a new feature that was being built instead the designers pulled together these boards which showed all the common UI elements and developers could refer to the wireframes refer to this style board and start working in the code straight from there. It allowed us for a very light touch to have a consistent level of visual design. Another thing that we did was actually make our own custom icon font. So again moving away from designers creating images which had to be changed every time we wanted to make a change in the software they would instead once off create a scalable vector and then we pulled all those together into an icon font which developers were then able to style those icons again the size, the color, the shape everything in the code which made it much faster to make those changes. We started using a live style guide as well. Because things were changing so rapidly and because things were new if we had a new UI feature that we wanted to build it may or may not be in the code already. So developers could refer to this live style guide and see has it already has it the visual design already been done. If so they could reuse that and keep that consistency and if not they would add it. And it really just saved us from creating that same orange button six times in six different places. Finally we pulled that together into a style gem. So creating a library where we could use those styles across products. We started to instead of having one product we had two and we wanted to share that visual design across the products again very easy to stay in sync make changes. So all of these things of making our tools more lean it was an effort to keep making changes really easy so that we could keep moving at that rapid pace keep learning and stay lean. As the team grew though it became very large we became a team of 50 people 60 people and even beyond now and we were distributed across the United States across Brazil and India it became more difficult to make decisions like we showed you before with everyone kind of huddled around a desk and collaborating together. It really became unrealistic when we reached this scale. So we needed to rethink the way that our organization was actually structured. So we moved away from having larger product or feature group oriented teams and split up into smaller empowered goal oriented teams. The teams were still united by a common purpose the kind of the company vision for education but they were able to set their own objectives and frame their own hypotheses as to how we could provide value. This allowed them to make decisions independently. We were still unified by our common purpose but the day-to-day decision making at the team level was able to be much more flexible. Also to that end we took our design group which as we had grown has started to kind of work as one coherent group and we split it up. We took a visual designer and a user experience designer and we embedded one each and all of these different goal oriented teams. We still brought everybody together frequently to talk about kind of the decisions we were making across the product and what our interaction patterns were going to be and also whenever we needed to do visioning for new feature areas. But for the day-to-day decision making the designers were embedded in the teams and were really able to help us get back to kind of that collaborative model of everybody working together that we had had at the beginning of the project. Also at this point we really started focusing a lot more on problem definition. So before we would do a design workshop we would have a problem definition workshop and so we would go through a process of really trying to understand the problem that we were trying to solve for our users and then we would use that to kind of frame up and validate all the different solutions we were creating. In a way this acted as a filter so we could understand which of our concepts were more effectively addressing the problem statement that we had set out to accomplish. All of these things kind of worked together to start to build in a lean mindset into our organization to move us away from just thinking about efficient processes even though that was still important to really focusing on value and to measuring everything that we did against the value that we anticipated to provide and validating that with metrics. So we've really been on a journey over the past year and a half with this team from doing lean to being lean. It's much easier to start doing lean processes using lean tools on the ground than it is to change the organizational approach to product strategy. However, as an organization grows and becomes more complex it's ever harder challenge to stay lean and avoid this upfront planning and design. And so we were continuously challenged to evolve our thinking and move this lean thinking earlier up in the decision making process to make us more effective at eliminating waste and creating value. It's not to say obviously that we stopped using these processes and tools but rather we built on top of this and the way that we did lean evolved as our practices matured. We moved from trying to build features efficiently to trying to build the right features. Lean product design is a great approach to developing a product but it won't work unless you kind of absorb some key organizational values. Even if you're an expert in lean and you're really excited about bringing it into your organization it's not always possible to jump straight into using lean to define your product. You may need to work slowly and incrementally to build trust in this way. It's scary not knowing up front what the endpoint of your product is necessarily going to be and you have to have some trust in the process to bring you to a good place. You have to have humility you have to accept that you're not going to have all the answers right away. It's not about being right or wrong it's about taking a scientific approach and always be willing to validate or invalidate an idea. And you can't do that if you're too attached to any specific concept or decision so you have to have humility. Finally you have to have courage you have to be willing to start anywhere you have to not be afraid of making mistakes you have to be willing to make a mistake if it's in service of doing the right thing. So we started off by telling you about amplifiers ambitious vision. They were thinking big about a new kind of product to transform the way that we teach and learn in schools. We had to take a step back from that vision and find a way to start small and move fast. And lean product design was a technique that enabled us to do that with great success. Taking a lean approach to product design is really well suited to situations when you're trying to develop a new kind of product or enter a new market and you have a lot of unknowns. It helps you to learn quickly about all of those things that you couldn't possibly have all the answers for up front. It helps you encounter all of those unexpected external factors that were always going to come into play earlier rather than later. And so from this vital feedback you can make changes early on in the product development and for a new product or a new market that can be really key to its success. So we hope that this has given you an entry point to using lean product design in your organization in some shape or form. Whatever phase of maturity it's at and whether it's big or small. Thanks. We're sure you have a lot of questions so as you can see we've left a lot of time. We'd like to use the second half to have kind of an open discussion and I open it up to you guys. Do we have the portable mic? I think there's a... Is there a mic on it? Oh, you have it. I guess if you could just give it out to whoever has a question that'd be great. How did the culture in the system that you're trying to build affect the hiring decisions as you're building the team? Do you know anything about how that process went in picking the right people that could rapidly adapt to the way you were doing things? Yeah. It's a big part of thought works the way that we hire people. We spend a lot of time and effort trying to find people with the right cultural fit and who are kind of open to working in these ways. So we definitely help try to encourage those principles and philosophies with amplifiers. They were hiring as well. There's also an aspect of, especially as a design team, I think we really tried to focus on people who could think in terms of simple and elegant solutions. That's something we really very specifically targeted when we were looking for people in interaction design especially. When we were evaluating portfolios that was definitely something that we had in mind. Is there effectiveness of reducing a complicated problem down to a simple solution? When you were changing the culture, what were some of the hurdles ... Sorry. When you were changing the culture, what were some of the hurdles and challenges that you had to overcome to get buy-in with the culture and flow changes? Yeah. That honestly was a huge part of what we've been doing. Probably over the last six months. It does take a long time. A lot of it was around being clear on spending very hard to talk about a product or any use of value that you're trying to ... One of the biggest shifts I saw is when we really started focusing on the problem definition workshops. One of our colleagues, Paul Sullivan, has really spent a lot of time thinking about how to frame up a problem workshop. Actually, if you're interested in getting more details, I'd be happy to talk to you about it afterwards. We really discovered that when we were really diligent about defining the problem that we expected to solve before we even went into the design process at all, it really changed the entire tone of the conversation, and it became less about people's opinions and more about how we were going to get more insight into the value area that we were trying to address. I know it sounds like a really simple thing, but even just going through the discipline as a team of starting by framing the problem and writing it out and putting it up on the wall and using that to define and kind of filter through the rest of the design workshop was extremely effective for us and changing the tone of the conversation around what we were talking about. Yeah, and also just that shift in thinking about the product owner not having to always have all the answers to have that pressure of always being right. That shift in mindset really allowed the rest of the team to contribute and for us to collaboratively try to arrive at the best solution. It also, I think, provided a path for the developers to be a lot more involved, when you're having the conversation on the level of addressing problems, it kind of opens up the floor for people to present different kinds of ideas. Like Natalie said, taking the pressure off the product owner or the design team to be the generators of these experiments. Two questions. One is that the level of six and the other one is the level of 60. So when you're at six, you mentioned scientific approach. In lieu of any statistically significant data, what kind of scientific approach would you use, right? And then at the level of 60, making very quick decisions is very important. And how do you manage that with so many strong-willed and knowledgeable people on the team? Who does this decision-making? What process do you use, if any? But at the small level? Sorry. Here, we'll take that mic. So your first question was about the jinx. When we're small, how do we validate without? Yeah, so I'd say the approach is scientific, but the conclusions are maybe not, right? Because it's about human interaction with software. Everyone's different. One of the things that we tried to do early on was build a better understanding of teachers and students. So we did a lot of user research, going out and talking to them, before we even had a product. So not testing out our product with them, but actually finding out, what are some of their day-to-day pain points and challenges and issues that they have? What are the ways that they think about teaching or learning? What gets kids excited by school, or the teachers or not? So really trying to gain that understanding so that we'd be able to better make decisions from that. And again, taking our product into the schools early and observing people using it, observing teachers or students struggling with particular things. So that's how we were able to get feedback quickly, even when we're small. So yeah, it wasn't, you know, about 50% of these people in this A-B test, it was very much, let's go and talk to the real people that are using it. The second question. Yeah, just from, I guess, like a kind of nuts and bolts practical approach, one of the things that kind of guided our research at that phase is, we would just talk to people until we started hearing the same things again and again. And that to us, I think was the indication that we had talked to enough people a lot of the time. We were at that, especially at that point in the product development, we weren't looking for statistically significant information. We were looking for a lot more qualitative data. We didn't really talk about research very much, because that's a whole other story and how that has changed and grown over the organization. I think we're more data driven now. It was important at the beginning to be more qualitative because as you identified, we didn't have enough people to have that kind of statistical insight at that point in the product. Oh yeah, decision-making is really a big second part of the question. Yeah, I'll repeat it. Yeah. So I believe the second part of your question was with a lot of strong-willed people who have well-informed opinions, as the organization gets larger, how do you manage decision-making? Roughly, is that good? How would you start? Again, I think one of the techniques that really helped us with that was the problem definition statement and having that focus on, it's not about individuals asserting their opinions and their ideas were better than other people, but it was about looking at what was the feedback that we were getting from our researchers at the schools and from the teachers that we were bringing in and seeing how people were using the software and trying to take that approach of moving away, detaching your individuals and their association and their fondness of their own ideas, I guess. That exact problem, though, was also part of what drove us to break into much smaller teams. I mean, it was really a direct reaction to that issue. We had started a sense that, especially in conversations about architecture and technical decisions, there were just so many decision-makers that it was hard to get conclusive results, but this is exactly what you would expect. So having teams have more specific focuses within the product and kind of be able to make decisions a little bit more independently really helped us with that. We definitely needed to make an investment in infrastructure to enable that. Obviously, like, there need to be certain systems in place for teams to be able to work independently. So a lot of that, like, infrastructure and plumbing between groups was important to figure out at that phase, which is, again, kind of a larger story. But that was a big part of why we made that decision at that point in time. And again, I guess, the goal-oriented teams was also an attempt at that moving back to the situation where a team can work really tightly and have, you know, very fast real-time feedback with each other and be empowered to know that the decisions that they're making and what they're choosing to do are right because they can measure them against these known goals. Whereas if it's building out a product backlog, you know, it can take a long time for kind of product vision to come down through the organization. I'm curious to hear a little bit about the makeup of the small teams that you created, who, you know, how many engineers, who's a part of the team, what's the balance that you found that works well? Sure. So it's a little bit different depending on the team. We have some kind of back-end infrastructure teams that, as you would expect, are a little more engineer-focused. But generally speaking, I would say that, like, for instance, the two teams that I work on are both six to seven developers, one to two QAs, a dedicated product owner for the team, and then, as I mentioned, a user experience designer and a visual designer. So I guess, like, we've been hovering around a three-to-one developer-to-designer ratio for a while, and I think we've found that that works pretty well for us. Anybody else? I'd be curious to hear how do you deal with the earlier phase of design being account for things that might happen later, and obviously, Leen, you don't want to do that, but then, as new patterns emerge and new needs emerge on common components, how do you manage moving those forward so that you don't have a product that has dissimilar behaviors across what should be common components? The question was around how to keep the product coherent, or it was around... One of the things that, yeah, we noticed early on was, as we were very first beginning to build this product, we had no UI patterns, right? So as time went on, we really felt a need to kind of create these patterns and make sure that we were using something, we were using it elsewhere. Initially, what we went with was the things that were kind of faster and easier to develop, right? Because we didn't know if we were going to be throwing out that feature later anyway. So we just wanted to kind of get something there that worked, that was usable but maybe not the greatest thing ever, kind of. And so once we kind of got the product to a certain point where we had examples of UI patterns, we were able to see, you know, through observation in the field which ones worked well or not and ones that worked really well, we then took those and we made those UI patterns and so our design wall was a big part of that, was having, you know, our designers kind of huddle around the wall and talk about, you know, I noticed that you designed this feature but it's really similar to this and this one here. How about we use the same pattern? So I think that the design wall sort of facilitated that cross-team collaboration and consistency. Another thing we did is, you know, there's a lot of conversation around tech debt, the idea that over just the natural time of building a product, you wind up with a lot of technical tasks that kind of need to be done to have a solid infrastructure that is good for the long-term. We found that we were building up UX debt as well and one of the strategies we employed is having a dedicated designer, developer pair every once in a while just to kind of go through the entire platform and make sure that everything was consistent again. And one of the strategies we employed is just kind of like, even though we would still write stories for that and track the work, we would kind of track that outside of the progress we were measuring and the rest of our teams and consider it just like you'd consider it to be almost like a tech task to pay down that tech debt, we would pay down our UX debt from time to time as well. Yeah, I'll just quickly add on that as well that that actually worked much better to do it afterwards as a UX debt because when we tried to bring it up front and bring that consistency beforehand, it just like suddenly, you know, slowed down the speed of the team greatly because one thing would change and it would change a hundred other things. So it really was much better than before, you know, particular launches that we went through and did that quick cleanup. There was a great comment in the keynote about engineers planning for reuse before they plan for use. And I think like the design equivalent of that I think is planning for consistency before you have even figured out what you need. So we, well, you definitely need to do a certain amount of forward planning. It's a lot easier to figure out how consistency should work in retrospect than trying to anticipate. At least that's certainly been our experience. I think we're out of time. I'm afraid you are out of time, but thank you so much, Joe and Natalie.


All right, so we've seen a couple of quantum mechanical problems so far, the free particle and the 1D particle in a box. Now let's consider a third problem, which turns out to be very similar to the particle in a box, but with some important differences. So that's going to be called particle on a ring for reasons that will become clear shortly. But mainly so that we can highlight the differences between the particle in a box and the particle in a ring. Let me just summarize a few things we know about the particle in a box, namely that if we take a particle and confine it to a one-dimensional box between zero and A on some axis, then the particle has some wave function. There's a lot of different wave functions I could have drawn. Each of those wave functions has to hit zero at the edges of the box. Those wave functions had a particular mathematical form and they have a particular energy and again each one of those different wave functions, n equals one or two or three, has a different wave function, has a different energy. So that's what we know about the particle in a box. Now let's consider a different problem. Let's consider still confining our particle to one-dimensional line, but now let's take that line and let's bend it. Bend that line around all the way until it forms a circle. So I could say still length A, but now that length is the circumference of the circle. So if I want to describe where the particle is, instead of describing where it is on the x-axis, I'll just describe at what angle I find it.  I'm still going to have a wave function. I'm going to have a wave function that depends on the coordinates of the particle. Now because the particle can be anywhere in the plane of the board, it feels like a two-dimensional problem, but because if I'm confining the particle to live on the circumference of the circle, r has to be constant. So the only variable that's allowed to change is theta. So it's still a one-dimensional problem, but now I'm describing where the value of the wave function is a function of theta rather than as a function of x. So that's one of the first differences between this problem and the particle on a box problem. We can see now why it's called particle on a ring. Particle is confined to this ring. We could have called it particle on a circle, or if we've got this string that we've now wrapped around into a circle, we could call it particle on a rope. There's a lot of different names you could give to this particle. We'll call it particle on a ring. Some people call it particle in a ring. So we're going to have a wave function that depends on theta. The boundary conditions, the thing that for particle on a box required the wave function to hit zero at the edges of the box, that's no longer required. The wave function doesn't have to hit zero at the edge, because there is no edge of this box. If I have a wave function that starts somewhere and oscillates as it goes around the circle, all I need is that when I get back to where I started, notice I made these two curves join up. So really what I need to have my boundary condition is that when I get back after one full lap, back to where I started, plus one full revolution of the circle, that needs to have the same value as when I started. Otherwise, if I came back and missed, if I came back at a different value than when I started, I wouldn't have just a single function. I need to have the same value on every trip around this loop. So that's the boundary condition that's going to replace the zero at the edges boundary condition for particle on a box. So that's another difference between this problem and the particle on a box problem. And I'm not going to go through the solution in detail, I'll skim through the details of this particle in a ring problem, because we've already seen something very much like it in particle on a box. But if we solve this problem, we still need to write down the Schrodinger equation. I'll write that down for us, and then I'll explain the differences between it and the one for particle on a box. So this looks a lot like the particle on a box, 1D particle on a box Schrodinger equation. In that problem, our second derivative was d squared dx squared. Here the variable we're interested in is theta. I don't just take d squared d theta squared, I take 1 over r squared d squared d theta squared. You can take my word for it that that's what the Schrodinger equation, the little plus sign looks like for this problem, or if you want to double check me, what this is is if you take for this two dimensional problem, if I write down d squared dx squared plus d squared dy squared, because I have two different dimensions, if I convert those x's and y's to polar coordinates, and then I require r to be constant what survives is just 1 over r squared d squared d theta squared. If that sounds like fun, pause the video and do that little exercise right now. If that doesn't sound like fun, like I said, you can just take my word for it. That's the Schrodinger equation for the particle on a ring problem. So again, it basically boils down to the same problem. Just like particle on a box, we're going to be looking for some function that if I take the second derivative of it, I'm going to get back the same function, but with some constants and a negative sign. So, like before, function, a function whose second derivative is the negative of itself is trigonometric functions like sine. So sine of some constant time theta is still going to be a good solution. Unlike the previous case, cosine also works. Cosine didn't work before because we needed something that would start at zero, right, the wave function had to be zero at the edge of the box. Sine is zero at x equals zero, cosine is not. So, sine was a good solution for particle on a box. Cosine's also a good solution for the particle on a ring. If you're not too terrified by complex numbers, we can also use e to the i times a constant times theta. That'll also solve this equation perfectly well. If you don't like complex numbers, we can put off thinking about complex numbers until a little bit later in the course. And you can pretend I never wrote that down. For the moment, let's just stick with constant times cosine constant times theta. The next step would be figuring out what that value of k needs to be in order for this to be true.  Or whether you think about it mathematically and think about how to obey this boundary condition, we can convince ourselves. Cosine of any integer times theta. So cosine of theta, cosine of 2 theta, cosine of 3 theta. Or in fact, cosine of 0 theta, or cosine of negative 1 theta, or cosine of negative 2 theta, those are all fine. So any integer, cosine of an integer times theta is a perfectly valid solution to this Schrodinger equation. And again, that just means the sine or the cosine needs to oscillate one full round trip as it goes around the circle. That would be cosine theta, or two full oscillations as it goes around the circle, cosine of 2 theta, or and so on. Or for 0, cosine of 0, then the wave function is just constant. It remains constant without oscillating as it goes around the circle. So any of those solutions are perfectly good. The next step, so each of these different solutions with a different value of n is a different wave function that we can call psi sub n, just like we did for particle on a box. If I want to know what the energy of those solutions are so that I can compare them to the particle on a box solution, we just need to plug these wave functions into the Schrodinger equation. So plugging into the Schrodinger equation, I can say minus h squared over 8 pi squared mass, 1 over r squared. Second derivative of the function, second derivative of cosine is negative cosine and I'll pull out an n twice. So I've got capital A and I've got an n squared that's been pulled out, cosine n theta. That's on the left, on the right I've got, that must be equal to e energy times the function, a times cosine n theta. Now this is just, I'm sorry, second derivative was negative a n squared cosine n theta so that pulled out an extra negative sign. I've got an A and a cosine n theta that cancels on both sides so what I'm left with is that the energy of this psi sub n wave function is equal to this collection of constants which looks like h squared over 8 pi squared mass r squared and an n squared. So again that's similar, this feels very similar to what we did for particle in a box. If I compare these directly they're not the same. I no longer have a box length to worry about. Instead I have an r squared in the denominator down here and a pi squared that survives that didn't previously. But aside from that the energies do depend on n squared just like they did before. All right so, and I guess the last step would be if I want to know what the value of this normalization constant is to normalize this function since we've not done that too many times. Normalizing the function requires that I take the wave function times itself or its complex conjugate and that has to equal one if I integrate everywhere. What that means for us is that if I integrate theta from zero to two pi, I'm sorry, if I integrate over theta from zero to two pi the wave function squared is going to look like a squared cosine squared of n theta. If I care about what the normalization constant is I would have to complete this integral solve for a and that tells me what the normalization constant is. It doesn't end up being squared of two over a as it did for particle in a box. What we get, skipping the details of doing that integral, this integral turns out to be a squared times pi so a results in one over squared of pi. So that's the normalization constant. The last thing worth mentioning about this particle on a ring model is why we should care. Why did I say take this straight line and bend it around into a circle? What in the world would we ever use this particle on a ring model for? The answer is if particle in a box was a useful model for simulating conjugated hydrocarbons that had the shape of a straight line if I take one of those conjugated hydrocarbons and bend it around into a circle then the particle on a ring model will be a good model for those molecules. For example, a conjugated hydrocarbon bent into not a circle but a hexagon would be benzene and there's a variety of other cyclical molecules that we could use as applications for this particular model. The reason it's worth talking about benzene as an example in particular, let's go back and look at what these energies are. If I draw a graph, a chart of the energies of these molecules, the n equals zero solution has an energy of zero times some constants. So the energy of the ground states down here, there's two states n equals one and n equals minus one that have the same energy as each other and then somewhere up higher there's two more states n equals two n equals negative two that have the same energy as each other so some of our states are degenerate, there's two states with the same energy at this level and at this level but the ground state is not degenerate, there's only one state. If I were to start filling these states with electrons notice that I fill up the first level after putting six electrons in, not coincidentally benzene has six pi electrons. So the six pi electrons in benzene occupy these lowest wave function energy levels that correspond at least approximately to these particle and a ring energy levels. So the reason benzene is a very stable aromatic compound is because it fills up the ground and first excited state. You may remember from an organic chemistry class, Huckel's rules about aromaticity specify that if you have four n plus two electrons a molecule will be aromatic and that is a consequence of this particle on a ring model. Two electrons per energy level, four n plus two electrons is like two n plus one energy levels so if I fill up to here six electrons gives me an aromatic benzene molecule, if I fill one level higher I would need 10 electrons to make the next aromatic ring size. So Huckel's rule about aromaticity is really an example of the particle on a ring quantum mechanical model.