Yeah, welcome to my talk on the great topic concerning distributions and a question actually to you as the audience. So we are preparing an open source project for embedded devices, which supports Linux-based embedded devices more with the needs of such devices. Open distributions typically support desktop or server use cases and we want to specialize on that. And I'm basically asking the question, do we need something like that and I will go into detail what we plan and hopefully we can, yeah, I can get some feedback whether you think this is a good idea or not. My name is Lars Geier-Blaumeiser, I'm from Bosch.io, which is a subsidiary of the Bosch Group and basically, yeah, this is an initiative of the whole of the Bosch Group. I'm only representing Bosch here for today. So that's what we plan to have an holistic approach for product diversity based on Linux. So it's more than just a distribution, it's also an infrastructure and all the stuff needed to easily build products based on Linux, especially as I said, embedded or IoT devices built on Linux. And there is already a blueprint for what we plan because there is already a project where Bosch also participates and invests a lot. It's called Apertis and here's the mission statement for Apertis, a versatile open source infrastructure originally tailored to the automotive connectivity needs. So it was built by a Bosch business unit around calm multimedia. So it's more about, yeah, multimedia devices for vehicles, but we saw that it's also fit for a wide variety of electronic devices. So we use this also in other business units nowadays and that's the potential we saw. And before we solve this internally, we thought perhaps it's a good idea to share this with the community and get other companies also into this initiative. What Apertis is all about, big important parts of Apertis are of course security because it's connected devices, they have to be secure and modularity. And I will give you a glimpse in the talk about what we mean with modularity. It provides a feature rich framework supporting add-on software and resilient upgrade capabilities. So, of course, updating is this topic. Beyond an operating system, it offers new APIs, tools and cloud services. So it's also the infrastructure and the services around the core distribution. Coming to Bosch, so I'm from Bosch and why are we interested? Basically because we connect everything. So there are so many different business units at Bosch who provide IoT devices. And here you see in the first column you see more tracking, management, device management, use cases. So, for example, we have all these here are products or projects we had so far. There is a project to manage connected filters which are placed somewhere on the lower left side. You see track my tools, it's for workshops who use a lot of power tools, drill engines and stuff like that. And they want to manage where are they on which place where they build something do we use certain tools? Eating systems are nowadays typically intelligent so you can also control them in your smart home environment connecting them with your house. We have lawnmowers or gardening stuff, household appliances which are connected nowadays. The third column is more on the industrial side. So industry for zero is of course a big topic for Bosch but also tools and systems in the logistics area. And finally on the right hand side also some nice topics in the agriculture or more agriculture areas, internet of oysters, monitor how your oyster farm is running. And finally of course the topic of software updates over the air is a big topic at Bosch. So you see a wide variety of things and this is really an amazing number. So Bosch produces one million things a day. So it's really a large number of devices which are connectable. Not all of them are then connected of course but which can be connected in principle. And that's why our chairman Mr. Denner said three or four years ago that he expects every electronic product to be internet capable to be connectable and for him the question is not whether this will be the case but only when this will be the case and that's when Bosch changed the strategy towards yeah being able to connect everything they produce and this is the way we are still going. We have lots of connected devices but still there are some to remain and of course new developments need efficient development methods. So if you look at typical demands of today's industrial devices we see that there is quite some change when we compare with Bosch products for a few years ago. Today's system evolve so if you have a not connected device development typically more or less ends when the product is raised to the market in the car area for example there will be perhaps some updates bugs that are detected in the field or for example changes to the legislation exhaust gas legislation for example requires changes in a project but typically it ends at the start of the product. Nowadays with a connected device development lasts until the end of the product usage so you always have security issues you have connectivity issues and you have to be able to maintain and develop the software over the whole lifetime of the product. Also products are integrated so we have a thing in the internet of thing is not a valuable thing except that it provides data which is then integrated into an IoT service and the value comes then with the IoT service so we have the things integrated and yeah they have to talk to each other and exchange data. We have topics like time to market which is rather short these days so products come into the market really fast with this means efficiency of development and of course that's also something Bosch observes that the highly inventive areas are typically nowadays open source ecosystems where collaboration in those ecosystems enables new trends to appear magically more or less so it's not one company that is providing the innovation but a set of companies pushing the innovation forward and finally what we see is that there are blurring boundaries for products so what you on the one hand see for example in the in the production area is the batch size reduction and the same applies for products so you have more and more specialized products which are related to other products but still they are specialized and fulfill very specific use cases on the other hand side you also see that by using existing products you can or by having existing products and the hardware of those existing products you can easily add new services a good example for is here cars where you can add new comfort functionality or or active or passive safety functionality on top of the existing sensors so the customer can buy additional services and they are installed only as a software product on his car so these are the trends we see and when you look then at the software what are the typical demands for the software one is of course that you have optimization for a specific hardware so there are of course standard hardware architectures standard reference boards that are available but typically the product then has a very specific board design with potentially hardware devices which are unique for this device and the specific optimizations for that are needed also size of course because you want to not over size your products optimized for the specific functionality this could also be for example environmental influence like the place where the device has to work for example in a car in a dirty area or somewhere where you have a lot of sunshine and it's very hot and stuff like that are very cold so also an optimization for the functionality of course the systems have to be secure so we don't want to risk security issues they have to be reliable they have to work without maintenance direct maintenance for a long time they might have a user interface but that's typically as you know from embedded devices simplified it's not a feature rich user interface it has to integrate into the relevant environment be it the connectivity be it the sensors and the actuators but also for example if you want to place it in a car it has to match the space in the car where the device has to be installed and of course you want to easily plug and play connectivity so whoever has built up a smart home knows what a pain easy or not so easy plug and play connectivity can bring and I mean many of those systems are not really maintained in the sense that there's always someone around to check what's what's going on so they have to be easily connectable and stay connected over the time. Security I already mentioned hardening against any kind of attack you don't want to get the data corrupted or listen to from from people that don't have the the right the right to to read the data but you also don't want to have your devices be the source of a denial of service attack for example so by having those millions and billions of devices in the world they're of course attractive to start such kind of attacks so they have to be hardened and secure to not allow such things. When you talk about embedded devices you also talk about commercial products components and proprietary components so you want to a mix of open source of course but also add some proprietary stuff that might be bought or developed in-house and for all those components be it open source or proprietary you need a life cycle management of the used components so you need to know which components are in so that for example you learn about a vulnerability that you can react and know where this vulnerability might affect products and can also analyze whether the vulnerability may have an impact on your product and if this is the case and you have to update of course you need a reliable update system so you don't want to have sorry you don't want to have updates breaking and and making the device to a prick that is not running anymore so you need a mechanism that doesn't atomize update but of course you also want to ensure that only those updates get to the device that are really really needed that are coming from you and not from anyone else and are not corrupted on the way and yeah we talked about efficiency so it's of course about the use of infrastructure and of general components things that are not non-competitive of course those you want to easily use and and don't want to invent more than once in your organization and finally one typical demand we see today is also that you have this two level of trust so you have a trusted base system which you control completely but you also want to enable the customer to add services applications on top which are not in the same trust zone as the base system but which are able to run reliably on your device and yeah the customer is able to to select which of those he wants on his device okay so that was basically what we what what the requirements are we see in in in the field and now there I want to talk about what we think is or is part of the answer at least a new open-source project basically founded on on top of of a practice we call it as a working title industrial great linux and I will come to you to what what this really means so the core idea is of course there is this multitude of open-source projects and all of them do something special and and and are making sense but in the end to use them in an industrial setting there is simply more than that and and the first step is to package them so that they that they fulfill a yeah a unique they have unique properties in how they can be used and that's for example something the debian universe has solved very well with the depth packages and and all the the infrastructure and and processes around that to select the appropriate open-source projects to pre-qualify at security policy management and and compose make compositions of them and basically we want to instantiate the debian process to even more narrow this down for the purpose of embedded devices IoT devices um so we would select the parts from debian that are relevant for that we would add stuff that's relevant for embedded also improving or increasing the enhancing the the infrastructure so that it supports also the embedded use cases to build the industrial great linux and that's then the thing that can be used by companies like Bosch and I'm showing the Bosch use case then on the right side can be used to build IoT devices so it's then again to enhance and reach the the open stuff with these special things that are needed for the different areas for example mobility or industry or residential areas so that we can build a multitude of IoT devices out of this core set of basic functionality and create products out of that so we widen up again and provide the necessary infrastructure and and a base system so that we can easily build those IoT devices so the idea is from a variety of open source coming to a very tailored distribution specialized on the embedded IoT use case which is then enlarged to be used in many many IoT devices and I talked about industrial strength so what do we expect from this interface here where we have this open components so to be able to use such components there are a set of constraints or properties that are needed to make them easy to use one thing is we want to have them as source or binary so there might be build systems like for example yokto which would rely on the sources but the standard build system provided in a purchase for example is based on on build packages optimizing them then during the image build we want to have them tested in reference systems so one idea is to provide a set of reference boards which are used or which are used to test certain compositions of the distribution and show that the software in principle runs on this reference systems and it's then part of the open source system to define which reference systems are relevant and should be part of this validation step. As BOM enables software bill of material enabled means that we want to have identifiable objects so even in the case that a project says I use a certain open source project but not as is I want to do some modifications the idea is to have the right working model the right process in place so that they can still fix this changed piece of software make it identifiable by having for example an a versioning scheme that allows them to state which component they're using which is the base component and adding that they have modified this component so that it's still manageable that we use this exact piece of software. Why do we need this basically for two reasons or there are multi more than two reasons but the two more most prominent are of course open source compliance so we need to identify which components are in a system and then which licenses applies to each and every component in this in this bill of material so that we can fulfill the license obligations of the of the whole system so but here we really need the reliable information so it's not only a declaration but it could also include the analysis of the sources so that we also find hidden licenses of files which have been copied into the project stuff like that and of course this the second reason for second important reason for the bill of material management is of course vulnerability management and this yeah is then relevant over the whole life cycle especially because vulnerabilities come in every time so anytime so you you don't know when and you also have to monitor your products in the field to be able to react on detected vulnerabilities and for that you have to know which product contains which component and all this this informations and additional stuff add to the quality of the components to the meta quality of the components let's say and make them ready to use so lots of of effort is spent in companies like Bosch to provide for example the open source compliance stuff and a lot of energy is burned doing that so having here efficiency really would improve the quality or the the speed of the process also the quality assurance the validation in reference systems that allows to get started fast and finally of course by having this this bill of material we are prepared for long-term support so if there are requests that come in later we know which components are in we can do any kind of update strategy be it a general update or be it for example only a small update in a certain component to make only minimal updates so everything is possible is transparent and can be used over the lifetime of the product the other thing what we expect at at this borderline is an infrastructure extension so there will be an infrastructure or there is an infrastructure in the open that does all this testing and all this this information retrieval and the infrastructure that is then used within the company to build the product should basically be an instance of this open infrastructure and it should seamlessly integrate into the open infrastructure so that there is no friction on on this move from the external to the internal usage of the component here again another picture on on the aspects of the project so in green you see really the software that is part of the distribution in blue you see the tooling around it and in gray you see the soft parts of the open source project the soft skills let's say of the open source project so in green we address basically four different pieces or types of components on the lowest end you have the hardware pack so that's really the software which is hardware dependent which supports a certain hardware to be executed with with this stack the next level the os pack is basically the the core of the distribution so the operating system but also all the packages that are for example the building the middleware and and supporting certain use cases libraries stuff like that and that can be chosen to be part of a product the third level then is um add on software in deployment so here we talk about applications that can be installed on on the device also after the product release for example we think about technologies like flat pack or snap or also even docker to allow to add software while the product is in in the field and yeah provide the services so that this can be integrated into a product that this is possible to to add software and finally on top cloud stack integration we also want to provide all the stuff that is needed to integrate into provide cloud backends iot cloud backends like the bosch iot suite but also of course the big cloud providers so that you can build the whole iot system easily by by using the stack and then add integrating the devices into the appropriate iot backends services so that's the software part we talk about also the infrastructure so we have the software development kit of course so everything you need to to build the software and to develop the software debuggers ide integrations documentation sample code whatever helps you to easily get into the the stack and make it usable for for you use case on the left hand side we have first the collaboration infrastructure and then infrastructure so everything continues integration services test base test centers all the stuff that is needed to run the project and to to execute yeah the project builds and the image creation and stuff like that and finally on the on the very left side security infrastructure this is tooling but also mechanisms for processes to ensure that you can build secure systems also that's needed to add security to your device that's basically then the the infrastructure part and of course there is there's also then the open source project so we have the os s community and something we want to build up and and maintain and and push forward and then again the working models within this community so the government structure the collaboration models so that it's easy for new intro interesting power interested parties to join the initiative and understand how it works and and can add value there and finally i mean such a distribution is of course nothing new there are also providers who do this but the idea here is to to have this this end user driven approach so it's a distribution that is driven by the using companies and of course we welcome engineering service providers because we don't want to build up all the the knowledge in-house we encourage service providers to to join the effort and to provide services in the ecosystem and but the goal here is to not have a render login at that point so we try to attract multiple such providers so that it's easy for for the end user for the using companies to choose the the engineering service provider they see best and or they like best and and and also be able to switch without changing the infrastructure without getting without losing all the the invest that is already in the system so in this picture i show you the the basically the the infrastructure and this is both the infrastructure for the external project but also the the the infrastructure used within the company so that's why some of the things here on the left hand side look weird when you look at an open source project but make sense if this is also used in-house so the core is here in the in the middle the the package repository so a repository of built components that can be used to create products and also of course the the storage of those products so the assembly of the components for a certain product and this is filled by the open-built service or which takes a multitude of input one are simply rooted through Debian packages which can be simply used as is they are updated and merged according to what happens in in the Debian source repositories but there will also be up there open source projects which are either not part of Debian because they are not fulfilling the purpose they have with with the components but also since Debian has a very conservative process and update strategy software that is for the use case embedded IoT devices needs a more frequent update so we would also manage such packages you know independent of the Debian part and and but in the same manner so we would use the same process the same package management process like Debian but would do this on our own in our own GitLab instance that is already existing for a purchase for example and of course there might be customer specific software and also binary only software that can be added to this this package repository these things are then of course not publicly visible they will be in parts that are private or even in private installations of those technologies within the company so that's how the packages are built and then when it comes to building then a firmware for an embedded device out of those packages there is then an assembly of the relevant packages and they are pushed through an image creator which creates the images and what is this image creator doing besides blocking all the stuff together it also optimizes the packages so it removes for example all the stuff that is not needed in an embedded device you know when you're using linux that it's of course very attractive to have for example all the documentation in place or man pages that help you in running a tool for example but those things do not make sense on an embedded device so this stuff will is removed during the image creation so that we get a minimal image that can then be flashed onto the boards that are relevant in the open the targets are standard reference evaluation boards that are then used to quality assure the the result in-house that could be also some more specific company specific evaluation boards but in the end it's the project board that's of interest for developers it's also possible to create images for virtual machines so that they can do some testing on their machine without having the hardware available and of course the image creator also creates stuff for for updating and here it's about images but we would not send of course the whole image but only a patch a delta so that only those parts of the device are sent that have to be changed in the end yeah and that's basically the technology the infrastructure behind and as i said the the idea is to have this in the open to have the quality assurance in the open and and all this stuff and the open source package repository in place but then you can use open stuff also in in a private manner but also have the local instance in your company to address then the product creation stuff another few on on the modularity so i mean of course we have packages and you can exchange all packages but this really important thing for for the whole thing is that it's modular and that you can that you can exchange whatever you'd like and this is what basically is shown here on this picture so the green parts are the reference system you see full and green are the parts that are provided by by the project and which you can use out of the box but of course you can exchange and remove whatever you like so you in this example here you have four different products which take only parts of of the whole thing even removing some stuff and even exchanging lower parts so you can also exchange the kernel in the bootloader if the one that is coming with the distribution does not suit you and on the other hand side you can also of course change the build system so you don't have to use this described image creation process for example if you are working in an in yachto based environment you could also use yachto as a base system as a build system and yeah step by step change your recipes to use the components of of a purchase or what the follow-up project of a purchase to have this industrial strength quality of the components or the appropriate meter data and and stuff like that which allows you then to to use at least this stuff out of the system so yeah we propose this so of course Bosch will be a driver for the whole thing but it's not our intention to to be there in a kind of master situation we would sponsor it of course in the beginning but the goal definitely is to to be one of many using companies which are shown on on on the top of this slide and in the end we don't want to be special here we want to be one of many it's an Linux foundation open source project so when you are used to Linux projects it would have been it would be standard so we would have the the steering and governance set up and there's already a proposal available which I will show in parts later on yeah and and it's basically then providing the package repository the the infrastructure and the working models to to the charter and all the stuff so the definition of how to collaborate but as I said earlier one one really really important thing is that the unique selling point of this is that we enable or we motivate a multitude of companies to support this and to support expert services to the using companies on on the top so it does not make sense to get into all the different details of of linux systems for for companies like Bosch it would be easier to have expert companies that where we can um yeah I get the information in a service contract stuff like that and yeah here we one thing on driver is of course to provide to prevent uh render login um but yeah um this we can only achieve if we make this as attractive to such companies um um that they are willing to to provide services here for example a purchase um is done in together with colabra um and I think colabra would also um yeah support us in in building this up but it's definitely not our intention that this is then only colabra supporting the whole initiative from the Bosch point of view of course I talked about the governance model and yeah if you know linux foundation projects this looks familiar to you so the idea is to have the the business and and budget side dealt with in a governing board to have also on the other hand side a more technical board the technical steering committee that is looking into the the collaboration with upstream into the different packages and then is discussing technical things architectural things the tooling and so on um and this is simply a best practice for open source projects to get the the commercial parts and the technical parts separated split it into two different boards we plan to have an outreach committee to market the whole initiative to communicate to go on events provide trainings so that it's easy to get on board into the thing and of course then the deep technical work will be done in working groups in technical working groups that are provided or that are executed within the context and again of course it's a model with with membership you can as you can imagine running such an infrastructure running the test systems the the services costs money and the idea is then to have to have a charter in the open source project a membership model on different levels and based on on your level of membership you will get more rights here in in those boards and committees on on the one hand side allowing to propose for example I will have influence on on reference boards and on on packages and all this stuff that's the one hand side so that's the commercial part so that we can run this and then make this a long lasting success story on the other hand side it's an open source project so everyone can contribute to to the projects to also to the package mechanisms and and all the stuff so there is no need to become member but in the end yeah we have to carry it in a way that it can last and it's a success a healthy project that has enough resources to to run the initiative so that's that's basically my proposal I'm coming to to a summary what we want to do is to control the complexity so the idea is simply we see this in-house so we have several business units which fight to get linux products onto the market and and we want to bring them together so to not invest redundantly in same in the same stuff and the idea is basically hey when we do this internally that is not really leveraging to the amount possible we can really get the lever when we do this as an open initiative let's get together and do the non-competitive parts together but also let's drive together innovation in this collaboration and and push linux for IoT devices forward the idea is to provide a collaboration infrastructure and to get industry strength packages that can be easily used out of the box without pains without additional processes needed that means on one hand side we want to enable maximal reuse so we want to provide basically everything that makes sense to be reused in this in this distribution on the other hand side you have absolute freedom to change whatever you like and also do this within the system so you can easily add packages you can easily change packages and maintain them still in the in the logic of the of the the infrastructure so that you don't lose the the management capabilities the lifecycle management capabilities but yeah can do whatever you need for your product to be successful component validation and reference hardware so we want to provide components that have proven to run in in a multitude of environments so that it's most likely that they also run in yours life cycle information life cycle management is very important so getting rid of all the pains with open source compliance with vulnerability management also providing stuff like export control information whatever you need to make yeah your your product ready for the market the thing is that in our belief this package-based approach so to to stay with packages until the image creation is really the key to keep the traceability throughout the product lifetime and and to provide long-term support to have identifiable stuff which is uniquely uniformly managed and being able yeah to to change it over time so that you can fulfill the the maintenance of your product throughout the lifetime and in the end what we want to provide prevent is really that you need to have a big bank switch so at some point you have perhaps bigger switches bigger changes but you should be able to make use of this and gain gain stuff by switching partially incrementally to to this logic behind behind the system and finally the project principles to to make them prominent in the end so reuse we want to be integrative and and we want to yeah provide available projects wherever possible so it's it's never that we will push something in this when there are open source projects available they will be available if this makes sense when we up contribute it's it's upstream first so we want to push it to the original open source project only if needed into the package management system and all the stuff simply because to prevent this maintenance hell of of having to patch software again and again modular you want to work in packages as building blocks and compose software systems of those so that's the modularity aspect by having the diversity so it's possible to change the components to the packages and to add packages wherever you need it it's a distributed development model of course you have to do this distributed because you have to support all the different parties collaborating here and we chose deliberately to use the Debian system here because this has proven over years that it's working in a large scale environment and we adapt this only were embedded aspects are required requiring to make the changes efficiency ideally you don't have lots of efforts or much effort using all this non-competitive stuff you can simply instantiate the base system and concentrate on your on your product specific parts that's the goal and yeah so it's for embedded devices so we think that embedded devices simply have lots of of special requirements and we want to have a free non-biased by by one company distribution that can be used to easily get embedded systems running so that's it that was that's what we plan that's our proposal and yeah as i stated in the beginning that is the gap we identified there are of course different systems like yokto like like the zip project who concentrate on on certain aspects but at least we didn't see that there is such an such a system such a logic in in the place to to really address the industrial concerns in using such a multitude of open source projects and that's the gap we identified and we want to solve and yeah my question to you basically is do you see this gap as well what's your experience with this just as an additional hint there will be a later on a panel session with basically the same title title where i will discuss with representatives from Colabra, Siemens, the Linux foundation and Bosch on yeah is this really an issue and does it make sense to to start this project so i'm really inviting you to also join us there in in this effort yeah to get more into and and also be able to phrase questions on yeah on on that topic i think now there is still some time so you can still ask more questions i will answer them and until the end of this session yeah and this means i'm at the end thank you very much for your attention and yeah looking forward to see you again hopefully in in the panel session and if you have questions please get them into the chat so that i can answer them thanks


It's a great pleasure to be here and to be able to talk about a subject very close to my heart, education and in particular how to be more scientific about it. Now some of you may already heard me speak before or watched my confessions of a converted lecture on YouTube and most of my talks on education, I start them with that picture on the screen and ask the audience what is it that is happening in that picture that was taken of me when I taught my introductory physics for pre-med students at Harvard? This is a very old picture, it's BC, before computers. You can see I'm using an overhead projector. The point I want to make with that picture is that the traditional approach to teaching focuses on information transfer. And what is important is not what the instructor does in front of the students, but what happens inside the brains of the students. And it turns out that's not very much. And I want to back that up with some data that were published recently in an article that was published by my colleagues at MIT in the Media Lab. They developed a very interesting sensor that you can wear around your wrist that measures electrodermal activity, which is a measure for brain activity. Actually, most of this paper goes to great lengths to point out that the electrodermal activity measured at the wrist is correlated, very strongly correlated with what happens inside the brain. The great thing is that you can basically wear that sensor for a week, then download the data and analyze how brain activity correlates with different activities during the day. So they had a couple of students wear them for a week, and they published in that paper without, I think, commenting very much on the actual activities. This graph that I'm showing here that shows on the vertical seven days at the bottom day one, day two, day three, day four. And you can see the electrodermal activity, the brain activity basically, as the trace that goes up and down. And labeled in color at the bottom, you can't read it, the letters are too small. Oh, incidentally, on the horizontal axis is time for a reason that I don't exactly understand. It starts at 16.00. I guess at MIT today starts at four o'clock in the afternoon, I don't know. And then it goes all the way to three o'clock in the afternoon. Now, you can't see it, you can't read it from where you are, but I want to draw your attention to the parts of these trace that are labeled lecture. Actually, they're labeled class, but, you know, they're typically lectures. The trace goes flat. In fact, it's very interesting to contrast that with the brain activity during sleep, which is highlighted there. In other words, students are more asleep in lectures than they are when they're in bed. You know, the French writer Albert Camus once said, some people talk in their sleep. Lecturers talk while other people are sleeping. I wish he could have seen this trace because it would have made the point much more salient. Now, there's only one activity. There's only one activity that matches class.  TV, exactly. Here's TV. And of course, that's something that people knew a long time ago because people had done, you know, using electrodes in the brain, scans of brain activity of people watching TV, and it's been known for probably close to 40 years that people are more asleep in front of their TV than they are in bed. If you stop to think about it, it would be so surprising that class or lecture, including this lecture here, is not that different from watching TV. And I think that's a very important message, especially as we move a lot of information transfer out of the classroom, onto the web, onto computers. Why would people be more engaged by watching a lecture online than watching it in real life or watching TV?  Today I have a slightly different message here. And the message is on the screen there, is that we should not abandon the scientific approach when it comes to teaching. A lot of decisions in education are made based on completely anecdotal data. And in fact, it's not that different in my own department. We often talk about teaching. I'm in a physics department surrounded by physicists, and you would expect physicists to only take decisions based on evidence, but it's kind of surprising to see that even very distinguished physicists, Nobel laureates, completely abandon the scientific method when it comes to teaching. My students like it when I do this or that. Lots of decisions get taken based on simply anecdotal evidence. I want to quote here Lee Schulman, former president of the Carnegie Foundation. He once said, and I love rubbing this quote into the face of my colleagues, he said, the plural of anecdote is not data. You don't use anecdotes for evidence even if there are many. And I know that in a sense because I based my approach to teaching on my own experiences. When I was an assistant professor, that picture is from when I was an assistant professor at Harvard, I basically started doing what my teachers had done to me. And I really thought that's how we learn. And I really thought that's how I learned. So I taught this win for many years. I thought it was a good teacher until I discovered that my students were not learning anything. So ever since then, I've basically used my classroom as a laboratory to obtain data. And I'm going to show you today three little projects that I've carried out in my classroom that show how we can use the classroom as a laboratory to obtain data to improve teaching. And I want to urge all of you to do the same when you have access to a classroom.  I'll talk a little bit about these are three separate parts. And feel free to interrupt me at any time and ask questions we don't have to wait till the end. I'll talk a little bit about gender issues and especially gender issues pertaining to science, education, a particular physics education. Incidentally, I'm a physicist, so I'm going to put this in the context of physics and you'll hear a few physics examples. Do not worry if you cannot follow the physics because the physics is irrelevant. The message transcends the field. So I'll talk a little bit about gender issues. Then I'll talk about lecture demonstrations and sciences we love doing lecture demonstrations. And again, this is, I think, very relevant for online education where a lot of people think seeing is believing. The same is true when doing demonstrations as a physicist and particularly as an experimentalist, I thought what better way is there to teach students how nature works than by just showing how things work? False. People don't see what they should see. They see what they thought they should see. So I'll talk a little bit about that. And then lastly I'll talk about confusion and the role of confusion in the cognitive process of learning. So let's start with gender issues. For many, many years I have administered and so have many other faculty across the world a conceptual survey called the Force Concept Inventory. It's a 29-question survey that tests students' understanding of the concept of force which is one of the first concepts to be covered in any physics course. And it's actually that Force Concept Inventory that prompted me to completely revise my approach to teaching because I found out the first time I administered it that in spite of my high evaluations as a teacher and in spite of my students doing well on what I considered complicated exam problems they failed to do well on this concept inventory which I thought was actually pretty trivial. But that's not to the point. I want to highlight another issue with this Force Concept Inventory. Basically the way it's administered typically is once at the beginning of the semester before physics instruction and once at the end of the semester so pre-test, post-test to see what the difference in score is and to measure how effective the teaching has been. Well the post-test scores show a disturbing trend namely that men outscore women by a significant amount about 15%. These are data actually from the University of Minnesota which is a large public institution in the Midwest. If we compare for a couple of institutions the size of this gap here about 12% of the University of Minnesota in my own two classes that I've taught at Harvard two different introductory physics classes which are similar and Woosta Polytechnic Institute which is a small liberal arts college also has a similar gap. So the question I'd like to ask is what causes this gap and can we do anything about it? First of all is it cultural? If you take all of the data from the US and put them together you get about 11 to 12% very statistically significant gap between male and female post-test scores. Now I have the benefit of getting a sabbatical every couple of years and whenever I spend that in a foreign country I try to push my host to administer the FCI in their institution. So a while back I was in Belgium at a pretty well-known engineering university and I decided to see if I can convince my colleagues to first of all translate the FCI into Flemish which is not that hard. I was educated in Holland so I'm fluent in Dutch so I could at least check the quality of the translation and then administer it and two things were striking. First of all I having been educated in Europe always thought that in Europe things surely had to be better than in the United States they weren't but secondly the gender gap was actually significantly larger than that in the US. A little bit later this was still in the mid-90s I was in Taiwan the whole first generation of my graduate students ended up becoming professors at Taiwan Dashi which is a national Taiwan university their flagship university in Taipei I spent half a year there and tried to convince the people in the physics department at NTU to administer it there. They looked at the FCI and they said no way we cannot administer that to our students they thought it was so simple so trivial that they refused to to even consider testing their students I said but look we even do it at Harvard and the students don't do that well well their respect for me immediately plunged and I was unable to convince them but fortunately near Taiwan Dashi was another institution National Taiwan Normal University which focuses on training high school teachers and there they were very interested in this test we translated in Chinese I had my graduate students translated back from Chinese into English to make sure that I could vet the translation and we administered it and not only did the Asians do significantly better than both the Europeans and the Americans on this FCI but on top of that there was no statistically significant gender gap now I was never able to really research what the cause was of this but there was one variable that stood out at this National Taiwan Normal University about half the faculty was female in Belgium at that university I'm not allowed to disclose which one it is after the dean saw these results he said he called me to his office and said I never want you to disclose these results so I won't tell you which university it is but at that university there were no female faculty at all in the sciences it was a completely male dominated department and in the US the number of female faculty hovers around 25%, 20 to 25% so maybe the environment has an influence on the gender gap what about if we go back to the US data where I have many many more data tens of thousands of data points what about the effect of pre-college education you would expect if a student has had some high school physics then that is going to affect their understanding of Newton's laws in fact I learned Newton's laws in high school when I went to university to study physics we didn't even discuss Newton's laws anymore the concept of force was assumed to be already understood well there is some correlation between students who've had between the score on the FCI on the vertical scale and students who've had no high school physics whatsoever students who've had a course in high school physics and students who've had a college level advanced placement test but it's not very strong which is kind of surprising right you would expect that to be a much stronger correlation especially students who've had an advanced placement this is for the female students here's for the male students and you can see that this gender gap persists across the pre-college level of education and it's actually even worse if you look at the numbers because women are underrepresented at the higher levels there's about the same number of students who've not had this is in one particular year in my class at Harvard who've not had high school physics 15 and 18 about the same number who've had one year of high school physics but then the advanced placement course is taken by twice as many men as women so what can we do is there's something we can do to mitigate this well there's plenty of literature that shows that women thrive in an environment that is more collaborative not competitive and also that focuses more on verbal and visual representations as opposed to purely mathematical representations and when I read about that I thought well what I'm doing in the classroom actually mirrors that let me tell you what I did in the classroom after discovering that students didn't learn very much from my highly regarded lectures what I did is basically abandon lecturing altogether instead of focusing on information transfer which is kind of ridiculous in an information age right ever since the biggest invention in information technology people have been saying that we should no longer lecture referring here to the invention of the computer or the internet I'm referring to the invention of the printing press and the book back in the 18th century I think it was Samuel Boswell who once wrote why are we still lecturing we now have books but for some reason people kept focusing on information transfer as I said before education is more than just information transfer so I like to see education as a two step process one information transfer you need that otherwise nothing happens two assimilation of that information or in Piaget's terms accommodation of that information it's what the learner does with that information in his or her brain that matters you need to connect it to the knowledge that's already there you need to connect it to the experiences I've often asked myself where did that happen for me did the aha moments the insights the connections did that happen while I was sitting in a lecture hall not that different from this one here listening to my instructors I don't think so I think most of that happened outside the classroom so if you think about it completely pragmatically education being a two step process one information transfer two assimilation of that information and you ask yourself which is the hard part I think we'd all agree it's step two so it's kind of ironic that in most of education we focus on the easy part information transfer leaving the hard part to the students on their own that's true in the information age two most uses of information technology focus on information transfer not on the hard part the assimilation thereof anyway when I realized that I thought what I should really do in my classes is throw out the information transfer completely ask students to watch a lecture online or ask students to read the book and in class I teach by questioning rather than by telling I walk into the classroom I put a question on the screen student think commit to an answer the whole clicker craze started basically in my classroom about 20 years ago and then I tell them now turn to a neighbor who has a different answer and try to convince your neighbor of your answer complete chaos but in the process students teach each other and then I have them vote again so there's both feedback which as many educators knows crucially important in education active engagement you can't sleep in my lectures because every few minutes your neighbor will start talking to you right and the message has shown to give much bigger gains in learning but I'm not going to focus here on this method what I want to point out is that it's a collaborative approach to teaching students helping each other and it's one that very much focuses on the verbal individual rather than the mathematical so I thought could that method help mitigate the gender gap because it increases both collaboration and interactivity so I decided to take the many many years of data I had collected both in the traditional lecture setting and in interactive lecture setting and to compare three different pedagogies which I had done for many years before switching as T then the first thing I did was make my lectures interactive by using this approach to teaching that I just described and which I later in my book called peer instruction I labeled that I and then a few years later I decided not only to have the interactivity in the classroom but also enforce that I don't like determined force but also encourage that outside of the classroom so in discussion sections in homework I encourage the students to work together increasing the collaboration there as well so let's look how these three different activities impact the gender gap so let's first look at the pre-test course for women for these three different treatment groups T, IE, interactive engagement and IE plus interactive engagement plus and there's no difference for the pre-test because they've not yet been exposed this just shows that over the years because we're talking about the 10th year span here the type of students that get admitted to Harvard performs equally which is quite reassuring otherwise this research would have been very difficult I put a little triangle on the far right side of this diagram to show you the average pre-test score for women here at the men there's a little bit of fluctuation there if we take the average of these three different groups we get that red triangle and the difference between the blue and the red triangle is the gender gap on the pre-test that's how the students get delivered to me in my classroom now the question is can we close that gap now let's first compare how women do across the three different groups so traditional interactive engagement interactive engagement plus and you see that women gain significantly more as the interactivity gets increased the men do too but the women gain disproportionately more and in the interactive engagement plus the gender gap is no longer statistically significant so that's good news that means that women don't discriminate this gap which shouldn't be there in the first place simply by increasing the interactivity in the classroom if you're interested in more data you can find that in the paper that we published about six years ago in the American Journal of Physics let me slice the data a little bit differently who are the low gain students I'm showing there a data point for one student who started out at a post test of 70% and ended at a post test of 90% I put the pre-test on a horizontal score and the gain the difference between the post test and the pre-test on the vertical score you want your students to be as close as possible to that diagonal line you start at 90 you gain 10 you start at 20 you can gain 80 that means you have a 100% post test score here are for a class I taught at Harvard all the female students some of them on top of one another so there are actually more women than there are data points here and here are the men and if you look at those who score below the mean you see a gender imbalance there are far more women than there are men this is in a traditionally taught class after implementing interactive teaching all of the data points move up which is great it gains more but if you now look below the mean you see that there is actually gender balance there are as many men as there are women so I think the verdict is pretty clear the interactive teaching actually helps overcome this gender balance so the points I want you to keep in mind for this first little part of my presentation here today is that in the sciences this gender gap comes from both culture and background you have to do something about it no question but even if there is such a gap it can be eliminated by introducing interactivity in the classroom I'm actually happy to take any questions now where we can wait till the end but feel free to raise your hand and shout out a question because we're switching now to something different namely demonstrations and I chose that topic because I think it's particularly relevant for people who are doing simulations using technology or putting lectures online in hopes of actually promoting some real learning I'm an experimentalist as I say a moment ago I always loved to do demonstrations especially if they're kind of spectacular so when I started teaching I would have this rocket car which was basically a fire extinguisher propelled car where I would shoot through the lecture hall I would hoist myself up to the ceiling of a lecture hall that was as big as this one using a set of pulleys if there was a cannon to be fired those were the best demonstrations and students would remember the demonstrations vividly but then I read this that was published at the University of Washington in Seattle by a student who had gotten a PhD in physics education research and as part of her research she had interviewed students and in one of these interviews she discusses with a couple of students who have taken introductory physics but not majored in physics their recollection of demonstrations if you've ever taken an introductory physics course you may have seen it it's called shoot the monkey or sometimes it has a more politically correct title but anyway you aim a gun at a suspended monkey it's not a real gun it's a spring loaded gun and then the moment that the projectile emerges from the barrel the monkey stuffed monkey not a real one of course is released and falls under the action of gravity but miraculously the projectile that is aimed exactly at the monkey still hits the monkey the reason is that it too is subject to gravity so they both fall so no matter how hard this projectile goes if it goes faster it hits the monkey higher if it goes more slowly it hits it just before it hits the ground well two months after seeing this demo the student correctly remembered that the monkey was hit was convinced that the person who had done the demo had adjusted the aim of the barrel in order to allow for this falling of the monkey which is not what happened because the point was just to show that you don't need to adjust the aim so I thought that's weird you show something and then several months later even though they've seen it with their own eyes they remember it incorrectly so I thought what is actually the purpose of demonstration is it entertainment or are people actually learning from demonstrations so I decided to do a study which we carried out over two years where we basically carried out first seven and later thirteen demonstrations in four different modes I had about 250 students in my class so I split the class into four groups and these four groups would get rotated each week through four different modes of presentation the first mode is very simple we would not show the demonstration at all that was our control group the second presentation mode we call observe we ask the instructor to do the best possible job showing and explaining the demonstration but it was completely show and tell yes the students were allowed to ask questions but nothing else in the third group before showing the demonstration we showed a graphical representation of the demonstration described what was going to happen and we asked the students to predict the outcome of the demonstration using clickers so we had a number of different potential outcomes and they basically had to click in their answers after thinking we call this the predict mode and then the last one is discuss we would give the students a worksheet the top of the worksheet would describe the demonstration then it would ask the students to predict the outcome free response not multiple choice so they had to write down this and this is going to happen then we did the demonstration and they had to record their observation so we were actually able to check whether they had observed it correctly or not and then they had to check a box my prediction does or does not correspond to my observation and then we gave them a few minutes to resolve any inconsistencies between prediction and observation with their neighbors in a discussion of course there is a different investment in time here no demonstration takes no time at all the demonstration the observed takes whatever the demonstration takes predicts adds another 2 minutes to it and discuss adds another 8 minutes to it so we would rotate the four quarters of the classroom through these different modes so if you were in my classroom one week you would be in the no demo mode another week you would be in the discuss mode in a totally random order and we actually never even told the students about this study they didn't know that we were part of a study we followed up with a free response test at the end of the semester we only took the demonstrations in the beginning of the semester so we had at least a month for the memories to settle in and we added also a number of exam questions where we would take the demonstration but put the same physics idea in a totally different context now I'm going to show you an example and again don't worry about the physics the physics is irrelevant although I'll explain it briefly but you'll still get the idea without getting any of the physics here's the demonstration demonstration consists of two scales one on the left one on the right and there's a plank that rests on both scales and there's an object in the middle so that each scale reads half the weight of the object then the demonstration consists of moving that object to the side and noticing how the scale readings change a lot of students, a lot of people think that it doesn't matter where you put it because the plank evens out the load this demonstration in a physics class is used in the context of teaching a concept called torque which has to do with forces making things rotate and the point of this is it makes a difference where the object is because you have to balance not only the forces but also the torques you can forget all of that just remember what is shown graphically on the screen here if you move it to the side one scale goes up and the other one goes down so that was the demonstration then two months later on a web-based test which was a free-response test we put this question we changed it into a numerical question so we said we had this drawing there and we showed that the two scale readings were 10 and 10 when the block was in the middle and the students had to say what the scale readings were when the block was on one end which is what they had seen should be 20 and 0 and then we extrapolated to something they had not seen in order to test their understanding, namely what would happen if the block were not all the way at the end but halfway towards one end this is something they had not seen in class well let's look what the result is 24% of the students gave the correct answers and correctly in their explanation which was free-response mentioned the concept of torque which is exactly what we wanted another 38% had also had the correct answers but used proportional reasoning rather than mentioning torque we'd rather have them use the concept of torque than proportional reasoning so we could debate about what is right and what is wrong here if we're generous we could say both groups are good if we're very strict we can say only the one on the left is right 20% of the students said it doesn't matter it's always 10 and 10 when I saw that 20% so that's one in five students at Harvard University okay I thought those must be the students who did not see the demo but we attract the ideas of students so we could exactly see in which group they were we had taken attendance so we knew exactly what had been in the group that had no demo or had been absent which means they're effectively in the group or been in one of the others no they were students who had actually seen the demo in fact some students actually wrote down and I'm quoting here as shown in lecture it doesn't make any difference where the block is placed I got so excited I went straight to a friend in the psychology department and I said I think we found something really really exciting here some really new exciting psychological effect he said really and I said yes and I told him about it and he started smiling he said Eric we've known all along about this he said why do you like physics? I said well I like physics because I don't have to remember that much he said exactly the brain stores models not facts so imagine you're a student who has the wrong model the model of it doesn't make any difference where you put the block now you come to class and a professor does or teaching assistant or whatever does the demonstration and you see contrary to your belief one scale go up and the other down you have what Piaget calls a cognitive dissonance you go whoa this is different from what I expected maybe you quickly scribble it down in your notes but and this is the key point before you have time to think about it the professor continues to lecture you have no time to readjust your mental model and what happens for many students especially students who are discipline after class to sit down and think let me sort this out what happens is that instead of adjusting the faulty model to the observation which is what we want to happen what happens is that the memory gets adjusted to the faulty model and you know psychologists have known this a long time because advertising agencies try a lawyer they all make their living on this principle now notice that if you're a student who will actually write down as shown in lecture the demonstration is actually rubbed in the wrong model right now you're using your adjusted memory as evidence that your faulty model is correct okay ten percent of students just used qualitative reasoning they didn't give numbers they said up and down and another six percent of the students the forces don't balance so if you add the two numbers all of a sudden the block has become heavier or lighter and two percent of other incorrect answers if we look at how this correlates with the presentation modes this is what we find and I want to draw your attention in particular to the observe mode those are the lowest scoring only eighteen percent got it completely correct meaning they had the right numbers and they mentioned torques so is just presenting harmful I'm going to quickly whiz you through an exam question and again ignore the physics here it's just so you get the picture and here this is exactly the same principle but a very different context I would like to argue that the real hallmark of learning is being able to take something learned in one context and applying it in another context we do very little of that as anybody who is teaching here knows after exams all these students coming to you saying professor we've never done a problem like that I'm telling my students well if you had done this problem it wouldn't be a problem anymore by definition right anyway here's the same problem in a very different jacket a plank suspended by two ropes and the tension in one rope is given in essence the scales have been replaced by ropes well you know automatically or you should know automatically that the tension in the other rope because of symmetry also has to be 150 Newton and therefore the weight of the plank is 300 Newton but now the question is if you move that rope from point Q to point R what are the tensions in the two rope they should of course still the sum should still be 300 but how the magnitudes depend on those torques which are not that easy to calculate this is a little bit harder problem 36% of the students gave the right numbers 100 and 200 was the correct reasoning then different amounts of students gave wrong answers based on incorrectly calculating the lever arms for the torques interestingly enough as I was scoring this this was a free response exam I saw lots of students who came up with very curious numbers 112.5 and 187.5 I had no idea where these numbers came from this shows that you know when you're a teacher and if you know the right answer it is almost impossible to wrap your head around the wrong thinking of a student which is why it's so hard to teach students who have the wrong model because you don't understand how they're thinking in fact I remember sitting there scoring this exam and each time I saw in total there were something like 16 students in my class or 20 students because I had 215 in total I kept seeing that same number I said where does it come from then I figured out 112.5 is 3 eighths but where does the 3 eighths come from and 5 eighths is 187.5 and then after probably scoring 180 exams finally there was one student who actually explained where those numbers came from look if you take the plank between the two points P and R and take the left half and you say the left half belongs to the left rope the right half is a right that belongs to the right rope and you also add the part RQ you get those numbers so they just you know imagine the plank being consisting of two parts I would never have imagined this and then of course they are here about 2 or 3 students who had 112.5 and that's because the part RQ does not need to be supported at all anymore it sticks out who would have thought right so interesting it is something normally when you grade an exam or when you look at incorrect work you look for a mistake and you stop thinking the point I want to make here is that often it's very interesting to reverse engineer the thinking of the students because it gives you a glimpse into their thinking and unless you are aware of their thinking you cannot address it anyway over all the demos that we did 13 demos you see that as you invest more time the number of correct remember the first column is how well they remember it goes up oh no no sorry this is the exam problem I'm getting ahead of myself here so here's for the exam problem so you can see that as the the discussion increases from no demo to the bottom the number of students who actually correctly explains it goes up dramatically now the aggregate for 7 demonstration shows the following and I'm going to show it graphically rather than in a table if we use the no demo group as a control group you find that just observing on the vertical I have the improvement in the students ability to correctly predict the outcome well in that case it's usually rather than a prediction right and the observe is barely statistically significant in other words the observe group does barely better after one month than the group who had no demo at all there's an increase with prediction and discussion which becomes even more dramatic when you ask for the students to explain the demonstration observe not statistically significant predict a significant improvement and discuss an even bigger improvement so the lesson that take home lesson for me was here if you do a demonstration at a minimum you should ask the students to predict it before they do it takes just you know a minute or two and look at the enormous improvement over the observe the same should be true for anything that is done online if you have a simulation have pause before you show it have the students predicted before they actually see it it makes the cognitive dissonance so much bigger that students are more likely to adjust their mental model to the observation rather than their memory to their faulty mental model so points to keep in mind before I wrap up here was my last part demonstrations without engagement are just not very helpful I always thought seeing is believing I think this is very clearly proven here that seeing is not believing what you see depends on what is already present in your head and the results can be dramatically improved simply by having students predict what they're going to see before they actually see it so anybody doing online visual simulations online demonstrations and so on should think really hard about incorporating such a step in their approach any question yes why don't we wait till somebody brings you a microphone this way people online can hear too so when I was doing air level physics we had to show our working out on the paper and we had to write them by hand our scripts and I don't know how you do them nowadays maybe they're online or on the computer or something but I know that must have been really helpful to expose misconceptions to teachers and to inform them as to why we got the answer wrong and I know we just didn't get a mark for the question if we didn't show our working out it was that strict so is there some way that that could be what's your ideas about that maybe using technology to reintroduce that into assessment I was educated just like you writing things out and in fact I still teach that way students have to work things out on paper and I look at it but frankly I used to usually not pay attention to the mistakes you end up spending disproportionately more attention to the students who get it right than those who do not get it right you see a student make a mistake stop reading the rest you stop thinking about why the mistake was made so it really takes some effort to analyze for misconceptions and wrong work and I would argue 99.9% of faculty do not do that they should but they don't do it now online distortion gets worse because most online systems only check for the answer not for the steps that lead to the answer I wish I had time to talk about assessment because I've asked myself lately a lot about what role does the assessment play and I think that the type of assessment that we have that we use mostly you know summative assessment high-stakes summative assessment is often the silent killer of educational innovation you you pushed a button here so I'm going to get sidetracked for a few minutes what is a real problem a real problem is knowing exactly where you want to get but not knowing how to get there you have a product you want to make it successful we have a problem in education we want to make education better in both cases you know exactly where you want to get a successful product or improved education the question is not the outcome which is known the question is how do you get there if you stop to think about it every real problem fits that pattern you know the outcome you don't know how to get there instead most assessment certainly in the sciences does not fit that pattern it's the opposite you apply a known procedure to get an unknown answer and the answer is checked I would argue that that's actually an inauthentic assessment one that does not really teach an interesting skill and one that actually a type of problem solving that can be done by computers and just as assembly workers have been replaced by robots so will any procedural problem solving be replaced by a computer so I think any assessment that checks the answer and not the procedure is one that is inauthentic and I would argue that presently most online systems you know mastering web CT you name it there are many different systems they focus only on the answer and therefore you know they don't expose any misconceptions they don't really put the emphasis where the emphasis belongs that's very unfortunate and I'm not sure there's an easy technological solution that's an online question let me see if I can read that does this have implications for the Khan Academy type of presentation is that the question yes so you know the Khan Academy to me and I know Selkan I've met him and he's an incredibly charismatic person there's no question about but frankly let's face it it's old wine into new bottles it's old wine that's not the hard part of education I said before education is a two-step process one information transfer two assimilation of that information that's the hard part I've been focusing on that second part because I think that's where we need to put our emphasis just taking a lecture and putting it online why would it become a better lecture than than a live lecture anyway any other question on demonstrations and the ineffectiveness of demonstration could we have the microphone can I ask the person there's somebody in the back I had my hand up I saw Selkan's keynote at Blackboard 2012 this year in New Orleans and he was very clear that his new wine old wine in new bottles is intended to be used as part of a more interactive environment in a school that's my first point the second point is that I think the fact that you're lecturing at us shows why the lecture is still a popular mode of information transmission because it's a way to introduce information to large numbers of people in a cost-effective way particularly when these ideas are new and I wondered if you'd care to respond to that oh absolutely I'm not trying to teach you physics here I'm not trying to do something that is conceptually very difficult so I think that's not the same I think and you know there are plenty of examples online there was one that was actually posted by Michael Pershan a few months ago look for M. Pershan P-E-R-S-H-A-N and there's another one by what is his name again is his Twitter handle Derrick Miller in Australia critiques of the Khan Academy and also ways of improving on the Khan Academy the problem is a lecture including my own lecture here or a lecture in the classroom or a lecture online is that it holds the mind captive there is no time when you're in the audience to pause and think there's no time to let the mind wander and the mind likes wandering you need to have your mind wander in order to make the necessary connections so I think the big problem is the continuous stream of information and you saw the trace that I started my talk with both in TV watching and in listening to a lecture the trace goes flat because there's no time to think and reflect that could be improved by breaking up things in questions but sorry we're breaking up presentations into small sections which the Khan Academy is doing in a sense but it could be improved even more by having speed bumps in between those sections where you actually engage the mind of the learner by asking for input and questions actually Sal Khan is starting to implement that too you had a question too just a quick observation in the early 70s I was at Sussex University and with colleagues we started learning to use programming to teach a whole variety of things linguistics, logic, mathematics and so on and by setting exercises where the students had to design things to do things they got feedback all the time because their program didn't work and they learned from that and they were asked to write about it and so on and I think that that is a way to get feedback which is highly productive absolutely I couldn't agree more about the importance of feedback I think the feedback is so crucially important where do you stand as a learner and that actually leads very naturally to the last part of my talk which deals with this topic of confusion in general instructors are praised for clear lectures in fact on most teaching evaluation it's a question that Professor Mazur give clear lecture on a like it scale of one, two, five and confusion is generally seen as something extremely discouraging nothing is worse than having a confusing lecture but on the other hand to wonder is to begin to understand we don't learn by just opening our skull taking the information and closing our skull you learn by thinking why is it that way and reflecting on it and then going oh yeah now I know that's the real learning process not just listening, storing and regurgitating later although I would argue that much of our assessment just as lectures are regurgitations of information so is a lot of our assessment regurgitating back of the same information one thing I noticed when I switched from lecturing to peer instruction was that there were no fewer questions but more all of a sudden you know there were lots of students who were asking after I had them think about a question and talk to each other there were lots of questions saying Professor what about this, what about that so and another thing I noticed is the following before each lecture I give students a reading assignment they have to read a chapter in the book and they have to communicate to me before class whether they found anything in the reading difficult or confusing and the night before class I press on a button on my website and I get the pictures of the students with their names and after it says what is difficult or confusing normally when you teach a large class you see a lot of faces but there are no names attached to the faces and when you check work that people do you see names but there are no faces this was an opportunity for me to connect names and work to individuals but the interesting thing was to correlate the students to their perceived confusion after reading the textbook I would have some students would say what is so confusing and I would look at the picture of the student and think this is not one of the best students in the classroom let me look up his record I am sure enough was a struggling student and then there were other students who said I am confused about this I am confused about that I am confused about this these whole essays this is one of the best students in the classroom I thought that is kind of strange it looks as if there is an anti-correlation in their actual level of understanding so does confusion indicate a lack of understanding or maybe the converse does lack of confusion indicate understanding so I decided to actually after using this approach to teaching to go back and carefully analyze the feedback I received from students on their reading assignments now if you are interested in more detail of that technique which is called just in time teaching there is this book published by Pearson Prentice Hall in 1999 called just in time teaching blending active learning with web technology and the idea is you give students an assignment typically it is a reading assignment but it could be a watching assignment you could watch a lecture too and that is followed by two content based questions difficult questions are not the type of questions where you can just look up the information you have to take whatever you have seen or read and applied in a new context most students cannot do that but they are told you are going to be evaluated on effort not on correctness you have to show me through the thought you put in the questions that you have done the reading or the watching and then the third question feedback question what did you find difficult or confusing I am going to show you one example out of a fluid dynamics fluid statics class and again you know there is some physics but don't worry about the physics and the idea was to analyze the understanding expressed by the answers on the first two questions and correlated to the confusion expressed on the feedback questions so let me give you an example I am going to give you the context here too so here is the first question consider the capillary rise of a liquid in a glass tube where you take a small capillary you put it in the water rises in it how does the pressure at point p at the surface of the liquid compared to the pressure at point q at equal height now let me give you the answer and I am not here to teach you about fluid statics but it is the same because if there was a pressure difference then there would be a flow of water into or out of the tube because that same height in a liquid the pressure is the same by the way this was a class taught to medical students capillaries are important right blood vessels understanding capillarity the textbook that we used was one that actually applied physics to the context of medicine and it made a very strong point of always having examples out of the medical world here is question number two you have a tube with a valve in the middle and you have two balloons identical balloons but blown up to different amounts balloon B is inflated more than balloon A what happens when you open the valve the intuitive answer would say the big balloon empties out in the smaller balloon but as anybody who has ever blown up balloons known right for your kid's party or whatever you take them out of the bag, you put them on your lips you start blowing you must have noticed that the effort in the beginning is significantly larger than at the end the first few breath that you have to get in the balloon you have to work really hard once there is air in the balloon all of a sudden it goes a lot easier the reason is that the elastic membrane when it is folded can pull in much more easily than when it is blown up actually the pressure in the small balloon is much higher than in the big balloon and something absolutely surprising happens if you do this we actually do the demonstration in class if you open that valve the little balloon empties out in the big balloon and it turns out this is crucially important in medicine for the alveola in the lungs the alveola in the lungs are not really little balloons but they are cavities that are similar to the cavity inside the balloon the main reason that prematurely born babies before 20, 21 weeks of gestation cannot survive is because they lack a substance called a surfactant which coats the inside of the alveola of the lungs which is there to even out the pressure because if you didn't even out the pressure then little alveola would collapse into bigger alveola before 21 weeks you don't have that take a baby or fetus expose it to the air all the little alveola collapse into the bigger one the bigger one's burst with disastrous consequences the book actually mentioned that so I thought it would be interesting to see if students could take it from the alveola to balloons and then the third question please tell us briefly what points of the reading are most difficult or confusing if you did not find any part of it difficult or confusing please tell us what you found most interesting that last part is crucial if you don't add that then students could get away by just writing nothing is difficult if nothing is difficult they have to tell me what's most interesting so let me give you an example of two students responses to these three questions one capillary action is due to the cohesion between water molecules and the adhesion of water to the surface of the glass tube this is a sentence just pull out of the textbook cut and paste more or less it's true I don't know how relevant it is to the answer but it's true negative pressures can result from the cohesive forces of water that is another sentence out of the book it's also true I don't know if it's logical but that's okay then at the same height the pressure inside the tube is much less due to negative pressures I can't follow the logic I don't know but it's wrong anyway so wrong then the second one the air flows from high pressure to low pressure that's right the fully blown up balloon has higher pressure than the half blown up balloon sorry that's wrong so the air flows from the fully blown balloon so wrong wrong what about the last question about the feedback nothing was difficult or confusing the sections on the surfactant in the lungs question 2 and the heart as a pump were interesting because they relate physics to biology okay so now for a very different example the water rises because of an interaction between the water and the walls of the tube that's the same idea that the other student said but now this student is trying to put it into his or her own words rather than using the words from the book the interaction creates an upward force which causes the water to rise the force is due to the surface tension between the water and the walls of the tube the pressure at the point inside the tube must be the same as the pressure at the point of equal height outside the tube because if there was a pressure difference then there would be a net flow of water into or out of the tube and the pressure difference is equalized fantastic the students own words well argued correct question 2 Laplace's law tells us that it requires a greater pressure difference to maintain a small sphere than a larger one so the pressure in the small balloon must be greater and the air will flow from the small balloon into the large one correct now let's look at the last question I found the explanation of Laplace's law and while I can understand the conclusion drawn I don't understand the reasoning which led to the conclusion so I thought what I should really do is take the first two questions for all of the 250 students in the class code them on whether they're correct or incorrect and then look in the third question for confusion expressed on the topic of question 1 or question 2 so this student would be correct correct confused about question 2 but not confused about question 1 and the previous one would be not confused about anything but incorrect and incorrect so let me show you the data which shows something really interesting of the students who said they were confused about capillarity 44% got the right answer to the question about capillarity of the students who did not say they were confused about capillarity only 25% got it correct and the same is true for the other question Laplace's law so confused students are twice as likely correct and the reason probably is that the students who say they're not confused have not even begun to understand they've just read the book passively without engaging their brain they are not even realized they're not even realizing the book is different from what they're thinking the same danger that appears in a quote-unquote clear lecture you walk out with a false sense of security thinking you understand it but in fact you haven't even begun to understand I'm sure there are many academics here and you go to colloquial seminars once a week from time to time you have these brilliant lectures and then an hour later you come across a colleague in the hallway and the colleague asks you did you go to the colloquial lecture yeah I went oh it was fantastic what was it about and that's when you realize that you can actually reproduce the subject right that's the danger of a clear lecture incidentally using a research-based text which is a text that prevents the students from reading through fast which anticipates their misconceptions and asks questions again shedding light on how we may want to do online education by just not just delivering information and having students reflect while they're reading it we actually are able to erase that difference between the confused and the not confused students but here's the key point there is more confusion among students who understand especially when they're not pushed to think so I think that we must recognize that confusion does not necessarily correlate with understanding it's also not necessarily the result of poor teaching you can have brilliantly clear lectures no confusion at all but there really is no understanding either because most people have not even begun to understand confusion is part of the learning process and should actually be elicited Socrates already said 2000 years ago we should teach by questioning not by telling why am I making this big point of confusion well you can go over the past 2000 years of history of educating people and you'll see plenty of complaints over the centuries of educators complaining that students are not learning and as I said 2000 years ago Socrates already said we should teach by questioning not by telling how come that we're in the 21st century and we're still mostly teaching by telling how is the move to Coursera Udacity Khan Academy and so on the focus is on presentation not on questioning I think in part it is because once you start implementing questioning you elicit this confusion and confusion is generally seen as something negative of course confusion in and of itself is not good enough you need to resolve it eventually right but unless that confusion is an essential part of the learning process it is going to be very very difficult to change the approach to teaching so the point I want to make by these three little distinct parts is to show to you how classroom data is vital to actually improving education and as I said at the beginning of my talk I see my classroom as a laboratory in my real laboratory I collect data on the interaction of laser pulses with matter in the classroom I collect data on education in the hope of finding ways to improve education so I want to end by giving you a link to my website the slides are not on there yet because my hotel internet connection was not very good but I'll upload them very shortly and you can also find more information now no need to write it down just remember my last name go to Google and then hit this button and then you'll have a copy of this presentation thank you very much