Welcome to the fifth webinar of the engineering rising to the challenge initiative from Purdue Engineering. My name is Arvind Raman. I'm the executive associate dean for the faculty and staff here in the college and also professor of mechanical engineering. And now this initiative was launched in May of this year in response to the National Academy of Engineering's call to action for engineers to tackle some of the challenges posed by the COVID-19 crisis. A Badaar initiative also looks in the longer term future to rethink and re-engineer the very systems that our modern society has come to depend on so that they might be more resilient to such shocks in the future while also serving society better. Now part of the initiative involves webinars where distinguished panelists unpack some of these challenges and provide us a glimpse into what the future might look like. And today's panel is about the future of online education and it is my distinct honor to introduce and moderator Mark Lundstrom, the acting dean of the College of Engineering and the Don and Carol Seepers Distinguished Professor of Electrical and Computer Engineering. Dean Lundstrom is internationally known for his pioneering contributions to nanoscale and electronic devices. His work in this field has been recognized by the IEEE Brunetti Award and the Semiconductor Industry Association's University Research Award. As a teacher, his definitive textbook Fundamentals of Carrier Transport was recognized by the ASWE by the Terminal Award. And for his student mentoring, he received the IEEE Leon Kirkmeyer Graduate Teaching Award and the Semiconductor Research Association's Aristotle Award. As a leader, he founded the Nanohub at Purdue, the Trail Blazing and Preeminent online platform for scientific collaboration, simulation, and education all around the world. A member of the National Academy of Engineering, Mark also received a Morrill Award from Purdue, highest distinction to a Purdue faculty for his impact across all three missions of our land grant institution, learning, discovery, and engagement. Over to you, Mark. Okay. Well, thank you very much, Arvin, for that kind introduction and good afternoon, everyone. And thank you all for joining us for this very important and very timely discussion. I especially want to thank our panel of experts from industry and academia for being with us at this very busy time, especially as academic years are beginning. So let me begin today by introducing the panel to you all. Professor Carrie Douglas is Assistant Professor of Engineering Education at Purdue. She studies how to improve the quality of classroom assessments and evaluation of online learning in a variety of engineering education context. Under NSF support, she's currently studying engineering instructor decisions and student support during COVID-19. Her research on the evaluation of online teaching has been widely supported by NSF and published widely in journal and conference presentations. She was a 2018 recipient of the Frontiers in Education New Faculty Award and is the 2021 Program Chair for the Educational Research Methods Division of the American Society for Engineering Education. Brian Gonzalez is Senior Director for Global Partnerships and Initiatives within Intel's governments, markets, and trade team based in Washington, D.C. Brian started at Intel headquarters in Santa Clara, California. And in 2000, in the year 2000, he took on a range of global leadership roles driving large scale national programs with partners to accelerate educational outcomes through technology adoption. His work has covered the full spectrum of learning scenarios, including university student innovation, teacher professional development, K through 12 student learning technologies, and personalized learning infrastructures. He's a frequent speaker on the impact of Industry 4.0 on transformational educational programs as a core competency of high performance teams. And he was awarded an Intel achievement award for his contributions to global education transformation. Our next panelist is Dimitri Perolis. Dimitri is the Michael and Catherine Burke Head and the Riley Professor of Electrical and Computer Engineering at Purdue. He is also, he also serves as the special advisor to the dean on online learning. Professor Perolis has been a key contributor in research on developing high quality reconfigurable filters and filter synthesis techniques for tunable, miniaturized, high Q resonators. He became an IEEE fellow for that work. He's also an outstanding teacher. He's received 10 teaching awards, including the 2010 Charles B Murphy award, which is Purdue University's highest undergraduate teaching honor. And our final panelist is Dr. Sanjay Sarma. Dr. Sarma is the Fred Fort Flowers and Daniel Fort Flowers Professor of Mechanical Engineering at MIT and the Vice President for Open Learning there. He co-founded the Auto ID Center at MIT, which I understand coined the term Internet of Things, and developed many of the key technologies behind the RS standards now used worldwide. In all of his endeavors, Professor Sarma has been at the forefront of technologies now known as the Internet of Things. MITX, which reports to Professor Sarma at MIT, delivers revolutionary, massive open online courses to millions of students around the world. Over the years, Professor Sarma has been involved in other sectors as well, healthcare, energy, automotive technologies, government, buildings, infrastructure, mining, and financial technology. He's also the author, co-author of the award-winning book, The Inversion Factor, How to Thrive in the IoT Economy. And he advises several national governments and global companies. So we're really honored to have this distinguished group of experts with us today to explore the future of online education. So the way we will run the panel is that I prepared several questions. Several questions have come in from you all, and we appreciate those. We'll go through these questions, but I hope we'll also have time for some questions from our audience as well. So let me begin. Well, let me begin here before I dive into the questions. Let me give you in with just a couple of remarks. They say that the pace of change in academia is glacial, but recently events demonstrate that academia can change quickly when we need to. Now, online learning and remote learning is now taking place for elementary students, graduate students, working professionals. Our focus, our discussion, is going to be focused on what this new world means for a college of engineering and for the broader society as well. So I have a set of questions prepared, and why don't we dive into those questions, and I'll begin with the first one. Online education has been with us for some time now, but the COVID-19 pandemic is giving us an entirely new experience. What should we learn from these recent experiences with online education? And let me begin by asking that question of Sanjay. Well, I'm going to be a little bit controversial and say that we're doing the lessons from the COVID crisis are not ones we should take back. I think we need to understand that first of all, the way to do online is to pre-record videos, make them asynchronous, maybe put them in a MOOC, highly produced, and what's happening right now is Zoom University. And in some ways, in my view, it's sort of the worst of both worlds because we're taking, you know, many classrooms around the world are sort of socially distanced to begin with, right? And all we're doing is doing it now on Zoom. So it sort of exposed the problems of the classroom, in my view, and it's not true online. It's live, you know, people are being heroic, but this is not what we should be doing. I hope what we take back from this is the need to, on the one hand, produce really good video content, etc., that is asynchronous, you know, well-produced, and on the other hand, when nature grants proximity back to us, that we use that video content to flip classrooms and make the classroom much more dynamic so that when we have proximity back, we can make it count. So that's my sort of provocative answer to you, Hark. Okay, thanks. That's interesting. Are there any other comments to that question? Brian, I wonder if you're seeing anything from an industrial perspective? Yeah, I would agree with Sanjay. I mean, from a standpoint of the technology, there's a lot to be desired still. There's a lot of challenges that I don't know that we should have foreseen when you need that persistent connectivity. You need the devices that really are going to be up to the task for that, you know, persistent requirement that we have today. And the part that I think is really open for improvement is the whole area, and we'll cover it, is the whole area of content and using the technology to create more engaging content. I think that some of the limitations when you go from the sudden tsunami and demand to the fact that there is a lot of, you know, lack of connectivity, that there is a reliance on going somewhere to get a device for access, and you realize that that's not an option anymore. I think it highlights an opportunity that we see in the industry and that we really need to address dramatically and differently. So, you know, just to add a comment here, recently I've had discussions with a couple of colleagues who are using Zoom lectures, and who are telling me that they're feeling that they're learning how to do this really well, almost to the point where they say it's almost as good as being in the classroom. I wonder if anyone is willing to speak up in defense of Zoom lectures. I want to say something about that. I'm not sure if I can defend Zoom lectures, but one of the things that I have heard and I have observed myself is that students like the multiple ways you can interact during a Zoom lecture in the sense that they're not always have to raise their hand and ask a question, but they can use the chat box. Other students can respond live. The instructor may also choose to respond live, and that's a new modality. I don't think we're very used to that modality, but it really brings an interesting way to handle classes, particularly when you have to do a very large course. If you have 100 or 200 students in your class, this actually does open new opportunities, and I think in the future we may have software as well as advanced features perhaps that may enhance this even further. Maybe if I may, having just thrown a little bit of a grenade into this conversation, let me also pick up on what Dimitri has just said, which is I think there is in fact a defense for Zoom lectures. Zoom lectures are more Democrat. I should say Zoom interactions. So if you have a classroom of 36 students, you have the panel, there are no back benches. The chat is actually very rich. A lot of students who don't, I think to Dimitri's point, don't feel up to asking a question live. We find they're actually quite vocal in the chat, and that's something we'll keep when we go back. In some ways, I think if Zoom is used as the flipped classroom, where the Zoom lecture is two-way as opposed to one-way, I think that's a win. And then when we return to the classroom, hopefully we will bring that to life in a much more sort of real way. So I do think they're benefits to Zoom. All I'm saying is that right now in a heroic way, they're sort of doing Zoom lectures. I'm just saying that's not really the best of all. All right. Thank you. Carrie, did you have something you wanted to add? Yeah, so I'm going to hop in here. And I have a little bit of a different perspective, I think, than Sanjay and maybe even a little bit more different, a little different than Dimitri. And that, you know, my perspective is that whether a course is taught online or in person, or whether it's using Zoom or a MOOC platform or some other LMS does not determine the quality of the learning experience or the learning opportunity, but rather how the course is implemented, the pedagogy is the curriculum, the instructional responsiveness, and how well those things are all aligned to the learner's needs. And that to me is what would determine the quality of the learning experience. And I actually do use Zoom to teach 120 students in a studio course. I had my first course yesterday and actually what we're doing is taking the best of best or the best of both. So what I mean by that is the students watched, produced short videos that were made long in advance. And then when they came to class, I was able to have breakout rooms. And so I, you know, we, so this is what's actually done through our whole first year program is how we're rolling things out currently in the current conditions because we couldn't de-densify enough to all be physically together. So we'll, you know, have a large group Zoom meeting and I'm, hi, I'm your professor, but then we have the students in the breakout rooms and they're working together on things. And so like in a course where we're doing, you know, learning to program in MATLAB, you know, they can share screens with each other, look at each other's code, and then as an instructor, you can pop in and out of those breakout rooms. You can look at the screen and see what the students are doing. And so there's a lot of ways that you can have an active learning experience while not physically being in the course. Okay. Thanks, Kerry. So good discussion. Now, let me move on to another question. So is it becoming clearer now what types of education are best done online? What best done in a residential setting? Will we see more and more hybrid types of courses? Dmitri, do you have any thoughts on that question? Well, let me start by saying that in my mind it's become clear that we can deliver high quality education online. And what I mean by that is that there's a lot of people who are already convinced, but also there was a lot of skeptics in the area. And I have had discussions with my colleagues who actually have been surprised with how well things are going. Of course, when you have time to prepare, you can get things much better done. And thankfully, we have had the summer to prepare for this fall semester. So I'm very hopeful that a lot of things will actually go well. But one thing that we did learn here is that you can meet learning outcomes and you can have good quality learning in an online environment. Now, which parts are done best online? Which ones are best done with physical face-to-face contact? I think it's a little too early to tell. But I do think it depends a lot on what your audience is. Are you talking to first-year students that perhaps this is their very first experience in college? Are you talking to master's students? Are you talking to working professionals? Are you talking to advanced PhDs? I think each audience will have a different response. And of course, it also depends whether you have to run a lab or not. So I think it's a little too early to tell on that. But I am optimistic and encouraged by how most people are responding to this. Thank you. If I may, I just want to throw something in here. I think that over time, we will be able to do, for example, chemistry simulations online. Will we ever be able to teach welding online? Perhaps not surgery. I probably don't want to be operated on by a surgeon who didn't apprentice in real life, certainly with flight. But the flight industry is very interesting that the U.S. has actually used flight simulators for almost 100 years from the 1930s. The Link Company was used. They made machines that trained pilots. And actually during World War II, the U.S. had pilots. They had planes, but they had also had pilots. Well, the Axis countries didn't have pilots because of virtual reality or augmented reality performance simulators. So I think that there's hope, but there's about a long way to go, I think, to get that stuff online. All right. Thank you. You know, let me ask a question that came in from the audience. And that's what about teamwork? How do you do teamwork online? We have courses where students compete, helping each other or faculty or mentoring them. How does that happen online? I know, Kerry, I think you may be involved in this right now with our first year engineering program online. Hey, I happen, but I noticed Brian was like right there ready to go. And so I'm going to let him and then I can open. Great. Thank you. Thank you, Kerry. I was just going to go back to the point on, you know, on virtual labs, which maybe this is where this could work. I think one of the work that we're starting to do now is, you know, if you look at NASA and the way that they approach their learning, we've been working with them and a company called Digital Ocean to understand what elements of that can we bring and help facilitate and put that online in a way that creates that, you know, engaging content. Because my previous comment was on the pandemic. We just tried to substitute what we were doing. What we need to do is to look at it differently. And that's where, you know, this idea of collaboration and the collaboration tools are great. We can, you know, I can connect with all of you now from Washington and maybe, you know, there are other ways that I would not have used this way, maybe if we didn't have this pandemic. So there's no good side to it. But there is a point where it forces us to be better at certain things. And I think the other opportunity would be around student-generated content. And that is one that we can see where the technology can actually help quite a bit. But I'll turn it over to Kerry now. Okay. Awesome. Yeah, no, I totally agree with your thoughts there. You know, I think it kind of piggybacking off of something you said. But I think it is a mistake to think that whatever we did in person, we can just do online. It kind of reminded me when the pandemic hit, it reminded me of, you know, when I teach introduction to engineering design. And in that course, one of the things that we know is that students are tend to be so focused on the solution or the one way of doing it that they don't take a minute and stop and think about what the, you know, the problem really is that they're trying to solve and what that need is. And I think our tendency when the pandemic hit was, you know, well, my solution is how do I do what I was doing in person now online? And what I think really needed to happen or what we have time for now is to revisit and go back to what's that fundamental learning need. And knowing all the resources and technologies that are now available, how can we create the best learning environment to meet that? And so, you know, engineers, so back to the question, engineers have been working in teams around the globe long before the pandemic. And they have been collaborating, you know, they, you know, engineers in different countries, different languages have had to work together to solve, you know, design challenges have been given to them or work on products together. So, when we're giving those opportunities to students, we're helping prepare them for that professional working world. And so, you know, there are a lot of collaborative documents. I mentioned, you know, doing the breakout rooms.  I mean, they have all these social media apps or messaging apps. And so, letting them pick how they're going to do it. And the same way we'll approach them and, yeah, they're going to do teamwork.  This would just, it was crazy to me. But I would have students in person with their back that were in a team, their backs would be to each other because we are in the studio and they would be working on the same Google Doc.  And they were in the same physical space. So, if they can do that, then they can turn Zoom on and they can have their Google Docs going. And if they need to say something, then they can. So, one thing that we've been really focusing on is what types of interaction are okay or best done through text and what kinds of interaction really need to be verbal. And so, thinking about utilizing the different means for different types of conversation. So. Thank you. So, this question of virtual labs has come up and, you know, and maybe there, maybe there are things like surgery that, oh, I think my power just went out. Can you still hear me? All right. My battery is okay. We've been having a number of challenges here the first few days of the semester. So, virtual labs, can everything be done with a virtual lab? I think Sanjay mentioned that maybe there are some things that cannot be done with a virtual lab. But I know Dmitri is working very hard in electrical and computer engineering to try to produce that virtual lab experience. So, what do you think about virtual labs, Dmitri? I think virtual labs is really an amazing opportunity that we are just beginning to explore right now. For a long time, we have been having labs done exactly the same way for literally generations of students. And I think every major university like here, we are faced with a big challenge when we have to innovate and when we have to, let's say, renew the content or the delivery mode of a lab. We, particularly when we're talking about basic undergraduate labs, one of the things that we have been doing actually for a number of years now is we have been experimenting with creating a virtual lab environment. And virtual lab can mean many things for many different labs. Typically, what we try to do is we try to identify what is the hardest thing that is there for students and then identify a virtual component that can actually help students achieve their goals. Like to give you an example, in undergraduate circuit labs, we have identified that the number one challenge that students have is debug the circuit. So in a physical setting, when you have an instructor in a lab with a few three, four dozen students, it's hard to actually spend the time to do that. If you can actually create a smart environment that will guide the students through, it can be a very effective way for students to complete the lab actually much faster, which means you improve on efficiency, you improve on learning, you give more opportunities and so on and so forth. But I think this is just step one. Step two, and going back to what my colleague said, is really rethinking of what labs should mean in 2020 and beyond. I think it provides us an amazing opportunity to think again, what should we be really teaching in a lab? Are all labs necessary? Do we need more labs, for example? Do we need fewer labs and what labs do we really need? So I hope we grasp this opportunity and we move ahead. Thank you. Any other comments on the laboratory experience? Well, I mean, I think that Dimitri's is right in the sense that there's a bunch of stuff you can do online. But you know, I think the finishing aspect, you know, you'd learn to fly many elements of flight, of flight, you know, from a flight simulator, but Sully probably got his, you know, most important lessons, apprenticing with a real pilot. But you can, I think to your point, Dimitri's maybe move more and more online. And labs are expensive and so on. And really focus on making that experience count. Maybe that's one approach we'll end up taking probably now. And do we feel that a freshman chemistry, wet chemistry lab can be the same or equal experience of actually being in the laboratory? Maybe some aspects, you know, some aspects, but not maybe the final piece. But I think we need to explore those boundaries a little bit. Okay. All right. So let's move on to another question. And this is one I know some of my colleagues were experiencing in the last spring when we made the transition. And the question has to do with how can we maximize student engagement and participation in their online classes? And some of us noticed last spring that there were some students who were deeply engaged. And once we switched online, we lost that connection to them when there was very little engagement. And on the other hand, I've heard faculty tell me that there were some students that didn't seem to be much engaged on campus, and they became very active and engaged in online. So what are the ways that we can engage students when they're online and not with us in the classroom? Carrie, do you want to start? So it's really interesting. In our research on students, we right after the semester to try to find out more about who was supporting them, what resources they had, et cetera, during the pandemic. One of the things that was coming out as we were asking them questions is students saying things like, I've definitely become a lazier student because of it. And I think part of that maybe was because students did have that option of the past, no pass in the courses instead of taking a letter grade. And so then we also had students who were complaining about they wanted to get a grade in the course, but maybe they were the only one in their team. So the other teammates were just going with the past option, where they wanted to actually get the grade. So I think that option decreased some of the interaction that students were in. Another finding that we had was that students were talking about the lack of personal connection that they had with people, and in particular with their instructor. So I can remember one student saying, you know, that just not having any human contact because four of the five courses that she was taking were asynchronous and had no contact with a faculty member at all. So I think it's really important that we foster ways that the students can connect with the faculty and with each other and also in informal ways. So another thing that the students were talking about is that, you know, walking to class together, they could discuss things with the person going with, or when they would get into lecture, they could turn to the person beside them and say, I didn't catch that, did you? Or, you know, how did you do on your homework assignment? Here's what I got. I mean, there are all these conversations that students were having with each other that really were supported to their learning. And interestingly, there were students who didn't have each other's contact information that they were taking class with that maybe when they were in class together, they were having these kind of conversations, but they didn't even have a way to get a hold of them once they were in the pandemic. That was just amazing to me that they could have interactions so regularly when they were in person and then they lost it the way that everything happened. So some of the things we're doing are, you know, creating Slack channels for a course. And it's kind of akin to having a discussion board like on the edX platform or one of the other platforms where, you know, students can post questions and other students can respond back to it.  If I agree, they can also direct message each other. They can direct message me. And then I have my, you know, the teaching team helps to man that, you know. So those are some of the ways that we can foster those informal interactions. But I think it really is very, I think it's very crucial that especially when we're talking about this age range of higher ed. So I mean, developmentally, having the expectation that they can manage all their time when there's all this freedom. I mean, that's really challenging. Yeah. Yeah, that's important. And I know our president used to equip that in the future, people will ask whether you earned your degree in your pajamas or whether you really went to college thinking that there was, they might be fundamentally different experiences. But maybe we can figure out how to make that online experience almost as good. Sanjay, I think you had some comments you wanted to add. Yeah, I actually wanted to pick up on something Carrie said, and maybe another way to look at it from my perspective, which is very consistent is that online is good if there's intentionality to the communication. So for example, if I want to attend a meeting, it's pretty good, I can set it up, but I cannot have the hallway conversation. And I think a lot of the informal conversations are low on the intentionality scale. And as a result, there's an informality, you can even ask a dumb question, like, Hey, I totally didn't get that.  But I think we need to figure out how to introduce the incidental low intentionality conversations. I don't think we cracked that not yet. Interesting. Brian, let's maybe we can hear from you. Yeah, I'm just coming from, you know, from our, from what we do at Intel in terms of like the engagement and a lot of it has to do with it could be translated, I would hope in education with the instructional design, which is going to be more project based and you have joint deliverables, you can create communities of inquiry. So they're ways that, you know, the students can help be more active in sharing their knowledge. I talked about student generated content and making that part of it in some type of shared way. And I think that will prepare them well for whatever they do next, because that's how we do everything at least at Intel. Okay, good. Dimitri, did you have something to add? I just wanted to add the aspect that a lot of students are actually very familiar of using online tools to get connected with one another. Keri mentioned slack and there are many, many others. I think one of the responsibilities that we have as instructors is to become also equally familiar with those tools. And I'm really wondering how many of our instructors are actually ready for that. There are certain things and certain behaviors that we need to engage in their social media interactions, for example, that are familiar to our students, but perhaps a little bit less familiar to some of us. And then the other comment I would like to make is that students remain engaged online when they find that they get quick responses to what they need. So it is really our responsibility when somebody's posting a question discussion forum that they get a rapid response back. When that happens, you can see that students become very, very engaged and they love that. So I think it's just a different environment that all of us will gradually embrace and hopefully that will help things improve a lot. What happened last spring was an anomaly, but I'm hopeful about the future. All right. Thank you all. So let's move on to a slightly different topic. Online education just naturally generates a lot of data that we can mine and analyze. And the question is, what have we learned from this data? Artificial intelligence and data science seem to be changing everything that we do.  Now, I guess we could be collecting data from students in the classroom as well. We aren't doing that right now. I don't believe we're doing very much of it. Should we be doing it? Would it be helpful? Are there privacy concerns that we need to be worrying about? So maybe I'll start with Carrie on these questions because I think she's been looking into some of these. Well, there are a lot of possibilities. I think when we start talking about data and looking at it with online education, it seems like the conversation tends to go pretty quickly towards either intelligent tutors or some kind of competency-based learning where there's recommended content to the learners. They can move through as fast as they want, that sort of thing. We've used the data to try to better understand patterns and how learners utilize the information who's using what. I think if we do it well, I think we can actually use that data to inform future improvements based off of evidence. But I think you make a really good point that, if I can say it in my own words, that online education receives far more scrutiny than in person does. So I think with the data, we collect a lot of data on online learning and it seems like people tend to just have a little bit of skepticism about it being online or sort of like there tends to be a lot of folks that have an implicit assumption that it's worse because it's online. Maybe that's because they're more aware of a lot of bad examples and not what could be or how it could be high quality. But I think through the data, we can determine what high quality online looks like and use that information to make continuous improvement and innovate new solutions. Okay. All right. Thanks, Kerry. Sanjay. Yeah, I just wanted to say that there are sort of two ways to extract value from the data. When you look at the data passively, you look for natural experiments to occur and you look for conclusions. And that has actually reached a plateau. You can get a lot of data. You can get things demographics, who's logging in when, things like that. There are privacy issues there, but you can figure out students are active at 2 a.m. or whenever it is they're active and officers are effective at this time. But there's a more active form of data that I think we should all think about and that is experiments where you do A, B tests and you say, I will do my office hours at 10 p.m. and one where the office hours are at 2 a.m., which class does better. There's some ethical issues obviously. We can do all this, but I do think the active insertion of an experiment makes the data valuable. And I think if you just look at the data come in and see their patterns, they're beyond the point, it's diminishing returns. I think that makes a lot of sense. I agree with what you're saying, Sanjay. Some of our work is sought to just do some benchmarking so that we even know what to expect in terms of learners' outcomes in an online environment, especially when it's like an open environment. And so folks can kind of have that baseline of here what the learning outcomes were and then as they innovate or as they try out new approaches, they can compare that to the baseline. And I think that might help someone who didn't want to do ABA testing or ABA testing, they could do some experiments but using a baseline comparison. I think you also have a point about being intentional about the data and the research. And one idea that we also have been thinking about doing some work in is I'd really like to see the platforms make progress on automating tagging of the content because if we can have good metadata structures on the data that we get from these LMS systems, then we can make more meaningful inferences coming from that data. So we can actually say this activity and this activity were all designed to support students learning this particular learning objective or these learning objectives and then how did they actually perform on the assessments that were also mapped to those learning objectives. And I think right now it's a very manual, painful process. Seems like it should be pretty, seems like it should be pretty simple but yeah the field just hasn't been able to crack them. Is there an opportunity and are we planning to do this? This fall we'll be teaching the same course to students here on campus and to a set of students off campus. We'll have many examples of that. Are we going to be looking at outcomes from those two different experiences? Dimitri and I are doing some of that, right? Yeah, I was going to say to some extent, yes. We do have in some of our last courses we have cohorts of learners that are fully online and cohorts of learners that are doing a hybrid mode and we are looking to see how those learners respond. What is their learning performance in the course? What is their satisfaction from the course? So I definitely think we'll have more knowledge than we have had an opportunity to acquire before. But I do agree with my colleagues who are still at the beginning phase of understanding how to use data to improve online education partially because we have not done it very well for face-to-face education so we have to reinvent a lot of things here. Okay thanks. So the next question is about validating the quality of online learning. How do you know that understanding has been achieved and is it as simple as giving tests? That's even challenging for residential students but then there are the added challenges with online tests and the concerns about academic integrity and questions related to this are also coming into us live now from the audience. How do we assess the quality of learning in an online environment? How do we give tests that we can have confidence in? Let's see, who should we ask that question? Carrie, how about if we start with you? All right, I have lots of opinions. I really, I wanted to see if other people wanted to speak up. I did notice Sanjay had his hand up. I'm always ready to talk about evaluation but Sanjay, I want to hear what you have to say and then I can respond. Well, I mean we've been thinking about this a lot and I mean I'll just make the statement that I think that exams are just one test of learning, right? If you're trying to, if I'm learning cooking and you want to see if I can cook, an exam is not going to tell you very much. You're going to have to do something more authentic, maybe make me cook and see if the food tastes anything like real food, right? So I'm hopeful and this question predates COVID obviously and I'm hopeful that this forces us to rethink the whole exam mentality and forces us into a paradigm where we're doing projects and you know just more authentic tests. You know, this is not something I've researched a great deal, this is something I've had a major concern about so maybe I'll take the question down, pass it on to Carrie, but I really feel that this is an existential question for us, you know, especially the gig economy, you know, people are going to work for themselves, it doesn't matter what the past test, it's going to be whether they get the next freelance gig, you know? That's right, you know, people, I noticed I think someone in the question asked something similar about online exams and I think this goes back to a comment I made earlier that we can't just do what we did in person and then stick it online, you know, in the real world, if I'm working on something and you know, I have awareness of what that issue is or, you know, I have my content understanding, but if I don't fully, like if I'm in the problem and I need to go back and reference my material, I'm able to, you know, if I'm working on a project, I can save you a colleague, what do you think about this?  And I think the type of knowledge that we're talking about is very hard to assess in a multiple choice exam. However, if we always are going to have large lectures with limited instructional support, it's going to be really difficult to assess them in any other way. So I really think that the quality, again, the quality of the online learning or how we determine that, I think it's really based off of our values. And what we value is being good, like how do we know a good course when we see it? And there are different frameworks for that. You know, I have my ways of thinking about it, but I think it really goes back to what is it that we most, you know, we most value, and then that's how we evaluate it. Right. Thanks, Kerry. Dimitri, did you have something to add? Yeah, I just wanted to say that my colleague's phrase, it's articulated so well, I don't really think I want to add too much more anymore. But I do want to say that it is an important question, in my opinion, it's a short term versus long term issue. In the long term, I really think we should completely rethink of how we evaluate students. In the short term, I'm sure we're going to go through some growing pains. But I also want to say that it's a little ironic to me that before the COVID data, many of us have been discussing with each other, saying, well, you know, exams don't mean anything, or you know, just because this person has such a GPA, it doesn't mean that they necessarily know the material and all of these complaints are back and forth. And then when the COVID hits, the first thing that comes to most of our minds is how do I replicate that exam I was just complaining about. So I think I do want to be optimistic and follow what Sanjay said. I truly hope that this will challenge us to rethink of how we do that. And I'm hopeful that in the future we'll come up with more meaningful ways to assess students' learning. Okay, good. And maybe I can ask Brian, how do employers view online degrees or online certificates versus one that takes place on campus? I think you guys covered a lot of key topics. I mean, we look at it in terms of what you can bring and how you can get things done through others. And the level of expertise that you bring needs to be complemented with end product results.  How can you do it in a way that is that we are really a learning organization? So how can you document what you're doing in a way that that can bring, you know, move us forward? But we, you know, we look at that what is the job at hand and what is the requirements that we have for that? And what can that prospective employee bring to the table? I think that obviously, you know, the prestige of the University and other alums, there's a lot of alums from Purdue and Intel. You know, obviously, there's a track record there from certain institutions. And that, but you know, if it's coming from that institution and if it's online or otherwise, you know, we have faith that they are going to come in well prepared and that we can help them at Intel to be successful within the constraints that within the organization. Okay. Thanks, Brian. I noticed somebody in the window in the chat. Dave said the my test at all. And I think something I was thinking about is, you know, there needs to be some way for students to demonstrate competency. That's the point. Like we need some way for students to demonstrate what they know. And in order for us to design assessments that are going to effectively do that, we have to first know what that competency looks like. We have to actually think about, you know, if a student is competent in this or they have learned this, what does that actually look like? And then we need to consider, you know, what kind of opportunities we can give students to demonstrate that competency. So sometimes it may be an exam. I'm not going to say it's never, you know, like sometimes that might be the best. Sometimes it might be more like a performance exam. So, you know, we're expecting students to complete this task. And like I said, going back to my analogy earlier about, you know, in the real world or working in my resources, if I can't do my job in an efficient manner, or if I'm constantly, you know, flipping through a textbook trying to find out what I need to do my thing, I'm not going to be a very good employee. But if I, you know, so I think, you know, if we can create opportunities for the students to demonstrate what they know, but not, you know, but in the same way, in an efficient manner, I think we can create sort of innovative approaches here. And maybe also with adding that there's a difference between formative and summative exams, right? I mean, exams are good for assessing, well, okay, as we all just said, at assessing capability, depending on the topic, maybe it's good in math, but not in music. But formative assessments are very helpful, which is where they use to promote learning.  And that's what the edX platform does, actually, which I'm sure that Dimitri's and Kerry have used. Okay. So is this another example of how my necessity now dealing with the pandemic and doing more and more remotely and online, we have these questions about how to do assessments that is going to end up changing the way we do assessments on campus when we're past this? Okay, let me move on to another question. You know, there is an increasing number of master's programs online, more and more certificate programs, the micro master's programs from edX. What's the next type of, is this what we're going to see more of in the future? Or is there a different kind of program? And related to this, will we someday see complete ABET accredited online bachelor's programs in engineering? So who should I ask? Sanjay, you're on my screen, so I'll ask you that question. I was hoping you would point to Brian, actually, because he's the recipient of the, he's the person who's going to give these people jobs. I, you know, for all my excitement about online, I happen to believe certainly at MIT, learning by doing is very important. And that's why we created the micro masters. So we did some, you know, my view is that there will certainly be new credentials like the micro masters, where you get some piece of the education online.  So my view is that there may be, there certainly will be successful undergraduate programs fully online, but I think we'll see the emergence of a number of new stackable certificates and credentials. A new, a new sort of language of achievement that companies like Brian's company will have to figure out and collectively we have to make these things real, I think.  Right. Brian, do you have a perspective from industry on that question? Yeah, I think, you know, in our case that technology, as we all know, is changing constantly. And, you know, today it's Python, tomorrow it's Anaconda, whatever comes next, right? So there's going to be that constant need to help us bring, you know, leading edge innovation. So we're going to look at those folks that are doing something meaningful in that area, not only that they, you know, have a particular expertise on it, but what have you done? What are you demonstrating? How are you sharing that? How are you bringing that to market? How savvy are you in connecting and collaborating? So I think, again, back to, you know, this crisis that has us all connected this way, if you can master this and do what's needed to create new value, those are the folks we want to engage. And they will constantly be learning new things and creating community of learning. And those are the folks that we want to kind of bring on board. You made an interesting comment earlier that surprised me. How did you describe Intel as a learning organization? We are. Absolutely. Every year we're challenged to come up with, you know, a more competitive product to an increasing competitive market. And every year we start from zero and have to increase inventories. And at the same time, we have to predict what may be happening two or three years. So we need folks that are constantly sensing, constantly learning, very self-directed. And again, we have a number of your alums helping us to drive that. But, you know, getting involved in a very collaborative and very global view, absolutely global view on everything that we're doing. Okay, good. Dimitri, let's turn it to you. I think the underlying question here is how do we really balance long-term learning objectives versus near-term industry needs? And it's interesting because from a student perspective, right, a lot of students think earning a bachelor's or a master's degree as a credential, that would be helpful to them, not just for their next job, but for their entire career. And universities, I think, in the past have been relatively inflexible in that sense. For example, in the master's level, we say that you really need to get 30 credit hours to get a credential that actually means something. And recently, we have been a bit more flexible on that. We're saying, well, not necessarily, you can get a micro-credential or a micro-master's. And it's encouraging to see that the industry does recognize those. And so that means that, yes, you can start unpacking this master's degree of 30 credit hours to smaller things and stack them together and get something meaningful. It's, to me, the next challenge will be thinking about doing the same thing at the bachelor's level. Right now, in my view, it's quite unbalanced. For example, if you get 120 credit hours on average, then yes, we say you have a bachelor's degree. If you have 117 credit hours, what we print on your CV is you have some college, right, training. And that seems quite not that I think to do there. So I'm not sure what that solution is, but I'm hoping that we can rethink of how we can perhaps unpack this at the bachelor's as well. Okay, interesting. You know, I'm thinking back to my days as an undergrad and I remember I received my BS in electrical engineering in 1974. And electrical engineering has been a rapidly changing field for a long time. I remember being told then that the half life of my education was five years in 1974. But I don't think that's quite the right analogy because things like the math, the physics, the chemistry, the engineering fundamentals, the circuit theory, they don't disappear in five years. They're there with you for your for your entire career. So, I mean, is there a distinction between what it can be well taught online and what and what is better taught?  But that assumes that you have an understanding of fundamentals and you can quickly learn these trend trending technologies. Is that the role for residential education when you need this extended semester long experience that dive deeply into fundamentals and less well suited to online education? Sanjay, I don't know if you're is it was your hand up from before or I did actually I've been this is something we've been giving a lot of thought to. In some ways, I think that and Brian forgive me for saying this, but a company wants to hire someone to do the job at hand. But our role as educators is to prepare people for life. Right. And so what that means is that we need to think of using an analogy from athletics. We want to make sure they have core body stand that they have good cardio. They have they understand how their body works and they can take care of, you know, it for the rest of their lives. But you know, the student is going to become a football player. We want to make sure that they have a good arm.  Right. So I think that the I remain a little I have I've retained some romantic views of what an undergraduate degree is.  And once the student is out in life, if they have to learn some new technique, this is where I think these micro credentials become important. Now it doesn't mean to Dimitri's point that they might that even the undergrad can't have, you know, to his point, you know, hundred and seventeen credits and you're you're incomplete. I think we can create gradations there as well, as we have with the associate's degree, etc, etc. I think our mission in undergrad residential is different. Thank you. Yeah. Okay. Very good. So let me move on to a different question about is online education, is it can it be equitable, fair and inclusive? I know we were concerned when we sent our students home in the spring about what kind of environment they would be in and some some were able to thrive in their home environment and some weren't. So, you know, what do we think about this as we do more and more online are the things that we need to worry about to make that all make sure that all students and all citizens can participate at a more or less equal level. Kerry, let me ask you that. You know, that's a great question. I don't I this is one I struggle with. So I want to hear. I know Brian's team's been working on these issues. So I'm really interested or not just but I'm excited to hear him respond to this. I do know that you know that the the numbers that the FCC has on Internet access are grossly inflated that the way that they do the counting on or sorry for Internet access. So the way that they do the counting on that is if in a given area anyone within that area has Internet access, then that area is stamped as having Internet access. And so there are areas where, you know, a large number of Americans do not have Internet access and we don't even know how many. So at a place like Purdue, a lot of our students, you know, if they're coming from out of state or they're coming, you know, from other places, maybe that's not an issue for. But certainly there are we have students from rural Indiana that may not have Internet access and we may have students, you know, needing to go outside of somewhere that offers it. And when in the pandemic, I think the challenge the additional challenge with that was that even public places where they might have been able to learn previously like the library to tap in, they no longer were able to do. So I actually I saw an interview with someone who was driving to a Taco Bell to access the Internet and was sitting in the parking lot of their car in order to access the Internet. That was because Taco Bell offered free Wi-Fi. So I so it's something I do I do think a lot about it. And I think if we're not wanting to see the divides in our society become even greater, we're going to have to be really creative about this. But you know, distance education has been going on for more than 50 years. I mean, Purdue has a very long history in distance education. And this comes back to I think my original point, which is it doesn't matter the mode, but what are we trying to achieve?  How can we we got, you know, how can we innovate that situation? But I want to hear Brian. Yeah, let me let me thank you, Kerry. And you know, there's again, Rick Chiviria, who's a Purdue alum drives Intel's overall strategy for our pandemic response. I drive the education side of it. And our focus from the very beginning was on reaching title one school students and families. Kerry, to all the points you made, I mean, if you had it tough, it got bad. If you didn't have anything, you have a whole different situation going on now. You know, we went from a digital divide to a completely new digital landscape that it's not going to go away, right? So we have to look at kind of new and permanent changes on the way that we bring solutions for education from a technology perspective. And we look at it from a solution, which is really a device, a capable device, you know, not any device will do. There's, you know, there's a misconception for years in my kind of standard role at Intel. I was working with folks that said, well, I can get it all done with a phone. You can get some of it done with a phone, but you can't get into a creation creation space solely on using your phone, you can't get into, you know, a collaborative, multi, you know, workload demand just with the phone. So we needed to look at the right platforms in terms of devices. And the connectivity, again, Kerry, to your point, and there's a lot of articles on it, it is, it is, it's just sad to see the lack of connectivity. I've traveled around the world and I don't have to go far anymore. It's just here in Washington DC and parts of the city that there's just no connectivity. And the kids go outside of the school that was connected to a home that was not connected for the kind of work that they need to do. And the other part that we touched upon is the new and more engaging content that we need. Even in LA Unified where we have a big project, the kids that can get online are, are checking out because the content is not engaging to them. So we've had a lot of discussion around that. But I mean, we cannot have enough. And again, the students that are going to come into Purdue, they're going to find ways to do things. They're going to be those folks that, that are going to, to go that extra mile to go find the connectivity. But a lot of folks in rural America are not going to have that and urban America do not have that. And we need to come with creative solutions that are affordable and capable, because you can't water it down and say, well, they just need any device. No, you need something that's the very best so they can learn and have all the advantages that we all have. And Intel's working really hard towards that. We're working with 45 school districts as a starting point around Title I schools, and we're engaging with experts, a company called First Book that all they do is the care and feeding of Title I families. So we're focused with that. We're providing content for the parents so they know how to engage the kid in Disney learning environment.  It's amazing, right? I'm here and I'm taking notes. But the parents are home now that are kind of taking this role. And I'm one and I have a kid in college. They don't have. They need to be engaged also with that. So there's a lot of areas.  We're working with Lego Education on a lot of cool projects that are going to make it more interesting and engaging. But at the end of the day, it's about helping the teacher and the educators to do the work that you guys are trying to do. Brian, I am so on board with what you're saying. Sanjay and I had a little discussion here in the chat that the opportunities we can provide our kids because of the privileges we have, our education and the way we're able to support the students is a real advantage that not all families have. And there's something really disturbing about an education system that would require the knowledge of the parents for the kid to be successful. Well, I know there was that National Academy's report on the K-12 education in this. And they said we have done a tragedy to our kids, a huge disservice with how things went. So anyway, I just wanted to echo that and say, you know, there's a lot there and it really, with the kids, the little ones, it really requires the adult being with them and helping monitor what, you know, that they're actually paying attention and engaging. So I've got a first grader that's doing e-learning and we're very cognizant that if left alone, like, e-learning is not learning. No, no. Sanjay, you were going to say something. No, no, I think you said it well. I mean, I think that essentially it takes a village. Some kids have the village at home. For some, the village is the school. And so the equity issue is multi-dimensions. Another one is you're at home. Let's say you have great internet connection. Your family has one computer. Okay. You have your own computer, but your kid's sister is crying in the background. You can't pay attention. I mean, sister, I think it's a very complex space that I've struggled with. This whole equity question. You know, one of the things with the students we were interviewing was that I didn't even think about, but there were college students who when they went home were having to be the teacher on hand for their younger siblings because their parents were still trying to work remotely from home. So, you know, our students were saying, you know, well, I have a lot more responsibilities at home, so it's a lot harder for me to stay focused. So again, there's just a lot of diversity in what home situations are and what's expected of students when they are home. And I think we have to be able to think of solutions that don't require setups like all of us have because we do have a lot more resources available than what, you know, a lot of families have. Really, those of us on this panel, you know, the people watching too, I mean, we all, you know, we have really great positions. We have good jobs. We've got a lot of resources and not everybody does. This is a big challenge. It really is. Say, I think we still have 20 minutes or so left. It's been a great discussion. I believe Arvind has been monitoring the questions that have been coming in and we probably have several from our audience. Arvind, are there some questions that you can summarize for us? Yeah, this is about 20 questions, Mark. Probably five buckets of questions. One has to do with, you know, exams and evaluation and online learning.  And C, can they be scaled up to large class sizes? Yeah. Would anyone like to take that one? If I'm a panelist, can I nominate someone else?  But Dmitri, how are you doing it?  You had it online. What did you do for assessments? Did it work? Did it not? How did it go? I can share my experience from this past summer. So this past summer I was teaching a large online class about Purdue, the university policy for summer courses was that they must be done fully online. And I had 200 students. This is an introductory course to circuits. And one of the big questions was exactly how we do assessments for something like this. So what we try to do is we try to break this big question into very small pieces. And we gave students a lot of different options and a lot of different scenarios. So we made cheating almost unnecessary. So let me give you just one example. We don't want to take too much time, but we made students take the exam twice. So first they took an exam, we had multiple exams actually. And sometimes we had the exam every week, but we make the students take the exam first by themselves. And we recorded the grade. And this was with no proctoring or anything like that. And then the students would get into a group that was prearranged. And they were stuck with the same group the whole semester. So they kind of knew each other. And we'd ask the students to take the same exam again, but as a group. So they would rework the problem. It was actually a great learning opportunity for them. They would have to come to consensus for the questions that they did not know how to answer themselves and then resubmit the exam. And the final grade was actually a mix of the two. And there was two interesting learnings from that. Three actually. The first is that we got the service from the students back. And most of them they actually said they really enjoyed working in groups and doing that. The second is that our average of the exams was actually very comparable to what we used to see in face-to-face settings. Actually almost a match one-on-one. We were really not able to detect any at least major issues. And we were able to avoid, you know, people, students posting solutions on check which actually had happened this semester before for the same class. So I would say that with some thinking and some innovative solutions and a combination of things, even with the existing tools we can provide, we can have at least reasonably good solutions even for low-level courses with a large number of students. That sort of gets back to our earlier discussion about how we need to rethink the way that we're doing assessment. It sounds like what you're doing. So one way, one of the things that I think can be used more effectively is increasing variance in tests. And so what I mean by that is kind of taking the idea that's used in designing standardized tests or large-scale testing programs where they will identify what the learning objectives are and then develop, you know, a few different or several different, in their case several different items that are tapped to measure that particular learning objective and then they can randomly assign them out.  So I think we could generate more unique exams. So I actually had a graduate student who last semester prior to the pandemic had been working in the first year courses and he had written a code in MATLAB to give each, to essentially give students unique exams because of the having a learning objective and then having different, you know, questions per student. So again, from my perspective, if students can access material and still get it done in the same amount of time, if they're not flat out cheating then, you know, if they're not, you know, if we're giving them projects or problems to work on and we're grading them, you know, that I don't, I think that's another way around it, but, you know, increasing the variance of the tests themselves so that they're not all getting the same one. And you can, and so one way of exploring objectives, another is giving students different data sets to analyze so they don't all have that maybe this question is the same, but you've changed the data so you're creating multiple forms of the exams the students may not know what they get. So it becomes real obvious that they're cheating. All right, thanks, Gary. Arvin, maybe we should try another question. Yeah, you know, a second bucket of questions is about practical skills and competence. And I thought I'd read you one of the questions from Marco Baldwin. As an engineer at TELS Tesla, one thing to value is resourcefulness and finding solutions. Competence is not necessarily knowledge known at the moment, but rather the ability to quickly learn and apply. What are your perspectives and how that could be implemented? And I assume it's in online education. And there's a similar question by Roger Willis about competencies, you know, online competencies as well. Well, I mean, I'm hopeful that we will get more into projects to do this. We especially with things like, you know, let's take data science, you could have challenge problems, you could have teams address them and so on. But I do think this is and actually I'm particularly interested in doing this in conjunction with companies. So for example, you could have an online course where the problem is solved in conjunction with Intel or some of the company, and you have an alum, you know, mentor the project, and you get a project rate. So I think that's sort of one approach where you address the difficulty that dimetries and carries just addressed of assessment and proctoring and so on. You sort of make lemonade out of it. And you go to this other extreme where you're being much more authentic in your assessment, but also weaving in some competency. That's sort of a quick response from me, but this is something I'm actually rather intrigued with. Yeah, this Brian, yeah, no, I think if it's not happening already, I'm sure folks at Intel would be interested in that. I think when you get into like the problem solving and looking at resourcefulness and how do you collaborate and bring that into the expertise that you can bring in, but also how can you learn what you need to know. And that comes from a lot of the experiences that they would have had, you know, online or on campus. And then they have an opportunity, every project is going to be different every day here at Intel to kind of have a lot of variety and work with that pace. So I think that would be an interesting thing. And Sanjay, I know you're with MIT, but maybe Purdue, we could do something too. You know, we're open to that. Yeah, just to add maybe a couple of thoughts on that. We see that a lot in our, let's say, higher level courses, you know, the courses towards, you know, focus more on senior design projects or things of that nature. And and it's interesting how the online world maybe even motivates more people to think about the resourcefulness and think about determination and greed and when they have to become more agile in what they're doing to get something done. So we saw that in a bad way, honestly, next spring, but I think in the future we can kind of put it in the design of these courses and make it a positive experience for the students.  They were, I brought them into a lot of staff meetings that I was doing on like key issues that I don't know if they would have had that exposure if they weren't as available online or if I wasn't available as much as I am online because I'm online all the time. So I think we're trying to make the best of it. I was actually surprised because I felt bad for them to be honest with you.  But I think we were able, so I think we're, you know, we get into that collaboration space, project base, accountability and responsibility for this is the piece that you're working on. You need to figure out how it's all going to come out at the end and that's the result. I think it was really engaging experience. I was pleasantly surprised plus the students were great, but I was pleasantly surprised. It's interesting. We had this experience too that attendance at seminars and committee meetings and things seems to be significantly higher when we do them online. It's just easier for people to get there than to get across campus sometimes. Also there's no such thing as a local seminar anymore. If you have or a PhD defense, you know, you've got people around the world joining. Right. Or you're on a plane traveling. I can't get a hold of you. No, they can get a hold of you all the time now. 24-7. So it's interesting. Work-life balance is, you know, that's what I used to like about plane rides. I could get away from it. Yeah, not anymore. Right. Hey, Arvind, what's the next question for us? You know, maybe it's time for a couple of more questions. One has to do with the difference in online learning needs of freshmen versus senior. There's different angles to it. One is, you know, are there differences? What can we learn from it? But a second one is also, when you look at it from the freshman point of view, how do you get to freshmen excited about engineering online? And how do you help these students grow interest in different areas of engineering or projects when they're all online? I'm going to hop in on this one. I think that's a great question.  I mean, it's huge. And we really have seen in the research that my team's been doing that there is a big difference in how the students at the different levels felt about the instructor's supports they were receiving during the time of the pandemic. And I would say that, you know, I mean, this is like, because we're still collecting, we're still analyzing data, but the preliminary things that my eyeballs are telling me that we're seeing is that, you know, having that, like the instructor, having that perception that the instructor is really present and really available to them is so important. And having those opportunities to connect with other students in their course informally is so important. So I think the cases that we've seen where the instructors, you know, we're utilizing the stuff we talked about before with, you know, creating messaging places and then participating in it in a rapid way. Like, I think one instructor who I've been studying because because a part of this was case studies of instructor decisions and student support. So one of the instructors, and the interview actually said that because so much was going on and there was so much chaos in the world and everything was feeling like changing, she felt like what she could do was provide stability and consistent feedback to her students. So everything else was going crazy, you know, but she could be there for them. And I tell you what, when I look at the stuff that they say about her, I mean, it was glowing, like they felt supported. They thought they knew that there was a professor at Purdue that cared about them.  Now, the more advanced students like when we were looking at the seniors, there was a lot of loss there. There was a lot of upset that that was their senior year of college and they had a lot of expectations about the way that was going to go and it did not. And for a lot of them, their design projects, they said sort of fell apart. They didn't get the feedback they needed. Maybe there was a industry partner who was supposed to be doing mentoring and, you know, that when everything happened that no longer was the priority of the person that was supposed to be mentoring the students, I think in some cases. So they didn't need as much of that like proactive engagement that the first years did because by then they already had their social networks connected. So they were able to still have their peers to interact with, answer questions and things like that, where the first years hadn't developed those connections and supports yet. So I think there is something where early on they need more attention and these things are really more important and then later on they've sort of acclimated to, you know, what interacting with faculty in a less frequent manner. Thanks. Mark, just a couple of questions that are related to something of interest to, since we have many academic institutions, you know, it's related to, you know, with the rise in online offerings and online degrees, you know, our university rankings go to change. Another related question is geography becomes less relevant with online. So how does that change the nature of competition among institutions? I'm going to say, I'm going to say real fast because I know I just talked so I'm going to lay rails, but I'm just going to say that current rankings and the way we evaluate courses on campus is a very limited approach. It does not give us information about how well we've taught. It does not give us much information about what we've developed. So I would be thrilled to see evaluation models that actually take into account, you know, the quality of the learning opportunity, what students actually achieved, you know, from a more holistic perspective, like just surveying students at the end about how their professor did. That's not an evaluation. That's a survey. Evaluation is taking multiple sources of information and then coming to a conclusion of sorts. So I know that's sort of a controversial thing. You know, there's lots of things about it. But I think for me, the rankings are too. The rankings are, you know, we're getting credit for getting better students, not what we do with the students we have. And I think that I see a conflict at times with our land grant mission because we are so focused on increasing our rankings because part of that's going to be the test scores of the students coming in. And we already know that if we make decisions based on those test scores that there are demographics that are going to remain underrepresented. So anyway, that's my, like I get real worked up about these things because I don't think that they are, they don't capture the values that I would have as an educator and what I want to see, like, you know, we be accountable for doing. Nonetheless, they are important. People pay attention to them so you can't ignore them.  And if you had asked that question 40 years ago, it would have been more or less the same. These things, these things change very slowly. But I wonder, are we in a time of disruption where or maybe things will get scrambled and there are some universities may react to it well and some may not be able to do it as well. Sanjay, do you have any thoughts on that?  But I do think that I think we'll probably have just pure online. You'll have the Coursera, the edXs or some universities. But I do think that when we come back to actually Kerry's point, when we truly look at transformation, I think the human touch remains very important. And that coaching, the way we that qualities maintain the transformative, you know, aspect of the education, whether Intel really values graduates from this hybrid entity. That will, if we can keep an eye on that, I hope it won't become a winner takes takes all. There'll be, you know, the people who can only do stuff online. But for everyone else, I'm hoping that universities will be able to hold their own if they focus on the whole individual, the quality and how they're transforming them, which includes coaching, tutoring, in-person stuff, maybe blended online in-person experiences so they can get the things they can't do online. Hopefully that'll keep the great universities if they can react fast enough also in the rankings. Okay, thanks. Thank you. So I'm getting a little bit sensitive to time here. We have a couple of minutes left. Arvind, was there one more quick question that we wanted to do or should we wrap up? I know you've got many more questions on that list of yours too. I think I captured a significant part of the Q&A's posted here. Yeah. So I'm sorry, Arvind, are you scanning the Q&A's or? I think we've captured most of the Q&A's here. I know you've got a few more questions than, you know, maybe you can ask those and then we can wrap up. Maybe a quick question from your list, maybe about the killer products. Let's see. I mean, we are, we're about at the end of our scheduled time. I just, I believe we want to just be a little sensitive to that, right? So is that correct? Five o'clock? Yeah, it is. Five o'clock is the end of... Let me ask one more question then and then we can wrap up. And that's sort of, you know, it's a different kind of question. But what's made the American Research University so special is this coupling of undergraduate education at scale frequently and high quality research that pushes the frontiers of knowledge and the feeling that both of those sides benefit by having them located physically together. Could we be at a time when these could be unbundled, where the online education could be much less expensive and engage many more people and it could be unbundled from the research mission? And if that worked, then would that be a good thing? Or do we need to keep the American Research University the way it operates connected together? Are there any thoughts about that? I'll give you a one-cent answer, which is I hope it doesn't come down to a dichotomy. I hope we can democratize education and keep the residential education, which benefits from research. So that's one-cent answer. Maybe I can add a couple of sentences. I totally agree with Sanjay, but I also wanted to mention that online education does not necessarily mean being apart. It means using new technology to improve what you're doing even when you're together. I want to bring a quote that I have, that I have seen, and you guys probably know it. It's from Tony Bates, and it says that people may overcome a poor choice of technology, but technology will never save bad teaching. So I think that the way I look at it is slightly different. I think the work that technology has to do is to enable the student interaction and engagement. It's been an honor and a pleasure for me to listen to the work that you guys are doing. At Intel, our mission is to try to enable and to provide the access for everyone. So thank you for that. Thank you, Brian, for joining us today and thank you all. This has been just a terrific discussion about a really important topic that I think most of us feel will profoundly change us in the coming years. So I want to thank the panel for a terrific discussion. I want to thank the audience for being with us and participating and sending in their questions. It's been a wonderful experience with a lot of challenges, but a lot of opportunities ahead. So thank you all, and I wish you a good afternoon.


Heaven is real above us, but we're not going to all stay there. Heaven is going to come to earth. I said, heaven's coming down to earth. I heard a great voice out of heaven saying, behold, the tabernacle of God is with men, and he will dwell with them, and they shall be his people, and God himself will be with them and be their God. How do you know that's true? Because Adam and Eve walked with God in the garden in a perfect place in the cool of the day and talked with God, and the Bible says he's going to make everything just like it was in the beginning before sin took over God. He's going to restore and regenerate this earth like it was in the very beginning. So what does that mean?