All right, I think we should probably get started although I see people are still joining us. Let me welcome you back to the final session of the virtual segment of CNI's December 2021 member meeting. I'm Cliff Lynch, the director of CNI and I'm very pleased that you joined us for this session today. I think you will find this very interesting and I hope that we will be able to have some conversation with our presenters after they've taken us through this because I think there's a lot of uncertainty. There are a lot of different points of view around this and yet it's a it's really quite a quite a critical issue for the way knowledge is going to be disseminated and how the culture of scholarship is going to operate in different fields going forward. It's easy to kind of, you know, minimize this issue, but it actually is quite a quite a seriously significant one. It also has implications for things like the health of some of our scholarly societies, particularly financially. Just a couple of quick notes before I turn it over to our presenters. Danielle Cooper will not be joining us today but Laura Brown and Dylan Rudiger will authoritatively cover the issues at hand. Danielle sends her regrets. We've got the this session as we have all the sessions for this virtual meeting in web meeting rather than webinar mode so you can in fact see who's with us by looking at the participants and you should feel free to drop messages to your colleagues and things like that. If you like also please feel free to use the chat for comments or questions as the as the presentation unfolds. When we finish up with the discussion at the end of this invited presentation. I'll say just a few brief words to close our virtual event for 2021. So I will see you on the other side of this and with that, let me welcome Dylan and Laura. Thank you so much for coming to talk about this and thank you also for the excellent report that you've put out, which I'm sure you will have a link to somewhere in your presentation and I have also shared with the CNI announcements prior. So, welcome, and over to you. Thanks a lot Cliff and and greetings everybody. Thanks for coming to this session on coven in the future of scholarly meetings. I'll start out by just introducing us. I'm Laura Brown, until recently I was the managing director of J store, and now I'm a senior advisor at Ithaca where I work across all our services, and on research and special projects. And my colleague, Dylan Riediger is a qualitative analyst at SNR and a former staff member of the American Historical Association. We've had some firsthand experience of what we're going to be talking about. So Dylan and I will be talking today about a new combined Ithaca SNR and J store labs project. This is the first time that this kind of collaboration has happened within our organization to help scholarly societies imagine the future of their academic conferences. The initiative actually grew out of a series of interviews that Roger Schoenfeld and I conducted with societies right in the midst of the pandemic. And the biggest concern we heard during many many concerns was about the effect of the pandemic on their meeting plans. This was the most worrisome issue, and several of the societies asked us if we had useful advice or research we could share with them about this. So this project is the result of that question. And thanks to the Alfred P. Sloan Foundation, we have funding to support the inquiry. We want to be as interactive as possible today. So we plan to make just a short presentation, outlining questions and issues and giving you a little sense of what the scope of the project is. And then we want to leave plenty of time for a substantial open discussion questions, any kind of feedback that you can give us it would be helpful in shaping this too. So, I'm going to turn this over to Dylan now to do our short presentation and then we'll invite questions and discussion at the end. So over to you Dylan. Thanks Laura, thanks for the introduction and the framing remarks. And we're going to try to kind of get in and out quickly today to maximize the opportunities for interaction here we really are in a learning mode and phase of this project and so we're really excited for the opportunity to talk with people who attend and organize conferences and staff that that go to conferences and who are immersed in the issues who can help us as we think through what this project is going to ultimately end up looking like. In terms of some some initial kind of framing remarks. I think I would start by saying that, you know, as of early 2020 the basic model for scholarly conferences reflected what I think it's fair to characterize as kind of aging ideas about what scholarly communication looked like. And as ideas about the composition of scholarly communities that were increasingly at odds with what the actual composition of those communities looked like. And so you had kind of conference models that really were a little bit long in the tooth and had been around for quite some time without a whole lot of change in many fields. And part of the reason for this is that steering a new course for scholarly conferences, especially the kind of big flagship conferences that many scholarly societies hosts is quite a difficult undertaking it involves a lot of complicated financial legal technical issues, and efforts to balance the competing interests of members of scholarly communities that all kind of collectively make it very difficult to radically steer new courses for meetings they have a lot of legacy policies and forms that once kind of they become entrenched are very difficult to disentangle. However, when the when the pandemic broke out, as Laura has already alluded to it was obviously a pretty substantial challenge for many scholarly societies but it also provided a rather unexpected opportunity to shake up a genre that was beset by a number of challenges and kind of growing tensions that we see across a number of different scholarly communities. Before the pandemic started scholar societies were already facing calls from their members to rethink their reliance on in person meeting. And the criticisms of these conferences were coming from a number of different directions. There were environmental concerns about the carbon cost of travel, which was probably the most significant voiced concern again and most often voiced concern among constituencies, particularly in fields in the environmental sciences but also in other fields as well. Where members were really calling upon societies to justify the amount of air travel involved in in person meetings and pointing towards virtual and hybrid models as a potential way to limit the carbon costs of academic gatherings. There were substantial and growing concerns about accessibility and equity issues related to in person gatherings. Some of these were related to issues of disability and ability of school of members of the community, but also a lot of issues related to economic inequalities that were making the meeting increasingly inaccessible or a substantial burden for different kinds of constituencies who met at the at the annual meeting. In particular, perhaps early career and graduate students who were who were attending. There were also, and have been, and I think this trend it's safe to say we'll likely continue fewer and fewer places where it felt safe to meet in person meetings. One example, the state of California has laws prohibiting the use of state funds for travel to a number of states, including places like Florida and Texas that might otherwise be prime meeting locations, but which become really untenable if California employees are unable to spend funds to attend meetings in those locations. And so when you think about the kind of moral and financial considerations that are involved in figuring out where to have an in person the kind of politicization of most aspects of American life or making it increasingly difficult to find places to meet. And finally I would say that in some fields. There's been trends over the last decade or two towards reduced attendance at annual meetings, especially the big national conferences that scholarly societies are often involved in organizing. And so the net, the revenue and the attendance associated with these meetings has been under pressure as well as people are kind of voting with their feet for various reasons about where they were going to spend their conference time and money. So even before cove it came out there were already a lot of kind of concerns percolating and bubbling up to the surface about the viability over the longer term of in person meetings. Despite those tensions prior to cove it relatively few associations had made meaningful moves towards virtual or hybrid meetings. Most of them were still holding most of their gatherings as in person affairs. There had been experimentation with new forms of meetings of course, but much of that was kind of tentative and on the margins, and not very common in a in a genre that was still very much focused on in person. In most fields. I would point to maybe one possible exception that I think is relatively important in this which perhaps especially applies to humanities and social sciences field and that is the decreased commitment that societies had made to the idea that job interviews would be a central activity that took place at their meetings. It was largely forced on societies either because hiring was declining or because zoom and other video formats were increasingly becoming the kind of normal way to conduct first round interviews. But I think if you start to think about places where virtual formats were starting to creep into the activities that we often associate with annual meetings in particular. It's those job interviews where you started to see some of these trends emerging first, and also that has implications for why attendance, for instance, was under pressure in those fields as well. But either way, COVID created a kind of unexpected opportunity, an unwelcome one for many societies to all experiment simultaneously with new possibilities for what an annual meeting or a scholarly conference could look like. And in the now, you know, 20 months or so since COVID has broken out. We've seen collectively a very rich body of experiments being conducted by societies really all over the planet about how to rethink what their conferences should look like going forward. And this research that Cliff alluded to that that if the case and ours conducted earlier this year has suggested that in many respects the the virtual conferences of the past couple of years have worked better than many people might have thought they would have the technological formats have proven viable to have meetings like the kind that we're having today. There's some emerging evidence that at least in some fields, the attendance for virtual conferences has been broader and more diverse than has sometimes been the case for in person meetings. This also, I think, been shown that in that in some instances certain kinds of presentation genres and I would point perhaps to the poster as a really good example of this have kind of been sites of particular experimentation and perhaps invigoration as the kind of recorded format for posters has given the opportunity to talk about their work, not just show it, and perhaps to have it seen by more people. But even though there's been a lot of successes and perhaps some surprising ones. I think it's also fair to say that the challenges remain fairly daunting. Long term financial models for funding virtual and in particular hybrid events are not clear at this moment. There are some models that would support them don't necessarily align very well with the current staff that scholarly societies have the quality of engagement has been, I think it's safe to say uneven at best. It's particularly true of networking activities and the social aspects of conferences the kind of things that go on between panels and at receptions have proven pretty difficult to replicate in online environments. And that means that the kind of, not only the networking functions but the community functions of meetings which is a really important part of what they do have haven't fared as well as perhaps the like scholarly communicative aspects of meetings. And finally there's emerging questions about how and if it's appropriate to kind of monetize the increasingly large body of recorded content that's available as more and more conferences have recorded and archived sessions there's questions that are emerging about who owns the property right to those presentations, how it might be used as a benefit to members how it might be perhaps sold and turned into content and what the implication of thinking about scholarly presentations that were once mostly kind of ephemeral. What happens when you think about them as a kind of permanent archival stream of content that's available for for later use. So there are a lot of questions that are still up in the air and remain to be answered satisfactorily. And these are the kind of things that the future of scholarly meetings project is really designed to address. And as Laura mentioned in her introduction. This is a project that the Sloan Foundation has funded and that if it can SNR and J store labs will be running together. And it's a new kind of experiment for our two units of Ithaca to collaborate in this way. We're going to be convening a cohort of scholarly societies beginning in January of 2022. For a year long project combining research, design thinking, and opportunities to share knowledge and explore together within a cohort environment to try to chart a collective future towards scholarly meetings. And throughout the project we're going to be looking for actionable kind of feasible immediate interventions into the conference space but we'll also be keeping a very close eye on the bigger picture kind of conceptual issues that are in play here. What is the purpose of meeting in the first place. What are we trying to accomplish when we bring scholarly communities together. What kinds of communicative formats are we trying to foster and encourage, and how to shifts in the modality of the way that we meet, create new opportunities to align purpose and format, or to use novel formats to create new kinds of purposes for what scholarship can be and what scholarly communities can look like. The features of this project will be a series of workshops and meetings that will convene representatives from 17 scholarly societies together for sustained exploration and conversation about topics of immediate strategic importance for annual particular but scholarly gatherings in general. Combined with a series of research projects that if the SNR will undertake that are designed to provide data to support data driven decision making about technological options about models that are emerging from either other scholarly conversations or meetings that are taking place in kind of academic academic adjacent spaces or even beyond academia entirely. And so to bring models that can that can make the thinking inside this conversations informed by data. And then finally, JSTOR labs will be bringing to the table their expertise in design thinking and in design informed thinking so that the product of that research and those conversations can then be put through kind of iterative design work that's designed to result in the creation of several prototypes of new innovations in the conference space that grow out of specific concerns that emerge from the cohort. And I think it's this kind of combination of conversation and a community of learners research and iterative design thinking that make this project so interesting and promising. We're excited to announce that just this morning actually, we have publicly announced the list of societies who will be collaborating with us on this work. This is a, you can see some of the logos for the organizations here. They're probably kind of small on zoom. I won't read them all but I will say that they range widely, both in terms of disciplines that are affiliated with the project we have a large number of stem fields but also social science and humanities fields and interdisciplinary fields represented in the cohort. We also have and I think this is quite important and will be really interesting to see how this plays out societies of widely different sizes. The organizations, the American Society of Civil Engineers, for instance, are quite large. Others of them, the Bibliographical Society of America, for instance, or the Oceanography Society are quite small and this makes this as important implications for how they think about the future of meetings. If you have a staff of around 20 like the American Historical Association does, you have many different options that are on the table. Then if you're the American Arachnological Society and you have zero full time equivalent staff and are in fact your conference is organized by volunteers. And so by bringing all these groups together, we'll have both kind of two cohorts of smaller and larger societies that can work with each other, but also will have opportunities for thinking across organizational structures and sizes that should produce really interesting ideas. The project's going to launch in January with an initial series of fact finding activities to gather some initial information about the experiences that the societies have had over the past couple of years. And then we'll convene a series of meetings over the course of 2022. These will all be kind of working workshops. At the centerpiece of these will be the design lab that JSTOR Labs will be running in the summer, which will produce some kind of iterative design work, and a final public report will be published at the end of 2022. And we anticipate that the innovations that the cohort will begin to collectively come up with will start showing up in scholarly conferences and annual meetings as early as 2023. So as I'm, as both Laura and I have mentioned, we're really at the initial phase of this project. We're officially launching it next month. And we're currently in a learning mode. And we took this opportunity and this gracious invitation from Cliff as a chance to really have a conversation with the people who are here today. And to, to gather more information as we approach the beginning of this project, and we knew that we would have here today people who attend conferences who organize them who have staff that they have to think about how to use conferences for professional development purposes. I really wanted to leave as much time as possible for open conversation. So we're going to open things up for the remainder of the session to a conversation with everyone who's here. We have a couple of initial ideas that we wanted to put on the table as possible topics for conversation. And it's meant to be conversation starters more than to dictate what we need to talk about some of the things that we though would be interested to hear is generally about your experience at COVID era conferences and what you think has been successful or less successful. So why not and to what extent and how your thinking has changed around when it's necessary for you or your staff to attend conferences in person and when it's more appropriate to meet in virtual or hybrid formats. So what kind of agenda items you'd like to see on the table as we start planning the workshops that will constitute much of this, much of this project so with that, I'll close my remarks and we'll open up the floor to people who'd like to talk and I'm going to do it. So do we have any questions Laura. I don't think so I think we're just very eager to hear some of the experiences. People have had and, and especially like where, where you all see there's opportunity that we, you know you, you think we should be focusing on I know one thing that came out very strongly in the interview process. Was the question of vendor exhibits and how they had, they had basically been done the same way for decades, and not too much taken advantage of in terms of, you know, what's possible with digital or some kind of a hybrid thing. But if you look at the exhibit, the raise on debt rates, it's bringing together what what the society brings is the community that the organized community of learners and experts into that space. And the vendors want to reach that community with their services and, you know what happens lots of times people are sitting in a hall for a long time, just, you know, talking about vendors talking to each other, and the opportunity to really interact is is limited. And, you know, what kind of reimagination can we do there that would be powerful for all those different stakeholders so those are the kinds of questions we have that I think people are starting to to wonder about, given what they've just been through in the last year and a half. But what we're interested in all those kinds of questions, where their opportunities. So you see we have a couple of questions and comments coming in on the chat. And I think, I think, I think one question is about about the necessity for specialized conference platforms as opposed to a general purpose tool like zoom. I think it would be interesting to hear what you're thinking about that and one of our colleagues had a comment on that scout. Another question that that that's kind of implicit in some of the things that Michael Seidel mentioned in his comment is the is in is international scope.  And that's time zones. When you're dealing with something that's mostly in in a couple of adjacent time zones like CNI. Yes, there are a few, there are a few individual outliers but mostly it coheres into a fairly compact set of time zones. And when you get these really international meetings where, you know, in the, in the in person times, everybody sort of made a pact to go away someplace, get on the local time zone for a few days and have the meeting. That seems to really not work in the digital environment. I'm wondering if you're hearing people think that went through it all. Yeah, I think that's a really interesting and important question cliff and it's something that I've been trying to think through I mean, one of the traditional features of in person conferences for better or worse and it cuts both ways is that they've been having multiple gatherings of communities in often kind of liminal spaces people get on a plane and they leave their house and they go somewhere else. And they're together for a concentrated period of time and it creates physical kind of manifestation of a community. It also creates a kind of temporal bubble for people to inhabit a conference in a, in a very kind of totalizing way. The virtual conferences often end up being integrated into life rather than kind of set aside from the rest of life. And so I think that as one of the one of the questions that I think is on the table is like, what's the value of the conference as a liminal and physical space that's kind of separate from a lot of the rest of life. And then what are the disadvantages of that. And also in relation to things like time zones. We're also seeing in a lot of instances, a trend towards lengthening the time of conferences. Some of this is about optimizing time zones that are convenient for as many people as possible. Some of this is about, you know, realizing that if you're not bound by hotel contracts and airplane flights that you can meet for two weeks or three weeks instead of over a long weekend. And as you stretch the kind of, as you play with time and stretch it out, you lose that sense of a conference as a kind of singular event that's happening, and it becomes closer to almost like, you know, you have dreams potentially some kind of like Netflix out there where you can just kind of turn it on anytime and see a conference happening somewhere or another. And so I think, you know, I don't, I don't know what the answers to these things are, but I think as you play around with the temporality of conferences, you gain advantages like not having to have 8am sessions, or being able to think across time zones in new ways. You also have to reckon with what's lost when you're not meeting in a kind of liminal physical space, if that makes sense. Yeah, I think that I just extracting yourself from your regular life to be 100% at a conference makes a huge difference to the success of the conference. On the other hand, the participation that can happen for so many different parts of the organization in a virtual meeting. People were really excited about that and you know getting students involved and you know lowering the cost and all that. I do think that one of the things we heard was that there was more interest in doing regional meetings. And that that, I don't know if you all are finding that, but the idea that they could be less heavy to stage and more focused on the concerns of, you know, that particular region. Or even that, you know, sort of topically focused, but that a more intimate meeting in a, in an online space can sometimes be more successful than a giant presentation mode. Especially, especially when the presentation mode is like the old fashioned, let me get up there and talk as an academic for an, you know, 45 minutes about my paper. And that on zoom, or whatever the medium is, can just, you know, be deadly in a way that the you up the ante on well if you're going to try and entertain people for that period of time. You know, in a, in a, in a medium that people usually they are entertained by that the idea whole idea of a presentation may need to be different. We need to be more highly staged and more thoughtful about audience and all that kind of stuff so I do think that the, the idea of smaller, I think somebody said here that they, they thought the more the smaller meetings were much more effective because they were more informal and actually more interactive. That that may be an opportunity that would be interesting. And when the big conferences happen that there may be like a whole new level that we have to get to, to present something that's worth people sitting there. Laura, there's a question that's come in that that has to do with vendors that I'll just ask you if you want to take a crack at. And it's from it's from Lisa Hinchcliffe, who says that you know one question is about which subpopulations of people are no longer attending virtual meetings. She says that she's heard from vendors that there are more attendees, but they're not the decision makers on purchasing who are coming. And so the overall effect is that, you know the vendors aren't selling as much because the right people aren't there. Has that was that something that came up in the interviews that you and Roger did. No, not so much. I think that what came up more from that was just that, you know, that a lot of vendors, I think would love the opportunity to rethink how to have the right kinds of conversations, what the conversation should look like, how to, you know, how to present things in some way that that entice people. So I do think that's a question of if there are if people aren't attend, you know, quote unquote attending. Time is very fluid in this environment that you can schedule all kinds of different ways to have the conference and so the idea that that's one of the things that has to be rethought is how do you get the decision makers to attend some of the virtual exhibit stuff that doesn't look like walking around virtually into people's booths and seeing what they're doing but something much more dynamic that way. I think that's right for conversation. But we didn't hear we didn't hear that we heard that more people who hadn't been able to go before were able to go, but we hadn't heard that, you know, sort of the leadership had decided not to go or I've talked over the past couple of months with several different vendors, and the issue that I've heard mostly is more about just getting people to come to the virtual exhibit halls is very challenging. The offerings that people are presenting are not getting a lot of views and are not the traffic is just down considerably. And also, I did some exploratory research earlier this year to try to figure out what I could find out about demographics of who was attending annual meetings in 10 or 11 fields. And it was pretty difficult to find out much specific information. This is one of the things that I think we're going to be in a position when we have the cohort together to learn more about because societies do have a pretty good idea who's attending their meetings. They often either don't publish that information or it or it gets published, you know, on a delayed schedule, but by bringing together a group of societies from a bunch of different fields. One of the things we'll be trying to find out together is who's attending the meeting. And, you know what their profile look like there's been for instance some evidence to suggest that graduate students and early career people are more likely to attend virtual meetings, but there isn't a whole lot of hard data to back that up. And this will be something that will be in a position to to understand it with much greater specificity and I would think that Lisa's question about like who's coming is the kind of thing that societies probably know and are not for one reason or another sharing in a public setting. There's a number of comments coming in the chat it's actually so many of them that it's hard to keep track of, I would encourage people if they have something they want to say to just say it. There's no reason that this needs to be the Laura and Dylan show so if you've got a comment that you want to chime in on please, please feel free to turn this into a real discussion. And then that'll also allow us a little time to scroll through the chat. If somebody wants to actually talk. Just go go for it turn your mic on and, or I can do flow can floor control if necessary if there are too many people want to talk at once. Well, I'm happy to say a few words because this is an issue which the iSchools deal with quite actively. We are planning right now for our third virtual conference, and one of the real challenges is something you said the social aspects of it. Dale was talking about that too it's one of the things that we don't always do well in physical conferences. We cover the planets that we have time zones from New Zealand to Pacific in the US and Canada. This is all challenging but it is learnable. We are happy to share our experiences with anyone who is interested. We also use a vendor that allows people to create their own breakout sessions, which I think is a big plus because it allows for conversations. I could go on for much longer but I will stop. Unless of course there's a cap. Well, Scout has a quest, a comment, which I think is just really important, and perhaps. She suggests you might speak to this Michael but I think others others will wait in as well. And that's about mentorship for early career people graduate students. And that's, that's as much a social process as anything else. And, you know, really seems to be in peril in the virtual environment. But it's so important. I will just say a word about this because this is something that has come to our attention to, we have early career and graduate students sessions. One of the things that we're really trying to do is to mix it so that the people in those also get to engage not just at a particular time and with particular set of people, but with more people. In theory, this is possible. The Japanese members especially would like not just to talk to other Japanese but would like to talk to Chinese to people in Europe to people elsewhere in the world. Enabling that all is a technical challenge, but we're working on it. We were learning many things and we need to learn more. The value of early career mentorship is really a subsection of the general question of whether what the quality of engagement is like, and how much of the value of conferences happens outside of the panels. And certainly I mean I'll say for myself that I have not yet seen a platform that has really inspired me to feel particularly engaged in the networks or receptions that I've attended. In fact, I found that those used to be my favorite part of academic conferences and nowadays when I see them on a conference schedule, I usually avoid them at all costs. And I think that's a really tricky thing and I don't. You know, I suspect this is something that our cohort is going to be very preoccupied with, not only because one of the functions of meetings is to kind of replicate scholarly communities to reproduce them over time and to provide by providing mentorship, but also because so much of the value that maybe particular the decision makers that Lisa was talking about earlier, get from coming to these conferences is who they can see. And that's are a decent, you know, sometimes at getting people talking but they're not the same thing as having a cup of coffee around a table, which I think Laura had mentioned this kind of interest in more regional gatherings I think that's one that some societies are starting to think about is kind of what I'm seeing what I've kind of calling like hub and spoke model conferences, where you might have an annual meeting, being organized in conjunction with a series of regional meetings you might have a virtual virtual meeting that's actually happening live at different cities around the country or around the world. So all kinds of questions about how to do this effectively the the many of them seem to. Many of those models are built around the idea of hybrid meetings and hybrid meetings are very tricky to pull off their super expensive to figure out how to make work. I do provide some kind of models for figuring out how to have opportunities for people to be gathering in smaller more concentrated groups that is attractive to a lot of people. So just from what I've heard and we've been following this, most of the people at Ithaca let us know when they go to meetings like what was successful and what wasn't. And there's probably at this point, more that wasn't as successful in the experimentation place. I was saying that particularly was, I think challenging was the idea of trying to match people up beforehand into. You don't know these people and here's your cohort that you can, you know, spend some time in little breakout rooms with and I don't think the breakout rooms worked I think people resented being, you know, match made match made in it. And in yet those are the kinds of things that do happen at meetings when you happen to sit down at a table with five people and you don't know, you know, two of them and a conversation starts and it's just, I don't know how we build that kind of serendipity into relationships just are created in these spaces, but that's got to be part of the challenge if we're going to do more of it online. And maybe it's just because it's smaller you get that in a way that you don't get it and can't get it in a very big meeting. Thanks everyone I just have a quick comments from a different aspect. You mentioned about the difference about the in person conference, you have like a separate conference time days right to focusing on whatever same conference. But right now I guess the big challenges online virtual conference integrated with your daily life and your daily job. So that, however, I, I was thinking about something I might advocate in my local institution for spring is what about if you made it as a virtual conference let's say this week, virtual say I do the same thing like I did before for in person conference just to block my calendar. This is my virtual conference. I'm going to just to focus on this topic. Why again this is as a regular in person conference I still catch my work, like during the break or after a night or something. So again, I was thinking about that aspect. If the society individuals organization, try to kind of building this kind of a new behavior, why you are like for me I still have a tons of a meeting during this week get into my calendar but what if we, we find a way to still isolate ourselves for this topic for me I think is a huge difference is if I do that, I will have my mental space to follow the conference, even including the virtual one or pre recorded the one to build that kind of a scholarly thinking strategic thinking.  It's very hard to sometimes if you sit there you're still doing multi tasks to do your work right so just want to offer that. Yeah, that's that's so interesting and somebody. I think somebody said earlier in the chat called it conference hygiene was that what it was. I don't hear but I think same kind of thing yeah conference hygiene. Stephen talked about. I don't think we actually have those habits right now. Most of us. And so we're multitasking, or we'll say, Oh, I don't have time to do this really I'll listen to this offline and you know and then you're in the situation where you're sort of half in and half out. I don't actually bring your, your full self to the conference which, which is not what you're getting most of the time, if you're there in person. Yeah I would, I would just add to that that this is one of the things that a lot of societies and conferences have found advantages into stretching the conference schedule out, which is one way of thinking about allowing you to access it, but also makes it easier to pop in and out of, and to think of it as less of a kind of all encompassing experience. At the other end of the of the spectrum there was a conference this spring the the Peter Palooza conference that met for 24 hours straight and experimented with radically compressing the schedule to try to make more of an event out of it. So Peter, for the kind of the kind of issue you're raising about like how do you make it feel more like a conference, how do you dedicate the space to focusing on it and not on other things. How do you feel of like radically compressing the time so that it's something really feels like a concentrated event might be a potential model for recreating that kind of like all encompassing, but relatively short duration event that people can justify taking all day Friday to go attend, because it's going to be over as opposed to one that's going to go on for three weeks. Zoom fatigue is getting more and more real for all of us so the idea though of being, being able to concentrate for X number of hours a day after you've been on zoom all day anyway and then you're just going into, you know, full zoom mode for the, you know, for the conference. I, I think that's going to get worse and worse for us if we're not careful so so so the idea of breaking up the time or thinking about, you know, way to, to fit more in, in, in how we're going to learn when we're not online when we are online as opposed to when we're surrounded by all kinds of stimuli at a meeting, and, you know, able to look at the other people and able to, you know, everything you do when you're sitting in a, you know, in a session and, and it's, and it's live. So, so I do think a lot of this kind of presentation and timing is going to have to be rethought if, if we're going to be successful. If we're approaching the end of our time. There is an incredibly rich set of comments I've, I've, I've never seen anything quite like this since we started doing virtual meetings I mean, people will want to be sure to save a copy of the chat from this because it's really worth spending some time on. Before we let you go, let me ask a final question or a near final question that calls for some speculation. Do you, do you think that we're going to end up with a relatively uniform set of best practices and approaches for conferences in the, in the post pandemic era. Do you think that there are going to be just like wild disciplinary variations in practice. You know where the engineers do one thing and a biologist do something very different, and the historians do something yet again radically different. I think that's a really interesting question I think it's kind of an open question right now. And my sense is, and that discipline and resources and staffing capability will likely play a significant role in how different kinds of organizations react to these kind of things. And some, some types of some fields have much more in the way of resources than others and that's going to necessarily shape their decision making. But I also think that some of it is going to come down to leadership and vision and willingness to engage in really creative rethinking. The conference is really a pretty venerable genre that hasn't has been very resistant to change over time. And absent determined efforts to see things through I wouldn't be surprised if inertia kind of pulls things back. But I think it will be the case too I mean I don't want to sound too, too much like a burn it all down guy because I think some societies will make smart decisions that they want to go back to something that seemed like it was working pretty well for them. But there were there are a lot of fields that where there was already a lot of dissatisfaction about what was what the conferences looked like whether it was declining attendance, whether it was concerns about carbon costs or equity or, you know, accessibility issues. Those issues are going to survive the pandemic and suggest the need for continued thinking about these things. It's going to be tricky because these are hard steps to ships to steer and I think my hunch is that you'll see a spectrum, but maybe not a super radical spectrum. But I also expect there will be some outliers that will be instructive. Yeah, and my sense is that, as with all things technology is going to play a big part. We don't yet have like the kind of standard platforms that tell us how to put on good hybrid meetings that are cost effective. Not what zoom has done to the way we do meetings now and the kinds of innovations that they've introduced that have changed me or even something like PowerPoint. I mean these these kinds of things tend to homogenize the way we approach these new territories, and in some ways channel the kinds of innovation that happen. And within those technology choices, I think you get a lot of idiosyncratic and an imaginative, you know, uses and processes. So I think we can't count out technology as as as something that's going to be more and more an innovation factor, driving us. And I also think that adjacent industries that are completely dependent on events are going to be doing things that tell us new ways to think about how to present in the academy and how to build community online the community. And we've got to be looking at them that's one of the things that this project is doing is trying to find out where those innovations are happening in adjacent industries. And so I think, even being exposed to that can help change the way we solve problems and an experiment. Well thank you there's a lot. There's a lot in there to think about. I can't resist just as we bring this conversation to a close, putting a big underline under something you said earlier about repurposing and all of the, the fact that we now have records of conferences in a virtual environment in a way we historically didn't in many cases and the implications of that and what happens to those materials. That's something that I've been watching very closely and I know you have to and it's something I expect we're going to come back to in in future CNI conversations, because I think that that's another very significant thread here that we really, it deserves an hour in its own right, at least. But thank you so much, and I really hope that you will keep us posted on what you're learning through this process because it matters a great deal to all of us I think. Thanks for the opportunity Cliff and it's been, we're very excited about this project and we feel like it's kind of green fields to understand. Yep. It's possible so. All right, and as I said, people, people definitely should snag themselves a copy of the of the chat from this because there's a lot of wonderful material. And with that, I think we are just about at time. We've kept Laura and Dylan a little long. But I really didn't have a lot to say to close the meeting other than to thank the team at CNI who have made it run so smoothly. And to thank all of our presenters and not just the presenters that you've seen during the synchronous meeting but also all the presenters who made pre recorded presentations, we those will continue to be available. And there's a ton of really wonderful stuff in there. So I hope that you will continue to to mine that collection of material and share it with your colleagues. I just want to mention that we will be sending out in the next couple of days. Evaluation questionnaire for for this virtual event, we're going to do evaluation separately for the virtual and the, the in person event. Since they, they have different sets of attendees. I hope I will see some of you next week at the in person meeting for those that I'm not going to see or speak with again. I wish you a good holidays and a good new year. And I hope everybody will stay safe and stay in touch. And thank, thank you all for participating and particularly for participating so so eagerly in this last session with all of your comments in the chat. I know they're going to be helpful to Laura and Dylan and honestly they're very helpful to me as well. And with that, I will just say thank you and take care.


Hi, everyone. This is Akamsha and I'll be moderating this session, which is preserving privacy while sharing data by God and have, and we'll be starting in another two, three minutes. I think we should wait for more people to join in. All right, then I think we are good to start now. I'm going to play the. We know if you are unable to hear of, or if there's some technical difficulty. So here we go. For the interruption, I'm going to try to reshare my screen so that you can hear him properly. Government employees in the US. And in some countries it's a norm that salary data is public. So you first of all have to determine what actually constitutes sensitive data. And this can also be complicated for reasons I'll get to in a moment. I'm a trusted aggregator in the last slide.  A lot of private organizations sell data for profit these days. And also, you know, there's data breaches. You trust a company to keep your personal records safe. There is also some specific technical issues that relate to aggregating data. And one of them is the lack of data diversity and K anonymity failures is one term that's used in this, in this area. And basically what this means is that if it's known that you are part of a data set, that by itself can reveal something about you.  But if you're in some particular health database, for example, that may reveal that you were tested for some particular disease which presumably increases the probability that you actually have that disease. And then, finally, and again, this is a whole field of studying into a lot more detail here, but there's a general susceptibility to certain types of attacks. And it actually turns out that when researchers study this kind of thing, a lot of aggregated data and anonymized data that I think the average individual would say, well, yeah, that seems like it's pretty anonymized and that ought to be okay. And researchers have come up with ways, some of which are admittedly probably a bit artificial in real world data sets, where it is at least in principle possible to re-identify individuals in those data sets in a number of different ways. And even if they can't do it perfectly, they can do it in a statistical way. So we can say, well, I don't know that this data point is this person, but there's a 30% chance of 50% chance. And in many cases, that's all that you need in some cases, say if somebody is trying to track down that individual. One of the particular attacks, and this is from a US Census Bureau slide, the US Census Bureau actually has a lot of good information if we'll dig into this kind of thing in more detail. And on the right hand side, you will have certain types of aggregation with statistical measures against those aggregates. The number of females, the number of males, the numbers of white, black, married, black females, etc. And none of those are one. They all are an aggregate of some number of individuals and we could increase the numbers here too. But really you can think of that as almost a set of simultaneous equations which can be solved for. And that again can be transformed then by solving that set of equations into some very specific data points that don't have a name attached to them. But our unique fingerprints, which at least in smaller data sets may make it possible to, again, statistically perhaps figure out what is the entire set of data that applies to this particular individual. We can also identify patterns. This is an online fitness tracker data point, instead of data points from some individual. And if you look at these patterns, you can look at that and go, hmm, this individual that this belongs to, I don't know their name, I don't know their address, I don't know anything else about them. But they seem to concentrate their time in a fairly limited area, which is where the bright white is in that screenshot. And we might infer the person lives right around there somewhere. Now, maybe they get in their car and they drive to a park and that's where they do their running. Maybe. But if, if, again, if somebody is trying to track somebody down, for example, or something like this gives them a pretty good idea. And certainly similar if this was done with GPS GPS tracks from automobiles or anything like that. Yeah, you know if somebody is always coming and going to this house. That tells us something about that is probably where they live. If they're going a lot to an other person another house somewhere well that tells us something as well about their patterns. A really interesting area and probably particularly relevant today and will be very relevant when we talk about differential privacy is this idea of linkage attacks so what we have here in the left is hospital visits for an individual. This is an anonymized in principle anonymized record there's no one's name here there's no social security number here. But it does include some relevant identifying information that date of birth gender and the zip code. Day of birth, we could fuzz that a bit by just changing that to a year, as I mentioned earlier. The zip code is presumably a five digit zip code not a nine digit zip code. Although even then zip codes can be pretty small when you get into more rural rural areas for example but let's assume for purposes of argument that we have a record that taken by itself is pre anonymous. Well, the same individual has other records that are out there and open data sets or even that may be legally required to be public for various reasons so voter registration in many places will have information like name address phone number. Well as maybe date of birth gender, almost certainly zip code. And again, you can argue, you know, how much of this is public in a given case but for purposes of our discussion here you know let's assume that there is some number of fields that might overlap with a send with a piece of sensitive data. We can start to do correlations and of course there's not just one public record there's probably many public records. And to illustrate this a particularly interesting example comes with number years back. When many of you may remember there was something called the Netflix prize and the idea with Netflix prize was Netflix released a whole bunch of records that were anonymized so no names, no user names, no, no zip codes, no. No information IP addresses that maybe could start to give a fingerprint for a user even if their name wasn't attached. And these records were basically, you know, Alice, really like these three movies hated these four movies, Bob loved this one movie hated everything else and so forth. And it was anonymized, supposedly. Now what these researchers did was they took that Netflix data, which was intended basically for researchers to develop machine learning algorithms to improve Netflix's recommendation engine, which didn't really work out all that well for other reasons to aren't relevant here. But in a case there was this large public data set. And what these researchers did was they looked at an other public data set, which is the, which is the rating information in the internet movie database. And again, you have users who like movies don't like movies and so forth. And in those cases, at least some of those users are probably able to be identified because they may have a name that they use across different logins they may use something that is their actual real life name their real true name. And by combining those two different assets, what the researchers were able to do was essentially look at everybody's fingerprint in the Netflix data and everybody's fingerprint in the IMDB data and discover that, you know, if somebody like these three movies and hid these four movies and maybe no one hates hardly anybody hates one of those four movies. And yet there's a very similar record in the IMDB database that has a very similar type of fingerprint. You start going, you know, that might very well be the same person and I'm not going to keep repeating this. This may be a, this may be a statistical inference we may not be 100% sure it's not a smoking gun, we can take the court. But it starts to be a pretty good indicator and maybe you can correlate them with yet a third database of some sort, which has a similar type of correlation pattern and we can start to de-anonymize the data. Now, nothing here is, you know, is probably, is probably not a serious matter if someone uncovers that, aha, he didn't like the Star Wars prequels, well, that anyone like Star Wars prequels, but that's a different matter. But certainly you can imagine this being a more serious case that we were talking healthcare records, for example. This is not a new problem and solutions to this are not new. Going back to 1930s, the US census, for example, stopped publishing small area data. And this is the sort of problem I was talking about earlier, you know, if you have a census tract that one family or two families lived in, and you publish that aggregate data, well, you're not really anonymizing it because there aren't that many people that could be in that small area. This, by the way, is a similar type of thing to what, say, Red Hat does with employee surveys where you ask, responds a whole bunch of questions, including questions about their manager, and then the aggregated data is published within the company. Data is also shared directly with the individual's managers, but is only shared with them if they have a certain number of direct reports, because after all, if they only have one direct report, the aggregation of the data of their direct reports is the same thing as how that individual answered. And even if you had two or three direct reports, you know, the manager in aggregate gets a bad rating, and they have a good working relationship with two of their employees and a bad relationship with their third employee. You know, they can make pretty good inferences from that. But let's fast forward to today and start and talk about formal privacy, which the US Census Bureau adopted in 2020. And specifically, the technique of differential privacy, which comes from a 2006 paper, so it's fairly recent, about something called epsilon differential privacy specifically, and the impetus of differential privacy is the fact that, as you have all these very large data sets, as machine learning has gone extremely powerful, a lot of the predifinals, in many cases some will add hawks, statistical disclosure limitation techniques, are really starting to be ineffective and intuitions about what was sufficient in their current anonymization, where becoming less and less useful. And the basic idea here is widely shared statistics over a set of data without revealing anything about the individuals that's the real objective here. And among the requirements, what be a formal model. So, again, not ad hoc, but having math against it, resist both these kind of linkage attacks I've talked to, and hopefully resist future attacks that we might not know about today. And importantly, and again is what main impetus behind it is to be effective in places where there may be a lot of these external data sets available. The basic way differential privacy works is injecting random data into a data set in a mathematical rigorous way to protect individual privacy. The value of the randomness trades off privacy and utility and accuracy. So, if you look at the right hand side here, you the idea is that you data is aggregated by a trusted curator Alice. You have a querier who you can't trust not to try and get at sensitive data, the querier puts query in to the curator, the curator comes up with an answer, and then the curator adds some noise that and that noise is mathematically guaranteed to be efficient to keep the querier from being able to de anonymize the subject of their query. And the other way to kind of think about this is that you have this real world computation so you've got a query request, do some computation, get an output. You have a different input that that doesn't include the data from somebody on an individual. The computation analysis of that output it the difference between those two is at most and a value of epsilon.  There are some limitations here base rate so basically, if I know certain public characteristics about you such as sex age and so forth. And for certain things about you, such as your likelihood to come down, come down with some disease without knowing a private information about you, having anonymizing data doesn't change that fact. The noise is something of concern in that there is this idea that you are injecting noise into data so at some level the result isn't as good as it could be otherwise. And this was a concern by a number of researchers with the US census use of technique for example. Subsequent research suggests that in general, you can you can strike a pretty good balance here between I get between protection of sensitive data, and the accuracy of the overall aggregated data set. And one of the most difficult problem here is the idea of repeated queries I mentioned this epsilon is epsilon value earlier. And it's probably a better way just to talk about is your you can set privacy budget. The thing is you use up that budget, every time you do a query against to save data, which is a particular problem in a lot of modern countries where you're doing digital queries rather than just having an some aggregated tables that you can that you can access to. And this there are ways to deal with this so for example, a randomized subset of data can be can be used. And once you read the privacy budget has exceeded you can have a different randomized subset of data so there are techniques there, but nonetheless it's a limitation. All of this is assumed that we have a trusted third party, but what if we don't. And that's where the multi party computation I mentioned at the beginning comes in. This is essentially collaborative analysis of silo data sets without trusting a third party. Essentially you have a protocol that is equivalent to an incorruptible trusted party and conceptually some of this is similar to how enterprise distributed ledger technology blockchains work. The basic mechanism here is that parties can jointly compute a function on their own, an input a share of inputs using a protocol without information about those inputs being revealed. The objective here is to preserve privacy and correctness. At the same time there, there is an assumption and there was an assumption that some number of participants will be trying to break the protocol will collude with each other. Exactly how you implement multi party computation does somewhat depend on the threat model that you're assuming so if you're assuming that one party may by design or by accident reveal data. If they could, that is a different threat model that if you assume 50% of the parties may include in a way to sort of pierce the veil so to speak. And then you also need to think about how much overhead is involved. It works by you have a protocol distributing encrypted, specifically AES shares of data. As I said the implementation efficiency depends on threat assumptions, and although there's different overheads with different threat models and so forth in general we can say that unlike safely homomorphic encryption, the compute overhead is fairly low. But there's a lot of communications overhead because all these cryptographic encrypted communications taking place between parties. One specific use of this came out of Boston University in the city of Boston, where participating companies shared their individual wage data, but in a way that was protected by cryptography so that they weren't laying any third party. They actually see the unencrypted data shares. If you are interested in this topic. We've actually written and had some interviews on red hat research quarterly and give you a plug this is what the organizations at red hat I do a lot of work with. I suggest you subscribe to our newsletter. There's also a related website that has a lot of information about ongoing research happening at Boston University and other universities that are associated with with red hat research. Boston University red hat Collaboratory work some of that's taking place. And I mentioned at the very beginning, Andrew Trask and open mind. If you're a start that likes to get their hands dirty and this stuff. There is a Python set Python libraries there that let you play around with differential privacy and multiplayer computation. Yeah, thank you. And we have some time for some questions. Does anyone out there have any questions. Yeah, as I did drop in the chat. We did privacy was actually the topic at this week's we split up red hat research day into some topic oriented shorter sessions this month. We, we had a session and privacy earlier this week and we'll have video up from that, hopefully fairly soon. So if you're interested in this topic, that'd be a great place to go for some more. Well, if nobody has any questions. I will sign off you can always reach me at g half at red hat.com if you have any questions and I'm also on Twitter with the same handle assuming I can get my Twitter working properly again one of these days. So thank you everyone.


This research focused on developing a new type of electrochromic pixel, ECP, using the Biochromes Anthomatin, XA. The goal was to create a device that could change colors based on the amount of electricity applied. To do so, they used a combination of Pettit, PSS, and XA to create a dynamic ECP that could be printed onto any surface. By applying low voltage, the device would change from one color to another, cycling through the entire spectrum of colors. This technology has the potential to be used in a variety of applications such as smart windows, wearables, and even medical devices. This article was authored by Patrick Isseleven, Daniel J. Wilson, Matthew Valin, and others.


from theCUBE Studios in Palo Alto in Boston. It's theCUBE, covering IBM Think, brought to you by IBM. Hey, welcome back, everybody, to theCUBE's continuing coverage of IBM Think 2020. I'm Jeff Frick. We're in our studios in Palo Alto and this year, IBM Think is digital. So unfortunately, we're not all together at Moscone or the Sands or one of your favorite convention sites. We're bringing to you this all from people's homes and we're excited to have our very next guest. She is Marina Winans, the CMO of North America for IBM. Maria, great to see you. Yeah, great to see you. Thank you for having me. Absolutely. So let's jump into it. IBM Think 2020, the digital experience. You know, we've been going to so many IBM events for so many years and you used to have them. You know, you would inherit them when you brought in new companies. So there was many, many IBM events. There's a long history of IBM events. It's a really important part of your mix, but then suddenly in mid-March you wake up one day and are told, guess what? We can't do IBM Think in a physical event anymore. So give us a little background on, you know, what IBM thinks of events and how important they are and what you use them for. And then a little bit of the behind the scenes, what happened in mid-March when suddenly this thing had to go digital? Yeah, I know. Thank you for having me here again. theCUBE has been an integral part of many of our events from our conferences. Way back when. So let me start by saying that events are at the core of our marketing strategy for driving value, for driving growth. I mean, they really are about helping us build the brand. Events help us foster and nurture relationships for new and existing clients. They help us really establish that leadership to help shape business and society. And it's a great opportunity for us to showcase our best of features, our new announcements, our product, our functionality through amazing demonstrations and labs. And more importantly, it's a great opportunity for our sellers to progress conversations and to close business. And we do all kinds of a variety of events from breakfasts to what we called road shows on the road, Russian events and our premier flagship event, which is the 2020. It's when the one time of year where IBM comes together and it's a great opportunity to really showcase one IBM. And that's what 2020 is about. That's awesome. So when you guys, again, woke up mid-March, we got to change this thing, good for IBM to actually hold the date. Most of the people that we work with, which are numerous, either flat out canceled the date, postponed it to some kind of future date where maybe things are a little bit more back to normal or went with the digital, but even with the digital, they postponed. So as you think about what digital means, what a digital event kind of affords you, I think there's way too much conversation about what digital isn't vis-a-vis the physical and it just isn't. It's not the same as being together. And yet on the other hand, there's a whole lot of new things you can do in digital that you just can't do at the physical conference. So how did you guys kind of search those things out and define what a digital event is and then really getting a position to deliver on that promise? Yeah, so let me talk about that in a couple of forums. First of all, we know that our attendees perceive value or event is really heavily dependent on the in-person interaction. So as you think about that, we've had to step back and say, how do we reimagine how we deliver this meaningful one-to-one engagement that our clients know that in a value that's really now through a digital environment and really reimagine that whole interaction while still maintaining that very personalized, best-in-class experience that our clients value. And that's what really excites me about Think 2020. First of all, we are, so let me just highlight a couple of things of why I think and I'm personally excited about Think 2020. First of all, we broaden our reach. We are hosting over 50,000 attendees that's clients, partners, IBMers, influencers, and we're reaching a much broader set of roles and influencers who may have not had the ability to travel before to attend an in-person event. Second, complimentary. This will be our first event that at this size that is free. So we have many more new accounts joining us to learn about IBM for the first time. And this really opens up the doors to new business and opportunities for us. Very targeted agenda. So what we've done is we've taken an event that was gonna span over five days to now two days and we went from over 2,000 sessions that we were gonna run to really now think about how do we feature 200 sessions that are very focused, that are very targeted and very much on point and messaging of really dealing with what our clients are experiencing today with the recent world crisis. Another very exciting thing is Watson Media Platform. We are leveraging our very own media and AI platform that can host over a million users at once and really increases the opportunity for engagement. So very interactive experience. Our lab are sold out. We are completely sold out of our labs. Some of our sessions are wait-listed and we're also running the opportunity to do one-on-one meetings for these two days. So while we're not in person, we have a tremendous opportunity to really offer a very interactive experience by introducing live chat throughout the sessions and really engaging clients with each other and with their sellers during the event. And then for those that cannot attend or cannot attend all the sessions throughout the day, we are going to have what this what we're calling content on demand. The content will live on. It will live on in different platforms and we will be delivering the content to our clients, making it accessible at any time whenever it's convenient for them. So super excited about what the teams have done to really kind of step back and reimagine this whole experience, maintaining the interactivity, the personal engagement and really the value add that our clients look for. Right? You touched on so many good themes there and I just want to unpack them a little bit more. The first one that you talked about right off the bat was democratization. And as you said, there are many people that just can't get up and go to a conference whether it's just bad timing for the week, they've got to run their own business, they just can't get away. So for people to be able to attend digitally, as you said, opens up an entire kind of new palette of participation for people that maybe couldn't come before. Another thing I think that's lost is this kind of separation of content creation, content delivery and content consumption, which is consistent with how we consume a lot of things today in our media. Very few people sit down even for big television events like the Super Bowl have really appointment television. We're creating this content now. It's going to be going up in the middle of IBM Think and people might watch it on May 5th or May 6th. They might watch it on May 10th. They might watch it May 10th, 2025. So I think it's a really different way to think about it. And I'm sure as a marketer, at some point you're going to feel a little bit more freedom rather than try to force fit, say all the product groups and all the announcements and all the sessions, all this stuff that's got to line up for these what was three days in Vegas or San Francisco, now can kind of be broken up. And if it makes more sense for some information to get out earlier, it can go out earlier, it can go out later. But a conference is defined by the bounds of that physical location for three days and how much can you guys get in there? Where now suddenly you don't have those binds anymore and it really opens up a much broader kind of opportunity for you to communicate your messages, keynote messages, product information, training in a very, very different and in some ways I would bet liberating way. Yeah, I think this, what's happening here is an opportunity to really rethink in so many ways how we do marketing and what we talk about when we say digital engagement. We did a recent CMO survey, which we call our C-suite survey and we interviewed over 2,000 CMOs. And one of the big conclusions that came out of it was that digital engagement is kind of the future and the way that really what it really lands on is the fact that really digital engagement is the fair trade-off of information and the exchange of value. So when you really think about that, for us as marketers, digital engagement and value exchange comes in many forms. How we show up on our own platform like we're doing here with Watson Media, how we show up during events, whether it's webinars or third party. Even more importantly, the way that our sellers are gonna engage virtually, think about virtual selling and WebEx meetings, which is really how they're having to communicate with clients today. And then how we engage socially. And that's really for all the social channels and the work that we do with influencers. So we're gonna learn a lot. We are learning a lot. We're gonna test. We're gonna iterate. But this whole new normal has given us an opportunity as marketers to really connect the digital in the physical in new ways. And in that, we have to really consider all the multiple facets and elements when we're thinking about digital marketing. And that to me is a tremendous opportunity because digital engagement, there are many elements and it's gonna be super important for all the marketers to really kind of step back and think about what are those multiple elements that are gonna make us, as we think about, how do we stay essential? How do we stay essential with our clients? And that is providing information when they need it, how they want to consume it, and then how do we stay relevant? I think also when you think about digital engagement, is you've got to show flexibility, the ability to pivot strategically and quickly, which is really what we've done with things. We were ready to move forward on a physical event, amazing event the teams have been working on. And with everything going on, we wanted to maintain May 5th and 6th, we had made a commitment to our clients and we had to pivot very quickly and be flexible of how do we re-imagine things from physical to now digital. In a digital world, user first experience is a must. It has to be compelling. It has to be an incredible user experience and the entire experience has to be digitally connected. Content as you said before, content needs to be delivered in unique ways and where the clients are, not where you want the clients to come. So how does the content find the clients, the relationships? How do we continue to incorporate the human connection, the human connection and build those relationships with the client and interacting with them in meaningful ways. And I think one of the things that we've really over the last couple of years in IBM is grounded in leverage, really our data and analytics to better understand behaviors and determine what is that best digital investment, that best touch point at that best time. Very targeted, you know, yeah. I'm sorry. No, go ahead. I was going to say it's just, is it a hook or crook or circumstance, but just kind of ironic, right? That when you guys were forced to make this pivot and I think it's a nice statement to IBM that you were able to successfully make this pivot. Again, one of the interesting things about COVID is, right? Nobody saw it coming. Nobody really had time to prep. Nobody put in plans or was no slow transition. It was this light switch moment. But at the same moment in time, right? You just got a brand new CEO who is extremely social. And, you know, Arvind, you know, announced his, his kind of welcome to the world and his leadership changes and some promotions and stuff via a LinkedIn post, which I found, you know, really fascinating that, you know, he's leveraging the social medium in a way that wasn't necessarily done before at IBM and the leadership suite. And then the fact that you guys are making this quick switch, you took the challenge to keep that date the same, but reinvent, think again, as a testament to kind of a shift in IBM's culture and ability to move quickly in the face of this new challenge, which, you know, nobody really saw it coming to the degree and the speed in which we've had to react. So it's a very, it's a very nice statement on your visibility to execute. Yeah, and it's exciting. I mean, I think when I think about marketing and I've been in marketing most of my career at IBM, it's an opportunity for us to really kind of, you know, kind of lean in and put our creative juices to work really team together and embrace, embrace the opportunity to kind of think differently. And I think when we look at how marketing will progress and the leadership that we can provide in this digital era, we're going to see a lot more opportunity to get much more personalized. We're going to see chatbot kind of being used very strategically as we're seeing them today. We're going to look at opportunities to infuse AI in the experience to augment it, to personalize it, dynamic content delivery. We're experimenting with many platforms to really do that more effectively and recommendation engines and really continue to automate the marketing process and marketing kind of experience all the way through to make it as compelling and as personalized as we can. So a huge opportunity for every marketer in the world and a huge opportunity that we've embraced in IBM. Yeah, it's just, wow, just the forcing function, right? Just automation, we've been talking about that for a long time, you know, taking away boring tasks so people can focus on higher level activities. But it all kind of comes back to something I want to kind of close on and that's leadership. And Michelle Paluso had a post a couple of days ago talking about leadership, you responded to it and we've been reaching out to a lot of the leaders in our community to get their take on from a leadership perspective, because I think, you know, the cream really rises to the top in challenging times. And this is, again, cast upon us, no preparation, ready, set, go. And it is really a call to leadership. It's leading your own people. It's really putting a voice of confidence out to your customer base, as well as just your broader community. So that's partners and all of the constituents that play. So I wonder, you know, we've been in it for a number of weeks now as you kind of think back on adjusting your own leadership style, down as well as the leadership within the IBM senior team, great example, being able to pull this thing together and completely flip it on its head, if you will. Wonder if you can share, what are you trying to do different? What are some best practices? Who are some of the folks you're looking to to give you kind of some tips and tricks to lead in, you know, basically from your home office? Yeah, no, I think challenging times like this. And I've always said it, it is with the opportunity for every leader to step up and rise to the top. Leaders are made during challenging times. Leaders become during these, turning challenges into opportunities. So for me, you know, as I think about my own leadership style, number one is you put the people first.  And this is, you know, back to Michelle's article in LinkedIn, she talked about, you know, team matters an awful lot now and connecting people and making sure that everybody's well-being, it's their mind, their wellness, their activity and they're staying connected. I think the second is productivity is we have to keep the business going. We have to make sure that the work from home, you know, environment is safe and productive and everybody's work environment differently. Everybody has their own challenges and opportunities and we have to acknowledge that. So productivity is important, but everybody is in their own and you've got to be aware as a leader of those circumstances. You can't expect everything to be the same for everyone and you've got to be ready for interruptions. You got to be ready for those kids in the background and dogs barking and, you know, certain routines that have to be adjusted to. And I think the third is around staying connected. We have various schools and we're very fortunate at IBM, especially in marketing, that we've really embraced the new as far as the way that we communicate using Slack, you know, looking at, you know, how do we leverage our tools to inform our actions in marketing? And, you know, staying connected is more important than ever and we're doing, you know, we're touching teammates more often. I know that I am doing one-on-one calls with my team more often than I would have done in otherwise. And then being agile, you know, whether it's the tools, the process of the culture, for us embracing agile has been at the core of our marketing transformation and it's more important than ever that we stay agile, that we continue to work, you know, in feedback and learning and testing and in the ways that we know best. And it's an amazing opportunity to try new things.  Be open to innovation, try fast, learn, ready to pivot, you know, and really what's happening in the world is giving us an opportunity to really step out of our normal and some amazing ideas are coming forward. And, you know, at a very fast pace and we have to be agile and we have to be authentic to our environment. And I'll close with one thing. One of the things that we did as IBMers is we all specifically as leaders from Arvin to Michelle to every leader at IBM, we made a pledge, you know, a work from home pledge that we will support each other and we are each given examples of the things that we are taking forward to support each other as we're living through this new normal and digital engagement. So an incredible opportunity for every IBMer and really a prideful moment to stay strong as an IBMer and to stay strong as an individual and a leader. Well, very good. Well, it shows that you've been able to accomplish this feat and pulling off this really big event in the same time window that you had allocated and completely flip it on its mind. So it's a great testament to you as well as the leadership team and everybody executing up and down the line. So Maria, great to catch up. Thanks for your time and really a pleasure. Thank you. Thank you. All right. She's Maria, I'm Jeff. It's theCUBE's continuing coverage of IBM Think 2020, the digital experience. Thanks for watching. We'll see you next time.