All right. Good afternoon or good morning wherever you're logging in from today. We appreciate you joining us. Thank you for taking your time out of your busy day. My name is Marcy Reedy. It is my great pleasure to welcome you to this afternoon's webinar on supporting open science and the promotion and tenure process. Lessons from the University of Maryland. I'm a community manager here at the Center for Open Science looking to promote reproducible and transparent research practices among the education community and to expand the knowledge and know how of the open science disciplines. Before we started, I did just want to share with everyone one quick link that will hopefully be of use to you as you move forward. And that is to a knowledge hub that we have developed on the OER Commons webpage. We are referring to it or call it the Open Scholarship Knowledge Base, and it is the place for you to go to understand the who, what, and how of open science. Whether you're a student that is looking to expand your open and transparent research practices or a mentor that is looking for material that you can use to support a very able and energized graduate student. So I wanted to share that with you. You'll have resources also supporting our discussion today. And then without further ado, we wanted to move forward because we have a very exciting discussion planned for the webinar today, and I do use the word discussion very deliberately. So we were talking earlier, we just wanted to take this moment to encourage everyone to feel free to set aside some formality standards and feel free to enter questions in the chat function as the webinar proceeds. We really encourage engagement and discussion and hope to make this as tailored to your interest as possible. So, so please don't hesitate to speak up and also share your experiences and your stories, if you feel comfortable doing so. So it is next my great pleasure to be able to introduce and welcome Dr. Michael Daughtry, the department chair at the Department of Psychology at the University of Maryland. One of the intros or as we were preparing for this webinar, we were kind of doing some crowdsourcing on Twitter, on social media, on who is doing interesting things around the promotion and tenure process. And Dr. Daughtry's name came up repeatedly. So we're so pleased that he could be there. And one line, and I'm just going to quote from your, your bio one line, if you don't mind, but one line that really stood out to me that I absolutely loved is when he says that the view of a mentor that that you had and one that is carried on that you've carried on, is that the process of doing science is more important than the results one obtains. And so I thought that was just a really interesting context and important framing method of what we're trying to think about as we move forward. So, thank you for joining us. Yeah, my pleasure, my pleasure. And thank you for inviting me at this is, I think a really great opportunity to share, you know, some of the things we've done here at the University of Maryland. Maybe impart some lessons learned for the rest of the community who is interested in really promoting and advocating but also moving beyond just advocating for open science but actually doing the hard work to make this happen at your institution and I hope I can provide some help. So at the end of this webinar, Marcy is going to put up a link to a Google form that allow you to reach out to me, and I'm happy to work with anybody, any groups, any individuals, any departments who who just, you know, want to explore what they need to do in their department in order to provide that survey put up here at the end. So, Marcy mentioned my bio online and there's another part that I really love about my prior mentor and I remember early on in my graduate career. We were talking about research and and some research projects we're getting ready a lot launch and he looked over his glasses and just said, Mike we just need to be fair to science and the sort of this mantra, we have to be fair to science. And that stuck with me my entire career and I frequently returned to that principle of, you know, science is a process but we have to respect that process we have to be fair to the process because ultimately our goal is to understand fundamental laws of nature. And we can only do that if we're fair to nature and for fair to science. And so that has stuck with me and it's a principle I think that also is embedded and underpins the open science movement. And I just want to give a quick shout out to the Center for open science, Brian Nasik and all of his leadership that he has has put into this over the years to really bring this this focus. I think to the center point of academia and to help those of us like myself really move this process forward, without that, that leadership. We would not be in the place we are right now where we can start to talk about policies, the policies that allow us to codify what it means to be fair to science. Okay, and not only that fair to our community partners are funders are taxpayers that make all this work possible. So, I hope to move everybody from that point of I'm an advocate for open science and I want this to happen to now I'm going to do something about that. And so we're going to maybe hopefully get you to that point of being an activist by the end here. Okay, that's my opening here here that's awesome. Thank you so much appreciate that one of the mantras, we have it at the Center for open sciences, make it possible for open science practices, make it normative, so that it is appearing everywhere and the lawyer peers, and then implemented into policy so that's exactly, you know, as we move along that's, that's very true. And one of the things getting to the promotion and tenure process that is so critical about that is that if you don't like align the incentive structures for the promotion and tenure process, you're never going to see it widely adopted. So, on the structure for today. We have generated through a brainstorming process, a series of questions that we had about the journey that the University of Maryland psychology department went on in developing new guidelines, promoting open science and their promotion and tenure. And so we're going to take the liberty of answering the questions as we move forward. But should you have questions that we don't address. Please, please raise them. So without further ado, we'll move right forward. The first question that we have or was hoping to get some, some input from you on was, what gaps did you notice in the PNT process before you made your changes, and what motivated you to create and adopt these new guidelines. Well, when I started this process, Marcy, there was a, we were long overdue for revision of our criteria, our criteria were last revised in 2006. And probably they reflected the historical perspective even before that right so I'm guessing that the revision in 2006 really was very similar to what they look like prior to 2006 so they were long overview and we are sort of at this point in time where they, we had to by university guidelines, revise them. So, the biggest gap was is that they were embarrassing, embarrassingly incomplete and outdated. And by that, I mean, they really relied on a lot of sort of historical views of what it means to do evaluate research and when we talked about evaluating research, the question oftentimes is are we evaluating the reputation of someone, or are we evaluating the substance of one's work. And what I think most of us will probably realize is that a lot of the metrics that we kind of rely on, really go back to that notion of reputation, right, and then we have to ask what drives a reputation and oftentimes it's things that are completely unrelated to the science one is doing. In fact, completely out of the control of the, of the scientist, what sort of reputation you have now of course you can get good you know doing good science bad reputation for doing bad science. But my guess is that there's a pretty bad correlation between the science and the reputation. And in fact, you can get bad reputation quite easily, as we now know. So, yeah. So that said, revising our professional or promotion tenure criteria was was really a priority of mine when I came on as chair in what was it. In 2017, it was one of the things that I spoke with my department about when I was quote unquote interviewing for the chair position. And much of what I was focused on at that time was issue surrounding responsible conduct of research and reproducibility, because you know, where who are the gatekeepers of this. And notably, there's some gatekeeping that happens at the journals at the review stage but we know that those are really imperfect gatekeepers. And we also know that the reputation factors can drive whether someone gets published or not, depending on you know who you are what your name is where you got your degree from. And so, you know, the more I got into this, I started actually gravitating towards sort of a more moral perspective on why we need to do open science, such as, you know, as a public institution here at the University of Maryland, or even just, you know, an institution of higher education or mandate really is to serve the public, and specifically, we do our work for the betterment of society. And it's really just difficult to see how our work benefits anybody other than ourselves, if it's not made maximally accessible. So if our work is locked behind a paywall, or our data is under lock and key, it limits its impact, by definition, if something cannot be viewed by others, it's going to have limited impact. If it's not accessible by the community, it's going to have limited impact. And so the goal here is really, what can we do to make that impact broader, make our accessible, make our work much much more accessible to the public. So that we can have a bigger impact on science. And we can think about this not just as impact of science, but sort of a more general principle, which I call impact as access. So, access is a necessary requirement for impact. You can have impact in many different ways. And it could be through scholarship. And it could be also through who you serve, right? What communities is your work geared towards and serving? How do we serve our community as scientists and administrators? Now, what do we do in the life course of our work to make our community of scientists more inclusive? And so those are all issues about accessibility, accessibility of research and accessibility of people to the research enterprise. So this idea of impact as access, I think underpins almost everything that we've done in our promotion tenure policy. And I think allows it to have a much broader impact as it's realized in terms of faculty work products and research. Yeah, I love the discussion of framing it not only practically with the impact and accessibility, but also from a morality frame of this is the right thing to do for science. Yes, definitely. So then the next question that follows, we've kind of framed the setting and a little bit of the problems that were experienced. Why did you feel that these guidelines were important? Why were they the solution that drew you to this quest for greater impact and accessibility? Well, there are many reasons, really. First, I really thought it was important that our faculty be evaluate on the substance of their work and not just looking at indicators that boil down to reputations or reputational criteria. I'm not going to go into all the research on this, but if you look at the work on bibliometric indices, there are some serious questions about their validity. And I think some research that would imply that they're actually either completely unrelated to the quality of the science or in some cases negatively related to the quality of the science depending on what aspects that you're looking at. And my guess is that those indices play an outsized role in the evaluation of scientists. Now, if you're an industrial organizational psychologist or in human resources, you know that it's not actually not so cool to use indices or criteria for evaluating employees that could create what's known as adverse impact. You know, impacting one subgroup more so than another subgroup. But there's some evidence that bibliometric indices hold the promise to actually have adverse impact on employees. And so, you know, there's some potentially I've never heard the legal arguments raised about these. But when I started looking at this literature, I thought, oh, God, you know, if the lawyers saw that these criteria that are in our promotion and tenure documents have been shown to have manifested biases in, you know, males versus females or racial disparities in, you know, how those criteria manifest, they would probably want us to take a very serious look at the inclusion of them in our documents or in our criteria. So, you know, returning to this idea of reputational criteria, really the goal was to move people away from those things that the candidates don't have control over to those things that the candidates do have control over. So, you know, you can't pick your reviewers for journal articles, right? And oftentimes it's the luck of the draw. Did you get people who are going to be sympathetic to your particular theory or viewpoint or open to new ways of thinking? Those are things that can determine whether or not a paper is accepted in your preferred journal or not. The other part of this is that, you know, if you're doing work on problem X, where do you send that? Well, you can try for the maximal impact factor journal of recognizing that it'll be a little bit out of place and probably won't be accepted. And so you kind of clog up the arteries of, you know, that journal, or you can send it to a journal that might be more field specific, but would target the right audience. And so we wanted to give people the power to think about where is the most appropriate place to publish my work. Without them really sort of worrying about in the back of their mind is, is this going to count the same as X? So if you're doing work that is important to particular communities, then it makes sense to be publishing in outlets where those communities are, you know, reading that work, right, or who are sort of monitoring those those outlets. A few, I guess, the other things that I don't want to sort of just kind of drone on here, but we really wanted to make sure that faculty felt empowered to do science the way they felt like they needed to do science. And this is both the questions they ask, the populations they're addressing, but also in particular for early career researchers to be able to carry out the science in a way they felt maximized the integrity of the science, right. So people who were interested in or who really felt that transparency, openness and accessibility were cornerstones of why they're doing their work. We wanted to make sure that they felt that they can continue to hold true to their own personal values and still do their work and have that work counted. Now, of course, we all know open science, working with underserved populations or hard to reach populations can come at a cost. It takes time and effort to do those things. And the faculty need to know that they're supported and they need to know that that work is valued and viewed as important. And so when we set out to do these things, we were really trying to, in essence, add a layer of protection and reassurance to our younger faculty, many of whom are coming in with this idea that, you know, I'm doing science because I want to have an impact on my community or I'm doing science because I really feel like, you know, I can have an impact on the scientific endeavor. And they have these sort of worldviews about how that ought to be done. And then they get into the academy and we say, whoa, whoa, whoa, whoa. No, that's great. Save that for when you have tenure. Right now, just pump out a bunch of articles as many as you can. And don't worry about those things because it'll be detrimental to your career. You know, as a young faculty, even as an established faculty member, that's demoralizing to say, don't do it the way you want to do it because it won't be valued. Do it this other way, you know, and then knowing full well that that's not the best way to do it, feeling like you've been cornered. And it's no wonder people want to get out of academia when they feel like they don't have the power or the agency to do work the way they want to do it. So those were the sort of key components of why I feel like they're important and why we really pursued this in my department. Fantastic. It sounds like really opening up and giving some freedom to the faculty to pursue their best choice. Yeah, we hope so. Pausing right here. Just wanted to see if there were any comments, thoughts, questions from the audience before we continue to move forward. Feel free to chime in if you have any experiences or thoughts that you'd like to share. So the next question. Oh, Sandra has a raised hand. Feel free. Sandra, did you want to unmute and speak up? She may need permission. I don't see you. Oh, is it Sandra? See that she needed permission like a loud talk. Yes, can you hear me now? Yes, we do. We do. Thank you. I didn't actually mean to raise my head. I just I was trying to type a chat message, but it said chat was disabled, but then I saw the Q&A. So that's where I can type. So now I know. I do have a question because you mentioned how people may not want to publish in the highest impacts, but not so suitable journal, but maybe in a better suitable journal that may have a lower impact factor. How do you end up evaluating that? Well, you know, my attitude is, is that you evaluate the substance of the research. So I mean, if what we're going to do is say, well, I'm going to evaluate you based on your impact factor. That's, you know, as a someone who's supposed to be evaluating the research, I'm sort of pushing that off on some assumed process that went on at the journal level of which could have had all sorts of things going on that determined whether or not that paper was accepted for publication. So my view is, is that you focus on the substance of the research, not on the outlet in which it is published. I see. Okay. Yeah, it makes sense. Thank you. So another question that has come in that raises raises a good point is what about what are what were the responses from faculty who had made impact through these publications. So I'm assuming this is from the previous status quo of researchers that were finding success within establishment journal articles, correct me if I'm wrong, but you know what kind of reaction was there among those faculty. Yeah, I can maybe touch on this a little bit later when we talk about we have some other questions coming up here about the pros and cons of the obstacles and so the bottom line is by the time we finished this process, everybody was on board. We had unanimous agreement over the criteria, the people who I thought might be a little bit more resistant to change at bought in and it's a process. So it's not like, you know, people looked at this and said, okay, I agree with these criteria. Let's move on. We actually took a good amount of time. I want to say four to five years from the inception to the point in which we have these done and you know, the question you raise here is, you know, the definition of impact. So are we measuring impact by citations. Well for measuring impact by citations that really put citations on the left hand side of the regression equation, as if that's the thing we're trying to predict. But I would argue that the citations part and should be on if we're going to use it should be on the predictor side of the equation. It's an indicator of something, not the criterion to be maximized. And I think that's a key thing to really keep in mind that we don't do science to maximize our citations. If we want to agree if we believe that that's an indicator of impact, that's fine. Of course, then we need to look at the research and say, well, is it a measure of impact. There's some questions there about whether or not it is, but certainly some some articles are highly cited some art. And there's a whole literature on that. So I spent a fair amount of time talking about what we mean by impact, why we should be thinking about citations in a different way. And drawing on the data, the literature to show that, well, hey, look, you know, if we're going to think about citations like this, we'd be prepared for the fact that there are gender biases, there are racial disparities. And these are not good. These are not good properties of a predictor variable if we're using it to evaluate people. Have other forms, other measurements of impact. Yeah, the broadening. Another question on real quick, we can go to this one before moving forward. I think this is a great question about portability faculty that if they weren't staying at the University of Maryland, how would they, how would their UMD measurements be applicable at another institution. And where their concerns expressed about that.  I am a big proponent though of preparing your career in a way which makes you mobile. You know, I think everybody from staff on up to faculty should really be prepared to make themselves mobile. But, you know, the other thing here I think that is really relevant to think about is there will come a day. And I think probably now we're looking at this in terms of, you know, a few years where these types of policy changes will be required by virtue of federal policy. And so, you know, I think the issue of portability here is probably going to go away. This is just a guess. But you know that this is a legitimate concern that individual people will just have to grapple with. And, you know, ultimately, you have to make your case that I'm doing my work in a particular way that holds true to my values, the values of my institutions, the value of my science, my discipline. You know, when we sign up to be academics, we don't do it because we want to maximize our citations. At least, I don't think so. We want to do it because we want to do good science. We want to impart some good on society. Right, yeah, right. Exactly. And I see we have one more question there about leadership. Yeah, we may want to hold, David Moore, this is a great question. We have some focus coming up on forces supporting and resisting the implementation of the guidelines that this may play into. So maybe if you don't mind, we'll pause on that question and revisit it here in a moment. That sounds good. So we've kind of talked a little bit about the problem, you know, with broadly with the incentive structures and, you know, what motivated you to go into these guidelines and why you felt they were important. But did want to spend a little bit time into the specifics on the guidelines and how they supported open science and the behaviors that they were designed to encourage just to give more need on the bones of what the guidelines were. So if you would mind speaking a little bit about that. Yeah, yeah, and I don't want to emphasize the word support here because I think that's the right term. I'll come back to this later when I give you some tips. We avoided mandates. I did not want to go down the road of saying there is one way and only one way that you must do your science but rather what I wanted to do was make it possible. And and to reward those people who were going to do it by recognizing the workload that comes with doing open science. So just out of the gate. I was opposed and no one actually proposed. Oh, probably not surprised there but no one proposed a mandate on on on open science. So we didn't want people to really feel trapped or in a corner but rather we just wanted to support those people. So the question here I think is really what's the swath of behaviors we wanted to support and so this isn't just about did you publish an open access journal. Or there's some problems with the open access models and ABC charges, some things that are, I think, publishers are going to end up probably manipulating for for economic gain. So it's not just about making your scholarship accessible, but really making the underlying work products accessible. So for those of you who who most of you probably are in tuned with what's going on with the National Academies Aligning Incentive Initiative and Helios. If not, you can go to, I think it's heliosopen.org and you can, you can look at that whole initiative and what's going on there. I was working on some of these these guidelines. The National Academies was organizing an aligning incentives workshop and I had the pleasure of working with a few people early on in this. And early on in that process they were developing some template models for evaluative products, things that are part of the open science work product model that lies behind all the research but typically isn't made public. So things like the data, the underlying data, the metadata, analysis code, statistical procedures with the work of the Center for Open Science, you know, pre registration plans, registered reports. You know, all these things we wanted to be able to document and to be able to capture as part of the work product that people were were doing right recognizing that a paper is just the words on a sheet. But there is all the work that goes into preparing that and that we wanted to be able to capture the swath of work and recognize those work products to the extent possible through our evaluative process. And so in our, both in our annual review but also in our promotion tenure documents we do now actually ask people to provide pretty extensive annotations of a set of publications so that we can look at, you know, data sets. We can look at accessibility of data. We can look at accessibility of analysis code codes. At first they might have engaged in to ensure reproducibility in one form or another, and really giving people an opportunity to say more about the research that goes into a CV line than what you get out of a CV line. Right, so it's a pretty nice little extension on a CV that just allows people to advocate for themselves but just describe the scholarship in a little bit more detail along the set of dimensions that capture open science but other things too. Actually I lost track of the question I was trying to answer. No, you were perfectly in line, you were perfectly in line talking about the behaviors that guidelines were trying to encourage. Yeah, that might be most of them, you know, making the scholarship accessible. You know, here's a place where I can talk about leadership, campus leadership. I'm fortunate to work at a university where my leaders recognize and value the importance of open science at the time we started working on this. We had a new VPR who had just come over from NIST and it turns out that she was a proponent of open science. And my dean at the time was a psychologist by training and fully aware of all the stuff that was going on in psychology, the reproducibility crisis, the need for really thinking about open science. And he gave me the latitude to work on this project and the encouragement. My associate dean put me in touch with the provost's office. Turns out the provost's office was like, you know what, this is kind of an important issue. We've been thinking about this from the perspective of responsible conductive research. And so it turned out that campus leadership was mostly on board. Actually, I shouldn't say mostly, they recognized the importance and they endorsed it. And they allowed us to move forward. Maryland may not be the same as other institutions because here at Maryland, units determine what the guidelines are. And the upper level administration sort of looks to the units to tell them what we should be evaluating in our field. And so we have a little bit of latitude due to that too. So, you know, these issues were already bubbling up on campus at that time. Very good. All right, moving right along, let's look here at some of the next questions we have. But just a broad question, you've already factored a lot of this in with the leadership. But what factors supported the adoption of the guidelines and which factors opposed them? You know, so this is an interesting question because, you know, when I first started getting into this process, my first approach was to take our existing guidelines and start to tinker with them. And, you know, trying to infuse some open science process. And mostly I got some looks at me like, you know, what are we doing? Like, what's, you know, why would the, they're like, okay, yeah, we could probably do this. But at the end of the day, what I realized early on in that process was that we weren't really going to make progress if we took our current standard and try to change it, that really we needed a fresh start. And so, through a variety of external factors, including this thing called a pandemic, I was able to put that on hold and restart at some time later. And I restarted it with a completely different approach recognizing that we weren't going to be able to simply tinker with the existing criteria that really we need to do a complete rethinking of what our criteria were. Otherwise, they were just going to be really bad and incomplete, but have the word open science. And that just wasn't enough for me. We really need to have clear guidelines that provided clear guidance to our faculty and clear guidance to our external reviewers. So, you know, there were, by the time we got to the point where we were voting on it, people had bought in. And I kind of realized this when we were in one of our faculty meetings and we were literally going line by line. What do you think of this sentence? What do you think of that sentence? And the people who were jumping into the conversation were the people who I thought, if anybody is going to really object loudly, it's going to be, you know, some of these individuals. But it wasn't like that. By the time we got to that point, I think people had seen and recognized the importance of what we're doing and had bought in and saw the value. And so kudos to my faculty, really, quite frankly, for being so collaborative on it. So, you know, I think one of the questions that came up in this process, you know, it's a factor of post. And this is sort of reading the tea leaves is, you know, the read of some of my faculty, my read of some of my faculty was, well, what are other people doing? So constant questions were, are we allowed to? What does faculty affairs say about this? What are they saying in the Provost's office? Are we allowed to do these things? And what about other people in the field? Are we going to be the only ones?  Someone's got to be first and I would rather not be last. Okay. And, you know, for me, it was a matter of these changers are coming and we need to be out front of that wave, or we can be trying to, you know, save ourselves at the end. And I felt that it was really important for us to be deliberative where we could take some time to do this right. Think through the process and make things happen now as opposed to waiting until everybody else did it. Or no one does it, even worse, right? And we find ourselves not having made any progress on this. But, you know, one of those sort of undertones was, you know, what are other people doing? You know, is this possible? Can we do it? And ultimately, I came back to that something Marcy, you said early on is, you know, we do it because it's right, the right thing to do. Right? Sometimes you say, you know what? This is the right thing to do. And we should do it for that reason and that reason alone. And we return to that sort of general theme time and time again throughout our discussions. What is the right thing to do? The right thing to do is not to keep our stuff locked up behind the firewall or to keep our work siloed and private. The right thing to do is to share our work as broadly as possible. And I'll just give you a little anecdote that I learned from my librarians. And that is, you know, when you have a publishing deal, you set up with a journal or with a journal company like Elsevier or whatever. There's usually wording in there that the research, like the articles or the products can be shared publicly. But the definition of shared publicly is anybody is allowed to come into the library at the University of Maryland to access that material, which is just the most ridiculous form of public access that you can even imagine, because it's not public access. It's public access for those who feel comfortable coming on to the college campus, which means you also have to have some transportation to come to the university.  And when you start thinking about it in terms of access to real, you know, real people's access to that material, you start to realize just how messed up the system is and formative access versus real access. Yeah, exactly. And yeah, and if you really endorse these views of equity, of inclusion, then there's really no other way to think about it than really breaking down those walls that have prevented us from sharing our material. That resonated with people. And, you know, I have good faculty in my department who's, you know, they know what the right thing to do is, and they were able to do it, so. Yeah, that does resonate. It hits home and it's a strong, powerful message.  You know, I think that's a strong motivating factor. So that's wonderful. And I was really inspired to hear, even down to the detailed line about feedback from the faculty as you were going, they were getting into the specifics to make sure that they were comfortable and fully understood. So, you know, I'm a biased party, but I'm sold and I'm bought in, you've convinced me, that's for sure. But next kind of transition. I'm kind of interesting. You made all of these arguments. Now they've slowly been adopted and implemented. What has been the reaction since, you know, as they went into effect? What have you seen? Well, first of all, let me just say that they've only been in effect for a few months. And honestly, there's a grace period here. I mean, people are allowed to be grandfathered in to the old policies. So we have those legacy policies that are still there. And if someone doesn't feel comfortable going up under the new policies, then that's fine. But eventually people will be going through these new policies as they're specified. So that the grandfathering sort of protects people who feel like they were, they aren't consistent with the way they've been doing science for the last five, seven years or whatever. So the reaction. So a few things. One is one of my senior colleagues commented to me sort of offhand that, oh, so you mean what we just adopted is progressive. Yeah, yeah, like, this is the way we should be doing it. And this is what people really aspire to do. But yeah, we're, you know, we just adopted what I would call a pretty aggressive progressive policy and his response was cool. Another one. Yeah, and I've had one or, you know, a small number of other faculty who have reached out subsequent to our adoption of policies and just thank me. For really pushing it and saying that it was just, in their view, a really important step for our department but also for the field. And so, you know, I think there's some acknowledgement that this is the way we should be doing it and probably the way we should have been doing it for a long time. And thanks me for making it happen. But again, it's, you know, it's, it's the community that makes it happen, not one person. So we're still, you know, feeling our way through these things. So I don't have a lot of reaction but, you know, generally positive, you know, there were no, nobody objected the policies. It was a purely anonymous vote, and no one voted to oppose the policies. So that says something, I guess. Says a lot. Yeah. Returning to some of the audience questions, this kind of follows. What about the practices now that you're building support and hopefully to see that mushroom? What about other departments in the institution? Have you seen this pop up anywhere else? Not as much as I was hoping. I know that there are other advocates on campus who are pushing for change. And, you know, and it's, you know, there's a lot of variation across fields, unfortunately, and sort of where people stand, or at least their knowledge of what open science is and what it means. And even open access, I think is commonly misunderstood as you got to publish in an open access journal that you pay an open access fee for. Well, no, there's other forms of open access. And, you know, part of this is really about educating people and making sure people are making sure people are aware. So unfortunately, you know, right now, there hasn't been as much uptake on this at our campus level, not as much as I hope. We also had some untimely turnover in our administration that sort of now kind of go back to the well and start, you know, talking with new administrators about, you know, what we're doing and whatnot. I will say that our campus just happens to be going through a campus-wide process in the next couple of years for re-envisioning tenure and promotion criteria. I don't know what that's going to look like for sure. I suspect there will be elements of open science. I don't know if they'll be fully endorsed like we are in our department, but I think that will be part of it. But it is a priority on our campus generally. How it will manifest, I'm not sure. Another great question too about the point of promoting diversity, equity and inclusion, you know, with these guidelines, you know, kind of looking at the evidence base for that. Have you seen anything or read anything about adopting these processes that would improve DEI in the academy? Do you see a conceivable pathway that you would envision? You've already spoken to that a little bit. Yeah, so this is really an important question. When we talk about my, I'll come to my tips here at the end about, you know, how to sort of think about framing and really broadening the conception of what our revision was about was, you know, not thinking about it as open science per se, but really thinking about it from the perspective of inclusion, diversity and equity. And so many of the things that we ended up putting into our policy, you know, wasn't just about making our science more open and transparent. It was about broadening our definition of inclusion. Open science is one piece of that. But our criteria were very intentionally designed to try to eliminate barriers, to eliminate biases that may pop up and to start respecting to a much greater degree that there are, you know, a lot of young researchers who are coming into the field who they want to do good and they want to do good in particular communities or with particular communities. And so we made a point of really building our criteria around making that possible, making it rewarding and or rewarded and making it recognized. And so it's all part of the policy. So that, you know, in an ideal world, right, those are policies that are comprehensively addresses that set of issues, open science being one thing. But the other part of it being really thinking about who is it that we study, who's invited into the research endeavor. What are the challenges that individual people have in the conduct of their work as they wish to do, and then really writing the criteria in a way that enables them to do that, and to recognize those challenges and barriers. Yeah, so moving forward, going through C on the the next question before we we run low on time here. But what about tips you might have for people or departments who are looking to move in this direction. So that's the key question. Yeah, okay, so I have seven tips. Alright, so bear with me. A tip number one I'll elaborate on these just a little bit tip number one is identify your leverage points I'll come back to that tip number two is engage your higher administrators were possible. Number three is think broadly about change in your P&T criteria, again, not just about open science. Tip number four, which really should be number one, which is identify your core values because everything should be framed around that set of core values that may differ from one unit to another unit may differ from one discipline to another discipline. But you really I think if you can frame up those core values, it's really really important. Tip number five is dig up the data find the data. Do the work to find the consensus documents that you can use to provide your arguments to your faculty. Tip number six is be prepared to compromise. And then tip number seven is avoid mandates. So let me just return to number one, identify your leverage points. So you really need to figure out what's possible in your department or at your universities, different universities work different ways. And so what we were able to do here may not actually be very easy to do elsewhere. We controlled our own destiny and I was able to work within my faculty to make it happen. And so my leverage points were my faculty and trying to figure out how I can herd them into the right direction to make this work. So you really need to know where at the university, who you need to leverage to make it happen and also having advocates in your corner. So when I was able to go to my dean or my associate dean or the vice provost and say, hey, we're doing this. Is there a problem and they could say, no, this is great. We look to the units to do this. Move ahead. I could go back to my unit say we have approval to do this. I've checked it three times and we can do these things. Number two related to that is the engage or hire administrators. For those of you who aren't familiar with Helios. Look it up a lot of institutions have sort of signed on to the Helios project. So there's a lot of stuff going on at the higher administrator level. One thing that I was really struck by in some of these groups is that actually literally floored that being some provosts are looking to the faculty to do stuff. And simultaneously, the faculty are looking for guidance from the higher administrators. And so we're sort of at this impasse of who's going to tell us what to do first. Who goes first? Yeah, right. By the way, there's also another project going on through the Council of Graduate Studies and the Association for Education Research, or the AERA. So CGS and AERA is also has another project going on, but many of the deans and provosts and campus leadership are involved in these efforts. And so there's things going on out there that many of you may or may not have any knowledge of. My third point was think broadly about change. I brought this up earlier. You know, what are we trying to do with with open science? We're really trying to build a more inclusive science and more accessible science and the principles of accessibility and inclusion can manifest in a whole bunch of different ways. And so I encourage you to think broadly about what sort of changes you can make. The more broad you think, the more people you can bring into the fold, right? People want to see accessibility and inclusion address in a way that is meaningful to them. Open science may not be the part that's meaningful to them. That's okay. But there may be other things that you could do that would be very meaningful. And so you can bring more people into the fold, I think, by really broadening our definition. I won't say much more about core values other than look to your university's mission statement and see what they say there. If you have a departmental mission statement, even better, use data. There's a lot of work out there that's available on shoddy indicators of research evaluation. Have your faculty read the Dora statement of SFDora.org. I did this early on. I bribed them with ice cream. Get them to read the statement so that they see that there is a community of scientists out there that are saying, don't use these summary journal indicators so much, right? Back off those and think about reading the substance. Compromise is important. You know, you're not going to be able to fix everything, but if you can move the needle, that's probably good. And then I just said earlier, avoid mandates. So engage people, energize them, make it so it's possible and it's rewarded and cross your fingers that it actually happens. Have trust in your people and inspire them. Yeah, I think that's really it. Yeah. All right, we have a couple more minutes left, but definitely this is a fun question I wanted to get into as well. But what's the one thing you wish you knew before you launched this endeavor? If you had to go back in time and tell yourself. I'll share two things. These contradict each other, so bear with me. First, I wish I had a better sense of people's understanding of open science and why it is necessary. So I, you know, kind of overestimated how much people were paying attention to what was going on in the field. I happened to be in my little bubble, reading a lot of work coming out of, you know, Brian Nasik's lab and other people's lab and and thinking, oh, my God, we got to do something about this. But I learned that actually that was me, but not a lot of other people. And so there was a period of time when I felt, OK, I got to go back and I have to sort of start sharing out some things. And so I did the slow bleed of information sharing to get people sort of acquiesce to the state of affairs and why we were doing it. And then the other thing sort of contradicting this perspective is just how OK people were with change and prepared in a number of people wanting it, but never spoke of it. And so so I wish I knew that there were a lot more people who were ready to engage in these activities and welcome them. So much to my delight, I had a number of colleagues who were very explicit in their support early on and recognize the need to change. Fantastic, fantastic. We had another and I did not realize this. This is interesting. A great question from an attendee member that says, in a Canadian court case, student evaluations of teaching were prohibited from P&T decisions at one university because of inherent bias. Sex, age, appearance, etc. Has there been any history of public complaints that existing the existing P&T policies using things like the citation metrics that we discussed were also biased and unjust? Not that I know of. I didn't find anything on this. But I will say that in my reading of this literature, to me, it's it's a it's an easy target because there is enough research now that shows that there are inherent biases. So I haven't heard of anything. I wouldn't be surprised if we see something sooner or later coming. Great. One final question. And I know we are down to our final three minutes. But any quick summaries or input you would have for things that other departments, things departments can do to support open science. Yeah. Okay. So what are the things can you do to support open science? You know, the P&T policies are really, really important. But I do think that there has to be a very consistent message and consistent set of actions that a department does to support these activities. Can't just all be about mandates or promotion and tenure policies.  I include or we include mention of open science in our job applications or our job ads so that people know that it's coming. That this is something that we value and it is important. I created a funding initiative and put a couple hundred thousand dollars behind it in our own department to support what I call broadening participation. So it's an initiative that we started here to really broaden participation in science that includes open science. It's not solely focused on open science, but it also includes things like open education resources. So we provide funding for faculty to create free and accessible education resources to create open science pipelines to engage in other activities that might broaden representation in science in various ways. So that was really important. I also include in startup a funding line for open access publishing. So that's a clear sync signal both with, you know, here's what we value, but also with the money to back it up to say, we want you to make your work accessible and find it important. So that's required in all of my startup packages in my own department. Sounds fantastic. We are running. We are at 259. I did want to take a moment before we leave to provide a sign up for. For those looking to follow up. From this discussion, if you're interested in collaborating, looking forward to how you may be in greater contact. Do you like to say something about that, Mike? Yeah, so about this, so I just want everyone to know. Yeah, so no, I'm happy to work with anybody in the departments and any individual who wants to institute change. So if you sign up on that form, give me your email, and I'll try to organize some working groups for those of you who are interested in doing this and maybe sort of help guide you through this process with my lessons learned. I'm happy to work with whoever signs up whatever your unit is or whatever your particular situation is. So go ahead, sign in, log in, whatever, enter your email and tell me a little bit about what you want. Thank you so much. The link is in the chat. Michael, thank you. You have been a real thoughtful, thoughtful advocate for this.  So just want to recognize you and say thank you and appreciate everything that you're doing to do this the right way. So thank you so much and thank you for your time today. You're welcome and thank you. Yeah, and thank you everyone for joining us. We're at three o'clock on the dot. So we will say good afternoon or goodbye and we'll be in touch soon. Thank you.


You look at a picture of a celebrity. There's five fakers in that posse just trying to be like, yo, what's up? I'm here. No, you're not. He's here. You just rolled up with him. This is a definitive guide of how to ball like a baller. I'm Andy Kess. I'm a doer. I'm a VIP host and a concierge to the stars, celebrities and athletes. I'm the guy you need to know. My clients range from college kids to corporate executives, two thirds of the NFL. I have a ton of basketball players, Saudi princes. If you're not a basketball player or a multi-millionaire, you can still be a baller through a false. You need money to be a baller, false.  First and foremost, plan ahead. People come to Vegas. They're like, okay, it's 10 o'clock and I want to go out to a club. And it's like, what are you f***ing thinking? It's the last minute.  Now you gotta pay the piper and you're gonna pay double.  If you're very gracious and you're very humble to people, you can get whatever you want. For example, if you're talking nice to the staff, they'll be like, oh, you know what? They're so nice, we'll make the exception. Yeah, we'll give them a big package. So what are some crazy things that I've been asked to do? One of our big clients wanted Taco Bell delivered at three o'clock in the morning into the club. So it's very difficult to get food into a club because they don't allow it. But when you're ordering 10 bottles of Dom at a time, you get a little bit of leeway. It was hilarious. Girls had mariachi hats. Yes, we made it happen. True or false, your girl-to-guy ratio needs to be on point, which means more girls than guys. Depends on where you are. If you're in LA, absolutely.  Big mistake. But New York or Vegas, not as strict as long as you're spending the money, they'll make sure their girls come to you. Do you want to be a baller? Dress like once. True or false, a pro basketball player can wear whatever he wants to the club. That, my friends, is false. Clubs have dress codes, and even though you might've won the NBA title, you still gotta look the part. Male ballers usually wear suits because it shows class, sophistication, great pocket square.  The hottest restaurant and the hottest nightclub varies from day to day, depending on what's going on. So be smart and use this definitive guide.


From Miami Beach, Florida, it's theCUBE. Covering VeeamON 2019, brought to you by Veeam. Welcome back to sunny Miami, everybody. You're watching theCUBE, the leader in live tech coverage. We'd like to go out to the events, extract the signal from the noise, and we're here at VeeamON 2019. I'm Dave Vellante with my co-host. This is day two. Peter Burris and I have been covering wall-to-wall coverage with theCUBE. Folks from NetApp are here. Jeff McCullough, who's the vice president of America's partner sales for NetApp, and our good friend Keith Norby, who runs alliances for NetApp. Guys, great to see you. Thanks for coming on. Thanks for having us. So Keith, let's start with you. Veeam has been a partner of yours for a while now. You guys go to market together. You have always been very partner-friendly, particularly when it comes to data protection, but what's the state of the partnership today? Yeah, this is something that we'd looked at a couple years ago and got into a very much more strategic relationship with Veeam. Over a year ago, kind of worked through a lot of ways to reconnect and establish a better together, and this is something that we think is a strategic opportunity is kind of backed by a lot of the data you see at this show. Talking about organizations are going to change, roughly 60% of the organizations are going to change their platform because of cost-complexity reasons, and together we've been working with Veeam to figure out how to deliver data protection for a data fabric, and IDC validates that in a number of ways that we can unpack here on the show or in the conversations with customers, and we've gotten great reaction to it. And Jeff, you lead America's partner sales, so North America, South America, the Hulket, and Kibbutl, talk more about your role. Sure, well, my responsibility is net app partners. I am successful when our partners are successful, so everything I do is all around putting our partners in the position of executing, being successful within that app brand, certainly being profitable, having profitable, strong businesses, and growing, growing and taking market share and helping them expand and grow their respective businesses. Well, you guys have dramatically increased the percentage of your sales that come through the channel over the last 10, 12 years, I mean, pretty significantly, and that's a fundamental part of your strategy, stator at this executive level, so. Yeah, for sure, channel is core to what we do. When we go to market with developing our products or executing our marketing plans, it's all around how do we go execute with partners, right? Whether it's the tools that partners need, the pricing, the programs to help them go engage in the market, the leads, demand generation, and we're at various stages in all these, but I think you'll see consistently from the partners that certainly will talk and talk about their NetApp businesses, we generally lead in profitability across our partner base, and we absolutely lead in terms of total profitability when you include things like services attached and how we go and execute on a partner-delivered services strategy. So, I always say NetApp is, it's not just a product category, it's a whole economy for our channel, and it puts people to work, it allows them to expand and grow their teams, and it's a critical part of many, many of the partners that are here today at Veeam and certainly at Veeam-On, and certainly in the marketplace. And you're partner-friendly in a sense that you don't have a huge services organization that's competing with your channel. I mean, that's got to be appealing for those guys. We put partner services in the forefront of everything we do. Keith, you talked about better together. What does that mean, just in terms of engineering integration, go to market? I mean, how did you sort of in the last two years get better together? What specific actions were you guys taking? I think you got to look at it first from kind of the customer in, the markets in, and you got to look at what's the dynamic that requires change, right? That sort of shapes what your PRD and your MRDs are to make a product. In this case, we've got platforms that have incredible snapshot technology, so to me, it really starts there with simplifying the way that you get the first copy of data, and then simply working with the strengths that Veeam has in their platforms, and making sure that we have great optionality between our replication and other snapshot technologies, their replication technologies, to be able to give a level of flexibility for this data fabric to come to life. No matter if you've got the traditional data center that's got these enterprise apps, like at SAP HANA or others, or you've built a next generation data center like on NetApp HCI, and you're building a scale out via more private cloud, or you've got the hyperscaler cloud with our cloud volumes. We have options on how we get data throughout the copy process of primary to secondary to cloud and tertiary data. So to us, it was about really making that as simple and as pre-wired as possible via the APIs, and then really making that easy for partners to go and grab on to to make it easy for someone to buy us, because you always want to build something that people want to buy. No one wants to be sold any of this stuff, and so building the right thing that people want to buy, the next step then with Jeff, and the reason why he's so critical to this is getting that ready for the partners to be able to have an easy process with their customers that frankly they love. People hate to be sold, they love to buy. Yeah, and let's talk about they love to buy. One of the challenges that the entire industry has as we move through the significant transformation is customers, user organizations, are themselves in the midst of huge transformations. Institutional transformations, technology transformations, relationship with their business transformation, mission transformation. Jeff, starting with this whole role that the channel has been playing and is going to play, how will the channel be an increasing source of value add in the deal? Yeah. How's that playing out to help these customers smooth their changes? Yeah, and I think, you know, I was just watching the news this morning, right? Target announced their earnings, and a big part of their earnings announcement was the improvement they made in customer interaction through digital platforms, right? The ability to order online, pick up in the store, or order online and have it delivered same day, right? And these are, and it's just one example, you can go down the list of customers that have really used transformation to change their business, right? And Chipotle, who's trans, they've transformed Burritos now, and a lot of their success has come through digital transformation platforms. So, you know, the evidence is overwhelming that digital transformation drives better results, and we've done a lot of study at this, right? We have lots of detail around customers that know how to use data, and you know, the basic fact is, one out of 10 customers is in a position to actually leverage data effectively, right? This is all of the research we've done along, you know, with partners, with other companies. The other nine need help, and this is where channel partners come in, and this is what I tell partners all the time, is this digital transformation wave is real, the results are real, and the customers need to move is real, and so they play a role, and can play a role in helping customers accelerate that digital transformation, and so our portfolio is all around accelerating customers and their ability to leverage data to transform their business, and partners through both of the portfolio that they sell, but then the partner-driven services that we promote and drive, you know, really stand out in the forefront of being able to help a customer execute these really tough strategies, and you know, the reason why customers love partners is partners bring choices, right? And you know, for us as vendors, we have to deal with the other side of that, which is partners have choices on who they sell, so we represent a portfolio that is forward-thinking, it aligns to where the market is going, it aligns to the tough problems that customers have, and it's, you know, and it's a position that allows partners to be profitable and make money helping customers transform and deliver their own success. It's got to be more than just partners create choices, and let me explain what I mean by that. It's increasingly, your typical CIO, medium-sized company, large-sized company, which is where we spend most of our time, is thinking in terms of what is going to bring me value today and also generate a stream of value for me in the future, so I need choice now, but options for the future that are relevant and meaningful, and so partners increasingly have to be part of that options equation, how are they going to create options for customers, and you know, one of the nice things about the relationship that you have with Veeam is that you are a partner to Veeam, and presumably you're going to help Veeam customers create additional types of options through this, expanding portfolio of value that you guys have, so talk about that dynamic, because it really requires an even greater dependency on that customer partner engagement, including the dependency that Veeam has on you guys. Yeah. Do you want to maybe start with just the Veeam partnership, partnership? Yeah, I think, you know, which we create the conditions with which I think a partner comes to life with what we've tried to do in the product, building solutions, and then trying to develop the go-to-market around the partners ability to go meet the market and what the market is asking for. In such, you know, the partners have natural services on the front side of the assessments, a bit like trying to help you plan your 401k, they help you like see what kind of data you don't even see, we have a wealth of partners that just have incredible skills there, and then as they take that through our solution, we do everything we can to make that process easy to match our technology to that design requirement, and then afterwards the partners always have these great capabilities for things like, you know, a one-call or a managed service to help take even more complexity off the table for people to just live with the ability to have data protected across all spectrums of where they have data live. So the partner equation is definitely getting more complicated, right, if you dial back, you know, half a decade, decade, you had guys who sold hardware boxes, they were box sellers, we love them, but they moved a lot of product, and they worked with you. Okay, now the cloud comes in, you guys are going, you know, software defined, so you can run your services in the cloud, you know, or you run it on-prem, you've got hybrid, so it's a complicated equation, much more so than it was in the past. So how are you seeing the partners evolve and transform, you know, beyond the sort of box-selling mentality? Of course, you know, VMware specialists, you get those guys, SAP, maybe Oracle, but it's even more than that now with cloud, isn't it?  If you think of kind of client server is the big first disruptive wave, virtualization, the second disruptive wave, and now cloud, just purely from a channel perspective, the third big one, and maybe the biggest, right? Because it is completely changing the dynamics and the economics of how partners operate. And, you know, and we've been looking at this for a long time, and certainly as we move our portfolio, as we transition our portfolio to be cloud-enabled and native to the cloud, it creates options. But, you know, the market is moving from, you know, deal-based revenue to reoccurring revenue, and what I see partners moving to is various degrees of reoccurring revenue strategies, whether they're setting up their own MSP business and they're opening up shop and they're doing data protection on demand, or they are doing managed services on-premise, and they're charging a customer, or they're buying out the infrastructure and charging a customer once a month, or they're selling services in the cloud. And what I think is also interesting, you can see the kind of the direction where the industry of the channel is going, is when you look at the acquisitions that partners are making, not only of each other, but of software development, right? IP, they're going out and buying software development because the long-term opportunity is not just selling the infrastructure, it's selling a solution, solving a big problem, right? Which could be this digital transformation opportunity, but it's more than just, sure, I can upgrade your servers. It's their digital transformation, right? I mean, that is their digital transformation. It is, you know, kind of cloud's not really a destination, right? Everybody thinks cloud's the destination, I got to get to it. You know, it's not a destination. It's a tool in the bag that your customer's going to use, and certainly a partner's going to leverage cloud to create a money stream, right? A business model that is sustainable and can grow, but it's super dynamically different than what we do, you know, what they're doing today.  I was just going to say, balance all that against, I think, where the volume, the mass of the volume is, even though the hyperscalers have a tremendous amount of growth, it is still VM-based, it is still kind of on-prem-based, and so there's still, in this two-year window of change, the vast majority of the opportunity is going to be on-prem, but you also have to factor in how you involve the cloud in that strategy as what Ratmir would call the second wave, right, of Veeam's strategy, and we're right in the heart of that. I mean, there isn't any greater strength than what we're doing as a company with NetApp than what we're doing with cloud, and it's just a natural way for us to extend, a partner's capability, a customer's ability to buy what you'd want to get from NetApp and VM together. Well, and what the hyperscalers have done is they've changed the way in which people consume technology. Oh, absolutely, and NetApp is a great case study of a company that's moving through that process from a product orientation to a services orientation, but Keith, I want to come back to this notion of how the NetApp relationship with Veeam creates new classes of options for Veeam customers as they try to think about data protection differently because, precisely because, as Dave said, you have expanded your portfolio, you are going to market with a different value proposition that you did a couple years ago. How is that playing out in your conversations with customers as they think about moving from a data protection that's focused on devices to a data protection that's focused on delivery of digital services? Yeah, well, isn't that a great topic to talk about? Where do you start with that? Organically, I think you look at the way people try to operate and deal with the operations of data protection. It really starts there because, cloud is really about IT operations. What we've done is really try to simplify that stack to get beyond it being one single endpoint of technology, so it's not just about how we take data sets from, say, a NetApp FAZ or a NetApp HCI and bring it through Veeam to another FAZ or E-Series and then off to the cloud. It's beyond just the basic technology. It's much more operational in its nature. So if you look at all the stuff they're talking about here with VOA and all the discovery elements that they're doing to help make it easier, one of the areas that IDC caught particularly in one of our benefit statements on taking complexity off the table is our ability to have auto discoverer VMs. It's ways that you can make much more autonomy and orchestration of operations kind of come to life as a way of you doing this technology together. That's only just one of the example points that we have on this better together with Veeam. Taking the heart of their core technology and where they're being pervasive in not just a VM-centric crowd, but also Hyper-V and some of the other things they talked about at the top of their rationalized stack and then bringing that down through the heart of our data fabric portfolio and saying, anyone point at which you're at, we're able to put these things together at the heart of the first step and we kind of map this customer journey and our presentation to the attendees here was this customer journey from the current form of complexities you have and moving that all the way through to snapshot integration, platform selection of which ones would make sense for what scenario, how we work through Veeam's data replication and management technologies, our data replication, our data fabric technologies to get from one end point to the other. And then ultimately, you got to be able to talk about the ability to restore or you really shouldn't be talking about backup. All right, we got a wrap, but I'm going to ask you guys each question, Jeff, from Trip Reports. So from your standpoint, you're talking sales momentum with partners. What are you going to tell your colleagues? And Keith, obviously the partnership with Veeam, what are you going to tell your colleagues when you get back home? Yeah, so for me, it's, you know, this is, we've talked about transformation. This, you know, I think our relationship with Veeam and the strategies that we're executing is all around transforming data protection, right? And it's really around this concept of simplification. And I think, as we were chatting before, you know, before we started taping the, you know, simple matters, right? Simplification or simple is really attractive feature. And, you know, our ability to simplify data protection for customers in partnership with Veeam, deliver a solution that's, you know, clearly world-class and, you know, NetApp bringing world-class technology to the table. It's a great partnership. It creates an opportunity for us to go and have conversations with customers that maybe never thought of NetApp before. And it's, you know, an opportunity for us to open a lot of doors. And certainly for me, what I care about, it's an opportunity for our partners to open a lot of doors. Right, yeah. Keith? I would just say, listen, we've worked from our joint CEOs together. So George and Ratmir, starting this, like, joint bond of alignment all the way down through product solutions, field geos, channels. We're going to have explosive growth together. You know, we're going to go address this market that is looking to change. We've got something we're bringing together and it is absolutely better together. Great, power players aligning at the top all the way down through the channel, through the partners, into the cloud, bringing you all the data here at theCUBE. Jeff and Keith, thanks very much for coming on theCUBE. Keep it right there, buddy. Peter Burris and I will be back with our next guest right after this short break. We're live from Miami at VeeamON 2019. We'll be right back.


The talk is about probabilistic degrees of symmetric Boolean functions and this is joint work with Shrikanth and Uthakrish. So, well, the class of Boolean functions that we are interested in are symmetric Boolean functions. So, just to start with the basics, the domain of the functions is the Boolean hypercube and using the hamming weight function. So, the hamming weight of any vector is the number of ones in the vector. We can partition the hypercube into layers where we have one layer per weight and symmetric functions are those which take exactly one value per layer. And if you consider polynomial representations for symmetric Boolean functions, then we get symmetric polynomials which are also multilinear. Multilinear in the sense all the monomials in the polynomial representation are square free in some sense. So, let us start with some examples. The majority function is sort of takes the value 1 when the input weight is at least 10 by 2 and 0 otherwise this is sort of the graph. So, black indicates 0 for us and white indicates 1 that is how it is. So, the degree of majority as a polynomial is going to be omega n. The r function is slightly different takes the value 0 at the all 0s input and 1 everywhere else and again as we know the degree is n. The mod function is sort of defined depending on the characteristic. So, we will choose a prime that is not equal to the characteristic of the field. So, if the characteristic is 0, we pick some prime and if it is a finite characteristic, positive characteristic then we choose a prime that is not equal to it. In which case the mod function gives us a graph like this and the degree is going to be again omega n. So, it takes value 1 when the input weight is 0 mod q and 0 otherwise. The threshold function is a more generalized definition which captures majority as well as r. So, take a threshold parameter t and takes the value 1 whenever the input weight is at least t and 0 otherwise and again the degree of threshold function is omega n. So, these are some examples of symmetric Boolean functions where the degree is omega n relatively high and what we would like to do is approximate them with low degree polynomials. So, that is where our notion of probabilistic polynomial or probabilistic degree comes into picture. So, start with a Boolean function and this is more or less how it looks. So, if you think of polynomials as the model for computing the functions then a natural complexity measure is a degree and if you think of the randomized version then what we get is the probabilistic polynomial which is the distribution of polynomials used to approximate the function up to some error and the corresponding complexity measure is what we call the probabilistic degree which will be the max degree of the polynomials in the support. So, more formally start with a Boolean function f and take an error parameter epsilon and epsilon error probabilistic polynomial for f is defined to be a random polynomial. So, it need not follow the uniform distribution it is some distribution of polynomials with finite support such that at any given input if you randomly pick a polynomial from the distribution the agreement with the function is with high probability. So, probability 1 minus epsilon in this case and the degree of the random polynomial, the probabilistic polynomial is defined to be the max degree of the polynomials in the support and the epsilon error probabilistic degree is defined to be quite naturally the minimum among the degrees of all the probabilistic polynomials that approximate this function. Let us go on to some history of yeah no no no we fix a field. So, since these are Boolean functions we are only concerned with the base field and we assumed that the characteristic is fixed throughout could be 0 or a positive characteristic, but we fix it correct yeah yeah right. So, this is different from sort of approximate agreement it is not 1 minus epsilon agreement on fraction of points it is at every point you agree with probability 1 minus epsilon. So, definition was given by Raspbore and the initial works of Raspbore and Smolensky led to lower bounds for Boolean circuits for the AC0 mod p class. Then the works of Tyrui and Begel, Reingold and Spiegelman led to probabilistic degree upper bounds for AC0. This in turn was used by Breverman to show that poly logarithmic independence holds AC0 with set as linear Nissan conjecture. And more recently the probabilistic degree of any symmetric Boolean function was shown to be Big O square root n by Allman-Williams when you fix the error for constant error. And this in turn was used to get a sub quadratic time algorithm for a version of the nearest neighbor problem. So, symmetric Boolean functions are also interesting in the context of other complexity models for example, AC0 mod p circuits of quasi polynomial size. The polynomial itself as the computational model, but the approximate degree over real is as a complexity measure and also constant depth per se prance. So, a fairly interesting class of Boolean functions and an interesting definition for the complexity measure. So, what do we do here? So, let us start by looking at how do we approximate majority. So, Raspbore and Smolensky initially gave us a lower bound of square root n when the error is constant. So, we will throughout the stock fix the error to be 0.01, 1 by 100. And very recently the Allman-Williams proof gave us an upper bound of square root n. So, this was a curious case of the upper bound following the lower bound. And Allman-Williams did more. So, they showed that for any symmetric Boolean function the probabilistic degree is Big O square root n. Some other examples are as follows. Raspbore also gave us matching upper and lower bounds of constant for the OR function over finite fields. Over the reals we do have a gap. We know that the lower bound is something like square root log n and the upper bound is log n, but there is a gap. There is a gap which we do not know how to bridge yet. And for the mod q function if we fix distinct primes p and q, then again we have matching upper and lower bounds. This is again a case of the upper bound following the lower bound. So, what are we interested in in this talk? We would like to get such bounds for the more general class of symmetric Boolean functions. So, the question is as follows, given any symmetric Boolean function can you find a suitably interesting upper bound and lower bound on the probabilistic degree? Let us fix constant error historically for mod q, for majority and for mod q. So, what are we interested here? So, this is the basic question for this talk. Given any symmetric Boolean function, can we find upper and lower bounds on the probabilistic degree for constant error? More importantly, do they match? And our answer is yes up to poly log factors in n. So, in order to get to the statement of our theorem, we will consider a decomposition of Boolean functions. So, for any Boolean function g, let us first define the period in a very natural way. So, these are symmetric Boolean functions and therefore, they can be thought of as functions of the hamming weight. So, more or less univariate function. So, the smallest positive integer such that g of i plus k gives g of i for all valid values of i that will be defined to be the period of the function g. So, there is a slight abuse of notation here. This is the Boolean function g as a function of the weights that is all. And standard decomposition of f is defined as follows. Look at the interval of weights n by 3 to 2 n by 3 and choose the function g which has the least period which agrees with f on the interval n by 3 to 2 n by 3 and take h to be the sort of the mod 2 sum of f and g. So, the pair g comma h will be called the standard decomposition of f. One example is as follows. So, if you take majority then we restrict our attention to the interval n by 3 to 2 n by 3 and in this region well the period will be n by 3 and when you extend it naturally to both sides g becomes this function and the mod 2 sum is going to be h. So, the point to note here is that h is going to be constant on the large middle region of the hyper cube and probably arbitrary on the periphery. So, these kind of functions are going to be interesting to us which are essentially k bounded functions. We call them k bounded functions which are functions which are constant on the set of weights k to n minus k and take arbitrary values at the periphery. So, it is quite possible that given a k there is also a k prime which is less than k such that the function is also k prime bounded. So, we want the best value of k and which is what we denote by b of f. So, given a Boolean function f, b of f is the smallest k such that f is k bounded function and why are we interested in this quantity? It is because we will be interested in this quantity b of h where h comes from the decomposition of f. So, start with a symmetric Boolean function over a field f with characteristic p, p could be 0 or a fixed prime. Take the standard decomposition h is going to be b h bounded and it is not going to be k bounded for any k that is less than b h that is sort of the observation here. The upper bounds that we get are as follows. So, if the period of g is positive and not a power of p. So, not being a power of p also includes a case where the characteristic is 0. So, if the period is not a power of p then we get an upper bound of square root n and if it is a power of p then it could possibly be lower than square root n. So, it is the minimum of square root n and this quantity period of g plus square root of b of h. So, one thing to note here is that we are only concerned with symmetric Boolean functions and b go square root n is already an upper bound given by Allman-Williams. So, essentially the theorem says if the period is not a power of p then we cannot do any better and if it is a power of p then probably we can use the finite characteristic property of the field and do slightly better than that. And the second part of the theorem says that the lower bounds match up to poly log factors. So, again if the period is not a power of p it is omega tilde square root n and otherwise the other expression we are hiding the poly log factors here and which is why we have the tilde. So, let us start with lower bounds. So, here we will consider a small concept called the restrictions more or less clear what it means, but let me just define it for this talk. So, start with the Boolean function should be f and take a partition of the set of variables as called it n, z and u and a restriction of the function f corresponding to this partition is essentially defined this way. So, n is the set of variables which are left unassigned. So, they stay variables z is the set of variables which are assigned 0 and u is the set of variables which are assigned 1. So, for any such partition the corresponding function on the smaller hyper cube the smaller sub cube will be called a restriction of the function. So, now let us start with the lower bound for thresholds and slowly we will build up to getting the final theorem. So, this is the statement for the threshold probability degrees omega square root t and how do we go about it? Well, more or less straightforward look at the graph of the function and we see that no there is a sub cube over which this threshold function is basically majority. So, we just restrict to that sub cube and get a majority function on 2 t many variables and therefore, the probability degree is at least omega square root t. So, that is how we get it. How do we go ahead from here? Let us say lower bound for h and here is where we will use restrictions. So, let us start with the standard decomposition and B of h is sort of denoted as k. So, this is a slight technical definition n prime is taken to be something smaller than n which is n by 6 plus k by 3, but these are the restrictions that we are concerned with. So, h i of x and certain variables are assigned 1, certain variables assigned 0, some of them are left free. The number of variables left free is k by 3 where k is B of h and now we can observe that the h i is when you think of them as a function on the weights as a function on the weights and written down as a row vector, they will give you an upper triangular matrix and then you can show that the threshold over n prime many bits with threshold parameter k by 3 is going to be a linear combination of these h i's and as a result, we can bound the probability degree of h on the lower side with the probability degree of the threshold because essentially what we have done here is we have reduced threshold to h. So, now the threshold parameter here is k by 3 which is why we get square root k and we are done with this case. Yeah, in this case as it is, for the threshold, no, no, no, nothing, nothing, yeah. So, more generally how will it go? So, let us start with an arbitrary symmetric Boolean function and B is denoted as the period of g, k is the B of h and so we look at cases if B is not a power of p, the first case that we consider more or less the same set of tricks work. So, we can show that the mod q function where q is chosen appropriately not equal to p essentially, the mod q function reduces to g as a low degree polynomial in the restrictions of g. So, it may not be a linear combination, but it will be a low degree polynomial. In this case, the degree will be log B essentially. So, mod q will reduce as a low degree polynomial in the restrictions of g and therefore, we get the probability degree of q as omega square root n and if B is a power of p, then majority will reduce to g again as a low degree polynomial in the restrictions of g and then again we know that the probability degree of majority is omega square root n. So, we are through. In the third case, the threshold will reduce to h as a linear combination as we saw just now and we have the probability degree for thresholds and therefore, we have covered all the cases for g and h need not be x or depends on the field. It is some linear polynomial in the h s. Well, the polynomial for f in terms of g and h will be I guess g plus h minus 2 g h is something like that. So, it is fine does not depend on the field it is g plus h minus 2 g h. So, then we combine these bounds and get the correct bound for f. So, that covers the lower bound side. So, what do we do with upper bounds? We essentially start with the Allman-Williams bound and the more general version of the Allman-Williams bound is this. Given a Boolean function f symmetric and error epsilon, the probability degree of f is big over square root n log 1 over epsilon as a function of epsilon. So, let us start with the standard decomposition and in this case if b is a power of p then we essentially write g as a polynomial in the elementary symmetric polynomials and then we can conclude that the probability degree is at most big O b. And if it is not the power of p then we just use the Allman-Williams bound do nothing. On the other hand h will reduce to thresholds as a sum of thresholds again and how do we do this we will see in a moment. So, we know the probability degree for thresholds as big O square root t and then we will get the probability degree for h as big O square root k and then combining both the bounds we get the construction for probability polynomial for f that will finish the theorem. So, just before wrapping up a few details more. So, for thresholds how do the upper bounds work? So, this is the more general expression for the upper bound for thresholds as a function of the error epsilon and the proof is more or less following Allman-Williams it is an inductive construction. So, assuming that we have probability polynomials for all n prime less than n we just start with an input vector x and sample n by 10 many coordinates from there and now we readily have a probabilistic polynomial for any error epsilon prime on n by 10 many variables and any threshold value. So, if the hamming rate of x is much larger than t then even after sampling we can get with high probability that the hamming rate of x hat is also going to be on the correct side of t by 10. So, the lower order probabilistic polynomials will output the correct value. So, and therefore, we just invoke the epsilon by 4 error probabilistic polynomials by induction hypothesis and output the values. In the other case when the hamming rate of x is certainly greater than t, but not too far away then we have this additional case where probably x hat jumps sides and therefore, the probabilistic polynomial applied to x hat below order 1 might give us the value 0 instead of 1. So, what do we do in that case? Well the hamming rate is still in the interval t to t plus square root t log 1 over epsilon and this is the bound we are looking for. So, we just ignore x hat and use an exact polynomial interpolating that interval and output the value. The only change from the Allman-Williams proof is essentially the base case where well Allman-Williams only had to consider epsilon less than 2 to the minus n in which case you just use the exact polynomial and you will get degree n, but in this case we have an additional case of epsilon less than 2 to the minus t, but we essentially reduce the problem to r and use its probabilistic polynomial. So, in summary what we do is, so we want to obtain low degree approximation for symmetric Boolean functions in a probabilistic sense via probabilistic polynomials. We make the use of standard decomposition and obtain bounds on the components of the decomposition and combine them that is how we go with the proof. For upper bounds the construction is inspired by Allman and Williams, inspired in the base case and in the inductive case it is the same and for lower bounds we use restrictions and reductions to functions whose probabilistic degree lower bounds are already known. So, that is our strategy and yeah that is our way. Suppose you add a finite quantization for the probabilistic degree for r. Okay. I mean is that the only place where the log factor comes in or are there other places? At two places. So, the log factor is coming in because of the non-tightness of r and also in the construction. So, we are writing the function as a polynomial in the restrictions. So, that is where an additional log factor is coming in because the polynomial is low degree. Although it is low degree it is still log b. So, that is where an additional log factor comes in. So, that is from the construction. So, I cannot say it should be there, but it is there by the proof. And do you also have to I mean the finance that you had to work with a different epsilon even though throughout this you are looking at this epsilon being a constant. Correct. So, we scale down the error by a factor of 1 by 4. So, all of it works if we do not scale down too much and that is essentially the idea behind the Orman-Williams thing.


The Court of Appeals stated in Kano State on Friday set aside the judgment of a high court that recognized Mohamed Abacha as the gubernatorial candidate of the People's Democratic Party PDP in Kano State. The appeal court, however, ordered that Sadiq Ali, who was initially recognized by the Independent National Electoral Commission, remain the lawful gubernatorial candidate of the PDP in the state. In a consensus judgment of the three-man panel read by Justice Usman Musali, the appeal court held that the law is clear adding that the National Executive Committee, NEC of the PDP, can conduct the gubernatorial primary elections of a party. The panel added that Abacha didn't participate in the primary election conducted by NEC that produced Wali and therefore had no local stand to challenge the said primary. Recall that the PDP in Kano State had been engulfed in a leadership crisis which culminated in the emergence of two parallel gubernatorial candidates, Abacha and Wali.