to Mike Jervic, who is risk management consultant at Trusted Systems Consulting nowadays. Mike has worked his entire career in the Silicon Valley, where he now heads up his own information risk consulting practice. He's the current chair of the Open Group Security Forum, which sustains the open fair industry standard quantitative risk analysis body of knowledge. Prior to that, Mike had 35 years of extensive experience, including developing and managing high-technology products at Kulit Packard, managing information security IT projects at Visa, and teaching college economics courses at San Jose State University, and leading risk analyses at clients such as the city and county of San Francisco. This morning, Mike's session is going to present the Open Group Security Forum standards and tools that help confront the risk and security issues associated with digital-first architecture. So a warm welcome from the Open Group, a virtual welcome from the Open Group for Mike Jervic. Over to you, sir. There. Terrific. Thank you. So it's easy. It's obvious to everybody that a lot has changed in the last, in a very short time, just the last three months. Locations have changed of where we're doing things. Platforms have changed to where we don't often know if maybe we're in kind of this emergency, fast turnaround time working with individually-owned devices that used to be corporate-owned. Other technologies we may not know because we're really adapting on the fly. And that adapting on the fly is occurring, you know, across every industry segment, whether it's education, retail, it doesn't really matter. A lot is really changing really fast. And a lot of relationships are moving on the fly, where we had one supply chain, as we've been talking about, we may be trying to invent new ones very quickly in order to keep our businesses running. So boundaries and segmentation of functional responsibilities have been changing really rapidly. On the economic front, a lot has changed as well. The IMF World Economic Outlook for June 2020, in its article, A Crisis Like No Other, An Uncertain Recovery, estimate is revised global output to decline by approximately 5% this year. And the estimate for 2021 is 6.5% lower than January of 2020's estimate. What this means is that this change, this impact, is a likely protracted one, one that's going to take quite a bit of time, and it's deep. And so people are in, you know, something of a survival mode and are making decisions that way during this time while a lot has changed. And in this chaotic environment, hostile actors are finding a really opportunity-rich environment in order to capitalize on what for them is probably a once-in-a-generation, if not longer, opportunity. But a lot hasn't changed either. So what stays the same? Well, companies in business still have the same liability to conduct their operations. They still have to comply with their regulatory mandates. And they still have to manage and govern their operations in order to keep the business running, even in an uncertain time. That becomes their job. And management's job is, even in the face of uncertainty, higher risk, they still have to forge ahead and achieve the objectives on behalf of their constituents, of their stakeholders. So what's happened, right? What's the digital security considerations in this environment? Well, it's increasingly important to connect governance, management, and operations in the right way where the language is consistent so that we can communicate clearly and hopefully maybe once. We need to have the right frame. To do that, we have to frame our questions and our issues correctly with the right objectives and key results supported by good architecture and execution. From that, what is the language and frame? Well, effective cyber risk management becomes the goal. Security, architecture, deployment, and operations are the means to the end of achieving that goal. So security controls are not the goal. Effective cyber risk management is. And we at the Open Group, our security forum, we have the mission of connecting architecture and design with risk and risk management. And it comes down to this fundamental precept. Measure what matters. Risk is the measurement of security. Security is higher when risk is lower and we can measure it. That's what the Open Group security forum is all about. And this is our vision for it. If our goal is effective cyber risk management, we have a management stack that you see on the left whose goal is to effectively govern and manage. To do that, we have to be able to make well informed decisions to effectively compare alternatives. To do that, we have to support those comparisons with meaningful measurements and accurate models. That is the management mission. The board of directors and the C-suite is to effectively govern and manage risk within their operations. At the same time, and you see that on the right side of the language mission bridge, is that management, even in chaotic times, is moving towards a continuous improvement model where they need the right measurement so that they can see what works, they can adjust and they can take corrective action when they're not achieving their objectives. So to do that, we have to support a plan-do study or plan-do-check-act kind of continuous improvement cycle made famous by Schuhart and Demick. What we have at the Open Group is a risk taxonomy articulated in the tree under the bridge that is the way that defines the language that we can use to communicate about risk and then translate that, use that to guide the functional security operations that are control-based and technology-based that use something like the NIST-CSF as a framework for thinking about technology and architecture through things like identify, protect, detect, respond and recover. And to do that, we have to have the concept of what does it mean, what are we trying to worry about? We're trying to worry about losses. How do we talk about that? And then how do we correct that? How do we control for that? That's what the Open Group does. We bring risk management and we bring technology and architecture together to holistically discuss what matters most. The measure of security that is your organizational, your risk associated with IT assets. So what do we do? What are our resources and activities? We have the open fair body of knowledge that consists of two standards, ORA and ORT. We have a risk analysis spreadsheet tool that helps, there's a good tool that's been used in the field to identify, to decompose and measure single risk issues in a way that brings new practitioners up to speed very quickly and can help people learn about quantitative risk analysis. We have a process guide, again, aimed at the new practitioner to help make that first risk analysis successful. And we have a certification regime which has already certified 775 risk analysts. So we have a vibrant community within the Open Group. To measure, quantify, cyber risk, help educate, help train, bring people up to speed with tools and support, and then help signal to the world a skill level through the foundation certification program. We also have a vibrant architecture practice. One of the more recent publications from that was the Axioms for the Practice of Security Architecture that gives a basic structure for how to think about security architecture. And then we have some relatively new activities in flight all around zero trust. So that as we don't rely upon the provenance of devices, whether your presence on a network or the network, how do we guarantee some sense or how do we think about security in that sense where we have to test for and validate security on a near every data transactional level. And then, and this was referenced but it was in really tiny print on Bob's slides. We have the Outsourcing Network Services Assessment Tool that is used to help vendors, to help you understand and evaluate the supply chain of and risk associated with outsourced network services. Outsourced services broadly defined. That's not only an open group guide for the documentation and the approach to that but also a spreadsheet tool and user manual that helps embody and concretely give you tools. So one of the things that you see consistently here is that we go beyond just standards and just saying what, but we have for the last couple of years really tried to help make it real to people so that we can support the practice so that we can support the profession in quantitative risk analysis, security architecture and network assessment. We said, open fair is where risk meets architecture and we do that through what we call a loss scenario where we form a sentence, threats, breach or impair assets that cause observable loss events that have direct consequences and may have reactions from others. And from that, we can develop the concept of what happens before the loss like a contact event, a threat event that becomes a loss event and the controls in place that can mitigate a contact event from becoming a threat event or threat event from becoming a loss event. And then we can argue and support how those controls map into a control framework that the security operations people already have some familiarity with so that we can map what management cares about where risk meets architecture to the architecture and the operations. And similarly, once a loss occurs, how do we mitigate it once it's begun? So we have to detect it, respond and recover and again through forms of loss and through those functional categories of operational activities. We can help people see the complete picture for how they can achieve management objectives through security controls. The axioms for the practice of security architecture gives you a frame for thinking about how you build an architecture for controls. And you'll notice that the top of that at the basis of it is that it's a risk-driven architecture that has applicability through a context, scope, intelligence and how to build trust and how you think about trust between within the architecture then gives a structure, an interface. The interface is where the human design comes in for clarity of communication. Is it usable and is it designed well? And then it gets into detail, much of which we have already seen a lot of in the security practice like defense and depth, least privilege, access management, secure communications with a few other things added like precedence and design sovereignty. And you'll see at the link here where you could get that today. The guidance to tools to assess outsourced network services is a very recent publication that worked on and really led by the Department of Defense. But the open group contributed to that and has published on its website the overall approach to this and also then how the tool is not, the tool is not hosted on the open group website but it is hosted as you see here and we recommend that you be able to pick this up. This will help people address the outsourcing of network services and can meet the needs of multiple sizes and types of businesses. So it's designed to be flexible and scalable and publicly accessible. So what are we doing now? Well, we've spent about the last two years revising the open fair standards and some of the approach that you'll see in those upcoming standards, you've gotten a preview to today. We've done a lot of work to try to make the standard more accessible and more clear. And that we can connect and help people grasp more quickly what open fair is about so it's more tangible and more operational, more actionable. And the other activities we are focusing on are in the Zero Trust area where we're collaborating with the Open Groups Architecture Forum and the SABSA Institute. From this, I just asked our audience members if you're interested in this, come join us. We have recently attracted several new members who are in both the architecture practice and in the quantitative cyber risk practice. And so we want to continue bringing these people together so that we can continue developing this vision of connecting the board and senior management to security operations in the right meaningful way. And with that, let's take a couple of questions. Mike, thank you very much. Thank you very much for sharing what's going on. It's a great summary, there's a lot of work but you've stuck to time and managed to just hit the high points there. I mean, I know you and I have discussed this before but I'm not a security professional but it was nice to hear Bob say that he trusted me so that was good to know. But I'm not a security professional but I hand on heart can say that the open fair standards, the body of knowledge there are very easy to pick up and understand for somebody who doesn't have years of training in this. It really is a very approachable body of work and I'm sure there are quite a few of you on the attending today who may not have the extensive experience that some of the speakers have in this area but I assure you it's very easy to pick up and I encourage you to do it, it's great. And of course, you've been teaching it to economic students, Mike. So- I have. And they get it. So a couple of questions. You talked about collaborating with the Architecture Forum and the SABS Institute. Is there an Archimate Security Architecture Model or maybe a work to achieve one? So I'll say not to my knowledge but as you kind of opened me up, I'm much more specialized in the cyber risk and quantitative measurement areas and so I've less connected. So I think it'd be a good thing to let's take that offline and get back to the asker about that because I honestly don't know. Yeah, yeah. No, I'm nowhere one either but we can try and address that separately. Can you please touch upon the certifications for open fair, Mike? Sure. And if there's any further elaboration on that question but the open fair certification program has been around since 2013, we have like I said about 750 people already certified in around the world. It's an 80 question exam based upon the two standards and if you, but I will say as I've worked with people and ranging from undergraduate economic students to practitioners, it's a little hard to study for and get all the details in place to complete the certification exam but I've seen it done for self-study. I've certified undergraduate students in it, about 30 of them over the years and so it's quite doable and it's a like 80 questions, multiple choice and it's quite achievable and we have an open group study guide. It helps support the self-study student. So I feel like with that, the process guide, the risk analysis tool, for someone who wants to come up to speed on fair, there are tools and resources out there from the open group that can make that happen. Absolutely, yeah. And we'll hear a little more about those later in the day, of course. And there's a question just come in as it gets somewhere between a comment and a question. It's a bit different from TOGAS then, which clearly it is. Yes, yes, I actually have taken TOGAS certified. So I will say it is different maybe because I know fair so well. I think the fair certification approach is conceptually simpler. Oh, and it's the more the foundational level than TOGAS is. Yeah, yeah. Okay, Mike, we are gonna leave it there, but thank you once again for a great summary of what's going on in the forum. And I'll repeat your plea. Anyone interested in these things, please come and get involved. It's a great community and there's lots to be done. So meanwhile, Mike Jervik, warm round of applause from the open group. Thank you.


three-dimensional scaling is the big theme. The audience today has 3D glasses. Now we have Jack Dome on here at 12 o'clock and we're gonna be asking about that. It's unprecedented. Nobody's ever done that. Scale out, scale up, scale deep. We'll find out which parts of that are real, which parts of that are marketing and go deep today. Yeah, I mean I think the sun is shining here in Silicon Valley on us. We're in a great spot here. But we are listening to a revolution with Hitachi because as you know, Dave, we've been asking all of our guests going back to the SAP press conference of the Oracle event and VMworld, is it scale up or scaling out as a fundamental difference? And we've heard different answers and I think what Hitachi's coming forth today with is an absolute message that you need to scale up and scale out and scale deep. Now scaling out is something that's being worked on by the top computer scientists at Stanford, MIT and all around the world. So that's not, that code has not been cracked yet. And Facebook and others are dealing with that. But scaling deep is a data-centric model. And I am really, really blown away again by the difference between the two mindsets. We heard from Oracle, very proprietary application focused. They do not want to be open. Hitachi is absolutely being open. And again, we're seeing this mindset, the cultural difference between the industry heavyweights of open versus closed. That's right, a critical part of Hitachi's strategy is to virtualize the storage, not only the Hitachi storage, but any third party storage as well. That's a unique capability of the industry. Only a few vendors do it. IBM does it with an appliance. LSI is doing it. HP does it with the LSI appliance. And there are a few others. But Hitachi is really the leader there in terms of virtualized storage and allowing third parties in an open manner, as you said. And a nutshell, the key highlights for me that I heard were it's a very customer partner focus. But that's, you know, all corporations say that. But their business model has been changing as they pointed out over five, seven years ago. They've moved significantly from a revenue standpoint where software is driving, you know, over 50% of that revenue. And again, openness. And that is very key. And the data strategy, the data strategy is very, very independent of applications. It's very cloud-based. He talked extensively, must have used the word governance and counted, I counted over a dozen times. It stopped counting at that point. So the notion of governing data in an application silos is a big message. And then content cloud, he said content cloud, I counted almost half a dozen times. Yeah, the content cloud really caught my attention. Hitachi used to essentially, 20 years ago, used to just copy IBM. IBM would come up with a product and IBM would come up with a little bit better product. Or Hitachi would come up with a product 18 months later. This is completely different. You know, EMC is the big competition these days and Hitachi's strategies from a product standpoint and a marketing standpoint is completely different from EMC's. I'm very impressed with that. The platform is different. The approach is different. And this whole notion of unifying content into the high-end mission critical systems is quite interesting and I want to learn more about that today with the guests that we have on. Yeah, we're gonna have a lot of guests on here. But Jack Tomei was very impressive. You know, he's not the flashy Steve Jobs, but he actually delivered great messaging and he's a very cool guy. I will say that he was wrong on a few occasions. Being new to Facebook, as he pointed out, Facebook has more than 400 million users that's well over 500 million. I'm not sure he's got his hand on the pulse with relative to that, but. Oh, he's new and I tried to friend him and he can't friend him, only he can friend you. Well, I mean, I think he doesn't have a real pulse of the latest and greatest statistics, but I think he did bring up social networking as a proof point. And the proof point is that there's two aspects of data that he was talking about. One was the existing current situation, which is enterprises and big, large-scale institutions like financial houses, telcos, insurance companies. They have data. Data that's tied to application silos. So he talks in that framework as a governance issue. Who's going to govern the data? How does this stay persistent? So that's kind of the existing current installed situation, installed base, if you will. Then he mentioned growth. And the growth areas, as he pointed out, was what we've been talking about from VMworld, our talks with CloudEra, and then recently at Oracle Open World with Green Plum, is unstructured data. I mean, he clearly said that this new cloud, this new environment is going to be unstructured data. And that's where he tied in the Facebook point and that social networks are throwing off this new type of data and the innovation of how to manage that data. So he, although he got the number wrong on Facebook, really had his hands on the pulse, in my opinion, on the core issues. It was only 100 million off, John. No, but I think more importantly, he knows the issues. I mean, the issues are unstructured data, massive growth in data management. You know, is that going to get easier? No, it's going to get harder. And so by focusing solutions on that, that was an interesting point I learned. And Latachi's messaging focusing on that is new and different, right? You think about Latachi, you think about the most mission critical, high-end OLTP types of applications. And now Latachi, a couple years ago, bought a company called Archivus out of Waltham Mass. That was a content play. And now you're starting to see product merge together. And that's something that's actually quite unique in the industry that, now NetApp obviously talks a lot about unified storage, but at this level, at this high-end mission critical level, you typically don't see that. You certainly don't see that from EMC in terms of bringing together the content and the mission critical OLTPs. That's different, that's new. Let's find out how real it is. The other thing that I observed is Hitachi's very proud of their culture. I mean, they started the whole presentation up with the drums beating.  Ishigaki-san is now on. Yeah, so I mean, the heritage, they're proud of that. I mean, they're not known for their marketing hype, but clearly here, they're putting a lot of effort into pushing out messaging that's really kind of game-changing. So I was actually encouraged to hear and see that kind of cultural, you know, they're not hiding from the culture, they're proud of it, and that was interesting. Yeah, 100 years is quite a milestone. Let's talk a little bit about the folks that we have on today. So we're kicking off the morning with Roberto Basilio, who, and we're going to take a product focus. He's going to tell us what's new in the product, what's different, and what their vision is, you know, and where they've come from. And then we've got a number of analysts on this morning. We've got John Webster from the Evaluator Group, Ray Lucchese is coming on our own, Wikibon's David Floyer, Mark Peters from ESG is going to be on, and then we've got a segment with the IDC analysts, Rick Willers and Laura Dubois from IDC, and then we've got some bloggers on. DeVang, StorageNerve, many of you know DeVang, and Rick Vanover are coming on for a session. And then we go in to the afternoon and Jack Domey is going to kick it off at noon. Really excited to have Jack. 12 o'clock is a big interview. Jack Domey is dynamic, he's the CEO. Having him on will be really, really fun, and we're going to ask him some tough questions. We're going to put his feet to the fire and have him come into the cube, inside the cube. And I really want to hear from him more about some of the strategy and tactics that they're executing in this marketplace. Obviously Hitachi is changing their focus, obviously from a marketing standpoint, but the product differentiation seems to be a unique thing, and they've got that product leadership from what we're hearing, but I want to see the meat in the bone. I want to hear from him. What is his cloud vision, right? We've had a number of big execs on recently. We had Joe Tucci was on, Mike Cappellus, we had Dave Hitz last week, we had Tom Georgins on. Of course, I'm going to ask him the question that we ask every CEO, is storage sexy or hot? Let's see what he says. We had a great answer last week from Dave Hitz. What do you think he's going to say? You know, I'm curious to what he's going to say. We've heard a different answer from everybody. Joe Tucci didn't buy into it, right? It's kind of like, you know, the question, what's the craziest thing you've ever done? The answer is actually very telling by the personality. So we'll see what he has to say. And then later on today, we've got some customers. The GM of Intel's IT is going to join us. The head of trading and infrastructure at Lloyd's Banking. And then the University of Utah will be on. And then at the end of the day, Brian Householder, who we heard from this morning. And really, Brian sort of is the overseer of all of the marketing and the strategy. And that's really his role. Brian has been very active in the strategy of the company. So he's going to be a great guest. And then we're going to try to get some other guests on as well. We're here at siliconangle.com, siliconangle.tv, continuous coverage of Silicon Valley. We're here at the Hitachi Data Systems, massive launch of their new product, the virtualized storage platform. They supported us to come here. Siliconangle and Wikibon are independent media and research organization. We're excited that they requested us to come down and helped out a little bit with some of the costs to do the cube. So we're really thankful for Hitachi for inviting us. But we want to get to the heart of the issues here. We want to dig in, Dave. And I want to ask you about Hitachi. And I want you to tell me about how they've changed over the past five years. Jack Domey mentioned it. But I also want to ask you about that in context to Oracle. I mean, I'm just seeing two different animals here, Oracle and Hitachi. Oracle is coming across as an evil, land-grabbing, telco-like environment. Well, Hitachi, first of all, is a $97, $98 billion company. I mean, it's just an enormous firm. They build trains and medical equipment and all semiconductors and disk drives. And obviously, HDS does the storage subsystem piece as well as some other things. What's been striking to me, and I've been following Hitachi before they were Hitachi data systems. And as I said, they used to copy whatever IBM did. And they've really strived to do innovation. I've been to Odawara a number of times in Japan, which is their innovation center, where they design all these subsystems. And they really try to leverage the other parts of the organization, the networking piece, the semiconductor piece, the disk drive piece, which they purchased from IBM several years ago. But I think you're absolutely right with regard to Oracle. What impresses me about Hitachi is how they're not just copying IBM anymore, not just trying to copy EMC. They're really trying to break new ground. And they've proven that for the last five to seven years. We're saying leadership on the high end, low end, mid-range, all the above. What would you say that, from there, putting their stake in the ground gives you the competition? Their legacy is unquestionably the high end. And of course, management several years ago saw that the high end was a managed decline business. And so they struck out, they've really developed a much, much stronger mid-range product called the AMS. And they have a partnership with Blueark, a really hot California company who's been growing like crazy. And they've worked together to really build up, using that, again, that content archivist platform to build a great vision. We heard a lot of it today. We're going to hear more of it. Relative to Oracle, Oracle's a whole different ball game. Oracle's going to take its applications and really build a big giant, ironically, hairball around the Oracle applications and the Oracle hardware networks. Let me dig into that. So Jack told me the CEO of Hitachi this was on stage saying, he wants to free up the data from the application. And he must have said applications as I was a few times. What you're saying is Oracle is application-centric. They don't want to do that. No, they want the Oracle applications to be the center of the universe and the Oracle slash Sun hardware to be managing that. So they're really not very, not all inclusive of other. In fact, Sun used to resell Hitachi's high end platform. One of the things that Larry Ellison has said publicly when he took over the management of Oracle, said we didn't make enough money off of the Hitachi platform. Great product, one of the best products out there, but we didn't make enough profit so we cut it. We're going to sell our own stuff. Really very profit-driven. Sun is now chasing profits, not revenue. And innovation. What's innovation with the Oracle versus Hitachi? I mean, which open, closed architecture, which one of those, in your opinion, is most innovative? Well, I mean, Oracle owns the database, right? So it doesn't have to innovate the same way that an Hitachi or a NetApp or other companies have to innovate or they'll die. Tom Peck was on the show, we talked about StackWords, but one of the questions we asked him at SAP Sapphire was breaking down the silos here. Jack Domey was talking specifically about applications silos relative to data. Although Tom Peck didn't talk about that, and obviously the CEO of Levi Strauss doing theCUBE a few times, an alumni there. What is your view on the application silos and then Hitachi's position, which is saying, hey, we want to break that down and bring data into this virtualized storage environment or content cloud or information cloud? Well, some of the themes that we've heard, and Hugh Yoshida now, the CTO of Hitachi is on the stage, but one of the themes we've heard from VMworld on really here is that 70% of the investment that IT organizations make is in keeping the lights on. I call it run the business. And only 30% is spent on grow the business and transform the business. And the big reason why is that IT infrastructure can't support that grow the business and transform the business because it's all developed in silos, the applications, the middleware, and the infrastructure are all silo-based. And it's very, very difficult to be agile with that type of environment. Virtualization is changing all that. One of the things that we talk about at the market level and the end user impact all the storages. Storages, as we always say, sexy, but it really does impact the end users. I wrote a post earlier in the year called Mobile Innovation Cycle. So if you go to Google and type in Mobile Innovation Cycle, you'll see a post that I wrote was really groundbreaking at the time. But now everyone's using it, which is data is the enablement of innovation and relative to the growth of mobility. But one of the things Jack Domey said on there, and we could unpack that for a good half hour, was he said, there are new forces, new environments, and cost pressures with managing the data growth. And he said, the new ways to manage data growth are in three areas, servers, applications, and consumers. He's essentially talking about the areas of virtualization that have a data-centric component, obviously server virtualization. We all know that market, application, virtualization. That's where the storage is separated from the applications and the consumers. The consumers having iPhones and iPads and mobile devices, they're going to get virtualization, which we heard at VMware loud and clear that there's an end user component of this. So he actually laid down a foundation I was impressed with that it's very, very relevant, that three-legged stool, servers, applications, and consumers. And I liked Dave Hitz's answer on his storage sexy. He said, if we're the plumbing, if we were a Hollywood movie, we wouldn't be the leading actor. We wouldn't be the star. We'd be the supporting cast. And that's really what storage is all about. And so that's not what we're hearing from Hitachi. Hitachi's saying today that they want to be the lead actor. Well, but he's saying we want to be the change agent with 3D scaling. Yeah. So the interesting thing, does NetApp have an answer to that? Well, first of all, you heard Brian Householder say, every customer wants to do more with less. OK, and so Hitachi as a storage supplier has to provide that. I mean, whether or not a company in the storage business can actually be that leading actor, I think remains to be seen. We're hearing some interesting things about the content cloud and bringing together virtualized infrastructure and the content cloud. That make it a leading role. Again, I kind of like Dave hits his answer. It's the applications. You call it the cold, clean water running through the pipes. You use the plumbing.   So bad data is bad water. So to me, what makes storage sexy is it's a growing and profitable business. And it's interesting. A lot of characters in this business. That's what makes storage interesting and hot. So do you think, let's talk reality now. Let's bring this down to some proof points. Do you think Hitachi has some legs to stand on with this big announcement? I mean, honestly, they're talking a great game. Let's drill down. And when we get the guests inside the queue, I want to ask them some specific questions. What questions do you have in your mind to kind of flesh out and bring out the reality of this message? Well, so first of all, I want to understand what's new. So to your point, Hitachi has a lot of proof points. Hitachi's been doing this for decades. And Hitachi's one of those companies that makes slow, steady, consistent progress. Now, that whole thing about 3D scaling. Scale up, scale out, and scale deep.   And I get the scale deep because they're allowing third-party devices to attach. And they can go deep, cheap, and deep with those third-party devices. It's the scale out, as you said. There's a lot of computer science around that. And I see bits and pieces around the clustering technology that Hitachi has that I think give us clues. But I really want to know what kind of legs that piece that Hitachi has because that's on the scale-out business. The scale-out piece, that's right. Well, there's all kinds of issues with scaling out. As I see this round trip times on the network, and you've got data coherency. All kinds of data issues around updating. Because you can spread the data everywhere. That's great. But there's real innovation and real math that goes involved with that kind of architecture. You're talking about the coherency. Well, just scaling out is putting data anywhere. It's interesting. And it's a really, I think, important part of the future. And we've had a lot of conversations. We've had Fusion IO on Dave Flynn. He talked a lot about these Web 2.0 companies like Twitter and Facebook and MySpace scaling out. And they need to scale out because they're moving so fast. They need building blocks that allow them to grow and not have it infrastructure needed. Let's talk about one of the key pieces that we liked. One of the things that I liked, I heard, was this openness message. And you just mentioned that any third party can attach. Can you elaborate more on what that means? We saw a Falcon store in there who's been on the queue. You mentioned Fusion IO. You got a hint of all the partners in there. Intel's in there. What does all that mean? What does that mean a third party can attach to Hitachi? And they get the benefits of that. Hitachi early on decided it had to differentiate from the likes of EMC. And so what it did is it came up with this notion of a virtualized platform, very powerful platform. And they had this innovation where they said, let's allow third party disk subsystems to attach to our engine. And a lot of people think, well, that's strange. Why would you allow third parties to attach? Well, Hitachi recognized that there's a lot of data out there that has to be moved, that has to be managed, that's stranded. And so what they did is provide a platform.  But nonetheless, a lot of companies will attach third party drives and use them as a cheap and deep storage. So it extends the life of your existing assets. And what Hitachi does with its software is it embraces those other third party assets. So it brings life, if you will. It breathes life into these old, stale, third party devices, whether they're from EMC or 3PAR or IBM or HP or whomever, they support these devices. And it was a very interesting strategy. You've seen IBM have some success with that, with its SVC product. LSI with a product called SVSP. Actually they OEM that to HP. LSI calls it the SVP, I believe. So a number of companies have hopped on this. Now EMC with Vplex has jumped into the game. So you're seeing confirmation. Talk about EMC, Ashley. We know EMC, they support us with a cube and everything. But you're talking about EMC is on the high end. Okay, Hitachi has market share numbers that show them as number one in virtualization. You wrote a post and I wrote a post amplifying the fact that EMC is number one in virtualization. So who's number one? Yeah, let's clarify that. So EMC is number one in VMware. And presumably, according to this, so ESG came up with some data that suggested that in terms of share penetration within VMware and other virtualized accounts, EMC was by far the dominant account. And that's not a shock, right? You would expect that, right? They work hard in VMware. They own VMware. It's a big, big area of emphasis for them. They get a lot of... So they're number one in VMware. So they're clearly number one. And in my opinion, and the data shows it in the US. Anyway, it was US data, but I wouldn't be shocked if it's overseas as well. In VMware environments, hands down. Now, that's different than Hitachi. Hitachi is number one, or maybe number two, but right up there in virtualization of storage, the virtualization of the storage back end, until Vplex, EMC really didn't have a solution for virtualized storage. And I would say Hitachi's number one there in terms of maybe licenses sold, I don't know about revenue. We'll have IDC on. Maybe they can give us some perspective on that today. But very clearly, Hitachi is leading. Hitachi's claiming number one on that, so. I would believe it. I think it's certainly number one from a function and a product standpoint. Whether or not it's revenue, it's hard to tell because they ship licenses with this stuff, and it wouldn't surprise me if they're number one. It's Hitachi and IBM with SVC right up there. And SVC's really an appliance that doesn't have storage in it. So I would tip the hat to Hitachi because of the revenue associated with the internals of that device. Yeah, the other thing that we heard from him was anyone's storage. And the idea of having efficiency and mobility of data. And then he talked about the key things of this, which was optimizing the infrastructure and optimizing discovery. Those two variables were key on Jack Dome's list of the areas of innovation around virtualization of storage. Which he did say, virtualization of servers have been amazing, but virtualization of servers is dependent upon storage. So you and I have been talking about the linchpin of cloud is storage. It happens to be true. He talked about that. But importantly, if you optimize the infrastructure and then the ability to discover the data, you can integrate that into data and applications. So that, to me, is a big focus. I mean, we talk about the Apple iTunes and the App Store, the world seeing all these apps, tsunami of applications. So applications are hitting us left and right on the marketplace. All the data from those apps cannot live in a silo. Yeah, and I think it's smart. It's clear to me. I think it's smart for Dome to message around Facebook, for example, because you think of applications in the enterprise and obviously you think Oracle and SAP and you think these big conglomerates, but there's so many emerging applications that are driving innovation. We've talked about the consumerization of IT. It's true for the applications as well, don't you think? Yeah, I think anyone who could take the applications and give them freedom around how the data is managed and accessed will be a winner because application providers now have distribution characteristics of big application developers all the way down to what they call the long tail of applications, the small guy. Someone who bangs out an iPad app or an iPhone app or an Android app and gets 100,000, maybe even a million downloads. You can have a three person company do that and make money, have a great life, buy a boat, buy a house, use the cloud. But do they want to build a backend data center to manage that data? No, they'd love to have a Hitachi cloud solution for content cloud. So I think this content cloud has legs. I want to ask them specifically what software is being innovative to make that happen.


My name is Trish. I work on the research and evaluation team and co-support, co-lead the Institute for the Scholarship of Teaching and Learning. And yeah, as Will said, as part of that role, I end up supporting a lot of teaching and learning projects, developing evaluation over the lifespan of their projects at various stages. And I can provide support at all of those stages or some of those stages as many as and as few. And that's me. I'll pass it over to Natasha. Hi, I'm Natasha. I'm also on the research and evaluation team with Trish. My focus is more on the subtle side of things of the scholarship teaching and learning. But as with Trish, we both consult on the TLEFs and the OERs and we're happy to help you with any any stage of evaluation for your ongoing projects. Perfect. And so before we begin, we would like to acknowledge that UBC, which is hosting this session, is located on the traditional ancestral and unceded territory of the Musqueam people. And as we're working somewhat remotely today, I'd like to acknowledge that here in the lower BC mainland, we're often on the unceded territories of many different Koso people. When I acknowledge being on the Musqueam territory, it's with the acknowledgement that as a member of the UBC community, I'm very lucky to be able to work and learn on a territory that's not my own. And additionally, as somebody who really focuses on open educational resources, I also really like to acknowledge that OER is often grounded in copyright and western notions of ownership, which can sometimes be in tension with indigenous and traditional ways of knowing. And we're not going to really go into that topic at all in this session, but I do like to acknowledge that those tensions do exist. And we did have a session earlier this year with Kayla Larson, the Indigenous Services Librarian at We Will All Library that did explore specifically that topic. And I'm just going to post a link to the recording of that session into the chat. So it's definitely worth checking out if you're interested in that. And then finally, before we begin, this is a very small group. So I would say feel free to be more chatty and we can run it more as a consult. It is being recorded though, just to let you know that. And then finally, if you're not currently an OER fund ground holder, we do have a grant cycle happening right now. And the deadline has been extended to January 19, 2023. And if you're interested in applying to get financial and support, all sorts of support for developing or using OER in your courses, please come talk to me. And there's lots of opportunities and there's still quite a bit of time to apply for that. And with that, I'm going to hand it over to Natasha, I believe. Great. So thank you, Will. So let's get started with going through our workshop.  So do please feel free to raise your hand or just unmute yourselves and ask questions or throw questions in the chat. We're happy to pause and chat on the sections that are most relevant and useful for you. But just kind of overall, the goals for our session today were we're hoping that you can understand how evaluation can increase the success of your OER project, help you develop some kind of clear objectives and how you might reach those objectives for evaluation and look at what potential challenges might get in the way. So as we go through this workshop, please do keep in mind that this is kind of just the start of the conversation. The idea is to get you started with your evaluation plan, but it will take more than an hour, an hour and a half to work this through. And just a reminder that Trish and I and Will as well, we're all here to consult with you and work with you on developing this. So just a quick outline of our workshop for today. We'll chat briefly kind of about the big picture about the evaluation cycle. We have a couple of short activities to kind of help you leave with some kind of practical, tangible outcomes. And then we'll talk a little bit about different approaches to evaluation, considerations, and tips and tricks, and then some questions, answer, and defend. And just I will put into the chat. So we will have a couple of worksheets that we will go through during our session today. And you can find those at the website that is in the chat. So just as we get started here, it would be helpful to get a sense for us of who is in the room. So if you just want to fill out this quick poll for us here, we're just wondering about kind of where people are in the OER stage. It's great. So I think we're more kind of in the stage of interest, considering applying, trying to learn more about what the OER fund is in general. So just keeping that in mind. Yeah, I do feel free to jump in at any point in time. So let's chat a little bit about why we evaluate. So the goal of evaluation really is to make sure that your project outcomes are being met. So making sure that the goals that you have, the plan that you've developed, is working as you intended it to be. So evaluation can help you figure out if the goals that you have stated are actually being accomplished. It can help you improve the design and implementation of your project. So it can help you figure out if you have these broad goals, you have these intended impacts for those broad goals. And evaluation really is about making sure that those intended impacts are happening. So checking whether things are going the way that you hope and expect that things are going. And then evaluation can also help you make informed decisions moving forward. So looking at what's working, looking at what's not working, really thinking about evaluation as feedback, and then making informed decisions to move forward with the next steps. And then in terms of the practicality of it, kind of the OER fund, project evaluation is a required piece of it. So we do expect that if you develop something with the OER fund that you will be evaluating whether it is actually accomplishing what your purpose is accomplishing and will provide that information in the reporting process. So the other kind of critical piece of evaluation is that it is an iterative process. So it is not a one-time, I have this goal, I'm going to check what's going on with it, and then I'm done. It really is kind of similar to the research cycle, where it's always a circle, it's an ongoing thing. So maybe we'll skip the second poll here of kind of which stage you're at, just based on not many of us in the room here. I don't think anyone in the room here is a current OER fund holder. So just kind of getting a sense of this evaluation cycle. We will be going through kind of primarily the first two parts of the cycle. So looking at what are we hoping to evaluate, what impact do we hope that this thing that we're developing will have. And then we'll talk a little bit about identifying those sources and methods. So the goal for the rest of the session really is to get us through parts one and two here, to get us ready for part three, or for considering how we might do that in an application. So looking at practice and intended impact. So what do we mean when we talk about practice and intended impact? So when we talk about practice in the context of the OER, that really kind of is the resource that you're developing. We use the content of these slides and this idea is built off the TLF, which is a similar idea, but there's a bigger range of things that happen there. Whereas with the OER it is more kind of tangible and specific. So in the context of OER, the question really is what would you like to evaluate? What is it that you're building out? What is it that you're creating? So for example, that might be a textbook or a website or some sort of additional tool that you are creating. And then the impact is what do you want that product to do? What do you want the outcome to be as a result? So examples of that might be things like motivation or increased awareness or increased understanding, student learning, all of those different pieces. And so we really encourage you to think about what is it that you're doing? What impact do you want it to have? And then let's evaluate those impacts and whether they're actually happening. So let's evaluate whether student motivation is really changing as a result or whether student learning or performance is really changing as a result of this thing that we have built out. So thinking in terms of defining practice or product, so this little chart kind of takes us through those steps that we hope that you will be mindful of as you're considering evaluation of the OER. So for this slide, you kind of already will know what your hoped OER is or perhaps we're just kind of exploring what the OER is. So we won't go into brainstorming that in detail today, but this is in the worksheet and we would encourage you to take some time to brainstorm the practice and product that you might be wanting to create. So it would be helpful for you to spell out exactly what that resource is and who you hope it will be accessed by, the context that it will be used in. All of these things are important considerations to keep in mind. So considering the classroom context in particular, the level of the students, all of those different pieces of what the context might include. And then once you know what your practice or product is, thinking about how to define your intended impacts. So in this example here, our sample practice product is developing open self-study quizzes for all sections of a particular class. And the intended impact for this example might be to increase student learning and knowledge as a result. So thinking through all of those potential impacts, and we'll pull up a slide in just a second on that, but thinking through all of those potential intended impacts and what is it that you really want this product to be doing. So here we've got our list and I think we can skip the activity. So typically when we have folks who are current OER fund holders, we do ask them to kind of give us an indication. But we often see a good assortment of different things here in terms of the different impacts that people are hoping their new resource will have. I just want to jump in. So yeah, the purpose of this activity is to just kind of help people really think about making that link between the resource you're creating and what you want it to do. And this is just an example of some of the things. There are examples that we have seen time and time again in TLFs and in the OER projects funded to date. So you can also think about this as sort of like a motivational slide, as you're thinking about developing your project and what goals it is that you hope the resource to have.  That's the goal, but you want the resource to do something for students or for your own teaching practice to be helpful in some way. So even if, yeah, for those of you in the room that, you know, you're not at a stage yet to necessarily map this out, the worksheets that Natasha linked to earlier in this slide is really a good place to start once you have an idea in mind for a resource to build of things that you might hope that that resource will achieve. Great. Yes. And as Trish mentioned, these are kind of the more common ones that we see. And we do often see folks check off a fair number of them. So it's not that you need to pick one impact. It's a question of what are the things that you hope will happen? And oftentimes that is a number of different things. So, you know, cost savings for students as well as student engagement and student well-being and different pieces. But as Trish mentioned, thinking about using this list as a starting point to really reflect on what it is that you hope this resource will accomplish. And thinking through those pieces should hopefully help you develop the evaluation questions that you want. So looking at kind of what are you looking to answer? What are the guiding questions? The questions should involve kind of your intended outcomes. And the answer to those questions can become the claim of your project's success. So looking at how do these, in this example, how do these quizzes impact student knowledge of the core concepts of the course? And so being able to pull out that information will help you determine whether the resource is actually doing what you hope it is doing, accomplishing what you hope it's accomplishing and serving the students in the way that you hope to serve the students. So again, we'll skip through the activity portion here, but a reminder that you can use the worksheets that we've provided on that link to kind of develop and track these different pieces. So thinking through what is it that you want to build out? How is it that you hope that that will impact the students? And then what is the actual question that you want to establish out of that? So I'm going to move on to the next stage. So after you've gone through and defined your evaluation questions, what the resource is and the impact you hope it will have, what you want to do is think about how you would answer that evaluation question. How will you collect evidence to know how the resource is impacting student learning or impacting student motivation, et cetera, et cetera? So I'm going to walk you through some common sources and methods that we see in a lot of projects, kind of the more common ones we see. First, I want to talk about different ways to measure, different measures for capturing this data. So again, working with that example we had earlier, how did the self-study quizzes impact students' knowledge of core concepts? How would you know if that's happening? So some measures we can think about would be things like performance on knowledge tests, which we might access via quizzes, or something like student confidence. So how confident are they, a judgment of learning, for example, of those core concepts? And that could be evaluated via surveys or focus groups. Again, in the worksheets that Natasha shared, the link that Natasha shared, there are some examples of different measures that might make sense, depending on what your impact area is. So really think about when we use something like we want to improve knowledge, that's a really big concept, that's a really big impact to hope to have. So breaking down, how will you know if knowledge has changed? How will you measure that? And then thinking about once you've decided that, the evaluation method to use. So we're going to walk through a few of the common evaluation methods that we see in TLEF and OER projects. This is not an extensive list. There are many other things that you could do, like look at different types of course analytics, so engagement with the material, and how students are interacting with the material, feedback around that, looking at things like observations in the classroom, or observations of use with the tool. So there are other methods. These are the ones that we see most commonly. So I'm just going to spend a few minutes talking through them and kind of giving you a sense of sort of the pros and cons of each. When you're thinking about this, oftentimes what we see is someone say like, oh, I'm going to do a survey for this to find out if this resource is working or not. We really, really encourage you to go through those earlier parts of the workshop. So really carefully laying out what, so like the measures and the measures based on the impact you hope to have before you pick a method. Because just because you're familiar with something like a survey doesn't mean that it's necessarily the best way to get the data that you need. So just really being mindful before you pick a method, being thoughtful about what it is that you're hoping to measure. So first, just going to walk through an example of interviews and focus groups. So interviews happen at a smaller scale, usually one-on-one. Focus groups tend to be larger groups of people. But the dos and don'ts around it or so the tips and tricks around it tend to be similar. So we've just lumped them together here. So when you're thinking about doing an interview or focus group, one of the reasons that it might make sense to do it is that you get a more detailed understanding of experiences. So you're able to really dive into the experience of what it was like to use that resource or how they felt while they were using it and whether it made sense, whether it was helpful, whether it felt engaging. You're able to kind of ask more pointed questions and have more of a conversation around it. That being said, you are only able to get data from a subset of participants. If you're using this resource in a class of 200 students, you cannot possibly interview all of them or do a focus group with all of them. So just being mindful that it is a narrow or scope, but you tend to get richer data as a result. Focus groups are really beneficial if you're trying to elicit more of a discussion around unique topics. So if I was in a focus group with Natasha and Will, for example, maybe Natasha would bring up, oh, you know what, when I was using the resource, I actually found it really confusing how to navigate this part. And I felt like I couldn't get to the self-study quizzes because I didn't know where to find them. And Will might bring, echo that and say, yeah, actually, that was really confusing for me too. I hadn't thought about it. But you bring up that the color scheme wasn't great or the dropdown menu was unclear. Or Will might disagree and say, oh, actually, I found that because there was a link on the home page, that made it really easy for me to use. So it's not necessarily about getting consensus, but it's about how one person's response might trigger someone to remember something about their experience or agree or disagree with that feedback. So it's really helpful in that way. On the other hand, with interviews, it's better when maybe the topic is a little bit more personal, so something that's a little bit more sensitive, depending on the content, and really drawing out an individual voice. Another case is if you're worried about things like power dynamics, so not necessarily wanting to do a focus group that involves both faculty members and TAs, but perhaps interviews separately with a faculty member and the TA for a course to get those perspectives. And just really, you can spend a little bit more time to draw out an individual's perspective when you're using an interview. Some general tips when doing interviews is to make sure you have a protocol. Don't just walk into the session with a vague sense of, like, well, I just want to know how they liked it, if they liked it. Do make sure you map out questions in advance, be targeted around what your goals are, be mindful of people's times. It's one thing to just have a conversation, which can be a form of evaluation, but if you're really trying to answer a specific evaluation question, making sure that you're thinking carefully about how to get those answers. Also, think about how to elicit discussions. If you're running a focus group with a group of six to seven students, you might be familiar with the lecturing concept of crickets in a room. You ask a question about what did you like about this resource, and no one says anything. So again, if you have a protocol, you can map out some sort of follow-up questions, some ways to get responses from people to get the conversation going, so that you're not just stuck sitting in a silent room for an hour. Also, again, being mindful of getting student permission and consent, recording the session. And I also recommend, especially in the case of focus groups, having a second person there to help take notes. You're not going to remember all the discussions that happened, and what happens in the moment and how you perceive a situation might not map on to something that you think about later. Oh, one student brought up this perspective, but I don't remember exactly what they said, or I don't remember the context that they were talking about. So having that recording can be really useful to remind you what the conversations were about. Next, talking about surveys. So surveys can provide a wider, larger sample of the experience with the practice. So again, in that example, if you're surveying 200 students, you're likely to get a larger range of responses. So it's a wider sampling of the population. Again, it's a wider sampling of the population, but surveys don't go into as much depth as an interview or focus group do. So you have to be mindful and think about the types of questions that you're asking. Surveys can also be really useful for comparing groups of participants. So if you're interested in looking at two different types of experiences, or you want to learn about, how is there experience using this module in the resource versus this module in the resource? You can compare. If you have a set of questions around engagement or motivation or understanding of the content, you can compare what was student engagement like when they were interacting with this content versus this content and see how it differs. Again, for something like developing a resource, it's really helpful to get feedback on the entire resource, not just one piece of it. Surveys can also really easily be integrated into activities or assignments. So you can be thoughtful about the way that you gather and elicit feedback from your students. So in the example of an OER, if you have them, the example of the self-study quizzes, so you have these self-study quizzes, they're built into Canvas, for example, you could easily add a question or two at the end of the quiz that says, what was your experience like with this quiz? Please rate on the Likert scale how easy it was to navigate the Canvas site or something around, did you find the quizzes to be useful for your learning of X concept? Strongly agree to strongly disagree. And that really ties it into the content itself. You're getting feedback in the moment while they're interacting with the tool and doesn't require a whole separate survey that you're sending out and you don't have to, a survey doesn't have to be dozens of questions. It shouldn't be dozens of questions. You want to keep it short and concise. So being able to integrate it into the activity that they're doing and gathering feedback in that way is really useful. Both for motivating students to do it and for really, you can make a case for, you know, like this, I've integrated something new into my course, I want to check in on how it's working for you. So some tips around surveys are, as I mentioned, keeping it short, focus on the questions that will help you answer your evaluation goals. So this is something that Natasha and I spend, I think a fair amount of time in our roles is reviewing survey questions. And it's always a great idea to, you know, build on a survey and you could put as many questions as you want at the beginning before you share it and then go through each question and say, like, what data will I get from this question? Will this inform whether students are motivated to stay enrolled in this course? Will it inform whether students are learning this key concept? Just kind of going back to those evaluation questions that you developed earlier, based on the impacts you hoped it would have and making sure that those answers are there. Also piloting a survey, results can be really dependent on the question. We see a lot of poorly worded questions or confusing questions or questions that use a lot of jargon that students might not be familiar with or that you as the person doing the evaluation have a good sense of, but the reader might not. So having someone just read through your survey, even, you know, if you have a TA in your course or a student that is involved in the project, having them read through the survey questions and just making sure that what you're asking is coming across the way you intended to. And finally, using a FIPA compliant tool. So at UBC, we have access to Qualtrics, UBC Qualtrics, which stores all the data locally and keeps the student information secure. At this point, I'm just going to make a quick plug for a workshop that we're hosting, Natasha and I are hosting in a few weeks, a survey design workshop. So if and when you get to the point of developing an OER and you'd like some support on survey design, or if you're designing a survey for another purpose, doesn't necessarily have to be the OER. I do encourage you to join our workshop in two weeks, less than two weeks, for some more tips and tricks. The other piece that I want to point out is we talked about surveys, we talked about focus groups, we talked about interviews, and we know that people have different levels of comfort with each of them and what they're familiar with, but we really do encourage people to take more of a holistic approach and look at data collection and evaluation. So something we often do and recommend is using both. So having some quantitative and qualitative feedback. So maybe you do a survey and based on the survey, you realize, hey, students are really struggling with module three and four. Now I'm going to do a focus group to dive in a little bit deeper and find out what it is that they're struggling, why that content is not resonating with them or why they're not learning the way that I intended them to. Or vice versa, maybe you do a focus group and you get some feedback on the general sense of how things are going. Then you can implement a survey that goes to the broader class at large to make sure, okay, did this feedback that I get actually, is this matching up with the general student experience? So using a combination of both can be really rich and informative. So this is just an example here of a way that a little visualization of how you could look at something if this example here of a new peer feedback system. So if a question on a survey that students were asked was, I'd like to see the name of the system or your resource used for giving feedback on other assignments in this course as well. And then students responded on a liquid scale of agree to disagree. So now we have this figure to show a lot of students are agreeing with the statement. And then you'd followed up with some interviews or focus groups. You can also include some short quotes, anonymous quotes, to provide some kind of richer context about like, okay, so students said that they would like to keep using this peer feedback system you developed, your resource. So we have this 60% of students say, this open education resource would be useful. They'd like to keep using it on other assignments. But what is it that they like about it? Or why is it working for them? By doing an interview or some focus groups, you're able to gather that richer data. And so just some examples of things that people might have liked about our imaginary resource, the format of the text on the... Oh, sorry, some things that they might like is that it's anonymous so they're able to give more objective feedback. And then what we see, sorry, on the left is the sort of positive feedback from that 60% who agreed. And then on the right, what you're seeing here is sort of room for improvement. So some people might say anonymity gives people lower quality feedback or the format made it hard to read. So you're able to gather feedback from both perspectives. And I really want to just highlight the importance of that orange or disagreeing or negative feedback. That's really the point of evaluation. Well, that's not the point of... It's one really important point of evaluation. The point of evaluation is to get feedback. And hopefully that feedback is good. But when we think about evaluation as iterative, it's because a resource, a project is almost never perfect. And doing evaluation helps you figure out what it is that students are struggling with or the team that you're working with is struggling with that you can improve and make changes upon. So in the example here, the format of the text made it hard to read. If you're gathering that data early on in the development of your resource, you can just change the format of the text, no problem. You can change it or give students options to select the format that they want. And then you can redo this survey and get feedback to say like, did you like the format of the text? Was the format of the text readable? Then you have that feedback where if you're waiting until you've built the entire resource, you're sending it out to your entire faculty and then you get this feedback that the format was not so good. You've wasted all of that funding period and all of that time not really honing in on what's working and not working and not being able to make those changes as you go. So again, just really sort of beating this idea over the head that evaluation is meant to be an iterative process and it's meant to be an opportunity for you to make improvements and changes to help boost those positive comments and feedback that you get. So I just want to take a break here just because we're quite a small group before I move on to the next section and see if anyone has questions. We're quite ahead of schedule so we have time for a little discussion. Anyone have any questions so far? No questions. Perfect. Okay, so I'll move on. We'll have some time for questions at the end too if people have anything to add. So next I want to move on to once you're at the stage of thinking about collecting and analyzing data, some important considerations. So as I mentioned already, piloting your questions and thinking about brev approval is another important one. So we'll just touch on those briefly. So thinking about now you've decided, okay, this is the resource I'm building. I hope that it impacts students in these ways. Here's how I'm going to ask those questions. Here are the methods I'm going to use to ask those questions. Okay, you're building out your evaluation plan. So some considerations that you should do is establish a timeline with milestones. So thinking about how will you know that things are happening the way that you want them to happen? So who is doing what? Do you have a research assistant that's going to help you design your survey questions or help you by running the focus groups or interviews? What are the things that you need? Do you need to book a room to do the interviews? Do you need some type of incentive for people who participate? Do you need to learn how to use Paltricks? What kind of skills or tangible resources do you need as you move forward in your evaluation? How will you know if a milestone is met? So people think those surveys are short and snappy and easy to do. I'll just keep doing a survey every semester. But at what point do you say, okay, this tool is good enough. This objective is being met. The impact that I wanted to have is there. What kind of data do you need to kind of give you that assurance? The next piece is thinking about other considerations. So some common fallacies that we see as you're developing your project plan. Again, this is around the idea of piloting. So just because you understand something doesn't mean that your participants will understand it. So making sure that you, for any protocol you have, so survey questions, focus group questions, interviews questions, or any other verbal communication you're making with other people is just really making sure that what you're asking makes sense to them. So reducing the jargon, avoiding things like double-barreled questions, just making sure that the questions are being interpreted the way that you intended them to. Because the last thing you want is to collect a whole bunch of data and then realize, oh, I was talking about this module in a specific way or this interaction in a specific way, but students weren't understanding that, so they were talking about something else entirely. Just because something is interesting doesn't make it an evaluation question. So again, with the activities in the worksheet, specifically activities one to three, really narrowing down what your evaluation questions are and being focused on those pieces for gathering your data is really useful. So sometimes people are like, oh, I'll just add this extra question because it might be interesting to know how this is or it might be interesting to know this demographic piece of information. There are problems with asking those sort of extra questions. One is that it can be problematic for your data set. So if you're asking students, for example, about their gender, but you have no intention of using gender as a factor, you have no reason to suspect that gender influences the way that they interact with your resource, don't ask that question. It can lead people to think that you're interested in it or it can influence the way that they respond to the questions because they assume that's the focus of the project where it's really not about that. It's about understanding if this resource is working. So just being really mindful about the questions that you're asking and again, that point of, for each question you ask, going back and saying, how will this help me answer my evaluation question? And having that evaluation question really clearly mapped out will be helpful in that process. Just around determining whether the question is worth asking, consider the relevance, the specificity, and the inclusion. So relevance, literally for every question you have, do you need this item? What information will you get by asking this? Consider the specificity. So how will you analyze this question? So making sure that each question that you ask has a purpose towards the larger goal, that it's not just extra data you're collecting for the sake of it. And thinking about inclusion. So making sure that you're able to capture different voices or that the wording of the question welcomes feedback from everyone in the group. Some final considerations or additional considerations is thinking about how to integrate evaluation to the course, thinking about things like flow time and cost to administer. So if you're doing something like a survey, you'll need someone to help you design those questions. You'll need someone to create the survey, presumably in ball tricks. You'll need someone to analyze or anonymize and collate that data for you. If you're doing something like a focus group or interview, those are much more time consuming. And you need to think about how that process will happen. So we always very strongly discourage if you're teaching a course, don't also be the one doing the interview or focus group. That's a huge ethical do not do. For many reasons, you're unlikely to get the data that you want. Students will feel pressured to respond. There's huge power dynamics at play. Just in keeping student confidentiality in mind, that's also problematic. So thinking about who else needs to be involved in order to make this happen. Thinking about the time required. So if you're asking students to do a survey, again, the idea of keeping it as short as possible so that you're not asking an hour or even half an hour of their time to get you feedback. Likewise with something like a focus group or interview, how can you both provide an incentive and also like a reward for participating as you're taking time away from them for the purpose of gathering feedback. Another piece around time is also thinking about when to administer the focus group survey interview. Avoiding things like midterm season or final exams. Depending on your evaluation question, this might differ, but oftentimes we try to say incorporate it as closely as possible to when students are using the resource. If possible, getting their feedback after they've gone through that self study quiz module as opposed to four months later when they might not remember that they used it at all. Things like that. Thinking about ethics. This is a whole other conversation on its own. Natasha shared some links to a number of different resources. Also in our resource hub that she shared earlier, there's a whole guide around determining whether you need Rev Approval or not. We have a whole workshop on this question. So I should backtrack. Rev is our institutional ethics that covers behavioral research. So it's the Behavioral Research Ethics Board at UBC. And for many of these projects, they likely won't require Rev Approval. Most of the evaluation around OER projects is about looking at sort of more of this quality assurance step evaluating whether the resource is working. But it is a helpful process just to walk through some of the checklists to make sure that you're on the right track. And so Natasha posted a number of links that have many, many resources. We put together a 10 minute video that kind of summarizes the whole hour long workshop we normally host. So it's an easy one to watch. The second or the third one here is a checklist that's provided from the Research Ethics Office that you could kind of just go through and get a little kind of peace of mind on whether you're on the right track of going down the research track or focusing solely on evaluation. The other thing I just want to highlight is that being ethical and requiring ethical approval are two different things. So if your project isn't deemed as research, then you don't need to go through the brev approval process that doesn't exempt you from being ethical in the work that you do. And so things like asking for student consent before you collect any data, anonymizing the data or collecting anonymous data when possible for things like a focus group or interview having someone who's not involved in the course be the one who facilitates those sessions. Things like that are all really important in considering how that data is being collected and used. Also, in terms of the consent form, just really being clear and upfront about how the data is going to be used. It's about being ethical. It's also just part of that is keeping students informed about what the purpose of the project is and how the data will be used. And I found time and time again in evaluation projects that just being upfront with students to say, I'm trying out these new modules or this new resource for the first time, it's really valuable for me to get your honest feedback on whether you like it or whether it's working for you. Just being upfront with them about that before you send a survey or start the focus group, it makes a huge difference. Just being honest about the purpose of what you're doing. There's no need to conceal. It's not this mysterious research process. It's about gathering feedback. And likewise, just being honest, this is how I'm planning to use your data. This is how the data will be stored. All of those considerations helps the students feel more confident and I think more willing and open to share their feedback with you. So moving back to that evaluation cycle, we're not going to spend too much time on these last two pieces here, but just again thinking about evaluation as an iterative process. So once you've collected and analyzed your data, taking time to share your findings. So that doesn't necessarily mean you have to write up a big fancy publication. It can, maybe it will, but it also means having conversations with your colleagues, sharing something at a departmental meeting, attending one of the many CTLT events that we host and sharing your results that way helps other people learn about what you're doing and can encourage new resources or adaption of the resource you developed. So perhaps you make these self-study quizzes and you hear that students really love them in the context of labs, but less in the context of the lectures. That could encourage another colleague to adapt them for their own course. It also involves thinking about how you can modify your pedagogy. So getting feedback from other people, you know, some of the most valuable sessions I'm part of will hear someone say like, oh, like I thought this was going to work. It really didn't, students really didn't like it. Has anyone had a similar experience or has anyone tried doing this in their classroom and having that conversation? That's part of the dissemination process is sharing what you did and what you learned and can help you making that next step to change what you're doing and make a small tweak, you know, change the font size or add a new link to the homepage to make the resource more accessible. Elements like that, which so that you're modifying your pedagogy and then you're starting this cycle over. So what new questions will you ask? What new feedback will be required to help you understand whether the resource you've developed is working for people? So I understand that no one's in the stage of holding an OAR fund funded project right now. But I just want to go over a few examples of what evidence should look like. So when you're thinking about collecting your data and you're thinking about, how will I know if something's happening? Just a few examples of the way that you would want to talk about this information. So sometimes what we see is, when people are talking about their evaluation plan, what they're hoping to do, they'll say something like, we conducted surveys to determine student satisfaction. So what you really want to do is elaborate on what you learned as a result. So when sharing your results with your colleagues in a formalized report, whatever the source might be, being as specific as possible. So some of the examples here, highlighting that you used a Qualtrics survey to understand this new rubric that you developed. That was your resource. You found that a lot of the students say that it helped them prepare for the final exam. That was an impact that you intended as a result. You had hoped that the rubric would help them feel more prepared. That's what you learned. Something we see often in the context of OAR fund is the new open text, which is the practice or the resource in whatever context, saves students a certain amount of money based on how much money they were spending before. So just some examples on ways to think about what you hope will happen as a result of your project. And you can use these as ways to think about how will you develop your evaluation questions. So this is just another example from an OAR project that was funded or conducted in 2017 by Christina Hendricks here at UBC. So this is just an example from a paper that was published on her resource, breaking down all of the research questions and the methods. So again, just another example here of how you might start thinking about the evaluation framework. Finally, just some final resources. I think we've shared all of these throughout the workshop as well, but just a link to them here. And we'll make sure these slides get shared with everyone after the session, so you'll have access to all of the links that were embedded in the slides as well, so you don't have to save all of them. But just some more resources. And then for anyone who's here who's still kind of thinking about what an OER is and whether they should apply for the fund and how to apply for it, if you're thinking about the evaluation component, please feel free to reach out to myself for my colleague Natasha. For all things OER, all elements of it, reach out to Will. And then we also have faculty liaison. So if you're in one of these faculties, someone targeted, Marie is actually one of our participants here today, so another friendly face that you can reach out to to get a little bit more feedback and a little bit more guidance. And now, we've come to the end of the session, so I'm going to stop sharing my screen. We've got quite a bit of time remaining, but we'll open it up and see if anyone has any questions for us. I understand. Yeah, no one's necessarily ready or at the stage to be developing a survey or a tool right now, but if you have questions about the process or something about your project that we can answer, please feel free to go ahead.