The study shows that TiO2 nano sheet with poor hydrogen evolution activity can be used for selective and rapid nitrate reduction to ammonia under acidic conditions when hybridized with iron thalocyanine, resulting in improved efficiency. An alkaline acid hybrid ZN nitrate battery was developed with high open circuit voltage and power density, which can power environmental sulfur recovery and hydrazine, the nitrate fuel cell for simultaneous conversion, and electricity generation, broadening the field of energy conversion. This article was authored by Rong Zhong, Chuan Li, Huilin Kui, and others.


Hi, my name is Dave Yousum, and I'm a professor of neuro radiology at the Johns Hopkins University School of Medicine. Today, I'm going to talk to you about a really exciting thing, and that is a remote fellowship course that we have developed through MRI Online. This remote fellowship will be the first remote fellowship at MRI Online to feature over 100 studies of these 50 cases that include both MR and CT. You will have the opportunity to submit case reports that will then be reviewed and graded by myself. For each of the cases that the student submits, he or she will be able to see images that have been annotated to direct them to the pathology, and in addition, there will be a discussion about the entity, as well as a recent reference in the medical literature for each case. Plus, as a bonus, you will receive also access to three 60-minute lectures, virtually live or on-demand, with the opportunity to ask questions and receive the answers throughout the course. To sign up today, or to get more information, please visit www.mrionline.com. I look forward to working with you in this fellowship. Thank you for your attention.


Thanks for coming. I'm going to make this very brief because I'm going to have a chance to speak later. We have decided to speak in this order, and I'll introduce myself later. I guess I'll say a word now. I'm Peter Suber from the Berkman Center, but that's all I'll say for the time being. Each of the panelists will introduce themselves and then give a few opening remarks, and after that we'll talk to each other, we'll talk to you, we'll take questions, and we quit at 3.15. So first up, Cable Green. Thanks, Peter. Hi, everybody. I think I know everybody. So my job is to just introduce the big concept of open policy. What is it? What are we talking about? Why do we care? Why are we even having this particular panel? So let me start with the big goal, and I'm going to say things that seem obvious, but hopefully woven together, they'll make sense. The big goal that I think most of us share is that everyone in the world deserves the right, and in fact, given the tools that we have today, we could provide access to an affordable, high quality education for everyone on the planet. We all know that we've got the Internet, we've got the affordances of digital things, we've got open licensing, and we've got a mass willingness to share among enough educators producing enough high quality materials that in fact, if we're smart about this and we resource it properly, that we in fact can make that dream come true. So that's my personal starting point. That's why I am here in Boston at the Hewlett-Main. So if that's the goal, then I think the next question is then how do we maximize these tools that we have? How do we maximize the opportunity of open educational resources, of open access around journals, of open science, open data, open government information, et cetera. And part of that, and I ask that question for two big reasons. The most obvious one is a resource question. This takes money to do these things. The good news is it takes significantly less money than the current proprietary commercial model and significantly less, but nevertheless it takes still a large amount of resources. And as we all know, many of us are beneficiaries of the kindness of foundations to spearhead this environment. The bulk of the money by far does not sit in foundations. It sits in public funding. And so part of it is a resource question. How do we move public funds all around the world at all levels of government so that the default that those public funds purchase is open and that closed or proprietary becomes the exception? So how do we get there? One reason is to the sustainability question. How do we have the funds to resource this movement in a way that it can accelerate quickly and we can truly pay for, distribute, et cetera, the highest quality educational and research resource materials? The second reason is that when you change policy, you change the rules and behaviors of folks that operate within those rules. So most recently in the United States there was the Department of Labor grant, this $2 billion grant. CCBuy was required. People who were applying to that grant and I talked to many of those community colleges, they looked at it and said, yeah, yeah, one more requirement, check the box, move on. And then when we went back to them and said, do you realize what you checked? Do you realize what that means? And even after they understood, they said, okay, fine, we don't really care. It's just a requirement to get the money that we're trying to get our hands on so we can do the work that we want to do. Fine, yeah, it makes sense that this is public funds. We should share what we build. To them it was really not an issue. They didn't want to debate it. It was that the rules had changed and to get the resources to do what they wanted to do, they were more than happy to comply with the rules because those were the rules. And the conversation, as far as they were concerned, was over at that point. They wanted to get on with the work. And so I think that's a very interesting and important aspect of open policy. So what is open policy? Very simply, open policy is publicly funded resources are openly licensed resources. Now, it's just, it's even better if publicly funded resources go directly into the public domain where nobody holds the copyright and we all have access, no attribution required. That's, I would argue, ideal. But in many cases, that's a harder argument to make than to allow somebody to keep the copyright and ask that they put or not ask, require as a matter of open policy that they put an open license on what it is that they build. So I'm going to wrap up with just a couple of examples that I've been involved with recently. As many of you know, for the last four years before I joined Creative Commons, I was at the Washington Community and Technical Colleges. We looked internally at our system built, realized that our curriculum was not only out of date but we were not engaged in the OER conversation at all.  We were also not sharing with each other and this was a just looking within our system was a massive inefficiency and our students were suffering for it. And the most obnoxious example that I use over and over again is our one highest enrolled course in just the Washington Community Technical Colleges was English 101 and just Washington, just community college students in Washington spent upwards of $9 million a year buying that one book for that one course and that happened year after year after year. And we spent about $1.2 million and built the entire first 42 courses, highest enrolled courses and had textbooks that were between the cost of zero and $30 and we gave it all away for free and it's all CC by license and now faculty are starting to adopt that. At the system level we adopted an open policy that said if you take money from the system and the nice thing about the policy in Washington Community Colleges is that all the state money, all the federal money, all the foundation money flows through the state board and then out to the colleges and the policy shift was if you take competitive grants from our system you will put a CC by license on you what you build. If you don't like those terms don't apply for the grant. So to be very clear about policies we are talking about requirements. These are not optional conversations. When we're talking about policies we're either talking about law change at national government levels or we're talking about strict policy changes at institution systems, etc. Because you don't want to have these arguments, you want to be very clear that public funds are open. And then one last comment and I'll get off is that frankly this is good public policy. If you sit down with public policy makers and you ask them, you are the stewards of the public taxpayer money that you're responsible for and they will say yes and you ask them, do you feel responsibility to be efficient and effective with those dollars, those federal dollars or those state dollars or those system dollars that you are responsible for? And they say yes I am. I am responsible for that. That's part of my job. And then when you point out the existing systems that they are funding and the existing massive inefficiencies. Again, in my state in K-12 they spend $130 million a year on K-12 textbooks for only one million students and the textbooks are 10 years out of date and there's only paper and the kids aren't allowed to write in them. And that's 10 years on average the textbooks out of date. Many of them are 15 years out of date. I say to the policy makers, do you find that acceptable for an investment of $130 million a year? And they say absolutely not. And the only way to change, not the only way, one of the major ways to change those kinds of inequities, those kinds of massive inefficiencies with the huge amounts of public money that's spent on education estimated at approximately $3 trillion globally a year is to change the open policies. Thank you. Thank you. So thank you, Cable, for framing all the policy discussion. I'm Carolina. I coordinate the OER Brazil project, but I'm also a lawyer working on the open movement for a long time. So I want to also bring a broader perspective and point that to you, but framing it on what's happening in Brazil and what is this discussion we have there, right? So at the multinational level, we have been taking this discussion to forums like WIPO, the World Economic Forum and the Inter-American Bank of Development. So we talk to these folks. So when they go and talk to our governments, we have some positive feedback on that loop. In some forums, more than others, of course, you can imagine how the discussion goes in WIPO, but it's something that we are trying to do within the discussion of exceptions and limitations there, right? And with the expansion of the IP norms through bilateral and multilateral agreements, we are having many issues that are impacting the reform of copyright law in Brazil and actually other Latin American countries that may even make Creative Commons illegal in some parts, right? So in Brazil, you may have heard that we have article in the bill that's being proposed just to change our copyright law saying that any license you would have to register in a notary, right? And Creative Commons doesn't work like that. So if you have that requirement and people don't register for Creative Commons license, that would not be enforceable. So we face these barriers in this broader context and we really need to take those contexts in consideration when we are talking to policy makers and even building alliances, right? We try to bring alliances from our foreign ministers and other ministries, not just the Ministry of Education. We also partner with other sister movements, right? And Brazil is trying to bring what is the positive of OER which is very similar to many other movements such as the open access movement, the open data movement and the free software movement. And Brazil is very strong in free software so of course we need to partner with that movement, right? So what are the other movements in other countries that you can identify and partner with because they already have been heard, right? Brazil, many states have laws that give priority to free software so why not get a ride on that discussion that's already in place and already had to identify the network and the political alliances there? And finally going deep a little bit at the government, state and city level as we discussed in other sections here we all need to understand better our education systems, right? And the question on how to start, start by understanding our education systems and finding similarities cross-border, right? By understanding the educational system in Brazil we could find similarities with US, Poland and other countries and try to cooperate with these countries framing the language for the policies that are being developed. And in that sense we are working with the legislative and the executive and of course you have to build trust and you have to build expertise in the legislative branch and you have to present the numbers. So when all of you say we need metrics, yes we need metrics and all the time and these metrics need to be presented all the time and reinforced and we need economic metrics but you also need impact metrics. So when I was talking earlier today about the Folhas project with these open test book projects from Paraná State they actually presented better grades and they are trying to analyze if the use of open test books was related or not with those better grades. But at the same time sometimes it is good to as Ronaldo also mentioned, think of ways to introduce who we are in ways that the politicians are not paying much attention. So when you have call for proposals or call for bids to develop for example, educational resources in mathematics there are people in the Ministry of Education that are adding at the last minute but you have to have a CC license. So how you build these alliances that at the last moment somebody can drop that tweak on this public purchases process. Finally at the state level Brazil has US. One of the differences is our adoption process happens at the federal level but that doesn't mean that the states are binded to adopt those test books so the states can review that list proposed by our Ministry of Education and then adopt whatever test books they want. So you have space also to work with the states, right? And the states have laws about it and they receive funding from the government the federal government to purchase test books but they also have a lot of money purchasing and developing educational resources. And I see that we were discussing with yesterday night how it happens in the US. So I think it's a cross between what you have in California, Florida and Texas with the open states, right? That sense. So we have some flexibility to work there and how we try to identify within the curriculum places we can insert or we are at these lists that the government put forward has materials that schools should be paying attention to should be adopting because the government said they are good materials to adopt. And at the city level, specifically there was a work with the executive in Sao Paulo city and again was that point that the politician, the secretary of education of the city really liked the idea and moved fast so suddenly we had a decree declaring that everything producing by Sao Paulo city which by the way is the richest city is one of the richest city in the country and has big purchase powers. The publishers are in Sao Paulo, some in Rio. So they have a pretty important domino effect in the system there. They turned the key and said through a decree that everything that's produced and purchased by the city needs to be OER. At the state level, as I mentioned to some of you we already got two positive feedbacks in terms of approvals from two committees at the state level and that's moving really fast so I'm very glad to see that moving. But I also think we have some challenges to face and that I'm here to actually ask your constant help. There's a lot of fear. All the laws that are being pushed forward in Brazil they have language bringing CC by non-commercial share alike and this is an issue for us. We tried to have that to be by but it was something that was really, really difficult to accomplish. Why? Because there's a lot of misinformation about CC. So I think that CC through the Open Policy Academy and the repository has a great role to play helping us to break this misinformation. I don't know if you guys know about the situation with our Ministry of Culture. We had Gilberto Gil, which was a great ally of Creative Commons, close friends with Flasig. They were all the time together all over the world and Gilberto Gil left and we have a Ministry of Culture that is spreading a lot of misinformation of CC and I say misinformation because many of us from this network of people that work in copyright and other areas, we have informed them but they keep the stories in the news they still are pretty misinformed. And finally, and they say CC is not legal in Brazil CC is against authors in Brazil so all those issues, right? And we also need to partner with publishers I think in many situations. So in Brazil the publishers we have like a situation with six major publishers they are multinational publishers and they have the monopoly in the test book industry and they will not move for many years. That will happen, we don't have the same pressure as you guys have in the US. We don't have the pressure from user test books. People do not adopt user test books. One minute? Okay, so the pressures are different so the weight of the publishers are different so we've been trying to talk to them about new business models so maybe some research on open business models in this area would be really relevant and they actually call us and say do you know anybody to work with us to do some beta testing on open business models and there is still nobody in Brazil on the business side on that so I think it's always important to see what is the role of everybody in this ecosystem and try to move everybody in a positive way. Thanks, Carolina and Cable, I think you've set the stage really well. I'm Catherine Gorge, I'm the project director of OER Africa and we work, it's an initiative of SAIDI and we work primarily in the higher education sector in Africa working with universities and trying to sensitize them about how open educational resources can be a means of improving teaching and learning. Most of these higher education institutions that we work with do our government funded so speaks to the point that Cable was making about releasing these materials as part of the public good. So one of our first tasks when engaging faculty and staff at universities on OERs to demystify the term because it means so many different things to so many different people and we explained to them that at its core OER is a legal concept, it's about how you license teaching and learning materials in such a way that they can be easily shared with others and that this is different than from copyright which actually closes such materials off from being able to be shared. And then we share with faculty some of the possibilities that OER creates, particularly within the context in which we work which are typified by scarce human and material resources to deliver quality education and ever increasing demand for that high quality education and an environment within which technology and connectivity are becoming slowly but increasingly available as tools for delivering higher education. And so we suggest to them that OER is filled with the possibilities. Possibilities of integrating quality OER into course design as a cost effective means of updating and modernizing curriculum. The possibility of using OER as a stepping stone towards the development of human capacity to design course materials in curricula and the possibility of using OER as an impetus to revisit and modify curriculum on a regular and systematic basis and thereby ensure that it's continually being renewed, continually being made relevant to the world of work and to border society as a whole. So over the past four years when we've been working with the universities in Africa we've come to see that engagement with OER does not necessarily need to be a process of volume terrorism or even something that we just do because we think it's a good thing to do. We've observed that OER, if it is not useful or relevant to an institution is just not going to gain traction. And OER can only be useful if it contributes in some tangible ways to improving both teaching and learning. We've found that when the process of development and adaptation of OER is guided by a very real need for resources and the relevance of the resources available then the use of openly licensed educational materials has provided very practical opportunities for learning or relearning pedagogical skills such as materials development and course design acquiring technical skills such as using search engines and properly labeling resources. Kathy was talking about metadata earlier so that you're improving the searchability, the findability of these resources and it's also served the need for students to acquire, for example, basic clinical or nursing skills where the relevant resources either didn't exist at all or didn't exist in the languages that they were required or they were simply too expensive to acquire legally. And in many instances, collaborative endeavor has been an integral part of these processes of creating these materials. So as we've advocated for the use, the adaptation and sharing of OER within and between institutions of higher education, we've learned the importance of being absolutely explicit about what an institutional investment in OER might comprise. On Tuesday, Barbara mentioned that with every opportunity comes a cost or a challenge and we have to be clear about that from the beginning. People don't like those kinds of nasty surprises coming up on them in the middle of things. Time is one of those costs. It needs to be accounted for. It needs to be paid for. Faculty time to create OER, to search for OER. Awareness has to be built. Links to OER networks, links to repositories. Using these networks and these repositories as a way of rendering visible Africa's intellectual capital and avoiding the marginalization that has so often been the bane of Africa's participation in the global knowledge economy. Champions, they need to be nurtured. They can be individuals. They can be units within institutions. But they need to have credibility. They need to have passion. And they need to have a place within the institutional hierarchy that actually allows them to either develop or to implement the policies that will allow OER to be integrated within the institution. And finally, those guidelines and those policies. They have to be designed in such a way that both faculty and administration know how to do OER in a way that does not contradict either the university's vision or mission or existing policies. And so we found that when we're working with faculty, we found that an over-reliance on lectures as a means of delivering the curriculum can be tempered by the judicious use of OER as part of a planned investment in resource-based learning. But an institution's strategic decision to use to employ resources, resource-based learning can only be sustainable if it's supported by a policy framework that recognizes and supports the use of faculty time in devising those sorts of materials. The sorts of materials that will encourage students to solve problems by demonstrating creative and critical thinking. And we've also found that it's imperative that those who are expected to make use of new technological initiatives are not just interested in working in new ways. They're not just keen and eager, but they're actually supported and equipped with the skills and the tools to make use of and exploit these new technologies. So for example, in many universities that we've worked with, the IT department has been that department that comes round to service your computer when it's on the blink, or to help you to sort out computer glitches. But more and more, if people are developing resources that they need to put online, they need to start thinking about how you design resources, they need to be thinking about how you stitch them into a curriculum. So these elements of pedagogy, of course, design that weren't necessarily part of people's training unless they were taught as teachers, rather than as subject matter experts, which is why faculty are usually hired, become increasingly important. So we found that the existence and the implementation of a policy framework that supports this progression of faculty from chalk and talk to this more facilitatory role of being a guide on the side, a coach was the term that was used earlier in the earlier days of this conference. These sorts of supports are imperative. And we've also learned working with our partners that unlike at a national level that Cable was referring to, at a university level, at an institutional level, policy is actually a set of guidelines. It justifies the allocation of resources. But it doesn't actually, it's not really a means of compelling faculty to do anything. It's very difficult to get university faculty to do things that they don't want to do. There are always going to be those faculty who are quite content to do things the way they've always done them. And so it's important then to be working with the willing, not to spend too much time working with those who don't want to move with you. And so we found that an OER policy framework needs to cover at least four main areas. We're winding down now. And the first is that it has to provide clarity on IP that is created during the course of employment or during the course of study. It needs to explain how materials created while you're working at that institution can be shared with others or cannot be shared with others. Policy needs to talk about, it needs to provide clarity in terms of the HR guidelines about whether, for example, the creation of artifacts like learning resources constitutes part of the job description of staff and what are the implications for staff development, for performance management, for remuneration, for promotion. Does publication in an open access journal count for the same as publication in a peer-reviewed journal? ICT is another very important area that an OER policy needs to touch on. And this is about access to a use of appropriate software, hardware, access to the internet, technical support and so forth. And the last area then is that of materials development and quality assurance so that you're ensuring that materials that are created and shared as open by your institution are actually appropriate and that they're of a quality that you are proud to share with the rest of the world. And we're clear that both at the level of designing policy and implementing policy, there'll be lots of successes and there'll be lots of failures and we need to learn from both of them. And we're also aware that institutional policy, however good to be absolutely effective, probably needs support at both a national level and an international level, the sorts of things that Stamenka and Sir John were talking about yesterday or the day before. In the past few days, we've heard OER described as a way in, as a Trojan horse, as a pathway, as a disruption and we feel that whatever term you want to use to describe OER if it's a way of getting us to focus on improving teaching and learning on making sure that our faculty and our students are equipped with the 21st century skills of critical thinking, problem solving, creativity, working with others that we're creating people who are able to constantly evolve as a global economy and the world of work is constantly involving then OER can only be a good thing. Conversely, the risk is that if we allow ourselves to become overwhelmed by the sorts of gaps that Cathy was just talking about within the OER infrastructure, we might actually just do nothing. So we have to do something. We can't afford to miss the opportunity to use OER and to use OER policy as a tool to harness the time and the resources needed to improve teaching and learning. Good afternoon. My name is Doug Levin. I'm an executive director of the State Educational Technology Directors Association. I'm very pleased to be here and actually to be following all of my colleagues here to my right. I will echo, I think, and maybe elaborate on some of the points that many of them have made. I almost feel apologetic to say that I work in formal K-12 education in the United States. It has had a bit of a bad rap over the course of the couple of days here. And the folks that I work with actually are the technology leadership in state departments of education. Jeff Mao in the audience serves in that role in the state of Maine. He's a very active chair of our board at the moment. And we are working in coalition with a number of Washington, DC-based organizations to help coordinate advocacy activities for those in the community. And Linda Pittenger from CCSO and Reg Lifty from Education Council are part of that team. Susan Patrick from MyNACL as well. And then certainly welcoming many other friendly faces in this room and also not in this room who are part of that. And really collectively I think our goal is to cross this chasm to make OER mainstream. So what I want to do is talk a little bit about why we believe that policy is really an important pillar of this ecosystem, the sorts of policies that we need, but not so much the arguments for why OER. I think we'd love to talk about that. We can certainly talk a lot about that or even about the specific U.S. opportunities. But I do want to share a few principles I think as we think about OER policy advocacy. So why policy? So K-12 in the U.S., it is a public institution. It is highly regulated industry at the federal, state and local levels. Very complex governance structure with layers of legislation, regulation and policy, some of which is circumscribed by the courts. And it ends up prescribing how resources are spent, whether that's time or money. We talked a lot about innovative business models and innovation and disruption, but at the end of the day, we're talking about the market for publishers. Well the government makes the market, period. So the publishing industry is selling what formal education is buying. So therefore if we want to change the market dynamics ultimately you need to change the behavior of the buyer, which is the government. And then thirdly, those who currently benefit from how the system is structured understand this deeply. They know how to influence, to seek advantage, and to either stop or slow down the pace of change. So broadly speaking, if you're interested in, it seems to me that we absolutely need a change in this policy set, broadly speaking. And really at three levels, at least at two levels. So first is no matter what, you need policy that even allows the possibility for OER to arise. And certainly in things like textbook policy, there's fundamental policy that is completely, if not disallowing OER is incredibly OER unfriendly. So that needs to be stripped away, just to even get onto a level playing field. Secondly, you could talk about sort of OER friendly policy, policies that would allow OER to sustain and flourish. But they may not be, quote unquote, open policies. But if any, an aggregator or an OER publisher was interested in jumping into the market, they would have the same opportunity to do so as a commercial publisher. And then finally, we've heard about open policy, right? So a policy that is explicit about changing the current model and mandating the use of open. So this may be the sort of holy grail of what folks are talking about. So a few challenges in thinking about OER advocacy from my personal perspective. OER is not the problem, it's a solution, and policy gets made to address sort of a crisis or a problem. Every crisis we define has a constituency that we're sort of taking on, saying you're not doing your job well enough, and requires then different allies to move it and make it happen. For many of us, including myself, I think the easiest hook in K-12 is textbook policy. I'd say that warts and all. It is not perfect. The textbook model, I would argue if at the end of the day where I would love to see things if I could wave my wand is not that we have a proprietary print textbook, and then it's just an open textbook. I think that's helpful. It's a step in the right direction. It's not about the changing of the teaching and learning and the whole process, and how students and teachers are interacting and continuously improving their work. That's not all the way there. I'd say thirdly that there's more than one way forward, and that language gets in the way and that there is no neutral language. Even Kathy's talk previous year, she says we shouldn't say free, we should say no cost. Well, somewhat semantics, Nicole and I were joking yesterday. We were like, let's invent some new words. Well, if it was easy, we couldn't do it, and it depends. The words mean different things to different people. In different conversations, you have different ways to talk about this. For some, it is a values discussion, particularly in education. It's very compelling to talk about open, and I think people get it like green. I get it. For others, the lead may be digital, and certainly my organization works with technologists. It's an obvious hook. We've heard a lot about technology here, and there's an intersection. It's absolutely not the same thing, but they're related. But there is no one model or no one way, and in the policy discussion, you need to be flexible about that. I would say that passing the policy is the beginning and not the end. It may be a really hard-fought, difficult fight to get it passed, but I think it was pointed out, well, it's hard to mandate quality implementation, or even implementation at all, in some cases.  There are always unintended consequences, and we have examples already of policies that I think would be very encouraging. State of Texas passed open-source textbook legislation, but has yet to actually get anything through that process of any substance. Virginia Legislature launched a Virginia Open Education Curriculum Board that is meeting for a couple of years and is now doing a lot of naval gazing. I serve on that board. It's a source of some frustration. And even the federal government has made statements and passed policy, but implementation is uneven. So at the end of the day, we have to remember the goal is not the policy change, the goal is the change in practice.  And so, next point, one bad example sort of spoils the bunch. Every time we spin up a policy initiative, whether it is really just a press release opportunity or even something much more substantive, and there are challenges with it, it does reflect back on to future conversations with folks. So we all have a stake every time someone steps forward and said, this is an OER policy, this is there's a speech, there's a press release, there's a policy pass. We all have a stake in that succeeding because we're all going to live with the consequences. Yesterday, there was even a conversation with Boundless, which is now facing a lawsuit. So it's still converges. That is going to reflect on everyone, no matter the specifics of the case or of that company. Two more points. Policy advocacy can be shockingly simplistic and how it works. But the details end up mattering quite a bit. If for no other reason than to understand sort of the devil that you're creating for yourself and the details. We talk about Common Core as a huge opportunity in K-12. I absolutely believe that it is. Get down to brass tacks. It gets awful complicated on what is actually common and how common it will be over time. Technical infrastructure is lacking and there's a lot of complicated things that need to happen there. We can't mistake a clear view for a short distance. The choice of the open license, Carlina talked about, huge implications. So these are really tough issues and in the policy process, you may not really be able to deal with them but you really do need to understand what you're driving at. At least from my experience, if you're not at the table, you're for lunch. So if the frame is sort of textbook innovation, anytime there's a conversation about that, the open community has to be at the table. You can't walk away when we say that it has to be open and they say, what are you talking about? No and you take your marbles and go home. It doesn't work like that. You need to come to the table, keep yourself at the table. The opportunities will present themselves. Relationships will be built. The example was given that, you know, the best case scenario is you have a policy being written and the last minute a little piece of language is inserted and it skates right through. That only happens from building the relationships that happen over time. So it's really important this engagement is really important and so the question was raised in the prior session about engagement with some of the trade industry groups. It's important to do. It's not always friendly. That's okay, but you need to have those lines of communication and keep going back at the Apple. So I know we have a lot to talk about, so I'll just stop at that point. Hi, everyone. Nicole Allen from the Student Public Interest Research Groups. I work with students and faculty across the country, mostly focusing on lowering the cost of higher education textbooks here in the U.S. and thanks, Doug, and to the other speakers for kind of setting it up. I'm going to actually continue on some of the themes that Doug brought up just talking about higher education, formal higher education and how policy fits in there and just some of the things that we as a community should think about in terms of actual advocacy and how we're approaching policy as a community. So I think for higher education it's a little bit different the way the market works than K-12. I always say it's equally as screwed up, but in a different way. So with textbooks in higher ed, the decision maker is the individual faculty member, sometimes the committee. So it's a lot more granular and demand is highly disaggregated. And each faculty member actually has a First Amendment right to select the materials most appropriate for their classes and that's a right that we always need to respect with policy and can't interfere with. So policies tend to be a lot more market based because to really change things fundamentally you do need to change the market. And they kind of fall into three categories. So one is creating supply. So supply of OER. Cable mentioned a few ways to do that, requiring materials to be openly licensed grant programs. There's legislation in California to create a library of materials similar to the library created in Washington State. And while that's great, making sure that those materials are adopted it needs to be part of that too. So the second piece is just changing faculty behavior and a lot of times this is like voluntary guidelines that faculty senates, faculty unions can adopt or institutions can adopt to promote best practices in the institution. And then the third piece is to kind of change the rules of the market or at least level the playing field so that materials like OER can compete with traditional expensive textbooks. And one example of that is the Higher Education Opportunity Act which passed in 2008 containing policies that required textbook publishers to disclose price information to faculty when they're marketing their textbooks. So OER doesn't have a sales force to go out there and put materials on faculty members' desks and the publishers have that. And as a result, a lot of times OER just never gets on the table. And at least by putting price on the table it makes the faculty members aware of how expensive the materials are. And then when an option like OER comes up or open textbooks, it's a lot more appealing. So I guess the point of that is that it's possible to have OER-friendly policy without even mentioning OER. It's about creating a market where OER can play in an environment where OER can flourish. So the other thing I wanted to cover is just sort of how we as a movement can be more effective advocating for OER, as Doug mentioned. You know, it can be challenging to go and talk to decision makers about OER because chances are most of the people you're going to talk to, like policy makers, have never heard of it. Or if they are, they don't understand it. So it's really important that we are very deliberate in our approaches. I mean, politicians deal with hundreds of different issues every day. At the state level, at the federal level, members of Congress, state legislators, government, like these people tend to have a kind of low-level understanding across many, many different issues. So to focus on communicating the important points without getting too deep into details is really, really important. So keeping things very simple, not using language like you should never go into a politician's office and mention, yeah, our materials are licensed, CCBYNCSA, they look at you like you're crazy. And it would take half an hour to explain what that means. And it's also very important to back things up with facts, making claims like OER is going to improve education or, you know, flipping the classroom will lead to better learning outcomes. If you don't have the evidence to back that up, it's really not going to help anything because when you walk out of that office, it's just rhetoric. It's definitely the fact sheet that demonstrates it increased learning outcomes 50% or 10% or whatever it is. It can be a lot more effective because you can guarantee that when looking at adopting a new model, comparing it to the current model, which isn't risky and safe and just demonstrating in cold hard facts that there's improvement. So another point, just a general principle that's very, very important for grassroots organizing especially, that you want to meet people where they're at. You want to start where they are and bring them to where you are. And chances are where they're starting is a long ways away from where you are. So one thing that we found particularly helpful in our advocacy work is to start with textbooks and textbook costs because that's something that everybody can relate to, especially anybody who has kids in college. And start there, talk about the problem, talk about how messed up the market is and how OER can be a solution to that. And then specifically actually focusing on the form of OER that is most familiar, which is open textbooks, which look, feel, and act like textbooks like they are now, accept their OER and can do all these other great things. And I think it's important... Am I doing okay on time? Okay. So it's important to remember with policy if only a well-reasoned argument was enough to enact policy. It's just, unfortunately, it's not that simple. Like right now we have all the technology we need to solve global warming. But we haven't solved global warming yet because policy is an incredibly complex thing to work on. So when we're approaching policy it's going to be a political process and to be realistic about that and think about ways to motivate decision makers to make decisions that will support OER policy is really important. Not just why is OER good, but why is OER beneficial to this decision maker? Why is it in their self-interest to promote OER? Just an example, I was trying to convince a dean at the University of Minnesota to launch an open textbook catalog and the thing that finally tipped the scales was offering to promote them in the media and get them a lot of positive media attention and something like that can be the thing that enacts the policy versus just a really good argument of why it's important. So just, I'm wrapping up, quickly you want to echo Doug on how important it is that policy isn't the goal. It's the thing that the policy wants to accomplish. That's the goal. So making sure we don't stop once we pass the policy and have a very clear vision for what happens on the other side of the tunnel. And then finally, just, there's a resource in this community called the OER Advocacy List Serve, which has a lot of the people in the room actually on it and just it's important for members of the community as you identify roadblocks in your work that policy could possibly solve to let us know about what those are and just ways that the community can talk about solving them. So with that, I will turn it over to you, Peter. Thank you all. As I stand up here, I realize I have six to eight minutes and the whole session has only 15. So we're just about to use up our time in these opening remarks. So I'm going to try to come in early and then open it up for questions so we can at least hear some of them before we have to quit and go to break. I probably know the least about OER of all the people in the panel, of all the people in the room. I've spent 10 years working on open access to peer reviewed research. And when I started, that was almost indistinguishable from OER and from open data. But since then, each of these separate movements or subtopics has been so successful that you practically have to specialize to be on the cutting edge and know everything that's going on. And I specialized in peer reviewed research. So my firsthand familiarity is dated. On the other hand, I've spent my time working on policies for open access to research. And I'm aware that there are some important similarities between these two movements and some very important differences. And I'm going to try to make some recommendations based on policy advocacy in the open access world where I think there is some overlap, where I think the similarities justify the recommendations. But if I'm wrong about the overlap, I'm sure you'll let me know. And again, I want to come in short. Six goals or six things that would be desirable to do in policy advocacy. First, know what policy you want. Have a model policy. At least have an idea of what's attainable today. While you're thinking this over, distinguish between the best you can get today in current political circumstances, giving all the competing interests, giving the level of ignorance and misunderstanding from what you would really like in the best of all possible worlds. Go for what you can get, but bear in mind what would be even better. Distinguish the different kinds of policies that might be appropriate for different kinds of policy makers. Governments can institute one kind of policy, and schools can institute another kind. One kind of policy might apply to grantees who are seeking funding. Another might apply to teachers who teach in the classroom. Have an idea of what would be a good policy or even a model policy for each of these policy makers, which of these targets or constituents. I don't think they all have to be the same. And don't think, just to repeat a point a little bit, that the best you can get today is the best you can ever get. And don't think that because you have to compromise to get something adopted today, that you'll never get what would be most desirable. Second, get some endorsements for your model policy or for your good policy ideas from organizations that actually carry weights with policy makers. If it's just your good idea when you're standing in the office of the legislator, it may not carry as much weight as if you have an endorsement from, let's say, Hewlett or let's say Creative Commons. And there are a lot of weighty organizations that would endorse a good policy if they saw the strategic value in that and if they agreed with you that it was a good policy. So amplify your voice with the voice of important organizations. Organizations that aren't just important to you and the subject matter, but who are also important to the policy makers. Third, get your policies adopted someplace. Examples are better than arguments. If you have a policy in place somewhere, then some school, some government, some institution, some school district is actually learning about how that policy works. It's learning how to improve the policy that leads to the next one, which is do some research on the effects of these policies. Once you have some policies in place, you can gather data on how well they work. You can write case studies on how well they work, including the ones that don't work very well. You can learn from the mistakes. Next, try to develop some best practices on how to adopt these policies. If you start to get adoptions across several different kinds of institutions, several different kinds of governments, schools and so on, how did you do that? What did it take? What arguments worked? What incentives worked like promising to publicize in the media? Things that other people might not expect as they go into the office of a policymaker to make their case. There's already a lot of experience in advocating policy, and therefore there are a lot of recommendations and best practices out there, but is anybody collecting them together? If you were starting from scratch with your school district or your legislature, would you know where to go to collect the best practices or to get the reflections and reminiscences of other people who have tried this before? Would you know how to learn from the mistakes of other people who have tried this before? Probably not. This is something that took a long time to do in the open access world. So lots of separate initiatives are necessary, but somebody should be collecting together the experience of all the people conducting those separate initiatives. And this includes collecting together answers to frequently asked questions, frequently heard objections, and frequently heard misunderstandings. I think one of the most common obstacles to OER is probably the same as it is for open access, which is ignorance and misunderstanding. Lots of the panelists have talked about mystification, confusion, simple ignorance. Who's writing the FAQ for that? If you are dealing with policy makers or teachers or administrators or parents who don't understand basic concepts, you could try to explain it yourself. You may be very good at it, but maybe somebody else has done it better. Maybe there's a compendium of these frequently asked questions that you learned from and prepare yourself as you go into these meetings. When you write case studies or do research on policies that actually work or have not worked, focus on all the benefits that they bring, the problems that they actually solve. Don't just focus on cost. Again, the panelists have made a good point saying OER have to work, they have to be relevant, they have to be useful, they have to be high in quality. If you simply make the cost argument, you're feeding into a misunderstanding, the cost isn't the only issue. We all know that you can slash costs by slashing quality. You've got to show that this is less expensive and either as good or better or capable of becoming better. Do you have evidence for that? Do the case studies show it? Do the data show it? As you do the case studies and the research based on the examples that have been adopted, focus on quality as well as cost. Focus on all the benefits that you can put your finger on. Finally, get some discussion about the buzz going about the actual examples that have been adopted. People should be talking about the policies that have been adopted because, first of all, they may not be as good as they could be. Who's going to identify those weaknesses if not the people who know the issues and who care? If they are good, who's going to spread the word about the fact that they're good? Who's going to persuade the next school district or the next state to adopt a similar policy? Only the people who are tuned in and only the people who care. If this buzz and discussion get going, all the other recommendations I've been making, advocating policy is not the only way to advocate for OER. You need motivated teachers. You need motivated administrators, motivated policy makers, motivated parents. Well, how do you get them? Well, you get that in part through this buzz. Policy change doesn't create this motivation. It opens the door to people who are already motivated. But if policies create buzz, buzz can turn around and create policies. So don't stop at adopting a good policy. As Doug said, policy isn't the goal. Changing practice is the goal. But you don't change practice simply through adopting the right policy. Get the discussion going, and then you can spread good policies elsewhere and you can improve the policies that we already have. Okay, I'm going to stop there, which I guess was just about on time anyway. And yes, Carol. It's just a rapid comment about the FAQ actually. That I think it's something really, really important. In Brazil, we are very few dedicated to the OER, and we are funded by the Open Society Foundation. And we really needed to find a way to replicate the impact. We cannot be everywhere all the time. So the idea was to build a FAQ. And I spent days looking in all these projects that are here in this meeting. And I couldn't find one fact that I could just, for example, translate it. So I spent days having to build... Of course, I had a lot of material that I used, but I had to spend days having to talk about is there quality? Are there business models? And of course, trying to point to a lot of bibliography to complement. But again, the bibliography is in English, right? And we had to write the FAQ in Portuguese. So I think if we are... We need to think about the FAQ that relates all this discussion to policy, because people need them. It's not just policy templates, which are also necessary, and Creative Commons is working on that, but also these fast, small things that we can spread around. I agree. My question is actually related to that comment, which is we're not going to come up with the strategy in five minutes, but since everyone's here, what are the things we could do that would help state and federal level, as well as Africa, Brazil, Europe? What are those things that would help at the issue level, and maybe not in a specific jurisdiction? And can we identify a few of those things? Concrete actions. Yeah. Is that what you're asking about? Yeah. Yes, let's hear about some concrete actions. Well, so it's a good question. I think the first thing, and it's been a little challenging for me even at this event, a lot of cognitive distances in the opening day, because we're talking about informal and formal and K-12 on higher ed and domestic international, and that's not how policy is done. So I don't know if we need perhaps working groups. So, Nicole and I, it's important for everyone to share information, because I think the challenges that we face at a general level are the same. The specifics are very different, and actually some of the dynamics and circumstances are opposite from higher ed to K-12. So I think a lot of part of it is let's just get more granular, right? And then I think the examples that, the sort of activities that Peter talked about, clear policy set, model legislation, case studies, research, it gets much clearer. I think at a generic level, it's just, it doesn't work, I don't think. Is that to say there's nothing that would help Africa, Brazil, Europe, D.C., state level? Okay, so let me jump in. First of all, if everybody wants to stay during the break and go get some drinks, come back, we'll stay. So let me say that first. So as you know, Creative Commons has a global network of affiliates. We've got teams in 72 countries around the world, and they came together last fall in Warsaw, and this was one of the things they asked for. They said, we need all the things that you all have been bringing up. We need support, we need slide decks, we need FAQs, we need talking points, we need examples of existing legislation, we need institutional policies, and we don't just need them in OER. We need them in open access, we need them in open data, open science, public sector information policies, because we as policy makers at different levels in different countries have different policy opportunities at different points in time. And we don't want to do what Carolina had to do, spend two days if somebody else has already built all this stuff, right? And this was, by the way, in response to me, only six months into my job saying, what should I be working on? And that was one of the answers. So Creative Commons is going to do a couple of things. One, you've probably saw, we've already pushed out, we're building, really replicating what the open access movement did with RoarMap, which is a list of really nice open access policies that you can go to and see if you want to build an open access policy. We're right now simply pulling together a list of the OER policies that are around the world, and then we'll curate that. It'll be on a wiki, anybody can go update it. That's not, that's part of the solution, but that's this much, right? We're also, we'll be putting together and we'll be working with many of the people in the room to build this an open policy institute, which really seeks to address a lot of these questions. And the idea of it, and I'll keep this very brief, but the idea of it is sort of big tent, we'll seek out the experts in the world who have particular expertise in their niche area of open policy. So for example, open access policy is really critical. There are, you know, the leaders in open access policy are people like Peter and the folks at Spark and others, right? Creative Commons is not going to try to say anything about open access policy. We're going to rope these folks in to have ask them what are the FAQs, what are the slide decks, what are the sample policies look like, et cetera. We're going to reach out to the, you know, the best folks in who are, you know, like, you know, Wayne sitting in the top row, right? Wayne has a unique experience and skill set in how do you create that bridge institution that floats between the open space and traditional institutions that seek accreditation, right? OER University is unique in that and they've got things to say about what sort of policies work there, et cetera. And so the idea is this will be a place that anybody in the world can come to and say, I'm an open advocate in Ghana and I've got an opportunity three weeks from now because I'm in the president's cabinet. This is a real example. One of the CC affiliates leads of Ghana is the Secretary of Education. And it just might happen that her policy opportunities, because as Doug said, there's different opportunities at different points in time. You've got to be at the table. She might say, I don't have any OER policy opportunities right now, but I have an open access policy opportunity and one on public sector information, right? And that's the play. What she needs is the slide decks, the arguments. What are the commercial publishers going to throw at her? What's Elsevier going to come do in Ghana to try to screw this up? And what are the counterarguments and what's the sample policies and show me the five-on-roar map that I need to look at and what speech should my president give when he stands up in front of the legislature three weeks from now? Can you put that together for me? And that's what we're going to try to facilitate. In the open access world, we call that the SWAT team. When a question comes up and somebody has a chance, you call the right SWAT team and you get the right advice very fast. Sir. So this isn't really a question, but I've been listening over the last two days. And I mean, I think that there's... something for me really interesting is about the proliferation of openly accessible content. I have it licensed, which I think illustrates the fundamental issue that because ICT has moved the tools of production outside of the traditional environments that we've had for the last couple of hundred years, which is the traditional educational institutions and the publishers, we've actually fundamentally disrupted the power relationships in education. What I think we're in danger of doing at the moment in OER is by making things like open textbooks the focus of policy change. What we're actually doing is we're helping people to feel as if the change is a familiar one again, which means that what we're actually doing is reinventing a policy environment that's educationally fundamentally the same as the one we came from, even if some of the business rules change because we can cheaply produce that textbook. Instead of actually focusing on the fundamental educational transformation that needs to take place because of that disrupted power relationship. And it really does worry me that the policy change focus is so exclusively on issues like staff incentives and open access policies and copyright policies. I'm not knocking any of those. Those are engagements. We have an OER Africa all the time and they are critically important. But it seems to me if we're not also tackling the much more fundamental problems of curriculum and the way in which we think teaching and learning ought to happen and what we mean by content and how content gets used in educational environments, then I think what we're really doing is we're allowing OER to be co-opted by the mainstream and regurgitated as something that's fundamentally untransformative. And so I think that if we don't tackle that policy challenge, in the next couple of years the entire transformative potential of OER is just going to be completely blunted. I'd like to hear from the panelists on that but first I have to make a little announcement. We are past 3.15 and I was just told that one consequence is that the session will no longer be recorded.


Today on the menu we have word classification. So let me start with a list of words for, well, foods. Here, there's grapefruit, cauliflower, green bean, gorgonzola, etc. Also I have a column that describes the category of food, fruits, vegetables, dairies, you get the picture. Now my goal is to make a classifier that, given a food, can predict its type. So I'd like it to tell me if a clementine is a fruit or a vegetable. And what kind of food is camembert? Now the first step is to load the data into orange. I just drag in the excel file and open the file widget. Now you can also use the datasets widget in orange, just look for food words. Okay, there I have 108 instances of data. There's a text column called food and an extra feature called category, which I'd like to use as my designated target. So I'll have to change its role to reflect this. Okay, now I just need to remember to actually press apply for this change to take effect. Just like in the other text mining videos, we'll have to represent our words as numbers in order to develop a classifier. Of course we already know that we can do this by embedding. Okay, first I'll need to tell orange I'm working with a corpus of documents. So I'll bring up the corpus widget. Here I'll use the food feature as the document title and a text feature. Also, I'll make sure that this is all in English. Okay, now I can embed my words with fast text. Then just check out its output in a data table. It looks okay, there's a category column, a text column with names of various foods, and then 300 numerical features representing each word. Now I'll do two things. First, I'll estimate how accurately logistic regression models my data. And second, I'll classify some new foods using the model built from the existing data. And just as a warning, I'll be using some widgets that we've covered in our introduction to data science series. So you can go check those out if you want a more thorough explanation. The links will be in the description below. Now let's get started with accuracy estimation. I'll use cross validation in the test and score widget and attach the data and the logistic regression learner. Now we already have some results. Fivefold cross validation estimates that classification accuracy of logistic regression at 98%. This means there's just a handful of misclassifications. So I should be able to take a closer look at them with the confusion matrix widget. And indeed, the model misclassified two vegetables. Now I can select the cell with misclassifications and add another data table to the output to see that our model had trouble with ginger and pumpkin. Okay, now remember, we only have 108 words or foods in our data. We could further improve the performance of our classifier by adding some more items. You're free to try this independently. Now my second goal for the day is to classify some foods that are outside my current dataset. Now I could create another data file in Excel, but instead I'll use the word list widget. So let's try to predict the types for date, Clementine, Camembert and Gouda. I'll send these words to the corpus widget. Yes, they are also in English. And then get their embeddings as well. Now remember, it's always a good idea to double check results in a data table. Okay, I have the words and their embeddings and also an extra unused column that displays whether the word was selected in the word list. Now it's time for classification. I'll remove the test and score and confusion matrix widgets and pass all my training data directly to logistic regression. I'll just let it build a model. Then I'll send that model to the predictions widget. Now the data I'd like to make predictions on will come from this other pipeline that starts with word list. Great. Date and Clementine are fruits and Camembert and Gouda are dairy products, just like you would expect. Now there are a few more things I could do here, but that's always the case. For example, I could use Chesney to visualize the data and see how the groups correspond to my categories. Or I could check if embedding with Esper changes the accuracy, which it does. Or I could try even more words to see where my model succeeds and where it fails. And these are all things I urge you to try for yourself. But in the meantime, I have to head on to text mining documents instead of just words. So I'll see you next time.


Welcome to the last talk on the last day. Aww. But that's okay, because we've got Tom. Aww. Tom. No. Okay. That's all the heckling you get. Come on. Let's take it down. So Tom Eastman is formerly of Catalyst and of Seistak as a lead security consultant type person. He's really scary and knows a lot of things, so let him scare you in a good way. Hi. Apparently, I am that scary. I don't know why. I'm not a security person. I'm a Python dev. I'm a Django user. I've been using Python for 15 years. Django for close to 10, and going to be talking about ways to try and make things a little bit less scary. But I'm going to do that by leading us down a bit of a thought experiment, which might not be fun for everybody. Who in this room works on web applications, either on the public internet or as a private internet thing or both, or pretty much everybody, because otherwise you'd be in there anyway because you wouldn't care. So the hypothetical question that I'm going to give you is, you get back from KiwiPycon and you are told that while you were here, your website was hacked. Your application was hacked. On a scale of one, two, many, and lots, how panicked are you? Okay. So what if you found out when you got home from KiwiPycon that you were hacked six months ago and only just found out? The central IT department, the assistant admins called you and said, hey, why have you been sending all this traffic to Romania for the last six months? I get a lot of panicked looks when I ask these questions to people. Usually that's because it's usually, usually they're scared because they haven't really thought about it beyond a sort of, oh, please, just don't let it happen to me. There is actually a correct answer to this question. Does anyone want to guess what the correct answer to this question is? I'm not sure if that was totally a guess, but why is this the right answer? It's the right answer because, first of all, it's not panicked. And it shows that you've at least thought about this. You've at least gone down that thought experiment. Imagining how you'll get hacked is how you protect yourself from being hacked because you actually have to be thinking about all the worst case scenarios in order to even work out how you're going to protect yourself from them. So that's what we're going to do today. We're going to take an imaginary web stack and we're going to just sort of start seeing how far down this rabbit hole we can go. We'll start thinking about the horrible things that can happen to you in your day job. The most important thing to realize is that about security is that it's not any one thing. There's no single security mechanism so clever that it stops all attacks from all attackers except maybe unplugging the server, melting it down into a slag, and dropping it into the Marianas trench. And even then the attacker will find the backup of your database that you left in your home directory on your desktop machine. Yeah, there's more than one kind of attack and so you have to think about more than one kind of defense. A secure system relies on lots of defenses in lots of places.  When you do this there's usually kind of a structured mechanism to it. You find yourself thinking about classes of attackers in terms of what their motivations are, what their resources might be, and their potential skill level. So let's just play this thought experiment for a little bit. Opportunistic script kiddies. These are the people who are probably trolling the internet for low hanging fruit. They're looking for unpatched systems that have zero days already installed on nice user friendly, click it to hack it, things like Metaspoit. They're very unlikely to use financial resources when they're just casting their net wide, but they're dangerous because they have all the time in the world. And there's effectively an inexhaustible number of them. They're in it for the lows.  Why would they attack me? I'm nobody. And the answer to that question is if they can get in, that's the reward. It's not like they cared about who you were. Organized criminals, these people are usually out for some sort of specific financial gain. They're generally very skilled. They have financial resources and they're organized to at least spend some of that on getting into your system. They often have very specific motivation, which is also for financial gain. So if there's no financial motive to breaking into your blog, then it's possible that, why are you two dancing back there? That's strange. On the other hand, they could just be out to dossier site for blackmail or extortion purposes. Discounted former employees. This is the one that people don't think about enough and they need to think about the most. Nothing at all. But there is a really important point. Oh yeah, I used to work for you. Hi, Nick. Actually, how many people in this room did I use to work for? Couple of you.  Because you can't actually control the mindset of your employees or your former employees. They have intimate knowledge of the system. They probably still have credentials on your system unless you've got really good policy about the exit plans for people. And their motivation is right there. And then you might have people who might want to make a political statement out of attacking you. And then you might also have nation state actors. This has been in the public consciousness for a while. So I wanted to throw it in here. They basically have infinite resources and they probably don't have a lot of motivation. I mean, if nation state actors really do have motivation to get into your system, you probably have bigger problems and you probably already thought about this stuff, hopefully.  And I recommend you do this when you're thinking about your site. You should actually make up little people. If you use agile development methodologies, you're probably used to the idea of personas and user experience design and stuff like that. Do that for bad people too. Make personas for the people who might be trying to attack you and use those when you're sort of gauging the interactions with your site. Just a very brief example of this sort of thing. I spent a couple of years working on a tool for the Ministry of Education and it was a tool for teachers. But the attitude towards the tool was quite polarized. A lot of people were very for it and a lot of people were very against it. There could be reasons to attack the site because embarrassing a government IT project is very in vogue in New Zealand. It's something that you get a lot of free press out of. But the number one threat to this system after all of our sort of analysis really just consisted of a student leaning over the teacher's computer and the teacher walks out of the door. So you have to know sort of what the likely threats are and what the likely things are that can happen. I'm going to recite the definition of attack surface directly. This is straight from Wikipedia. The attack surface of a software environment is the sum of the different points, the attack vectors, where an unauthorized user can try to enter to or extract data from an environment. That's a bit tedious but it's just a point of saying basically every moving part and every exposed bit of your system is a potential point of abuse or attack or throwing random data at, see what happens. What I want to do is dig through a web stack. So your web stack is made out of layers of different technologies. You don't have to think about protecting only the outermost layer it's devastating if you limit yourself to that. There's more places where they can get in. Sometimes they can skip one of these and go straight to another. At almost every layer of the application stack you can perform the same thought experiment. You can say how might this layer of the stack get compromised? What are the potential consequences? In what ways can you reduce the attack surface there? And in the event of a compromise how can the consequences be mitigated if at all? Or at least what would your plan of action be if it had happened? Summarized as hindsight. Find a way of doing your hindsight ahead of time. I don't know what they call that. I think there's a P word for it. Planning. Sorry, I'm rushing really fast because this is a 45 minute talk and we have a 30 minute slot. So moving on. Your web server is probably some combination of Apache or Nginx. Generally speaking. Both have very large communities behind them. Both are mature and well tested pieces of software. But it's also obviously the most exposed part of your web app. Pretty much by definition. It'll be hit by everything. Any bugs discovered in either Apache or Nginx will be automated and exploited quickly. That's just the fact of life. And it'll be anything that happens on the internet will happen to your server at some point. SSL bugs, like the ones that we've seen in the last year, Heartbleed and Poodle and stuff like that. They're the sorts of things that will end up always being tried against your servers. So how do you minimize the attack surface of your web server? You start by disabling everything you're not using. In my experience, that's actually a lot easier with Nginx because it tends to start with a minimal configuration anyway. Apache kind of starts with a whole lot of the modules that it thinks you're going to like to use. And these days, you probably don't use those. You probably don't use ModCGI anymore. You probably don't use... You probably do use... Well, if you're running a Python stack, you probably don't use Mod Rewrite except for a couple of things. Actually, no, you do because SSL redirects. Sorry, I'm making stuff up. One of the best things you can do with configuration is delete big chunks of it. SSL configuration is hard. Jeremy went through this in his talk. And this tool... If you're not a crypto expert, or if you're just someone who actually has a life and needs to get their job done, the myriad of SSL configuration options and things are just going to be overwhelming to you. So get some best practices advice. And this is a really good place to start. Really experience this admins that I know have a few complaints about this thing. They have like, oh, I would change that cipher. I don't really like that one. But they're the experts. And if you're not, lean on experts. So let's say... That's minimizing the attack service. But what if someone got in? This is what we want to do. We want to think about what will you wish you had done? Or if it has happened, what can you do? One of the best things that you should have done is keep the web servers separate from the app servers. If you have any kind of budget at all for this, and you're not just running it on your Raspberry Pi, separating out either in virtual machines or actual hardware or whatever, the web servers from the application servers is a really valuable thing to do. And that's one of the main reasons why I'm moving away from ModWSGI and all my things. ModWSGI for deploying Python apps is really mature and really stable and really good. But it does couple your application server to your web server in a way that probably just ain't that good an idea anymore. But I gave this talk at PyConAU and Graham Dumpleton, the author of ModWSGI, pointed out that he's working on a system that might flip that on its head. ModWSGI Express, I don't think it's ready yet, but have a look at that if you are using ModWSGI and are interested in what options you have to make things safer. Operating system lockdown is important as well. Who here uses Ubuntu a lot? Cool, I'm basically talking to an Ubuntu crowd. Learn more about App Armor. App Armor or SE Linux, if you are using Red Hat, provide a lot of operating system level protections to prevent code exec, for example, on a web server from being devastating.  So how about we just don't let it? It's a really, really good layer of internal protection to prevent that initial compromise of potentially bad unpatched Apache code to get to the operating system level. I'm gonna skip some stuff. Oh, I have things, cool. A lot of people skip this step, but why does your web server need to talk to the outside world? Don't let it. Firewall off everything. Your web server shouldn't be able to talk to anything except the server that it gets its security updates from. Why would it be able to talk to anything else? Why would it be able to initiate connections to anywhere else? Okay, so let's talk about your Django server, or your modWGI server, your application server, or whatever. Because application servers are usually written in a high-level programming language like Python, you generally don't have to worry about buffer overrun C-style attacks like you do with Nginx or Apache, but if something does happen, an attacker could achieve arbitrary code execution on your application server by compromising any system commands that your application server runs if you're shelling out to anything as a part of your application server. You're using some magic tools to resize images and you're passing it command-line arguments and those command-line arguments aren't fully sanitized. But the more likely direct attacks against your application server code probably involve compromised login credentials. They probably involve someone getting a level of privilege in your application that they shouldn't have otherwise gotten. Stealing the accounts for the Django admin user, right? Or there could be an indirect attack on your application using cross-site scripting and we'll talk about that if we have a little more time. How do you mitigate this? You need to make sure your application server can't do any more damage than it needs to be able to do if you suddenly lose control of what it's doing. And to be honest, that brings you back to AppArmor profiles again. It's a really good idea to just read up on those and make sure that your green unicorn Django server doesn't need to be able to read files from slash dev by and large. Actually, locking down... I'm talking up AppArmor and I still recommend you do it, but AppArmor for locking down Python things is a little bit of a puzzle game because Python tries to do a lot of things that maybe your application server doesn't need to do, but it's still worth the time. Decent things to do include making sure it's running as a user who doesn't have the right to do anything else. Your application probably needs to talk to the database over a network. It probably needs to talk to the memcached layer or maybe Redis or whatever. Does it need to be able to write to any file on your system? Not for logging. Use a logging mechanism for that. Use this log for that. You can pretty easily write a web app that does not actually need write access to any file on the operating system. And then that's just one step away from having some kind of alert happen if it tries to write one file on your operating system and you go, hey, I didn't expect it to be able to do that. Credential theft attacks, that's a lot more dependent on your application setup, but IP address restriction on admin accounts is a really good start. If it suffers a service or something that you don't need to provide admin people to have access to, two-factor authentication is a wonderful idea. Having some extra restriction on there so that just plain old credential theft won't happen. What are the consequences if your application server does get hacked? It's not pretty. You have to think about these things, but it's not pretty. You probably have to assume that all of your data stores, your database, has probably been fully compromised because generally speaking with one of these web apps, you've got full database create, read, write, access. Customer information may be lost or tampered with. Incident response and forensics will become critical because you'll need to as much as possible determine the timeline and the extent of the breach because at this point it very much is getting in touch with your customers, explaining them what happened, telling them to change their password, but also offering free credit checks, blah, blah, blah, blah, Ashley Madison, topical stuff. Moving on, we have a lot of stack to get through. It gets uglier and uglier and uglier. The database server sits behind the application layer. You'd like to think that they couldn't get to that unless they had already compromised the application layer. Who here is familiar with the concept of SQL injection just to get a rough, excellent. Why are you guys even here? You already know how to be secure. If there's an SQL injection vulnerability in your application code, then it might be possible to compromise the database without requiring a compromise of the application server code. That's the risk, right? Any kind of SQL injection vulnerability, they just bypass all your protections and have full access to the database. The database could be exposed through lateral movement inside your network, or the database backups could be compromised. I won't ask for hands up on this one, but how many people, just, you know, grimace to yourselves if you have a database dump of your system sitting in one of your home directories on one of your staging servers? Cool, I saw a couple grimaces. Oh, yep, it's a thing. I probably, yep, I'll stop talking. I have a microphone. I have to be careful what I say. Database backups could be compromised, but yeah, much more likely is a compromise could happen on a development staging server that's using a copy of production data. If you use production data, you load it into your dev database. Your dev database is sitting on your laptop because you are a remote worker or you are traveling to a conference right now. Grimace to yourselves quietly if you have a copy of production database sitting on the laptop that you brought to KiwiPicon. You don't need to put your hands up. There's not a lot you can do if they get as far as SQL injection, but there's absolutely no excuse for them to be able to get to SQL injection. SQL injection is the number one thing on the OWASP top ten in terms of security vulnerabilities in the world, and it is also one of the most preventable. The techniques to prevent it have been spelled out for people. If you didn't put your hand up and you don't know what I'm talking about with SQL injection, Google it, find the OWASP wiki and read up on it a little bit. It won't take you long to learn what the problem is and you'll find... Yeah, it is very fixable and at this point there's no excuse for it to still be in your web app. Find a way to not use production data on your dev environments. Use test data or whatever. You probably have to use production data on your pre-prod if you have that sort of setup, but don't let it spread and here's the thing that everyone forgets to do. Audit where it has spread to. Just somewhere in your process of these systems are the ones that have production data. If you know it, you cannot protect it if you don't even know where it is. You cannot protect production data if you don't even know which of your employees have it on their laptop. How can you? That laptop gets stolen at a conference. Stay away. Yeah, know where your database dumps live. That's actually really, really, really easy. One of the consequences, the data's been breached. Actually, Madison, blah, blah, topical, topical. It can be a little bit embarrassing is what I think I'm trying to say, but it can be an awful lot more embarrassing because you probably don't even know it's happened. I am going to skip an entire section of this, but I love talking about cross-site scripting and I love talking about content security policy, so bug me afterwards because, and if you don't know what content security policy is, put that on your list of things to Google after this talk. Content security policy, I'll just buzz through that really quickly. Content security policy is a web browser feature that an extra header goes into your response and it will tell you, it tells your browser that only certain JavaScript is allowed to be run on this web browser from this site. So you can say only the JavaScript that is from my CDN is allowed to be run on this web page. And not only that, it can also report if there's JavaScript embedded in a place where it shouldn't be. So your web browser can become a warning mechanism if someone has executed a cross-site scripting attack on your site. It's only a mitigation, not a cure because only the most modern browsers support it. But there's a fantastic feature, which I just mentioned, which is CSP violation reports. I am skipping through, I do apologize about that. Web browsers that support content security policy also have a reporting mechanism where you can tell that browser, your server can tell that browser, hey, if you see JavaScript on my site that shouldn't be there, send a post to this URL and report where it was. So suddenly your user bases up-to-date browsers become your line of defense against cross-site scripting attacks that could affect the older browsers. The older browsers can't protect themselves using content security policy, but using content security policy can benefit everybody nonetheless. So that was the easy stuff, right?  All too easy. What if someone gets your infrastructure as a service keys? I heard that. What happens when someone gets your AWS keys? Who has ever accidentally committed AWS keys to a GitHub repository? You can grimace or put your hands up. So it turns out that this happens all the time. Automated scanners are constantly searching public repositories like GitHub Stack Overflow and Pastein-like sites, looking for strings like Begin Private Key and AWS Key and stuff like that and AWS Secret. There are 100 different things. They'll find them and they'll use them right away. And the good news is that when that happens, usually all that will happen is they'll spin up 100 machines on your AWS credentials and just launch a million Bitcoin miners.  That's the joy of the automated tools. But if someone who actually doesn't like you, like disgruntled former employee or just anyone who's targeting the hold of those keys, what are the consequences? What was your entire data center in AWS? Is it all in elastic Lambda-e services? Is it all in Route 53 or whatever the AWS product is? Are your backups all in the same place? Are they all backed up to S3? Do you have just a now's the time we wind down the company? Does it work? So, AWS, at least, lets you make lots of sub-keys with limited privileges. And you should be using really, really strong paranoid two-factor plus password-blocked-in envelope for the master keys and then have as little privilege as possible for the keys that are actually managing infrastructure or do your deployments happen from laptops that are here at the conference? I promise you I'm not earmarking laptops to look at afterwards. But, yeah, please, for me, just don't have all your eggs in one basket. Have a backup of all your stuff buried in a vault somewhere or maybe on your laptop, but maybe be careful with that laptop. I don't know. Maybe I'm just teaching you to be really, really careful about laptops. Okay. Final topic. What will I wish I had done when I had the chance? This is the thing that you should always be thinking. If something horrible happened to this part of the system, what is it that I really wish that I had done? What is it that I will have wished that I had done? You'll wish you had better logging.  If you're using log rotate, that, frankly, implies that you're using log rotate the files in viral log on those servers, but if those servers were compromised, you can't trust the logs anymore. Auditing is, auditing and is the key piece of incident response. You have to know as best you can what they took and when. And if you've been watching the Ashley Madison stuff unfold, one particular thing that seems clear is they just don't even know. They didn't even know. They denied that it had been breached. When it turned out it had been breached. They clearly could not even audit what might have been happening. Who does that? I want all of you to assume that your backups have not worked since last time you tested a full restore. Your backups have not worked since last time you tested a full restore. I want you to internalize that. Unless you've been doing full test restores, assume that your backups don't actually even work. Assume that they have not worked since the last time you did that. When was that? Never, probably. Modern cloud orchestration has had at least one benefit where a lot of the machines nowadays are ephemeral.  And that's kind of helped with the repeatable infrastructure aspect of that at least helps the backup situation a little bit because you don't have to worry about all the files in certain places. To prevent the worst, you actually have to imagine it and plan for it. Thank you very much. We don't actually have any time for questions. If everyone could make their way over to track one like now because we've got closing and maybe maybe I should check that first. One moment. Sorry, the full version of this talk where I do cover the stuff I had to skip for time constraints is available on YouTube. I gave the talk at PiconAU. So by all means feel free to check that out. They're not quite done over there yet, but I just said what Jen told me to say. We can do questions if we want to be mean. Well, okay, don't be mean. It can be a little mean. Questions? Okay, sorry. The question was if does content security policy prevent you from using inline JavaScript? The answer is the default configuration for content security policy does block all inline JavaScript. And that is its single best feature, but what that means is that you have to plan for it. So back porting content security policy onto existing systems often means you have to turn that off at first and slowly refactor possibly quite a lot of your code to strip out all of the inline in your pages. Every green fields project, every new project you start you should have content security policy enabled with a strict policy from the start and that means that you will put all of your JavaScript and JavaScript files and you'll be far, far safer that way. So yes, that's the biggest sacrifice that you have to make and it can make refactoring it a bit of a pain, but totally worth it. I'm not familiar with that. It's been mentioned to me before. He mentioned Google Tag Manager. All I can tell you is that because content security policy is becoming more popular, it's becoming libraries and JavaScript libraries are catching up and making sure that they're compatible with it by and large. So for example, when I was first using content security policy a lot of stuff in the Django admin section of my site broke because some of those little things are inline JavaScript, but these days it works well. There's one question up the back. When I first tried to use that feature, I pointed it at a page that had like a thousand little bits of inline JavaScript and this was an earlier version of Mozilla where it was still kind of provisional and it tried to send a post for every single violation in the page. That was less than ideal. You tell it which endpoint to go to and so you probably want to make sure that that endpoint at least will be able to handle whatever is coming into it. It's probably safer than that nowadays, that's for sure. I have a question. Yeah. If you've got an infinitely sprawling VALOG and you're not supposed to put it in S3 or Glacier, where are you supposed to store all your server logs? I'm not familiar with the current state of play software as a service log-ketcher type things. I imagine they exist but mainly what I'm thinking is if you most of my deployments have been you build like 10 VMs and you do some stuff rather than fully sort of Amazon type deployments and a separate logging server which uses separate SSH keys to get into that is basically just harder to lateral move into is probably a really good start. What is it called? Logly. That sounds promising. But as long as if you're contemplating the prospect of the application servers or the web servers or the database servers getting hacked or getting compromised their logs should go somewhere else and the log server can be kind of a black hole if you are careful with its SSH credentials, they're not just sitting on your laptop here at Kiwi Picon then it can receive stuff, it can write stuff to disk, that stuff can go to some sort of tertiary storage system, nice thing about logs on tape is that they can't be tampered with. So maybe, yeah, there's bound to be a million in one way of doing it but the main thing is just to make sure that a compromise of this end doesn't give you the logging server for free. I would imagine using something like syslog to live stream logs, yeah. Yeah, as opposed to just SCPing them across, yeah. We're pretty much out of time now, so if everyone can give another round of applause to Tom. Given the time pretty much