solutions for cyber security for the smart grid, so again, Jeff, Colin, Galen, Chris, thank you so much for being here today and help educate us on this important topic. First a few words about myself and what I do, I'm in the Computer Science Laboratory at SRI International, which is an independent non-profit research organization. And my group does research in security for critical infrastructures. We also do R&D portfolio management for our clients, in particular for the Department of Homeland Security and their Science and Technology Directorate. So at SRI we say that we work on important problems, not just interesting ones, that's something we take pride in to address what's really important and we believe that cyber security for critical infrastructures is one of those really important problems. And this has a lot to do with the possible consequences of a cyber attack on an infrastructure. There are lots of interdependencies between our infrastructures, between oil and gas, telecoms, electricity and so forth, but everything in our modern society is dependent on the electric grid. So therefore, dependability, resilience of the electric grid is something that's very important. And this is because the consequences of an attack could indeed be catastrophic. So when we make the new infrastructure, when we put the smart grid in place, put smart devices, computing devices, data communication in place everywhere, we have to make sure that smart doesn't mean vulnerable or hackable. And unfortunately, that can be challenging because security is really not an easy topic. If cyber security was easy, we wouldn't have to be discussing it, we wouldn't have to be working on it, we wouldn't have to read about it in the newspapers every day about cyber security problems. For many reasons, many of which we'll hear about from our panelists today, security is a difficult problem. And also, it's not necessarily a topic of interest to many people. I'm very happy that you are here today, but it's a fairly sparse audience today and it's always interesting to come to these energy conferences, for example, everyone's excited about new technology and what we can do with it. And then we security people come and say, oh wait a minute, not so fast. But what I want us to hear about today is how we can actually use technology to also make security better and make sure that we become less vulnerable, not more vulnerable. So end of my introduction here and I'll ask each panelist to come up here, give five minutes of an introduction. So we'll start with Jeff. Please go ahead. Great, thanks very much. I better start my clock or I'll go 25 minutes. Okay, so I'm Jeff Gooding with Southern California Edison. Southern California Edison is a utility just to the south of here. We cover 50,000 square miles and serve about 14 million customers. I am the manager of smart grid systems engineering, so I come at cyber security engineering from a little bit of a different perspective. Prior to doing that, I was the chief architect of our smart meter implementation, 5 million smart meters. So if you have any tough smart meter questions, feel free to shoot them at me. Not literally, right? I know we're in Northern California. Anyway, so from my perspective, you know, we are aggressively pursuing sustainable energy goals. And what that means is we're introducing a whole range of new technologies, energy technologies to the electric grid that don't have the typical characteristics of generators that we've had in the past. So for example, right now, the grid remains stable largely through the laws of physics, large synchronous rotating mass propagates a robust waveform that goes all the way through to the end customer. And that allows the grid to ride through transient events, lightning strikes, all sorts of things that might happen out there to wires on the grid, all sorts of faults you might see, you know, most of the time. And that's what makes it highly reliable. What we're seeing with the introduction of wind, solar, power electronics, even some of the distributed generation is you're seeing inertia, which is generated by that large rotating synchronous mass, dissipate across the system. And the effect of that is you see frequency stability, which is really the job of the California ISO and the WEC to kind of manage frequency and voltage become more fragile, managing voltage is the job of the distribution company, which is my company. So what that means is the time scales that we have operationally to actually respond to these emerging events so that they don't cascade into larger problems across the system are becoming much, much smaller. So the way that we actually address that problem is with technology, right? Wright Brothers Plain, governed by the laws of physics, had a very specific mission. B2 Bomber, massive amounts of capability, inherently unstable in flight, and technology is used to keep it moving straight and deliver all those extra capabilities. So that analogy doesn't stand up 100%, you know, with the electric grid, but it's a good way to think about how we're going to use technology to keep the grid stable in the future. So when am I going to get to cybersecurity in my long-winded five-minute talk? Well, basically now. So the introduction of all of this automation, all of this connected equipment, does facilitate vulnerabilities. And those vulnerabilities actually expose us to threats that need to be mitigated by security requirements and fed back into the design, hence the system engineering approach. So what we're doing about that in Southern California, Edison, you know, similar to what we did on AMI, when we came into AMI and we said, okay, we're going to put a ZigBee radio in this, we're going to put a disconnect switch in it, and we're going to put a communications network that goes all the way back to the utility. How do we make sure that the network cannot be attacked from a single and seized from a single endpoint? And so we looked across other industries and we started talking to the Department of Defense and specifically the NSA about using some of their SweetBee technology, and that derivative, a derivative of that, you know, was used on AMI. Coming out of AMI though, we realized there were some weaknesses there, very vendor-specific solution.  This cannot happen in a smart grid because all of these multi-vendor devices have to be able to talk to each other and communicate, and the relays have to be able to very quickly send communications back and forth on the state of the system. So we started working again more with the federal government to transfer technology, and this is a HAPI's derivative if you're familiar with the cybersecurity standards in the defense and intelligence industry, but this is a HAPI's derivative technology, and we redid our architecture in the grid control center to make a common service. So a common cybersecurity service where any device at the end could access cybersecurity services in the grid control center, and we had a way to distribute that logic. So if we lost communications, there would still be a set of policies and a set of keys out there. So that's the work we've been doing. Now I'm looking down and it says zero. So I'll wind it up here. Anyway, that's the work we're doing at Edison. It's in Department of Energy, smart grid demonstration right now, running on a lot of radios, and we hope to be rolling this security out in response to NERC's version five requirements in the utility industry over the next maybe year and a half. So thanks. Good afternoon everybody. I'm Colm Lennon, and I work for Honeywell Building Solutions, and I'm the global offering leader for our service and energy lines of business. At Honeywell, I can tell you that we're obsessed with cybersecurity, and it has become an obsession because we do believe at all levels of our organization how big this threat is. I don't know if anybody has seen the recent report that came out from the Department of Homeland Security, but the first six months of this year, it's been 111 attacks on industrial control systems on the electric power grid. It's actually, it wasn't the electric part as much as the natural gas distribution network that some of these hackers were really going after. That's 111 in the first half of this year, and it was only 88 in all of 2012. So there is a significant increase, and so our obsession with cybersecurity is related to this, not only the traditional threats of hackers doing it for fun, doing it for fame, doing it for fortune, but now it's some of the emerging threats. Cyber security has become something that we have to worry about in terms of cyber terrorism, something we have to worry about in terms of cyber warfare, and so you can think about how important it is to build security right into everything that we do in Honeywell, because what we do is we build out the control systems for buildings like this, but also industrial controls that go into refineries and petrochemicals and power distribution networks and things like that. We have demand response automation solutions that we sell to the utilities to help them manage their demand response programs. So you can think about the potential impact that a hacker could have if they took over one of our systems. So that's where the obsession comes from, and we've gotten industry recognition. We are on the top 10 of the Security 500 benchmark, and our chairman and CEO has been at multiple executive summits at the White House on cyber security, is very much helping the federal government with setting policy and making sure that we're thinking through what needs to be done to protect our critical assets. And so throughout our entire organization, it is everybody's job. It's not just the job of our security folks, not just the job of our product engineers, not just the job of the people that go out and make sure that we're patching and updating our systems all the time. It's everybody's job in Honeywell. And what we want to do is we want to make it more of an ecosystem responsibility. Jeff just mentioned it very well, is you got the smart grid where you have all of these different technology players that are inventing technology and they all have to communicate with one another. So it's important for us as an ecosystem to make sure that we're addressing the cyber security problem and working together because the hackers are working together. The hackers are investing billions upon billions of dollars to get into these networks and wreak havoc. And so we have to make sure that we're working together and bringing the best and the brightest together, sharing our knowledge and our best practices so that we can get always a step ahead of our quote unquote competition, which is a very nasty group of people. So with that, thank you. All right, good afternoon, everyone. My name is Gail and Rasha. I'm a technical executive with the Electric Power Research Institute. And I manage our cyber security program for our power delivery systems there. And so if you're not familiar with EPRI, we're a non-profit independent research institute based here in Palo Alto. We actually do a collaborative applied research institute for the electric sector. And our research programs are funded by utilities from around the world. And cyber security is a very, very important issue to them. So why is that? Well, they view cyber security as being critical for protecting their current systems, but also as an enabler for deploying the next generation grid technologies. Because the fact is, if you don't implement good cyber security, you really can't or at least shouldn't deploy a smarter grid. So I'm going to ask a quick question here. Who here works in cyber security? So one person quickly dropped his hand. Anyone else? All right, well, hopefully you'll leave today with Ben understanding some of the basics of cyber security and how they apply to this market. And it's a very, very challenging area. If it was easy, it would have been solved already. But there's a lot of differences between IT systems and operational systems that make cyber security more challenging. And you also have issues of legacy systems and other equipment that's in the field that doesn't support cyber security very easily, that can make it very challenging. But I think the main thing to keep in mind today, though, is that even though cyber security can be challenging at times, it is still a key component of grid resiliency. So just keep that in mind. Like I said, it can seem like a showstopper sometimes if you try to bolt it in at the end. But if you plan it correctly, it can really needs to be that key component to help maintain a resilient grid. So we've heard about some of the cool technology that SCE is deploying with their CCS architecture, very advanced security architecture and somewhat Honeywell is doing. I decided to talk a little bit about a challenge area that's not quite as exciting, but it gets left off the discussion sometimes. And that's assessing and monitoring risk. So the fact is, right now, there's not a commonly used industry accepted methods for assessing and monitoring risk in the electric sector. There's been some good work that's been done recently on creating more common ways to do security assessments for utilities. So for example, in 2012, there was a White House initiative where they worked very closely with DOE and DHS to develop the electricity subsector cyber security capabilities maturity model, which it's a bit of a mouthful, but it goes by the acronym ESC2M2. And what that did is it provided utilities with a common set of guidelines to measure how mature their cyber security processes are in key domains, such as configuration management, asset management, identity and access management, threatened vulnerability management, situational awareness, all great areas. And so you could take that and see how mature your utility was in these different areas and compare that from one utility to another. But what it doesn't do necessarily is actually tell you how to assess your risk and monitor your security posture in real time. So I think that's a very big gap right now for the industry and one that we need to be working on as a research community and applied research community as well as working with utilities in that process. But one issue there that holds that back is also not having a common set of security metrics to be able to measure what your security posture is. Now, utilities have in general taken different processes from the IT space and adapted it to their particular situation. But once again, there's not this common way of measuring what that security posture is across the electric sector. But if you had that ability, then it'd be possible for the executives of utility to know where do they stand right now. And it'd be possible to have a much better idea for an electric sector as a whole. Where do we stand as a critical infrastructure on our security posture? And right now that's a very difficult thing to measure. So if you don't know where you stand, you can also still create and deploy great architectures and good incident monitoring technology. But it's hard to know exactly how secure you are without that. Good afternoon. I'm Chris Viller out of the California Public Utilities Commission. Thank you all for coming and inviting me to be on this panel. So before I begin, here's my usual disclaimer. Since I do work for a state agency, any opinions that I may express are my own and not those of the commission of which I represent. So on cybersecurity, the commission really got really involved, or the commission staff at least got more involved in cybersecurity beginning last September when the staff of the PUC issued a white paper outlining steps that a state commission, in particular California State Commission, may choose to do in ensuring the security of the grid going forward. And it outlined several reasons for doing that, as well as several challenges on how to make that happen. So I'm just going to talk about the challenges side of it. So the first thing to remember is that the PUC only regulates the distribution aspect of the electric utilities business. So all the wires you see driving down on the streets, those are the ones that we regulate. The big ones you see on the highway, we don't regulate. That's regulated by the Federal Energy Regulatory Commission. So right off the bat, we have a jurisdictional issue. On the transmission side, there are longstanding cybersecurity standards promulgated by NERC. There has not been anything on the distribution side done by any state regulatory body ever. So this is one of the big challenges is as things start to show up on the distribution grid that are smarter, that interconnect with other things, the challenge from the state perspective is, how do we make sure that those new things that Utility wants to put in are secure? What steps should a commission do to make sure that the Utility is considering cybersecurity on these new investments? One of the ways that we do that is as Jeff was talking about all of his fun little projects he wants to do is that we have to approve the budgets for those fun projects that Jeff wants to do. So if Jeff wants to do a fun little project and it gets rolled into the right case and we review it and we say that looks good, but it's too much money at this time of the day, this time of year. So that money you want, we're going to cut it in half.  So as we start thinking about how do we understand what the fun project that Jeff wants to do, we have to start having an understanding of what does the impact of that? How do we know that the programs that Jeff wants to do, the risks that he wants to take and the way he wants to measure is how do we know that the money that the PUC has to fund for those programs is worth it? How do we measure the success of those things? And Gail is talking about metrics and that's one of the things that we are still struggling with here at the state level is how do we make sure the money that's being spent is appropriate? How do we know it's not gold plated? How do we know it's not enough? How do we know we're just lucky? And so that's one of the challenges that at least on the state side we still have going forward. Another one is of course staff expertise. The PUCs headquartered in San Francisco and we basically paid Sacramento money. So it becomes very difficult to retain or even hire the requisite experts that are needed to go into Edison, review their audits, make sure that their cyber security program is appropriate. We don't have that technical staff that can do that. For example, the three people sitting up here as well as Ulf, if you read their bios they have very long education backgrounds and long history of work. If you read mine I have a BA in history. So here I am a history major talking about cyber security. That's just an example of sort of the challenges that many states have. As going forward, so we have our white paper that talks about the electric grid and for the most part we're talking about electric grid. But the PUC regulates more industries than just the electric grid. We regulate the natural gas companies, investor owned natural gas companies. We regulate the landline telecommunication carriers and finally we regulate the water companies. So as we start thinking about this C2M2 or the risk management protocol that was also developed by DOE and we are focusing on electric, let's not forget the interdependencies amongst the industries. So without electricity does water get pumped and without water can many power plants run effectively. So how does cyber security policies that are developed, for example at the state level, how does that impact and how can that be implemented across other industries that are seeing similar risks. Water company has SCADA networks just as the electric utility does. Water company may be about 20 years behind on the technology investment but that's just, that's the perfect time to do it. You know we're talking about building security in to the investments. This is the exact time we want to do that. As we do smart grid, as utilities eventually, as the water utility especially eventually start catching up on technology, making sure the security is built in provides more cost effective solution for the rate payers, for the customers and for the utilities that the commission regulates. The last thing I just want to bring up is California despite having our paper and being in the press all the time is not the only state doing cyber security. There are many other states that are struggling with the same issue and investigating the same questions. State of Missouri has a rulemaking opened up last July asking the same questions. Now one of the challenges that we have is how do we have a public record for determining these policies. The Missouri PSC issued a series of literally 70 questions to their utilities to answer and that filing was all filed under seal which is good for the PSC staff and utilities were together but not very helpful for the public or the other experts that are really necessary for developing good policies on this. So as we go forward you know California the PUC is doing a lot on cyber security and we're getting a lot of good press but there are other states doing just as much work if not slightly more than us that as an industry we should all be aware that what happens to Southern California or PG&E is very likely to happen to other utilities and the ability of sharing that information across utilities with one another and across PUC is going to be a very valuable tool going forward and one that we should be working together to support. Thanks. Now we're going to go over to the discussion portion of the panel and I'm going to start by asking our panelists some questions and we'll have the one who feels most compelled answer the respective questions and we'll get a discussion going. So we heard in your introduction your respective introductions that the cyber security for the smart grid is facing many challenges. So I want to start by asking what are the most important things that your respective organizations and your partners are doing to make sure that the smart grid is as secure as it as it needs to be both looking at today's problem and also looking ahead into the future. So who wants to take on that? Jeff? Yeah I guess I can start. I think you know a lot of times utility companies approach problems like cyber security by looking at the minimum set of requirements they need to meet in order to just comply with a policy or a regulation. It's not just the utilities. Yeah and so it's like an epidemic across our industry. Anyway so you know I think when we start looking at how to solve these problems and how to bake a solution into and how to bake a cyber security solution into a new capability or a new solution we're developing we're really not just looking at meeting the minimum set of requirements anymore. I think the most important thing we're doing is trying to look ahead and figuring out how to design systems and develop architectures that are flexible enough to accommodate new requirements in the future that we don't really know. And that means from a practical standpoint what that means is a lot of the solutions we're developing are more like platforms that host policies. They're policy based types of solutions that allow us to respond to the changing threat environment. So aside from the investment and aside from all the money that Chris is going to send us I think you know really taking that focus that strong system engineering look ahead and making the most of every dollar that comes our way so that we don't have to rip things out and replace them in the future is really critical in meeting these challenges. Yeah I think I think that's key Jeff and one of the things that we always struggle with is our product engineers always thought about securities and afterthought so all of a sudden we'd be you know well down the product development path and somebody would say you know what we haven't taken a look at the security of this thing yet let's have a let's have a look at what's under the covers and then they'd get angry when we'd find 5 000 vulnerabilities on it and back to the drawing board and it was the security guys fault for finding all the vulnerabilities right so so over the years we've had to really teach our guys that security isn't an afterthought security is an upfront investment and that architecture is so critical and so it's it's kind of like building a house or building a commercial building I mean that foundation is necessary and absolutely critical from our perspective up about getting off to a good start and not making security an afterthought so with that solid foundation then you can build products and services that are going to be secure in their nature because you started off on the right foot and then after the fact one of the important things is just making sure that you've got process and governance in place so that you can check back and make sure that that security architecture has been implemented correctly because all it takes is one engineer to go down the wrong path and to take their own version of the architecture that you've set forth and you can open yourselves up to a lot of vulnerability so it's really important that everybody is engaged everybody understands what that architecture is and how it needs to be implemented and then having the process and the procedures and the governance in place to check that everybody's doing their work appropriately it's a very good point and since most of you in the audience perhaps do not work in cyber security but maybe you develop new technologies you oversee them you fund them you support them you promote them it's a very important point to make sure security is there from the beginning because we've seen so many times where something is readily developed and now let's look for security problems and guess what we usually find a lot of it all right and I'll add on to the end or the second part of the security cycle then so so we're talking about how you can build security in and protect protect devices but another key area that we've been working in is how to manage incidents and so that's how you detect respond and recover from cyber incidents because I think even with really good security protections in place there are going to be cyber security incidents there are going to be security breaches and that's mainly because it'd be very very expensive to implement all the security controls necessary to make every single system be impervious to any type of cyber incident it's just not cost effective to do that so we've been working with our utility members on this since 2012 and this year we've been focusing more on transmission distribution system and enterprise security as well and right now I'm even focusing on how to help them develop an integrated security operation center and probably nobody here has been inside of a security operations center but think of how to describe one if you may have seen a scene in a on a tv show in a movie where if you were going to this room with have maybe 20 monitors on the wall there's usually a picture of a world map on there somewhere and all these operators with four four screens in front of them with all these alarms and logs and things you know going by and things flash red on there you know and they're supposed to go do something about it then so that's kind of a gives you a visual image of what a secure operations center is or how it how it functions now we're not advocating something as grand as a global global secure operations center for everybody but but right now one of the challenges is that it can be very difficult to correlate incidents within a single utility that occur on the corporate systems and also on their ot systems so what's happening the control centers on their transmission systems their distribution systems their am i systems their physical security systems right now these are usually handled by different power systems groups and so they can be siloed there and so if there is an incident it can be difficult or take some time to go track down the information you might need from each of those domains and so some of our members have on their technology roadmaps now to build out an integrated security operations center and so then if you have that you can have this a much much better situational awareness for what's actually going on within the entire utility and it's not just connecting these systems if you think about how physically spread apart also you know the power systems are i mean you know you're really looking at systems over a very large geographic area so again let me let me stop you there and move to to a related topic of and that chris also mentioned about jurisdiction and the many stakeholders so in security for the smart grid we're talking about the federal government state governments the utilities the industry groups system vendors like honeywell here research labs like epri and sri and many other stakeholders so i'd like to hear somehow we we look at the respective roles of these stakeholders and also you can describe how you're collaborating with with some of these other parties today and also if you have ideas for how we can get better at collaboration information sharing and so forth sure i'll start so the usual mission statement of a regulatory body is is we oversee the utilities to make sure they provide safe reliable service at a reasonable cost one of things the white paper proposed is adding the word secure so we regulate the utilities to make sure that the power supply is safe reliable and secure at a reasonable cost for indeed if it's not secure your power can either be safe nor reliable um so that's one of the cultural challenges or issues that that the commission is being is that we're being addressed so what do we do is that we have we do work with the utilities i come to conferences like these i get on phone calls with a number of national standards development efforts that people talk over my head and i just focus on the policy side and how it implicates the utilities we regulate um that that runs the gamut from all sorts of things and the funny thing is you know we're talking about how we're at a technical conference like today with all fun all sorts of tech vendors out there and we talk about securities the guys in the room that always be that no one wants to talk about we'll also do privacy and if we think securities are the what blank you should try privacy talk in the big data people um the other thing is you know jeff was talking about flexibility and rates i hate talking about rates i'll be this i'll do this very quickly utilities come and ask for funding every three to four years so we have to budget for the utilities on a three to four year rate cycle i don't believe people hack utility networks on a three to four year rate cycle so how do we let the utilities and work together with utilities to make sure that that they have the appropriate flexibility to respond to these threats accordingly as they evolve over time because we can't wait three to four years for them to come back say oh we had something that happened three years ago we need money to do something to respond to that that is way too late and the pace of technological change is going to be one of these challenges i think we're all going to have to address at some point so chris you mentioned timescales and jeff you talked about that too and we see this challenge where it technology evolves so fast but we're we're looking at infrastructures where technology investment has to last for a very long time and even things like the am i the smart meters if they cost a hundred dollars each and you put them out to 10 million customers that's a that's a fair investment that you're probably not going to do every two three years when you need to upgrade them so this is a keeping up with all the big changes in technology and vulnerabilities in attacks in security technology and so forth that's a real challenge for for these kinds of of industries where you can't just throw out the server and put in a new one so what any of you like to comment on that yeah so i i think you know good am i lesson learn for me was you do want to try and keep the utility technology that's deployed and selected a fair distance away from fast cycle technology meaning fast cycle technology is dominated by consumer adoption of new technology and when you look at where those two things interface you want to make sure that you don't as a as a utility company pick a take take a really hard stand on a technology and i think that was a good lessons learned from selecting zigby and putting them in the meters now the world is going to be dominated by wifi zigby is not going to be the home automation standard we have five million zigby radios out deployed so what are we going to do we're going to put gateways out and it's going to be more expensive than chris and i had originally thought about to actually get this market off the ground and you need to understand that when it comes to fast cycle technology and consumer adopted technology you know the utility companies really are unable to participate in influencing those markets on the other side of the meter so what that means is we need to get comfortable using any communications pathway we need to develop company competencies that allow us to connect to the customer and their devices in a way in whatever way is available so if that's through the gm on star communications if that's through the internet to get to an inverter to get services that allow us to you know send a price signals down so that or some sort of signal down so that an inverter can behave to contribute to the overall health of the system or at least not do harm to the low voltage transformers then that's you know sort of the approach that we we have come to settle on i don't know if that exactly answers that question but no it does it's a hard problem yeah it is anyone else want to comment on that the only thing the only thing i'd add to that too is the the challenge is all over the place right it's in controls it's in you know critical infrastructure it's in servers it's in networks it's now everything and the pace of change for technology the pace of change in the security community in terms of their capability and the amount of money that goes into their capability is astronomical so it goes back to again what what i talked about earlier is that the ecosystem has to work together and combine forces more and more to stay a step ahead what that means is standard bodies are going to take more of a critical role in making sure that we're staying a step ahead uh companies like honeywell are going to have to work with some of our technology partners and our customers to make sure that we're implementing technology that isn't going to require a significant upgrade so that you can get more security benefit from it that we're making these systems in this infrastructure that's patchable and updatable so that we can stay ahead so it's it goes back to all of us need to be working together all of us need to be putting the best and brightest on solving the problem because the problem gets worse every day it doesn't get easier unfortunately i'm sounding very doomy gloomy today it's been that kind of a travel week okay we can switch to something positive well the funny thing about security though is that it's it's not all about technology and people related issues play a big role and sometimes they play a bigger role than technology issues we're all kind of technologists but would you like to comment on the on the people related aspects of security perhaps on the consumer facing things or what you do for for awareness inside and outside your organizations right well i think jeff could probably talk to this um very directly as a utility but you know one of the trends that's been happening um across the electric sector is um a couple of staffing problems so one is that i think in the next five years there's some very large percentage of of electric utility staff they're expected to retire you know so we're actually losing a lot of the um capabilities just on the power system there um but then also there are not very many people that have a very good solid understanding of how the power systems work how communication systems work how networks work how cyber security works you know that there's not a lot of people that that walk around with good capabilities in each of those areas and so i know that do we has been investing a lot in various programs to focus on workforce development for the electric sector there and i think one effort this out there is sponsored by the national board of information security examiners and they were looking at what are the security capabilities that you want operators to have and different roles within an organization to have and how can we um test for those capabilities you might have something like this cis sp which we have folks more on it systems what is the equivalent of that you know for an operator for the electric sector and so that's been an ongoing effort um but it's not not complete yet so what i'll say is that we have there there are three the three points i want to make on this one one is um you know the chat the people challenges you know as we've heard all day today in california there are many of the policies on the energy side is pushing more and more things on the distribution grid on to things beyond side customers homes so as we think about the people's issue the utility is going to have less and less control over this so how are the people setting up their distribution grid or their uh dg units their storage units things that are on the side of their house or inside their home that utility hasn't control over so utilities have to handle that uh two is an issues as again we're talking about people we have supply chain questions still you know the utilities and honeywell and they have to are relying on vendors to provide them with things and how sure are we that those things that they're buying are secure how secure is this supply chain of those vendors um so how we determine that level of comfort is going to be a challenge and the third thing um the commission had a thought leader's area this year and what was repeated over and over in this conference is that the number one cyber security threat is still phishing getting an email and someone clicking on something they're not supposed to be clicking on so the phishing and then allowing the malware to to warm their way into the utility networks just through email is still what we're told is the number is still the most likely successful means to hack into a utility network it's amazing what people will click on yeah but um one more thing to add to that about security awareness uh in honeywell we've implemented a security awareness program that's very comprehensive and every employee is educated on a very consistent basis about not only things that impact honeywell but things that impact their personal lives like how do you protect yourself from phishing attempts how do you protect your your identity and and how do you safeguard your your family and your children from security issues and so that combination of uh training related to security in the job or in the workplace with making it personal and providing them some value in their home and in their everyday lives has really gotten people's attention I think and focus and awareness of every day that security is something we need to be thinking about so that's one of the things that I would recommend in all your organizations is to think about how you make it personal not an email that you ignore but an email with a newsletter or an email with a link to a webcast that people really think about going to because there's some value in their personal lives right very good so I hope everyone here knows what what phishing is about it's when you when you get that email that says you won the lottery or it's from your bank or your airline click here just enter your social security number and everything and your your debit card pin and you'll be fine don't click on those all right so we're going to open up uh the questions from the audience just before we do that uh is there any uh sort of key takeaway messages from the the pattern for the audience today that you want to make sure that we state before we go into the open question session I mean I guess I can give a quick one just on the last question it takes me so when I hire an IT cybersecurity student it usually takes two to three years to get them trained up on how the electric grid works and teach them about embedded systems I have a much easier time hiring electrical engineers and training them up on how to figure out how to secure a lot of these systems so somewhere along the line cybersecurity got off into its own silo and and from an education standpoint it would be great to see people that had multi-disciplinary capabilities in systems engineering you know add it into the cybersecurity mix because it it does take a long time and a lot of investment on our part to bring those folks up to speed good point and I think it's an excellent point I would say the same is is to make that don't make security its own thing don't make it something separate make sure that security is everyone's challenge and everyone's opportunity to keep our systems in our in our network safe and to make sure that not only within your organization but outside your organization that you're working with your ecosystem partners to make sure we're sharing those best practices making sure that we're developing standards that are going to help our overall industry and keep us safe all right um I've just had um that you know sounds like there are a lot of challenges still in cybersecurity and there are one to emphasize so that there's been a lot of work that's been going on in this area since I'd say around 2007 2008 or so you know so there's a lot of good reference material out there and methodologies out there like the you could find the nister 7628 which is cybersecurity guidelines for the smart grid you know so if you find yourself in a position of needing to look at cybersecurity for the smart grid don't start from scratch there's a lot of good material out there make sure to make sure to use it so the the one thing I would make sure you all are aware of is that the relationship between the pvcs and in this instance electric utilities are actually really good on this topic it's not inconceivable for me to say that two even two years ago jeff's regulatory affairs people wouldn't say no way will you be on a cybersecurity panel the regulatory representatives so the fact that we are that we are able to and routinely now beyond panels together talking about cybersecurity I think is a good showing that we that the utilities and the regulatory bodies are aware that this is a solution that we are both having to work together to to implement and to address excellent great makes us all feel better thank you all right so if you have any questions please walk up to the microphone here so everyone one can hear you uh it would help some security conferences people are anonymous but my name is paul grant since 2006 I've been an independent energy technology consultant here in the valley prior to that though I had a 40 year that's 40 career with IBM and that was followed by 12 year career at upree as a science fellow actually well listening to this I remember in the 1950s as a 21 year old programmer for IBM working on the norad system that we actually did some encryption of the teletype communications between this is way before the internet by the way between the radar stations and central computers now this is on the cyber security question I'm going to ask now uh it's physical security and after 2000 after 9 11 at upree we put together a informal red team and our red team launched an attack on the transmission substations in california 10 of the largest we armed ourselves with hunting rifles and we knew where to shoot at the 300 me the uh transformers and we calculated we could bring down the state of california for months so my question is what's the condition of the physical security of the infrastructure in california and the united states given the fact just a few minutes just a few months ago some amateurs were able to compromise metcalfe does everybody know what metcalfe is okay to the extent that you can answer this yeah i mean it's 500 kv substation uh in pg and east territory uh where somebody shot the uh luckily they didn't know exactly what they were doing like in your simulation um but a lot of mineral oil was was spilled out of the transformers and they were able to get it back up and running you know so from 10 years ago from that study that epri did that you participated in we had the large transformers sitting out in the yard the spares some of them take three five years to build uh and from that study uh we got rail cars out there and and at least moved the spares uh off to a more secure location so they weren't all sitting together in the yard just waiting for you know if one fails then you you take the line you just hook up to the the next one um you know i have to say because the infrastructure is so distributed and big we still do not have adequate defense against somebody with a hundred more rifle who actually knows where to hit uh a transformer other than those spares uh and um we do have fiber optics now we do have communications we do have alarming um however as far as protecting against the attack itself physically um you do not see the really high walls or anything like that now maybe maybe because of this situation in pg and e uh we're going to get ordered to uh uh do something more physically secure we haven't a lot of the security problems we've had in the past 10 years have been people stealing copper and sometimes not so successfully unfortunately for them um and so we have a lot of thermal imaging and that sort of stuff out there video cameras but nothing like a security force uh you know that that's protecting the broadly distributed stuff just one more question in the study the red team study we did at apri one of the issues was cascading failures if you took out exactly substations now is that been treated now with better communications yes actually um so we we do have the the RAS schemes remedial action schemes that are in place at the substations uh have been there's projects underway to centralize some of them so you do have better protection phaser measurement units we have about 20 of them uh in which actually allows us to see whether we're suffering uh from a wide area uh problem or whether it's localized so a lot better simulation and modeling is going on inside the utility company right now so and then we're pushing that down to these automated systems so i'm feeling a lot better about our automated protections than even five years ago um but again these areas are still you know under development some of them and not as broadly deployed as we would like and we continue to actually you know move forward on that front so all your points are well taken and it sounds to me like these are areas where the new technology actually can lead to better security uh we heard about about the power electronics the secretary too was talking about this morning uh with that they replace the big physical transformers they would be easier to to protect uh and with the increased monitoring you don't have to wait for a phone call from a customer to figure out how the grid is doing you can actually see that in real time it's true but there's still very remote locations with very adverse weather conditions where the physical security as good as you can make it is still not good enough we've got a customer that has a wonderful perimeter fence and some great security cameras but when they got 10 feet of snow you walk right over the top of the fence on top of the snow i won't say where it is i won't say what the customer was but let's just say you can get yourself into areas of the grid across the united states and some very adverse conditions that you're not as well protected as you'd like to be so i'm glad to hear that something california Edison is in good shape but let's just say that the answer to your question is it depends all right next question hi i'm gary farhama phd student and management science engineering here stanford um guess my question is how will cyber security change the business model for utilities and if i was asking the cpuc i would say how should it change the business model for utilities and i'm gonna give two two sub branches for the question one is that are we if we're if we look five or ten years from today are we gonna look into more centralized more regulated more controlled kind of regime or are we gonna look at more deregulated where we have some more competition and thus some more market incentive to drive innovation into the utilities this is on one side and then on the other side is that does it does cyber security um or should cyber security be addressed by the utilities themselves or should we allow external parties to address the cyber security issue of the of delivering electricity so pjne can just deliver the electrons and we can count on somebody like google to make sure that our lines are secure chris that's all yours um thanks i have to admit i don't know how to answer that question that there's so much wrapped in that question that it impacts not only the cyber security policies that we would regulate over but just a simple market delivery of electricity um um so the way i would say it right now is that this is gonna be a very legal statement unfortunately is that current legislation does not allow the situation that you're talking about right now the utilities are the provider electricity unless you happen to live in a cca area the community choice aggregation area so the the delivery of electricity services itself is something that is subject to state law and we would uh let me strike that last statement uh it's it's subject to state law and that's sort of what our restrictions are right now now as we think about the business model how cyber security changes the business model i don't know my suspicion is that as things become more distributed i'll use that word um the utility challenges change and the ability of the utilities end to recover appropriate rates to fund those new services they have to or the new programs they have to implement becomes very challenging if there is not a sufficient rate base to support those investments so that becomes that becomes a very different issue and one that you know i think the utilities are all facing as we hear i believe secretary to yourself said that the shrinking rate base becomes a problem as more and more customers come off of the utility off the off the utility quick clarification on the legal issues so rather than you know the utility going and hiding somebody and training them for three years to become really experts in cyber security is are you saying that we cannot basically have a 30 like pgne contract with a company like google to actually take care of the cyber security for them well so maybe i can just answer what i've seen in other industries there's some aspects of security that can you can outsource for example monitoring even the security operations center but if you're an organization that's responsible for operating critical infrastructure you have to have some responsibility of the security for that that cannot be outsourced well part of our monopoly charter right as utility companies is to ensure the reliability of the grid so while we might be able to hire um you know the best brains in the world which we often look for in order to help us solve some of the these problems we would not abdicate our responsibility for maintaining the reliability of the grid because that is our charter as a business the grid wise architecture council actually is doing a lot of research on something called transactive energy so you might want to take a look at that as far as distributed energy resource integration with the grid because that that might be an area or a model for what the future of you know the business model migrates to in the utility sector so actually i have a very brief comment but before i get to the question you can outsource to google because the hardware software integration is is intimate period but i when i was listening to you now i speak to stanford professor um i had a previous job where i had to know a lot about cyber security i mean we guard the nukes we have national labs um and my question to the panel is how many of you have actually have security clearances and have gotten briefings anybody okay so um let me let me just say that it's a lot scary and you think and then there are various levels of security reasons okay even the lowest level security brief from wolf i will you will be convinced it is a lot scary and you think uh and the higher up you go the more scared you get so audience if you weren't scared enough enough already uh so so so um i'm and when we were working with homeland security in trying books we have a lot of the very high tech computer stuff again you know some of our firewalls have to be all the stuff having to do the nuclear weapons has to be very very good uh and as we start working with them and trying to figure these things out this has come up that uh because you know the financial guys have are very concerned because you know this is billions of dollars switching by a keystroke but the energy infrastructure should have at least equal concern and so i'm you know and so this is something where we were trying to figure out how to get the utilities to get a security clearance type of thing that you begin at least get a glimpse of it when google got a glimpse of what was really happening and surrogate bren in particular he just oh my god it's it scared the jesus out of google so okay yeah i mean this is serious there's nothing so the most frustrating meeting to your point the most frustrating meeting i had in the last year was when my ceo came to talk to us who were working in cyber security in the research department about the briefing the classified briefing he went to he couldn't tell us anything except just vague generalities about foreign militaries attacking investor-owned utilities and i'm like well we cannot if a foreign military government with all their resources decides to attack an iou infrastructure we need help from our own military so this is this is a work in progress because homeland security uh and you know the administration not this particular administration any administration really has to begin to work with the utilities because most chances the utilities won't have to wear with all i guarantee you yeah no and yet and there are certain things that you know but and but then it's once you know what's possible it also makes it easier for people to get you know it's got a double-edged sword to this but but i would say not to make anyone in this audience nervous but you should be nervous on that note i hope you'll all sleep well after hearing this panel let's thank the panelists


Howdy, y'all. Welcome to the very last session for today, the one that everyone really wants to be at dinner to already, and I don't blame them for leaving. So we're going to talk about media ads and performance today, and it's a Vox story. So hi, I'm Ian Carrico. I'm a web performance artist at Vox Media. You can find me online at I am Carrico on Twitter, GitHub, Drupal, IRC, my website, et cetera, et cetera, et cetera. Also, this presentation was half written by Jason Ormans. He's the other performance engineer at Vox Media. He is Okor all across the internet, and he is actually on paternity leave right now, so we're all hoping for the best. So what is this presentation going to be about? This presentation is all about two performance devs working on a legacy code base. We have a code base that's been around for a very long time, the Cora CMS, that we have been charged with making the entire site more performance, or all of our sites more performance. It's practical tips on what we have altered to make the site more performance and what we've done to enhance performance across all of our verticals, as well as some overarching lessons that we have learned in this venture. I kind of want this to be a frank discussion of the successes and failures that we have had, as well as kind of what we've taken across. A lot of performance talks, including all of mine in the past, have been based on the idea that when you're starting anew, you can have performance budgets, and when you're doing a redesign, you can do this or that. We don't have that ability anymore, and a lot of people in a lot of sites don't, because we have code that we're going to be stuck with, whether we want it or not. So let's first talk about the constraints that we face when we're working on performance. The first is that we cannot do a redesign. We have a bunch of different sites, and while occasionally we do do redesign for sites, we have not the ability to start from the ground up, migrate to a new system, or to start everything from scratch. We have a huge amount of users and a huge amount of sites that need constant attention, and we can't just stop supporting them for some way, shape, or form. We have eight verticals. So we run Vox.com, Espionation, Curved, Racked, Eater, The Verge, Polygon, and Recode. All eight of our sites. Anything that we do needs to work for all of them. We can't easily single out a single site and say we're just going to work on performance of this. We need to work on performance on all of them. So at any point in time when we release code, we're releasing it to all of those sites for the most part. And finally, we have advertising. I'm sure a lot of people use ads in their sites. Our business is primarily advertising driven, so we don't have the option of just saying, oh, no ads. There'll be a few accountants coming at my door wondering what I just did if I chose to do that. So we have our money pixels that bring in all of the money for the, you know, what time it is. There we go. So our plan. We stepped in and we started almost a year ago. Actually just over a year ago. So we started April in 2015 and sat down and made a plan on how we want to go and attack performance for all of our sites. We decided to kind of do a triage as you would in many different situations. So we start with the big ones. We start with the things that we felt we could make a quick impact and something that could easily, you know, show both the higher ups and other does, you know, the impact of what the foreign team can really do. And that started off with images, redoing how we work with images across all of our sites, how we do fonts across most of our sites. And then after that, we went to the more time consuming options, the ones that we knew we're going to take a bit more time, a bit more finagling, more work with other teams within the organization. So things like ad calls where we had to work with all of our revenue team, all of our sales team, all of our design developers on many different fronts. More image work. There's always more image work to be done that will take a bit more finessing. And also updating our JavaScript and how that is being run. So we're going to start with images. I'm going to slow you down right now. I did not do any of the image work on our sites. This is actually all Jason Orman's work. And if you're interested in learning more about this, again, his Twitter handles Okor, and he has gone deeper into images than I knew was possibly available. However, at first, we looked at delivery of our images and the different ways you can do that. So the primary methods that you can send off images to a site. You have a regular IMG tag, which we use for a lot of things. You can use lazy loading or JavaScript where you load it as it comes to a viewport, or you can load it after the page load happens, but something where you pull that image in with JavaScript after the initial load. Or you can use the picture element, which allows for various different sizes or types to be pushed forward to the front end, depending on your viewport and a bunch of other various variables. And we use, we actually use a little bit of all three of these, depending on the type of image and where the image is and why we're doing that, which we'll get to in a little bit. But we also want to make sure that the images that we're sending to the user are the best images for that user's screen, et cetera. We have opted to do a lot of real time processing. So almost every image you will see on any Vox property has been sent to a server, resized and reformatted to be right for your screen or screens very similar to yours. So there are a couple of options doing this. You can use software as a service. There are some things that do this for you. We use an open source project that we run on our own servers to resize, reformat a bunch of images. You could also, in theory, do pre-processing and have images as they get uploaded to the server, your editorial team puts it up there.  We don't do this because we have hundreds and hundreds of articles going up every day. Thousands of images getting processed every day. Most of our articles get looked at within the first 48 hours. So we don't want a huge amount of images stored on a server that won't be used past the first week or so. So, nope. So the open source solution we've been using is Thumbor. This has turned out to be amazing. We can send off different sizes through any single image, different formats. We're working right now on, I think he's doing JPEG 2000 support for this and making sure we can get that for some browsers. The cool thing that Thumbor can do is it can send off either a WebP or a non-WebP version depending on if the browser supports it. So this is kind of the before and after. The left is loading the Verge homepage without WebP on mobile, 3G. The right is loading it with WebP. As you can see, we save, you know, 24 seconds of load time just from loading WebP images. And Thumbor does all of that work for us. We do a little bit of script on the front ends to confirm whether or not a browser can use WebP, but that's it. Everything else is done completely by Thumbor to deliver smaller, faster, more efficient images to our users. Look how you can hear my straw. This is a very hot mic. And then we also looked at the timing of our images. How each and every image loads and when that happens within our load order. So we want to load our critical images first. We want to load the logo. So the logo is usually an IMG tag. We want to load the hero images as fast as possible. The hero images are usually picture elements because that is handled by the browser and sent off immediately. And then secondary images are delayed.  Those are done by JavaScript when you're close to the viewport. So that, one, we don't load extra images that users aren't going to see if they don't scroll down all the way. And, two, we don't accidentally have an image that's further down, possibly halting an image further up that we definitely want a user to see. Then we also use the preloader meta tag to connect to the right subdomains for our images so that when a user comes to the page, already a connection is being made to that server so that we know those images will definitely be downloading as soon as possible. And that's just a meta tag in the head. So just altering the timing on espionation. Again, the left is before, right is after. But we save visually for that first paint. Let's see here. It's six seconds, roughly, more or less. And it's not stopping the full page load will still take longer if you're including the images down below. But by lazy loading the images down below, by making sure that we have that preloader, and by making sure that we use picture element and the most important tags showing up on the right side, the user feels like they're having a much faster experience, a much better experience. And it's a whole lot better for UX. And this is our first overarching lesson, which is tackle the straightforward issues first. We found this is the best for a couple reasons. One, we showed the executive team right off the bat that we're useful, which really helps sell our cause for later work. They were already on our side and that performance is a good thing. But this is a great way of showing them. This is exactly why it's a good thing. But also it kind of builds up team morale right immediately off the front of, look, we can make an impact fairly quickly. So let's let's hit those ones that images were one of those things that we have so many of them changing a few things around like using the picture element for the here images was fairly straightforward, compared to other work that we could nail out within a month or so. So the next thing we did is fonts. And this is what I did it first. We use font face observer by Bramstein. It's a wonderful library that will check to see if fonts are loaded within the browser or not and alert you if they have. You can also use the document fonts, the font events already in native browsers. However, we use, I guess we'll get to this second, but font face observer is a small little JavaScript library that we send off the fonts that we're loading for a page. It returns back with whether or not they've been loaded with some promises. So we set that up to monitor with whether or not fonts are loading on Vox, Eater and curbs. We haven't rolled out some of our other sites because of the fonts forge that we're using doesn't support. They do things weird. Then we set up with Google Analytics, where we would send out to Google Analytics whether or not fonts loaded on pages or not. And we just sent off a quick event. Fonts loaded, fonts failed, or fonts are available. And we looked at the analytics data of, okay, what browsers are failing, which ones are succeeding?  There was no issues. But then we use that Google Analytics, and we sent it to Bramstein. We figured out what browsers were failing the most. And we did some debugging with font face observer on some older browsers on how to fix those ones. But also we found out the one that was failing the most often passed, like IE7, which was Google Chrome. Oddly enough that font face observer was failing most in that browser. So this is where we then figured out, okay, if we're failing most in the most modern browser in this, for reasons that we cannot necessarily ascertain, let's use native font events. So for any modern browser that has native font events within the browser, we use that first.  This allows us to load our scripts asynchronously. It allows us to push our font loading further down the critical rendering path to make sure that we have content loads first and then fonts get loaded in whenever they are available. And it also allows us to avoid, for the most part, the flash of invisible text where fonts go away and then reappear on your page sometimes. I say for the most part, we have a very minor issue that's popped up on curbed recently because we have updated some other things. But then we also worked with our design team in every site that we implemented to have better fallback fonts. I'm sure many people who are used to just fonts loading up on our page immediately because they're part of the rendering path don't always look at what the fallback fonts look like. And our design team hadn't, because we weren't loading the A-Sign or A-Sign before, like they kind of sort of thought about what they should be, but not what they look like. But we did a lot of work with them, a lot of QA work to make sure that all of our fallback fonts are as close as possible so that the UX jar is as small as possible. And that took coordination between several different design teams as we're working on several different verticals at the time. But the end result is that things like the Verge and Eater and Curbed and Vox all look pretty good without the design fonts within, with just using the native browser fonts. And as close as possible. The nice thing about Font Face Observer is we set it up to add a class to the body of fonts loaded once it's there. So when that's not there, we can alter the font sizes. We can also like it's not just we have a good fallback font, but we can change the size to be a different pixel size, not just changing the fonts, making it so much closer for all of our main fonts. And this is the next overarching lesson. We could not have done this font work if we did not have buy in through not just our team, but design team editorial. I actually did have some editorial people freaking out when they heard that fonts might not be there immediately. Then once I explained to them, they're like, Oh, right, that's logical, but editorial people don't like the design or the visual aesthetic to be off. The design team doesn't like that. But once we have that culture of performance is an important thing, reaching out to these teams has been so much easier. And to this day, I have a bunch of key people in various teams that will reach out to have like, Okay, we want to change this or that or we want to make just recently, I was talking with editorial teams about forcing HTTPS on our site. And like the buy in for that is so much easier once you have that culture of this is just as important as design. This is a piece of design. So you'll find a huge amount of talks all over is that culture is something you need to do. And this truly is in our experience, an important thing. Also, even with culture on your side, there are still some things that we can't or it's harder for us to do, because it's a little too much change, I guess. So the next thing we attacked, and this is something I just finished a heavy bit of work, actually, last week, Monday deployed is advertising. Our advertising system was built. I forget the original, like for a system pre DFP, that was all synchronous. So we moved to DFP. And before another team did a lot of work to make sure that DFP could load asynchronous eventually. So this has been a long heavy bit of work to go from a very synchronous, slow version of loading ads to a more efficient way.  So first, for ads that we built internally, Vox not only creates content, we also have actually an announced advertising agency for our sites that we build sites. We have a bunch of people who are creating ads and they use templates and systems that we've built internally, but they don't, they're site builders effectively that don't necessarily know, okay, is this a performance ad, is this slow ad or what have you. So what we did is we have a set of a personal web page test server that runs a test on every single ad as it's published. And then the creator of that ad will get what we call our speed rocket, which will be either green, yellow or red. Green is great. Yellow is okay, but could use some improvements and red is bad. And what ends up as is if it's red, then they come to an engineer and say, okay, something's wrong with this ad, can we take a look at it? And sometimes it's, oh, there's some template work needs to be changed on how the ad is working. Sometimes it's, oh, let's just optimize an image a little better. But we use that system for an easy way for our site builders to know, is this something that needs to be looked at again, or is this something that's good to go? And generally speaking, most of our ads already were pretty good. And this helps us find the little ones. We also use header bidding. Has anyone used header bidding on their sites? It's a relatively new technology. Okay, I'll make this section short then. So header bidding is a way for third parties to compete with Google's DFP servers. So it's a way for us to increase our CPM, our revenue from ads. But it means that in the header, we are sending off requests to three or four different exchanges for ads. So not just DFP, but also other ones. The problem with header bidding is you have header scripts that are going off to third parties. When this was first started, and when it was first added to the page before I was focusing on this, it was all done synchronously.  And if any single one of those were to fail, or their CDN were to fail, then our site would be delayed heavily. So we have since reworked that heavy amount to be mostly asynchronous. We have like one main script that is synchronous, and the rest are all asynchronous. We set up timers to make sure that if your response time is over, such a thing where you just don't care about your bids. And a lot of this is done by Tecnerati, but we have a lot of work to do with this, and internal work to this as well. And we have some more work hopefully coming out soon to make it even better. But also our ad calling. Our ad calling again was synchronous. So the work that we just pushed rewrote our entire ad JavaScript from the ground up to be instead of one, because it was synchronous in the forest reliant on jQuery. Because it was synchronous, there was a bunch of other little mini libraries that we did. We rewrote it so that it could be run entirely independent of any other scripts on the page. So that all of our JavaScript on certain sites are now completely in the footer. With the exception of our ad JavaScript, which runs in line. I'm not quite sure if it will stay in line. It may move to be something else sooner rather than later. However, having it in line means that we have no internal JavaScript calls in the header until, at all, the only JavaScript calls that are being called in the header are all done asynchronously via our ad cs scripts. And our results are, this is one of the videos that we shared, before is on the left, after is on the right. But we save out 45% roughly on the start render time. We're currently just baiting it on curbed alone. But as you can see, we have a much faster site just by moving our scripts of the footer, by loading our ads more efficiently, and stepping back and rethinking how we did in moving code that was legacy or may not be needed anymore, or there might be a better way of doing that. Now, I joke this is also where you can see we have the flash of invisible text is coming back because we have finally made our critical rendering path so small. Now we actually are having some faunce issues again. So we need to go back and improve that a little bit more. But this is the biggest lesson that I was very appreciative of. For a long time, we were kind of focused on quick result tasks, things that we could show come out. The entire ability of moving our JavaScript to the footer and rewriting ads came out of allowing the performance team to tackle the projects that we want to tackle. Doing this took about two and a half, three months from start to finish. And it was a specific decision by our managers after a spay and we had a lot of work to do. So what we wanted to do was to step back and say, okay, you have all the time in the world, what do you want to do? And this allowed us to tackle work that other front-end developers have been wanting to do for years. Other teams have been wanting to get done, but allowing a single developer to have that freedom, it allowed us to actually go out and really make a change and very large changes that will also actually improve the performance team. There's another team that is so excited that we've done this because it will allow them to write JavaScript better. And the last thing we have is testing. Is all of this actually working? Our very first meeting of the performance team was actually looking at how to test our sites and how to set up testing across the board. We've looked at a variety of vendors. We've built some internal tools to do testing in very certain ways we can look at to know is the change that is being actually a good thing or is it something that's not good. We've settled more or less on speaker. Speaker allows us to monitor all of our sites. It's a wonderful service. I personally love it. Because we are a site, we are somewhat unique in that we're looking at not just one site, but I mean in reality hundreds of sites, but eight main verticals. It's a little hard to get all of our data organized in a logical way as well. But I personally love it. But the biggest issue is finding the signal through the noise. When we have as many third party scripts as we load between ads and analytics and other targeting systems and stuff, and you see one of those things could be causing our site to be faster or slower on a given day. So we have been working heavily on creating some custom analytics. We use the timing metrics within the browser to know when ads are loading, when ads are done finishing, when third party scripts are done loading, all so that we can find a signal to know where along this path are we actually having issues. Which section of scripts might be causing us slowness or is it something that we've done, is it something that we just committed, or is it something that's completely out of our control? We actually started with a friend. I ignore the entire page load time because that could be altered immensely from just a single ad pinging a server repeatedly. It's a useless metric when you have as many third party scripts as we do. But our start render time is something that I look at very directly and that's what I look at heavily. That being said, data is really hard sometimes. It's incredibly important to look at the data. It's incredibly important to make metrics of how you're doing, but looking at straight numbers on a database basis can be very difficult on any site that you're loading, anything extraneous. Do it, look at it, heavily look at it, but recognize that this is an issue, especially with media sites that is not solved in any way, shape or form. It's a very, very difficult problem. So those are all the main things. I want to say thank you. I want to lose some time for questions today. But yeah, do we have any questions, concerns? Anyone working on media sites who found something interesting? Not supporting what? Define road blocking of ads. Yeah. Okay. So the question is if we have, for using DFPA in asynchronous mode, do we not support road blocking ads or ads taking over the entire page? We do. And what we end up doing is if I remember this correctly, we have it in asynchronous mode, but we don't actually run a call to get the ads until all of the ads have been defined completely. And then once DFPA has all the information, then we do a call to actually call the ads. So we, I forget what it's, we're still using single request mode. There it is. And that is something that we have to have on our site. Because those are great ads and they bring in a lot of money. Mm-hmm. Constant more. Mm-hmm. Yeah. No, and that's one thing we don't have infinite scroll in any way to reform. The ads that we defined at the beginning that we are in the markup are the only ads that we have. We've been looking at altering that possibly in the future. But at the moment that's not a concern that we share. But yeah, I don't have a device on how to, oh, yeah, that's impressive. For devices, so, repeat, how do we monitor performance for devices was the question. Trying to get better at that. So our speed curve will monitor not for different devices, but for different speeds and for different browser windows. So that will at least give us a view into relative speeds on devices. But actually for, we don't have any standardized testing set for different devices. Something that we do do is we have a device lab and I personally use our sites. And I personally use our sites not as someone who wants to use our sites, but as someone who travels enough that I will go in different areas with different speeds and purposely look at how our sites are just stealing. I'm a huge proponent on data and numbers and making sure that you have quantifiable data behind what's going on. But at the end of the day, none of that matters as much as how your sites actually feel. So using devices is what I personally recommend over strict data. The question was if we've noticed any changes in site usage. So as far as the things that we have done for like image and stuff, those were all kind of sporadically pushed out. So like one small piece was changed and another small piece was changed. So we didn't notice anything directly because of those. The ads change that we made, we just pushed it last week and next week we're going to be sitting down and looking at the data impressions and traffic and things like that. So I can't really answer that question yet. Hopefully once we have all that data done, then I'll be writing a blog post about it. But we just, using a single week of data isn't enough to know if we're having more or a different type of viewer or if just the latest curb article was really hot that week. Yeah. So for our programmatic ads, we use a system called the media trust. And the media trust goes through our DFP calls. And they, I forget the exact percentage, but they do a subsection of our ads and they run them for not just performance but a bunch of different metrics.  So they go through and they scan a subsection of our ads and anything that does not meet our performance or other security standards will immediately be rejected and disabled from DFP's perspective. And I don't know how deep that goes into if like that entire publisher will get removed or what have you. But yeah, we are constantly scanning all of our, all of our creatives, third party creatives. Yes sir. Were we using to share content? What modules? So this is the fun question I get to answer. We do not use Drupal. I'm a reformed Drupal developer I'm going to say.  So we built our own custom CMS many many moons ago that we're still using called Chorus. And that is a Ruby slash Rails thing. Although none of the, none of the specific work that we, or I advocate for is specific to a CMS or any back end language. It can be done with anything. If you have questions on specific modules that can do something like this, all of these guys out in green right here could answer the latest, anything else? Well hey, thank you all for coming. I'll be around all day all the first day. And tomorrow if you guys have any more questions I want to come in and talk. Also I would be remiss if I didn't say the slides are at imceco.com slash media perf. They'll also be on my GitHub in a week or so. I need to make sure some back end changes are okay. And then there is feedback on the page for this for how well this session went. I'd be happy if you filled that out. There's a short link but it's also on the New Orleans Drupalcon site. And yeah, it's a presentation by Vox Product. Thank you so much.


What's going on everyone? Sam here with late in sports cards ripping four boxes of twenty-two Pidini Immaculate football hobby for Adam M. So good luck here Adam, let's do it. Corey Davis base, seventy-four of ninety-nine for the Jets, potentially having Aaron Rodgers coming in throwing to him. This is cool. Uh, towel is fifteen of fifteen Shrey McBride. It's half of the Cardinals had there too. This one there. We've got a Shrey Lance the standard eight of forty-nine for the Niners. Got clearly Immaculate's is Austin Eckler eleven out of ninety-nine. Got twelve of twenty-five Hassan Haskins on the rookie three-color iBlack Auto for the Titans. And nice one here we've got seventy-one of ninety-nine three-colored RPA Jumbo Malik Willis. Congrats on that one there. Adam, a shield out of this. That'd be cool. Another Corey Davis this time thirty-four of ninety-nine for the Jets. We've got first hit ninety-nine of ninety-nine is Gabe Davis four-color patch for the Bills. Ninety-nine of ninety-nine. Got clearly Immaculate's numbered ninety-four of ninety-nine. That is Drake London. Drake London clearly Immaculate ninety-two of ninety-nine Isiah Spiller. A rookie iBlack Auto. Nice one here you can make this one too. Twenty-one of ninety-nine that is Christian Watson and the next one. Christian Watson iBlack RPA for the Packers. And nice one here Jersey number RPA three-color Trayvon Walker five of forty-four for the Jags full sig Trayvon. Congrats on that Adam. Another Corey Davis base eight of twenty-five this time. This is cool. Seven of forty-nine Kyle Long for the Bears. It's gonna be Cleats Adidas patch. We've got Trayvon Walker. Congrats on that one there. Very nice. Seven of fifteen Trayvon Walker on the Cleats patch. Congrats Adam. Very nice hit. Full Adidas patch there too. We've got Tyquan Thornton. Is ten of ten on the Jumbo Helmet Tyquan Thornton. Stick that into his sleeves where I'm looking for. Eight of forty-nine that is Jalen Waddle Immaculate Records a hundred and is at four receptions in twenty-twenty-one for the Dolphins and twenty out of twenty-five Jersey number we got a Breeze Hall three-color two-color RPA. Congrats on that Adam. We'll get that into a mag for you buddy. Nice Breeze Hall coming your way. Never mind. Last box of iMac for you here Adam. Good luck buddy. Always fun ripping some ripping some personal for you my friend. See if we can get you something else for kind of nasty out of this too. Get Wahoos. Honestly man. Cooper Cup fifty-five of ninety-nine for the Rams. Clearly Immaculate's T. Higgins. That is fourteen of ninety-nine for the Bengals. T. Higgins. Hall of Fame jerseys. That is Lydanean Tomlinson. Thirty-eight of forty-nine for the Chargers. Nice one here coming up. Really nice one coming up here. It's gonna be Russell Wilson. That is four of five on the laundry tag.  Congrats buddy. Nice Russell Wilson coming your way man. Mag this and the next one potentially. We've got that is eighty-five of ninety-nine George Pickens on the iBlack RPA. We'll get that one into a mag for you as well. Very nice. Very nice. And ninety-nine of ninety-nine Jaylin Tolber on the True RPA. Nice one there. Congrats. Empty empty. That'll do it for your break there Adam. Thanks again for the support. We'll get those right out.


St. Lucia has recorded two new cases of COVID-19 in as many days. Wednesday, October 21, 2020, Case 38 was confirmed. The individual is a 53-year-old female from Grosé-les. On Tuesday evening, 20th October, Case 37 was confirmed. The individual is a 43-year-old male from Grosé-les. During the contact tracing process, he was identified as linked to cases 33 and 34 and was placed in quarantine. He is now isolated for care at the Respiratory Hospital. The latest case, the 53-year-old female of Grosé-les, was also identified during the contact tracing process as linked to cases 33, 34 and 37. She has been in quarantine and is now isolated for care at the Respiratory Hospital. Dr. Sharon Belmar George is the Chief Medical Officer. The new cases for this week have all been identified through the contact tracing process and that tracing process continues this week. All efforts are being made by the public health team to rapidly respond to contain further spread of cases. We anticipate new cases as we continue the process to identify, test and isolate further contacts. As we proceed with the necessary public health interventions, a serious appeal is made to the public to cooperate with our health officers and to practice daily the behaviors which can protect your health and safety. We remind every individual that 80% of persons who have the virus experience mild symptoms. As such, it is important that we all remain vigilant and respond promptly to any symptoms of concern to avoid further spread of the virus. The Ministry of Health, we take the opportunity to remind the public to adhere to the infection prevention control measures, ensure that a mask is used always when going out in public, maintain a safe physical distance from others when out as well, wash and sanitize your hands often throughout the day. If experiencing flu-like symptoms, seek medical care immediately at the closest respiratory clinic. We also want to remind the business places to ensure that all of the protocols that are in place to keep your patrons healthy are sustained.