rators and LA it looks like in Santa Monica. Consumers, taxpayers could have an app, a mobile app where they can see all the scooters available from all the companies when then a quarter mile walk of where they are. So that would be really interesting and useful. I look forward to it eagerly when you allow, when you have an API endpoint for the data you're collecting. And the second question is so what is the women on a number of cars in LA or Santa Monica? And I guess if there's a woman on a number of scooters and there is and I get to understand it and it's a political reaction partly. When we have a discussion about how many cars are owned by individuals or leased by companies in LA County and the problems they cause and how they compete with scooters say for parking spaces that could be used by scooters in the public right away. It's a great question. To the second part, I think I just saw a headline was it in the LA Times yesterday or two days ago about how LA regionally really needs to start thinking about how to either charge, driving more or so I think it's a relevant point and we're not quite there yet with MDS. This data spec so far has been focused on the micro-mobility, the shared mobility. However, there are conversations ongoing right now about how to, on GitHub, yeah. And it was always kind of the idea to expand MDS to cover Uber's, Lyft, taxi, that kind of thing. So we can start to get an idea of how many vehicles are on the streets. To the first part of your question though. Well, one, just a quick addition. One of the reasons, just like a quick, I can hear the policy person I work with telling me this answer from, they did not want to spend Saturday at a Linux conference, shockingly, but you got us instead. But we're really interested in the governance and historically the city has had strong interest in the governance of fleets. Governance of individual, like if you want to buy a scooter off of Alibaba or Amazon, like it's not the city's prerogative to really, but if you wanna leave things in the public right of way, like ambulance, like when ambulance has come in, when taxis come in, when TNCs come in, the government has always had a much stronger role in regulating and the capping that number. If you wanna like, you can Google the somewhat sort of history of TNCs and attempts by cities to regulate them, and how we've been somewhat preempted by state law on that one. But on making sure that this data is available to consumers is, as part of complying with MDS, every one of these companies is making available their GBFS endpoints. That's a public-facing information feed about where these are at any given time. Some of the information contained in the MDS endpoints that the city consumes is sensitive, you know, somewhat sensitive information. It's not super personally identifiable, but we don't necessarily want that. But in terms of like being able to build app integrations, we do wanna make sure that there's an open marketplace for people to come in and work with GBFS style data about what's out there in real time versus more historic. A lot of the MDS stuff we talked about was about like more reporting and analysis, but there's a lot of really great stuff you can do with real-time stuff with GBFS and all these companies have that. Every company that is operating in LA, I can tell you, as a condition of their permit has shown me a working GBFS endpoint. We're gonna be issuing permits next week, and so that list will be published online. Yeah, so I guess what I mean rather is the dozen companies you're pulling in data from, right? So yeah, I get they all have, I guess, publicly accessible in theory endpoints you're pulling from, and I could too. But you're both pulling in all the data from all dozen companies, why not or when rather will you create one endpoint from each of your cities where developers can connect and pull in all that data in one big feed rather than having to connect to a dozen or more endpoints from each contractor or provider, so. So this is one where I'm gonna say that things you should never say as a government employee, which is that our peer cities are ahead of us on this one. So both the cities of Austin and Louisville have used this to release this. We are working on it. We are just resource constrained. They permitted three companies. I have 11 to go through and write integrations with. So it just takes time. But you can actually see all the trips as I showed that little dashboard that Austin put together, but you can go download all the Austin trips available as well as Louisville. I think the Louisville data actually ended up in the New York Times story recently. So yeah, so here's their sort of data. And then this is just available. We're gonna make some amount of data. We're still figuring, you know, there's process. It needs to be time, but we're gonna make it available at data.allycity.org at some point. I have a quick question, which I'm gonna use my role as moderator to just hop in right here and totally hop the line. Yeah, no, I'm cheating, straight up cheating. My question is, do you track crashes and do you cross link those to the hospital reports so that you know, because these are self reported crashes, right? So you know that you're going to have at least certain amount of injury based on what the hospitals are telling you. Yeah, that's a great question. And one that comes up a lot, right? The public safety aspect of all of this. I think that's what a lot of our jobs are about is ensuring that these devices are used safely and our streets are traversed safely. So yes, our police department, if they respond to a call, whether that's a crash or something happened and it involved a scooter, they take a mark that they note that there was a scooter involved in that. So we're collecting data on injuries from scooters, the citations that we're giving, both the users and the companies. When it comes to merging that with hospital records and it gets a little dicey and this is kind of, you know, pulling up the blankets of government, the police department, believe it or not, does not always share data with the fire department. And the fire department is often responsible for transport to the hospital or even providing some medical care right there at the scene. And so linking that data together is a little bit more difficult than just, oh, we know a crash occurred at this intersection at this time, we should be able to pull up those records. Not quite as easy as that, but it's definitely, yeah. You can't track rapid sudden deceleration in the GPS signal? I mean, that seems like a better, I don't know. Yeah, I was just curious, are you guys sharing anything with like open street maps so people can look up like where the scooter stands are or whatever, you know? Cause you said like you can find the data feed, you can see where they are in the city and you can see where the operators are, but I don't know if you guys, like you know where a taxi stand is at the airport, but I don't know if you know where the rental stations are or whatever, so people can look up at the layer of their open street map, where they are. Part of the issue with that is the nature of these operations are that they're dockless, right? So we don't have a list of here the stations. Now some cities, I was just talking with Paris France the other day, are looking at creating specific parking zones and kind of that's the model that they wanna operate under. You're not allowed to park wherever you want. There's a bunch of zones and so in that model, I think probably they would be looking at how much parking is occurring there. In Santa Monica, we've put out a number of drop zones that the city has painted kind of, you know, squares on the cement. We're asking the companies to figure out ways to incentivize their users to park there, but it's not a requirement, you know. We want this to be as free-floating and as useful as possible. Yeah, we're about to pilot our, I think we're gonna go stencil next month, number of drop zones as well. I don't think there's one of the things that I would like we're gonna figure out, especially on the agency sides of APIs, how can we express all of our drop zones, whether you're Paris, France is like, you can drop here, you cannot drop elsewhere, whether you're ours, which is in downtown, you need to use a drop zone, but outside of downtown, or if you're more than 0.2 miles away from a drop zone, you don't like, all these rules are like getting increasingly complex and like publishing that as, and sort of what I kinda hinted on the thing that's more in development is this agency API, or if you're Santa Monica's where you have, we want an incentive in the drop zone, you know, these are all different policy responses, neither of us are policy people, and should not be asked to like decide these things, but our policy people have come up with different opinions. So we're just trying to figure out, well, how do we make sure that the technical work, whether it's integration with OSM, integration with these apps itself, integration with future third party, you know, stuff like transit or Google Maps, that any constituent doesn't really have to know the laws of eight, whatever city they're in, can just use their mobility device. And that's the beauty of this technical spec. So you've mentioned one time in response to the question about making data accessible to just general developers, you talked about some data that was potentially sensitive, and I'm curious what your sort of privacy policies are. For example, for some of the trip data, do you only store it for a certain amount of time, or do you have any policies on how to both protect and limit the storage of this data that is potentially identifying? Which has been, I'm sure you're aware, there's been a lot of very interesting research about taking this data, trip data in particular, that seems like it might be anonymous at first, but then tying it to individuals. So the city of Los Angeles has an information handling guidelines policy is promulgated, so we have this huge, we have an information security policy, we have a chief cybersecurity officer. It is like a fairly robust team of people who work on this and just tell me what to do, and I trust them to do their job, and they do a great job. With regards to re-identification, so essentially that's one of the things is we have, we're, the administration and government has a retention schedule, subject to typically your open records law, which in this state is the California Public Records Act. I can see like my government IT peers like nodding along about this, because this will like dictate how long you have to store emails for, how long you have to store records for. So we will set a retention schedule for this. I do not know what it is going to be yet, because we only started getting data from this in September. So it's not, typically that retention schedule is generally two years, but there are exemptions in certain laws. The other thing to be aware of is like, we don't necessarily share like this freely across things. We've done a lot of work to try and limit our risk while preserving the public safety aspect. So obviously we're not going to collect like, it'd be interesting from like a planning perspective to get people to use names and like see all their trips that they took or their zip codes. We're not getting that, because we immediately said like, that's a bad idea, let's avoid that. In terms of longer term like preventing re-identification, one is being careful when we do an open data release to make sure that re-identification isn't easy. You saw this when like NYC Taxi did that. The other thing is re-identification always requires some other piece of information. So it's making sure that there aren't any obvious other sources to join, especially given the shared nature of these bases. A lot of that research is based on devices that are not shared. Typically a lot of the papers I've seen have been phone traces or personal automobiles is what they've been reconstructing with. The papers I've seen, you get into this like re-identification is a probability and I like can talk people's ears off about this, but at the end of the day, we think the risk is substantially lower based on the fact that these are shared devices rather than personally operated devices. But we're gonna be very careful about this issue because we again operate in the public's trust. And I'll just point again to this Austin work. They did a lot of work. Obviously they're releasing trip data here. So this is the sensitive piece of it, but they're anonymizing the trip, the location paths actually. So they published a lot of the code and how they're doing this. And I think we're all kind of trying to figure out if and when we do release this data publicly, just generally publicly, how do we do it in the safest way so that we're not risking re-identification or other things like that? Yeah, it's definitely, we have these conversations a lot, particularly with the companies and we like to remind them that cities are not, this isn't our first foray into sensitive data, right? Like we have license plate readers. They operate the LAX airport. Like there's all kinds of sensitive data that cities deal with every single day. I have a colleague who operates a system for our police department that is the list of domestic violence abuse survivors, where they experienced abuse and where their abuser lives so that they can respond in an emergency as quickly as possible. That is so much more sensitive that when I went to our cybersecurity officer with like this re-identification question, he almost lasted me. Cause he was just like, I'm used to holding stuff that is like, truly if this got out, like people's lives would be at risk. And that is, you want government in the business of solving problems where people's lives are at risk because we're owned by you. Like it's a democracy. Like you're here. We're here to act in the public's trust. So, you know, again, we're treating this seriously, but this is not the first time that any of us has seen super sensitive data in our organizations. I was actually curious about what your data flow, I mean, how much data are each of you getting for a day? And if you've looked at formats other than Jason. So the city of LA is about to permit 40,000 or so devices that we'll be launching in the next, we're going to be completing this permitting cycle by March 15th. Each of those typically generates on order of five-ish events a day, depending on use, maybe depending on the company and their operating model. We do not store the JSON. We only really sample every 12 hours worth of JSON. We store it in Postgres is our stack and we do use Postgres and PostGIS. So we do some ETL. JSON is very good for the interchange aspect, but for like long-term storage, it is not, we're not just like infinite blobs of JSON. I think we actually do keep it for historical because just throwing it on S3 is cheap and certain companies have changed their data after the fact, so it's very useful to keep time-stamped records of what they sent you. So always saving things is cheap, but yeah, our stack is pretty much hit the API every hour, get 12 hours worth of data, ETL it into Postgres, and then everything runs on top of an ETL version of the data. And our stack is very similar and I was just looking at kind of our complete data set so I have some numbers in my head. So we've been collecting data since I think, let's just say the beginning of November, so from November 1st through the end of February, Santa Monica has four companies. They're each permitted around 750 devices. So for that time period, for that amount of companies and devices, we have approximately 700,000 trip records and approximately 4 million or so status change records. Four months, four million records, so about a million records per month across all four companies. So it's not a terrible amount of data. One of the things that's challenging, and this is kind of where my, I've run up to my knowledge on Postgres and PostGIS and I've had to do a lot of research, is when we're looking at the trip paths, of course, each point is its own data point with the time and the location, all that stuff. So when you're unrolling a trip path and trying to do something like classify all the points and which ones are in downtown, which ones are in, that becomes a processing task. So I think, if I can remember the numbers correctly, we have, once we unroll all those trip points, it's about 40 gigs worth of just trip point data, aside from the other. So it's a considerable amount, but not unwieldy yet. There's also, I kind of reference this in the architecture. We got concerns from companies when we started this, that this would not be able to scale. And then we sort of explained to them, well, we have this nice sharding strategy called there's 88 jurisdictions in LA County. And then there's 50 counties in the state. And like, there's a very simple way to shard this into ways that the data size is manageable, which is nobody has the global picture, except for these companies. And you know what, they can hire engineers who make more money than me to solve that problem. Yeah, since we don't have any fixed drop zones in other places, so the question I have is, have you come across any lawsuit or anything which has been filed on account of a disabled person having an accident with these lying around? If it happens, who is liable? Yeah, not a lawyer. Somebody will do it. We'll get precedent. It will happen, I think. Any other questions? All right, let's give another round of applause for them. And the next presentation is in 34 minutes. Test, test, test, okay. Test, test. Hello, hello, can you guys all hear me? Looks like it. I see some heads popping up. We'll give it a few more minutes in case people are running a little late. It's 4.30 right now, but rather than start right on the dot, we can, I see some people trickling in. We'll give it till 4.32. I feel like two minutes is a good, you know, slob. Say it again. I'll get the rerun on YouTube. They could, they could, you know. But, you know, some people like to be there for the live event. It's like I work at Ticketmaster and I think about these things. What are those Ticketmaster people at scale? Never heard of them. What do they do? They're here? What? So, yeah, I'll just get started. My name is Mark Rodin. I'm the moderator for the open data and the open government tracks. And I have been trying to organize all these talks that you guys have been seeing for Thursday, today, tomorrow. And one of the things that, one of this talk came out of a very lengthy conversation that Dean and Kenneth and I and Erin as well in the back had over open voting in Los Angeles. And what happened was, about six months ago, the LA Times published a very quick snippet article about three paragraphs long that said, Los Angeles County is going to be making their voting system open source. And that was about it, that was all they said. And I said, well, that's really interesting. And one of the kids that my son goes to school with, his mom works for the LA Times. And so I reached out to her and said, who do I talk to? And she said, hold on a second. And like two weeks later, she was like, I finally tracked it down. You're gonna talk to this guy, Dean Logan. Here's his email address and just go to town.  And he said, yeah, have you heard we have an election in November? I'll talk to you after that. So that was pretty cool. And I could be sympathetic to that. I had heard that there was an election too. So I was pretty good with waiting. After the election happened, I reached out to them again and they were like, what just happened? We just got blown up. But yeah, and then they met with me and Chris Smith is actually sharing the development track here as well. And the two of us, we sat down and our first question was, okay, so you wanna open source the software? Why don't you just throw up a tarball on GitHub? And they looked at us like we were retarded, just like we did not understand anything about the world. And they said, well, what happens if there's a bug found? What happens if somebody tries to steal the software? What happens if somebody expects us to support it in the state of Oklahoma and we're not Oklahoma, we're a county like there's all sorts of laws and we don't have any support. And I was like, oh, this is all actually really complicated and really hard. And I had no idea just how complicated and hard the situation actually was going to be. But one of the things that was actually an interesting confluence of events was the way in which they were describing their system and how they wanted it to work. And Chris and I sat there and we said, so what you're trying to do is you're trying to make sure that one person goes to an event and gets to get in once and then they get to experience what happens in that event and then they leave and nobody else gets to do it a second time. And I said, yeah, I was like, so that sounds a lot like ticketing. And they said, oh yeah, I guess that is a lot like ticketing. And so that sort of formed the foundation for us to be able to talk about a lot of things because then we could start to understand just sort of the relationship that people would hopefully have with this new system that these guys have helped to devise and develop over the course of the last little while. So we should actually back up a little bit and actually, so we wanted to do this as a Q and A type of session where I was going to ask these two guys and hopefully shed some light on this whole process and hope to bring this into the open source community and see where this can go. But that doesn't preclude you guys from asking questions. The only thing is is that we only have two microphones. The one I'm wearing now and the one they're gonna be swapping between the two of them. So while we're recording, if you guys have questions and not totally interrupting the flow of everything, just raise your hand and then we can take the mic out and then I gotta bring it back. So it's a bit of a back and forth but that's what we got. So I think we really should start with asking and I guess I don't know which one of you wants to handle this but why did you guys feel that Los Angeles County needed to have a new voting system? I'll take that one. So thanks Mark for having us here and I did not recall that full story so that was great introduction. So I'm Dean Logan. I'm the registrar recorder County Clerk for Los Angeles County. And so I can sort of set the context for the project that we're talking about. Kenneth is our project director and senior member of our IT staff in the department. So when it comes to the actual part of this that I'm guessing most of you are interested in in terms of the programming, Kenneth can take most of those questions. I also wanted to point out just real quickly, Steve Vo who is here from Digital Foundry. Digital Foundry is our partner for the development of the tally system component of our new system and then Aaron Navarra's in the back is our government and legislative affairs manager in the office and a key part of the project as well. So just a little bit of context that the nexus of this or the genesis of this project really goes back all the way back to the 2000 presidential election, the infamous Bush v. Gore election where we as a nation saw the results of a razor thin outcome in the presidential contest and we began to see very vividly the warts on the process of how we conduct elections and in particular how ballots are counted and votes are reported across the country. At that time, for those of you who are around and who were watching what's happening on that, the focus was really on Florida because that's where the ultimate outcome was determined by the votes in Florida. Again, not just razor thin nationwide but razor thin in the state of Florida and much of the blame or the diagnosis after that election was based on the use of punch card technology as the means of voting in Florida and the flaws with that. The infamous dimpled or pregnant chads and whether or not operationally they had checked those cards and if there were votes that were dislodged or votes that didn't get counted. So what followed, I'll try to make this quick to get it in context of what we're doing here in Los Angeles County but what followed was a historic change in policy in the country where Congress for the first time in the nation's history decided that the federal government would make an investment into the infrastructure for elections and the focus again was on replacing punch card voting systems because that was the primary method of voting that was used throughout the country. Not the only one but it was the primary one and it was seen as the problematic piece in the 2020 election. So Congress appropriated a whole lot of money that was distributed to the states in 2004 under a piece of legislation called the Help America Vote Act. There were constraints with that money that included things like making sure that every polling place had at least one piece of voting equipment that was accessible to voters with disabilities for their ability to vote independently and without the assistance of another individual if they desired. There were a number of requirements for things on the policy side for fail safe methods to say if you show up at a polling place and you're not listed in the roster that you aren't turned away that there's an opportunity for you to cast a ballot. A number of those types of things. Also voter registration was moved from the local level to the statewide level. It required that states adopt a statewide voter registration database. But staying focused on the voting technology, the actual voting equipment, what followed is most of the country quickly moved to very fast engineered direct recording electronic voting equipment, paperless direct electronic recording equipment. And then fast forward to 2004 you had the next presidential election. All this money was spent, all this movement to these touch screen paperless voting systems and lo and behold, there were still issues on election day in 2004. It wasn't as close as it was in Florida in 2000 but there was certainly a doubt about the viability and the reliability of the touch screen voting systems and the lack of a paper trail. So to bring that back locally what happened here in Los Angeles and some of this predates me is it's Los Angeles is a very unique jurisdiction from an election standpoint. We have 5.2 million registered voters here which means as a local government we have more voters in this county than 42 of the states have statewide. So we operate in a marketplace that produces voting systems in this country that really doesn't build something for the capacity of a county like Los Angeles because the customer base is much smaller counties out there. So when we did the assessment after that new federal law passed and here in California there was a voter initiative to replace punch card voting systems in California. There was no system available on the market that could meet the needs of Los Angeles County. So we retrofitted our punch card system into what if you're a voter here in Los Angeles County you may be familiar with the Incavote system. We basically switched the IBM punch card system to an optical scan system where you fill in the bubbles with ink and we run them through retrofitted card readers that use infrared to read the mark on the ballot instead of the old chat. That was always intended to be a temporary solution waiting for the market and the regulatory environment to expand and provide an opportunity that would represent the new modern voting system for Los Angeles County. But as I said nationwide, those new systems when they hit prime time they kind of flopped. And here in California the Secretary of State actually decertified those new systems that that federal money was spent on and counties reverted back to their old system. So we sort of dodged a bullet here in Los Angeles County. We were on one hand, it was unfortunate there wasn't a market solution for us. On the other hand, we didn't spend all that money because there wasn't something that would work for us. We waited a few years for a solution to present itself and for the regulatory environment to open up to new more modern approaches to this problem. That never happened. So we decided in 2008, shortly after the 2008 election that we couldn't wait any longer. That if we waited any longer the current system that we had just local elections jurisdiction in the country found themselves unable to conduct an election because we couldn't get the parts for our voting equipment and we couldn't support the software changes that would be needed for new legislation. So we launched this voting systems assessment project that had morphed into the system that we're talking about today. And we sort of went out on a leap of faith and said if we're gonna do this on our own and do it as a grassroots effort, then we want to free ourselves of the regulatory constraints, the fiscal constraints and even largely to the technical constraints and start from the perspective of the voter. If we were gonna vote a voting, build a voting system and a voting experience that met the needs of the 5.2 million voters in Los Angeles County, what would that look like? So we spent a lot of time talking to voters and prospective voters about what they liked about the voting experience. Currently what they didn't like about it. We talked to people who were registered to vote but who don't vote and ask what is it about the process? Not the politics, but what is it about the process that you see as barriers to participation in the voting process? And that morphed into our introduction to a team or a company called IDEO in San Francisco. They're an international human-centered design company and we partnered with them to again go on this journey to design a voting experience, not a voting system. We purposely didn't look at this as solely a technology or a system project. We looked at it as a voting experience project. There were a few things that pertinent to today's discussion that were really important to us about that is one, we wanted to do this in a way that whatever work we did here could be leveraged by other jurisdictions. So other states and other counties could move in this direction as well. We wanted to expand the market because the market was too small for our needs and it really was limited and still is limited to a very small number of proprietary private sector firms that basically provide the voting equipment that is used throughout the country. We wanted in LA County to have a voting system that was publicly owned and publicly operated. We're large enough that we have our own IT resources to support a system in-house and we believe and still believe that the voting is fundamental to our participative form of government and that that process, if there was any system that ought to be long to the public and be owned by the public, it ought to be the voting system. And so in the course of doing that design and coming up with this new voting experience, we also wanted to do this in a way that it could be shared but could be shared in a manner that didn't require significant amounts of money by other jurisdictions. We've made a big investment in this project and it's an investment that we believe wasn't made when the country moved from punch cards to touchscreen technology. And we want to be sure that our investment can be used to leverage that system beyond our borders. So where we are today is we are on the eve of the launch of this new voting experience. We will go live in March 2020 presidential primary election with our new voting experience. We launched the new vote by mail portion and the new tally portion of that system last November went extremely well. And beyond the technology, the whole experience will be different. So we're moving to vote centers, we're moving to multiple days of voting, separating the tally portion from the equipment that marks the ballot. And you'll see some of that as we go forward in the presentation. The user interface on our voting equipment and our tally system have been built on open source platforms, open source stacks. One of the challenges and Mark alluded to this and I'm sure Kenneth will talk more about this is how do we now move a regulatory environment that has always looked at voting systems as end to end systems with proprietary software? How do we create a governance model that allows for the sharing of that data and the ability to get contributions from communities like this to enhance it, improve it and develop it as we have new legislation and new needs, but do that in a way that also meets the security needs associated with conducting elections. I mean, as I think everybody here knows that every day since the 2016 election, we've still been talking about the security of voting systems and the nation state actors that want to hack into voting systems and that type of thing. So there are a number of layers involved in that. We need to get it right for LA County first. We're committed to doing that. We don't want to become a voting system vendor ourselves. We want to put it out to the community. We want to put it out to the industry and have it grow and evolve, but we have to meet the needs of our community first. So there are certainly people in the open source community who have been frustrated with the project because I think they support the open source development model, but they've been frustrated with the inability to interact with it and for it to be shared. So we want to get there, but we need a governance structure to do that. That's what led to the conversation with Mark and others. We've established an open technology working group that we hope will help us do that and we're in conversations with the regulators on that to the Secretary of State's office and other people. So that's probably more context than... You gave me like a list of questions to ask you and you went through like 10 of them. So it's cool. I will forward a question from a gentleman in the audience, which I think you're going to touch on, but I want to make, this is I think an important point that a lot of us here are thinking about. So we'll go through the list of questions, but I think that one of the things that a lot of people have top of mind is trust. So like, do I have a tactile feedback? When I do the Inca vote, like I can see that the vote is there and now we're moving into a touch screen, like how do I know that my touch is going to actually be validated and registered? And maybe every 20th vote gets flipped by a nefarious hacker instead of every third. And so it's really hard to tell. So I know we'll go through the list, but I think that the trust is the thing that we should definitely make sure to talk about. What do you want to talk about that now? Because the next question is talks about why does VSAP have to be an open technology? And maybe you can maybe trust it would be part of that discussion. I don't know. Yeah, I'll try to hit both of them. I wanted to go back to the two points that Dean talked about was wanting to share and then the security. When we launched the program and we established an advisory committee and a technical advisory committee, we established a set of principles that we're gonna, oh, closer, that we're gonna guide the development of the solution and guide the program more generally to its successful completion. Two of those, of the 14, one of them was security and the other one was transparency. And others were accessibility. Others were cost. I mean, there's general principles we were trying to meet. But what we found though, as you go forward, is that some of these things are intention and sometimes trying to, for example, make a system more accessible runs up against the desire to make it more secure. And throughout this project, we've been trying to balance these. But the transparency principle of the program was one of the key drivers for selecting an open source stack for the entire solution. All of the components are on open source stack, even open source programming language, everything. So that we knew that in addition to being publicly owned and something that we actually had the right to share, we also knew that the software that it was built on was something that could be shared and it could be seen. So that was one of the driving reasons why we decided to go with an open source technology stack. Really based on that principle, now what I think we're actually realizing that it's a lot more complicated just than making it open source. When our tally solution was certified this last November, right away we got requests. You know, where is this software? Or can I find it on the internet? And it kind of surprised us a little bit that people would be asking just so quickly. But we actually have to, we have to actually take into consideration other factors like the solution has to be tested, it has to meet regulatory requirements. Some of those requirements are actually pretty outdated but we still have to meet them. And so the idea of actually allowing the solution just to be out into the world and to evolve, while I think we recognize that it's a good thing, there needs to be a way to ensure that however it does evolve it stays in compliance with the regulatory environment. And that's a problem we haven't really solved yet. It's a problem we're actually trying to solve through this open technology working group. But I wanted to get to the question around the security and the trust. One of the other main principles of the solution was that there would be a paper ballot, that the voters selections in an election would be recorded on a piece of paper in an indelible way that can't be changed. And at the same time, and so that sort of speaks to the security piece of it, at the same time we know that technology is actually provides great opportunities for voters with disabilities, wide variety of disabilities, to participate in the voting experience. And so the goal was to try to find a balance between the technology and then making sure that we rely on a paper ballot that we know is secure and auditable. So the ballot marking device which I think you see some slides up here, that's for the in-person voting experience. That is the, what it does is it allows you to insert your ballot into the device to vote electronically and have all the assistive technology features of a electronic voting system. But then it prints that ballot back out. It's verifiable, you can read it, you can put it back into that machine, you can actually take it to another machine and it'll read it. That ballot is the official record, the legal record of the voters vote. And it's the printed text on that ballot. So there's printed text and there's QR code that actually makes the ballot machine readable. But the printed text is what is the official record. There's a linkage between what's printed on the ballot. There's a code next to the candidate that is selected. That same code is inside the QR code. We're using an open, as you all know, QR code is an open format that can be read by anybody's cell phone. You can read it. And the vote by mail ballot, of course, is also a paper ballot where you vote traditionally still in the bubble. So that's the key security aspect of the system is that it's grounded in a paper ballot that is auditable. Let me just make one other, I think a key distinction is the difference between the paperless touchscreen voting systems that I talked about before is that this device is a ballot marking device. The device does not retain any of the voter data. The tally system is entirely independent of that ballot marking device. The purpose and the security around the ballot marking device are around usability, security, and accessibility. It solves a lot of problems. So it eliminates the ability to make ambiguous marks on your ballot, which when people have paper and pencil, they do very interesting things with ballots. And that becomes an issue in recounts and in razors and outcomes. So this solves that issue. It also allows in a jurisdiction like LA County where we have to provide our voting materials in 12 or 14 languages in addition to English, it gives us the ability to present that in a way that's not just compliant with the requirements for language accessibility, but actually functionally usable for people who need those services. So for instance, we learned in our user testing that there are voters who function just fine on the contest, the names of the candidates and the contest labels in English, but when it comes to the initiative and ballot measures, which are very complex and very legalese, that they prefer to hear those in their native language. So this device allows you to toggle between those two things. So again, the security and the usability of that device is focused on the voting experience. The tally system is entirely independent and those paper ballots come back and are tallyed centrally at our facility and we're able to then apply a different layer of security to the tally system because it functions differently. The tally system isn't about the usability for the voter. The tally system is about the accuracy of collecting that data and accurately reporting it. So there's a system independence factor there that's critically important. So when you look at it, it may look very much like the touchscreen voting systems that came out after 2000, but the functionality of it is entirely different. So again, we're now at like question 12. So. You don't have to follow up. Well, oh yeah, if we're not gonna stick to these questions, I'm gonna ask the question that, so we're all in this room fairly technical people. You have touched on some technical concerns like separation of concerns. You know, you talked about the tally system and I know you don't have it in your slides, but could you very briefly maybe using air quotes or something like that sort of talk about the different components of the systems and how they might link together so that as a technical audience, we can get a sense of well, the tally system does this and your ballot system does that or if you have a presentation like that, I can connect it up, your call. Well, first of all, we have on our website and lavote.net is the registrar's website. There is a vSAP section of that website that has a lot of information about all the components. Up here, we have some brochures. We have a brochure that kind of describes the overall solution, the different components of it and we also have a one pager on the system security but there's basically five components to the vSAP solution. The first is a ballot layout solution so it takes information from our voter registration database. It takes it from our candidate filing system, compiles all this information and lays out the contests on the ballot by their appropriate ballot style and does that in all the mandated languages that we have to support. It also generates several files that are then distributed throughout the rest of the system that configure those different components for the purposes, the purpose that they serve. The ballot marking device, which we talked about a little bit, that device is managed through another device appropriately called the BMG manager or called the BMG and that'll allow us to actually configure the devices for an election and do it very rapidly for us to be able to deploy the solution to LA County across approximately 1,000 vote centers in March of 2020. We're going to have to be able to program and test and secure about 31,000 devices in the space of about seven to 10 days. So we have a network that is pretty sophisticated that we're building. It's isolated network, it's air gapped. For those of you who are concerned about that, it's not on the internet, it sits there by itself but it allows us to very quickly push election configurations and do diagnostic testing on all of these systems over a short period of time. We have an interactive sample ballot which is essentially a website that allows voters to look up their ballot on the internet to see the contests that are on their ballot to actually mark that sample ballot in the same way that you would mark a paper sample ballot that you received in the mail. You can mark this and say, well, these are my selections. You can change it. There's different things you can do for certain voters. You'll cover voters of military overseas voters and voters with disabilities. They can also use this solution to actually mark a facsimile of the ballot and send that back into us so that they can actually leverage their assistive technology on their home and send us back a facsimile that then gets remade back in our offices. And then there's the tally solution as we talked about earlier. And that's a central system. We're using approximately 20 IBML scanners. I don't know if you guys are familiar with scanners. They're the fastest scanners really. There are on the market and they're used by the IRS and banks and things like that where they have to process high volumes of paper in a very short period of time. That those scanners will scan the ballot to a digital image and then the tally system does image processing on those images to either detect the mark on a vote by mail ballot or to decode the QR code on the ballot marking device ballot and then does the tabulation of those and the reporting. One thing I wanted to add about the tally system which I think was really important to us, our system that we have today is very old and we've been barely keeping it going. There's a lot of constraints that we've been doing workarounds for to keep this system going but it's an old system written in C and COBOL and assembly and super fast, super lean and we use IBM 312, Hollerith, the old card readers that used to be your primary data input. Those things, those are still today what we use for our vote at poll ballots. This is what we're gonna be getting rid of, right? That system unfortunately is very, it was definitely not written, it was not programmed or designed for any kind of usability or transparency. It's a hard system to understand. People wanna see it all the time. They wanna know how it works and we show them this blue screen. It does work too. Oh, it works very well, works very well and it's very fast but it's not, it's really hard to explain to people what it's doing. So one of the design of the new tally system, we really designed it so that you could actually see how a ballot goes through its life cycle in the tally system. So we actually have an interface so when we're doing auditing, we can actually trace the record in the system that we use for tabulation. We can trace it back to just the marks that were detected or the votes that were detected on the ballot back to the digital image itself and then actually back to the paper because these IDML scanners allow us as they're going through, we can apply an ID to the ballot. So we can actually trace an individual ballot from the paper all the way to the end and we can see how the system treated that ballot, how it applied business rules in the case of a vote-by-mail ballot, you can have overvotes. Voters can actually vote for two candidates and vote for one contest. Well, that has to be, we have to apply business rules so that to ensure that it can't be counted when they overvote it can't be counted. So we have to turn that from a voting for two candidates to an overvote and actually count those overvotes that you can see in the system and it's very usable. We actually did apply some usability principles to it because we want certainly auditors and we want anybody else that we have to explain how the system works. We wanna be able to do that very clearly and make it easy to understand because knowing that the system is accurate is of paramount importance and being able to demonstrate that is the most important thing. So the audience has asked, several people's audience have asked me several pointed questions. And so we'll get to those. And some of them should be given by asked by the people who asked them so that I don't play the game of telephone and screw things up. I think one question that we can go to, this is the last two questions they have on their thing while we talk, before we talk about Yeager and how they're implementing it now, right? Open tracing joke? No, okay. What are the next steps on opening this technology and how can this community help? I'll answer from the kind of organizational standpoint and then, Kenneth, if you wanna talk about it technically. I think, as I said, we obviously have a timeline. We have to have a voting system up and running for the March 2020 presidential primary election. And from the inception for us, this was about replacing the voting system and modernizing the voting experience for the voters in Los Angeles County. We've gotta get that done first. And it's gotta work and it's gotta be reliable and we're confident that it will be. Then we need the governance structure for where's the appropriate place to, where's the library to put this code in? And how do we govern review and acceptance of modifications and how do we govern the licensing of that to other jurisdictions? And keep in mind for us, there are multiple sets of code. The beautiful thing about this system is it's not an end-to-end system. So ideally, somebody could get licensed to take the tally system and use that on totally different hardware and a totally different ballot layout system than what we use in LA County. Same thing with the user interface that operates on the ballot marking device. That can be used on different hardware. It's not exclusive to the hardware that we've had designed for Los Angeles County. So we need to figure out how to put that in a governance structure that allows for it to remain certified and secure, but also allows for other jurisdictions to access it and to adopt it and not to be limited by the limited commercial for-profit markets that we're in. And so again, we've established this open technology work group. Marcus is gonna be a part of that. I think about a dozen people that we're gonna pull together. We're gonna do that work now in the background while we're getting ready to run the March election. But we also think it's important that we do run the March election and we have success with that so that people will want to adopt it and use it. And it's gonna be, I believe, easier to do that once they see it function than just having it out there and wondering if it's gonna work. Yeah, I don't have any. There's one, there's a gentleman in the audience who actually has a very pointed question and I will let him ask it, so. Well, hi. So I am a member of San Francisco's open source voting technical advisory committee and everything that we do is public. O-S-V-T-A-C dot github dot i-O. As you can see what we've been up to. The elections commission itself has now put out contracts to get the work done and of course we discovered very quickly that the Secretary of State's press release about there being an open source voting system was inaccurate. I think the communication between you guys and the Secretary of State was a bit off. But we've put out a set of recommendations about what the system should be and I think that probably you'd find a very high overlap to Dean's point. We really believe strongly that you have to use an open source, an OSI approved license that complies with the open source definition. Not one of these things with funny clauses added to them. And so, we're watching what you're doing and we're encouraging it because you guys are a bit ahead of San Francisco but I think there's probably some room for sharing ideas between the two cities just as the usual love of San Francisco versus LA. So thank you for that but I mean I think, let me just say first that we have been watching what San Francisco has been doing for quite some time and we welcome opportunities to collaborate. We have had contact with various members of the team in San Francisco. I think San Francisco took a slightly different approach than we did in terms of I think San Francisco started with wanting to build an open source voting system. We came to wanting to replace a voting system and came to a policy and philosophical decision that open source was an appropriate way to do that. You're right, we learned quickly that and maybe not everybody here would agree but I learned quickly that there are lots of different ways that people define the term open source and for something that's open, the definition of it often feels proprietary because everybody wants to constrain it by their definition of what open source is. So we're much more careful about that but as we've said, we have used open source stacks for the development. It has always been our intent to share and hopefully that will be beneficial to San Francisco. San Francisco has the ability to retain its existing voting system until they have the open source product. We don't have that ability. We're at the end of the life cycle of our current voting system. So we have to have something up and running for March of 2020. So I don't think the projects are at odds with each other at all. I think that what we've done can be beneficial to San Francisco and once it's available and open for review, I think the expertise of the team in San Francisco will benefit our project as well. I will conceive you on that because otherwise we will be here. I understand. And I would say, and I think I said this in the beginning, I think for LA County on our project, the top line, what was most important to us was public ownership. And so the tally system that we got certified by the Secretary of State is in fact the first publicly owned voting system in not just in California, but I believe in the nation and we believe that that is a model that can and should be represented. And I think the other thing, I'll pass out the mic but I think this is something that's worth mentioning is that the people in this room are going to be technologically capable of assisting in this project. And that's one of the reasons we wanted to have this presentation here at scale is the reality is that LA County is LA County, right? And there can certainly be collaboration with San Francisco, but the goal would be that we would have an open election system that is owned by all people and that all people can be participating in that. But in order for that to work, we need to have a viable open source community, right? And one of the things that I am very concerned about is that I run the fraud team at Ticketmaster. My day-to-day job is dealing with people who are trying to steal your money. That is literally what I deal with all the time. And my concern would be is that if we open source this or they open source this, because all I am at this point is helping, if it goes open source, then companies who shall remain nameless but who have a very clear financial benefit from polluting that well will do so immediately unless there is a very strong body that is maintaining and making sure that the code is being kept in an inappropriately sanctified way. And at the same time, there are also state actors who are very willing and able to pollute our elections as we know, right? We see state actors at Ticketmaster trying to make passes at us. The number of times I see people in Vietnam and Russia trying to steal people's tickets is ridiculous. The same thing will happen here. The same thing will happen. We will still have people who are trying to make, who are trying to influence our political process. So we have to be very careful that we don't just sort of, this is the point that was made to me, is you can't just toss this stuff on GitHub and then call it a day and then hope that somebody's gonna come along and you're gonna get, like, if you guys heard Mitchell's talk this morning for HashiCorp, right? Dude, were people there? Yes? No? Right? Like the way that he did that was excellent and it was amazing and it was a new product. And I gotta tell you the first time I ran Vagrant up, it was a magic moment for me. But that's not the way that this can be done here. We already have a system and we can't be sitting there saying, oh, 500 people downloaded and then you have to, do we do VC? Do we do thing? It's a different process. It has to be owned by people. Let me get the, enough of me pontificating. Let me hand out some mic. Two quickies. On the Tally program, you mentioned it reads the QR codes which aren't human readable. So how do you make sure that that says the same thing? You're not gonna scan the characters. And the second question before I turn it back is, can you detect whether somebody puts some ballots through the scanner twice? Second question, yes, we can detect that. To the first one, the code that we count in the QR code is printed on the paper ballot. So there's an internal integrity to that ballot. That allows, that marks that as the official vote of record, that code, right? So we're actually using a base 30 numbering system to keep the number small. But that code is what is the official. But we don't use optical character recognition or research into it says it's not accurate enough. It's just not accurate enough and especially at speed. And it's as much as I think we'd all like to go there and the system could eventually, it could evolve to actually just go straight OCR when and if it ever gets to the point where you can get to six sigma levels of accuracy, then we will do that. But we can't do that. The beauty of the QR code is that it is machine readable. It is an open format and it also, it's signed by the way, for those of you who want to know it, it will be digitally signed. But it also, there's redundancy in it. You can't actually miss, if you can't read the QR code, you just don't read it. If you read it, you read it successfully. And that is actually far and away better than what you get from a vote by mail ballot, right? Where you're trying to detect a vote mark that could take any number of shapes or forms, depending as Dean mentioned, on what the creative voter wants to do on that ballot. And so that's actually extremely difficult problem to solve. The QR code is actually very efficient and it's very accurate. I just want to say there's a lot of people who have a lot of questions. So I'm going to do my best to get through this. We have 15 minutes left. You guys are welcome to stay for like the 30 minutes after. And then there's a talk about PyTorch, which very different thing, so. I have two questions about, seems like most of this talk is focused on the vote, the in-person voting system. But first, the registration system. And second, the security of the tally, the vote counting. In the 2016 primary, there were a number of people who were registered to vote, that when they showed up at the polling place, they were told they were not registered to vote. And it happened in certain, mostly urban areas. The second question, so I'm curious about whether this system is related to the registration systems. The second question I'm asking is about for example, we had a recent California Assembly Delegate election in which votes, scores that were posted after the manual counting differed, or the vote that was posted by the party three weeks after the manual counting differed in some cases by 35% from the votes that were taken on the day of. And how does your system address that? And also is there gonna be chain of custody? This is three questions, sorry. Chain of custody, what's the chain of custody for the, okay the tallying, the tallying question. I'll try to run down those really quick and then we can follow up with you later. The issue with the registration system in June, it actually applied throughout the county and it was a compatibility issue between our election management system here locally and the new statewide voter registration database. So it wasn't that people weren't registered to vote, it was that their names just weren't printed on the roster. So their registration never dropped out of the system, it affected the printing, it was chaotic and we had the fail safe method of provisional ballots, all those ballots were counted and it was an unfortunate situation. The new model, not the technology but the new model of vote centers and multiple day voting actually will provide a lot better access and ability to navigate a situation like that should it happen again. But this particular system, the voter registration system is independent of that, that's a statewide system that's owned and operated by the Secretary of State. The tally component, oh this is the tally component to, so just so if it wasn't clear from the question, the party delegate election that he's talking about was not run by the government, that was run by the party. So this system will be subject to audits, required audits before the election is certified and those audits will be done actually looking at the human readable ballots. It's also subject to recount and again the paper ballot and the human readable printed output on the paper ballot is the official legal record. So part of the certification process when we take the tally system through certification with the Secretary of State is doing volume testing and checking the accuracy of it. So a 35% variance in the outcome of a manual count to the electronic count would not be acceptable and would not be certified in this case. The audits are required, they're legally required, we do them today. Well they were in Los Angeles County and there's a clear transfer at record of that. So there are several questions over here and there's several questions over there. I'm gonna go in the order in which I saw hands going up. Hi, thanks so much, this is great stuff. I live in Orange County. Last week I believe the county board of supervisors approved moving to vote centers. It's great I think, I'm concerned about, well I'm concerned about clothing, precinct, voting, general election but we'll see how it works out. So as an Orange County taxpayer and resident, I am very eager and I think many other taxpayers and other jurisdictions around the country will be very eager for you to please the code you have to be developing for years. This is a thing called Github, it's really handy. Security through obscurity is not. So if you want a way to actually audit your code and test it and have folks in San Francisco and Orange County and elsewhere help you figure out the bugs in it and protect it from potential malicious state actors who may already be in your system anyway as you know, unfortunately. Put it out there and publish it. I'm not a lawyer but I don't know of anything that would stop the county from publishing the code you have that you spent taxpayer money developing. Thank you so much for doing that, it's great stuff. So please put it out there. I would love Orange County to be able to see it and use it, the machines you're talking about and that Neil Kelly's talking about ordering are probably totally different.  So as fast as you can get it out there we will all, many of us would love to jump in and tell you any holes we find and stuff we can fix. The state or the state, the Secretary of State I'm sure there are great people and very competent technical folks but you can't be the competency of a worldwide community of hackers who want to have a great, secure voting system and not only in the United States. This would be very revolutionary. I would save taxpayers in the US alone potentially a whole bunch of money. So please publish it. Put it out on GitHub like Monday. Well, yeah, I'll let you guys speak to this but I have some thoughts, I think. I think we've already spoken to that so we can. Yeah, I think in the long run I think it's something that we aspire to. It's just that we have, first of all, we have our board, our board of supervisors has to give us the authority in the direction to do that. Also, we have the California Secretary of State is a certifying authority. So I know it seems in principle, it seems very simple and it is but there's a lot more to it. There's concerns around just optics, right? So if somebody does get on, they look at it and they call the LA Times or the New York Times or whatever and say, this code is terrible, right? Without any vetting, without any sort of ability to respond to that, you create an optic situation which is really, really hard to control and that's just another dimension of it but we want to get there. We want to get there, we're just trying to figure out how to do it. I don't want voters to understand this. My question has to do with the move to more centralized voting centers. Currently, vote reporting happens at a precinct level. What granularity can we expect to see in terms of vote reporting with the move to the more centralized centers? I think that was like six people's question. So great question. So the precinct level, the viability of precincts as geographic and reporting units will remain the same. So vote results will still be available by community, by precinct and all that. The difference is voters won't be constrained by having to go to a single voting location for that. So voters can go to any vote center and vote but their vote will still be recorded associated with their precinct. Just to keep, I have this one, here, here, here. Okay. Yes. Yeah, so my question is just a quick follow up on that because that was going to be my question but what is a precinct then? Maybe I don't know, maybe it's just the same as it is now because yeah, maybe we can just talk a little bit about that. Yeah, there's a legal definition for how precinct lines have to be drawn. They can't be, they're constrained by number of registered voters. They're constrained by contiguous geography and that type of thing. And again, for the purpose of being able to show that maybe the outcome was that Ken has got 60% of the vote county wide but in my neighborhood I got 70% of the vote and we need to be able to show that. I have been involved with the elections in the Florida Fiasco 2000 and was basically in company of partnership with lots of vendors till 2008 election or 2009. So my first question is regarding your certification by SO, Secretary of State, is it not necessary to get it certified in NIST now? So there isn't, right, so the certification of voting equipment is handled a couple of different ways. There is a, nationally there is a set of voluntary voting system guidelines that are adopted by the US Elections Assistance Commission with the assistance of NIST and there are independent testing labs that test to those standards. However, there's no federal mandate to use those standards. California, because those standards have been outdated because for a long time the US Elections Assistance Commission was not functioning because the president had not appointed members to it so they never adopted the newer standards. That's finally changed, but California passed a law that bifurcated us from those federal standards and adopted the new federal standards before the federal government did. So we actually have a higher standard in California and it keeps us independent so if we get into that political situation again where we're admired in partisanship at the federal level, that California can still move forward. So our standards in California are the base standard, are the latest version of the federal standards that NIST was involved in and Kenneth actually served on the working groups that created those standards. They're under review at the federal level now but they're adopted in California and the certification process requires that our system be tested by an independent testing lab to those standards before the Secretary of State certifies. So in 2002, I think 28th October when the Hawaii Act was passed for $3.8 billion, Georgia was one of the first state which came up with a single state wide solution for I think $156 million and because of a single state wide solution there was an option of being hacked quite easily. So some of the states they've mandated that we have different vendors, different solutions. So what's your thoughts on having a unified solution versus different solutions across the state? Yeah, I think Georgia's actually a good example of that. So Georgia is one of the states that actually statewide remained with paperless touchscreen voting equipment and it was all the same equipment across the state and they had some very close election outcomes in this last election and people are still talking about the viability of those systems. So I think that while there are benefits to a shared system across the state you also have to look at the risks of that that it's a single point of failure or attack. You also have to look at the difference. Again, even in California, this jurisdiction has 5.2 million registered voters. The next largest is one-third the size which is Orange County. The majority of counties in the state of California are much smaller. So their ability to support a system like we're talking about or to have the resources to take something like this on is very limited. So, and I guess the last thing I would say is I think that the record of the statewide systems in California sort of speaks to itself. As a software developer, I'm just curious, are you able to share any kind of details about your testing suite for kind of the software development life cycle that you went through? Which are vendors here, I was gonna- Are you phoning a friend? I am phoning a friend. No, I have a list of some of the software, but I don't know if it includes the actual, the suites. So we have two vendors, we have two vendors or two partners that are developing the solution for us. They have systems implemented to support continuous integration and continuous testing. So it's sort of beyond my much familiarity to be able to explain it, but I can tell you that they have test packages that they create, they go through their test cases, the reported back out bugs out to a bug log, they're using JIRA to support backlog management and user stories and things like that. So they have a pretty sophisticated development environment that is organized around agile methodologies and running through a pretty disciplined series of two-week sprints over the development life cycle, but I can get you more information later if you really wanna know the specific software that they're using to support that. That's a great question. I don't know the answer to that question. It's never been asked and I'm not sure, I guess we'd have to go back and talk about that. Yeah, yeah, that's a great question. And we do, as Dean just mentioned, we are putting together an open technology working group. We're gonna be having a kick off here fairly soon. These are the kinds of questions that we're gonna be asking this team of people to help us figure out how to open this solution up. So, other questions? This goes back to opening up the source to this. There have been cities, and I believe DC has done this, and they've put their actual legal code on GitHub. And yeah, that enables you to take a really good look at that code that takes, I mean, it's the actual, these are their laws that you could follow a similar pattern to that to releasing the code maybe a little bit earlier than otherwise you would be able to. We can look at that model. Yeah, we can look at that. Again, I think to the earlier question, that's something that we aspire to. I think we just need to understand how that works in relationship to the regulatory environment that we have to operate in. I think the worst part, the worst case scenario there is that everything gets released. So, I had this problem where I wanted to develop a system in-house, and the way that Ticketmaster works is you have to get funding for it, and you have to, it's like a small company and a company, it's a whole big deal. So, I would go to different teams and I would say, this is gonna solve a lot of problems, we should build this. And every team looked at me and said, great, and I said, would you wanna build it? And they said, no, we don't have time for that. And I go to each team, they'd be like, wow, that's awesome, you should build that. I'm like, me and my zero coders, how am I supposed to do that? Right, and I think we're in a similar situation here, like, a lot of people in this room are very passionate about making sure that this gets out. So, I think the really big question is, okay, but who's gonna actually do the work? Like, is this gonna be a situation where it's like, who's gonna help me get the flour? I won't, who's gonna help me get the chocolate chips? I won't, who's gonna help me milk the cow? I won't, who's gonna help me eat the chocolate? Like, I will, you know. But hey. Please. You can hear all the regulatory stuff that they gotta deal with, all the regulatory stuff. And I think the most important thing to be sure is, some of your customers are talking to the policy and the laws, if you'll be able to do that. All of the California Constitution, all of the bills, they're in the legislature today, their text is out on the California website. Sometimes you have to know where to look under a department that has their own particular set of codes that they put together after the law was written establishing their organization. But all of the stuff, if you just work at it, is out there in text today. It is searchable, it is findable. It has changed, it has changed control with the Constitution and the stuff that's in the primary group. They did it right. They did it long before GitHub was there. Not gonna disagree. But before we get mired into discussion and back and forth, which we've already kind of done a little bit of, does anybody have any other questions? I see one over here. And then I'll come back over here. Okay, and no, I'm not gonna ask for a Braille paper ballot on the way out of the booth, but I am gonna ask about the speech aspects. I've been voting on an accessible machine since 2000 and I gotta say the interface and the layout of the machines are clunky. And the poll workers don't understand how to use them. I had one friend of mine who actually waited five hours at her polling place until the accessible machine got there. So what's gonna be with the layout of those machines and the speech? I mean, the segments, they sound like they're spliced together and the levels are all different. It's not very consistent. Thank you. I'm really glad you asked those questions because that really goes to the fundamental purpose of the way in which we've done this project because again, and I mentioned this earlier, we have intentionally gone beyond just simply complying with the minimum requirement of the law which says have one piece of equipment in each voting place that allows for accessibility. And by the way, it assumes that there could be one piece of equipment that just suddenly addresses every possible disability. We know that that's not true and we worked very hard on that element of that balance between historically in this country, that has always been the trade-off as the trade-off is accessibility for security. We from day one on this project said that we were not gonna do a trade-off on those and we did user testing. We involve people in the disability community in the design of this equipment. And so the speech pattern that you're talking about has been developed using people who rely on that. The physical nature of the device has been designed. We actually worked with the United Cerval Policy Center of Los Angeles County on the design factor with people who have severe physical limitations and who have never been able to vote independent without the assistance of another. We went with them with foam board cut-out prototypes and then developed the system, took it back to them in electronic format and watched as they were all able to cast the ballot independently. This device allows you to print out a paper ballot and deposit it into a ballot box without ever having to physically touch it so that you can vote truly independently and in secret if you so desire. So that is probably the most significant change of this particular design and something that needs to be duplicated. You're absolutely right. It's unacceptable that we ask people to go in to the corner of a booth, a corner of a room and vote on a device that the poll workers don't know how to use, sometimes don't set up and it takes three to five times longer to cast the ballot than any other voter. And this, a lot of effort has been put into that process with these devices to ensure that that doesn't happen. You referred to your ability to trace an individual ballot all the way back. What is the demarcation point for anonymity? Is it when they hand over the paper ballot to the ballot box? The ballots are anonymous. So once the voter casts the ballot there, you can't trace anything. Yeah, basically precinct is the closest you could get. That's why there's a certain, there's actually a legal requirement to consolidate up to a certain percent so that you can, and restrictions on reporting out when you don't have, when you have, yes, yeah. So yeah, so the, it's a great question. So the ballot marking device doesn't transfer any, any information that can be used to determine who that voter was. It doesn't even, it doesn't record the, you know. Well, it doesn't know who they are. Yeah, it doesn't know who they are at all. Is there an ability to at least know which machine it came from in case there's. Yes, okay. Yeah, machine ID is recorded. And you have a way to make sure somebody doesn't vote two or three times, now that you have anonymity being preserved, yeah. That's through our electronic poll book. This is a quick one. Can a person change the ballot after the ballot hasn't been printed? Yes, so the question was can, can a person change the ballot after it's been printed? Yes, up until the point that they cast the ballot, which is the point that they, that they hit the button and the ballot goes back into the ballot box, then they can change their choices. Now, once it's printed, they would have to void that ballot and they'd have to be reissued a new ballot from the election board. But as long as the ballot has not been placed into the ballot box, the voter controls the ability to, to change that. And there are at least three points of verification in the, in the user interface for them to, to look at that before they, before they cast the ballot. The QR code, is that readable by say a phone QR reader? So I can take my phone out, check that the QR code on my ballot matches. Absolutely. I can decode the whole thing and see what it says I'm voting for. You can do that on your phone, but you can also do it manually because of the codes that are next to the selections. You can actually tie it back to a, to a glossary that tells you what those codes are. Oh, so it's printing a QR code like for each race or something? No. It's printing a reference point for each, each content, each outcome or, or selection that is embedded in the QR code. It's one, one QR code, one, one QR code is printed on the ballot. It contains, it's, it's just a string. It's a series of the codes and it's only, it's only recording the code for the candidates because the tally system can actually, because it's a unique ID, it can trace it all the way back to the contest and do all the other counting that it needs to do, but it's just one QR code and it, it inserts in the QR code the, the unique ID for the selection that you made. It also prints that selection. So theoretically what you could do is you could use a QR code reader, open it up. You would, you could see all the codes that were in there and you could match it to the ballot. That's what I mean by sort of an internal auditing of the ballot. You could do it right there just to make sure that the QR code has without having to rely on any of the hardware that the county provided. Exactly. Thank you. So we know there's actually the PyTorch group is starting to come in and so, you guys we have, we have 18 minutes. So real quick before you go I just remind people that there are materials up here and then there is a website. So, so what is published is all of our research, all of our diagrams and, and a vast library of information about the project which is that the website is vSAP, v-s-a-p dot l-a-vote dot net.  Thanks for that. This is, this is our logo for it. Yeah. Can I have one?  Well, I know the guy who wrote the challenge. Oh, okay. That's why I said it worked. What do you mean in 1999 that they all brought it up and built that law? Yeah. In 1999, no. Yeah, we, okay. That's when I was doing it. Okay. Well, there were only two and a half hours, two and a half million voters when we did that in 1976. Wow. That became our move. Yes, yes. No, I think it's a little, I've heard that. That's probably my thing to be sure of. Yeah. Hi. Thank you very much. Thank you. It won't be anything. Leonard Pannish was the red star when. Yes. When I was doing that stuff. It was not fun but, and after it passed, that was prop 13. So they just, they went, well there's no raising. So they kind of wrecked it. Thank you very much. Thank you. Thank you. Thank you. Thank you very much. Thank you. I think you know how to create a little bit. What's the worst one? I mean, I think you just did. Oh, it's hard to take this one. I will tell you a funny story that when I was in, they'd watched it. I mean, they had to recount for a governor. Yeah. The newspaper published a picture of a ballot where they had the broken arrow format with not the oval to actually build in. And people have connected all of the right hand of the arrow to one of the left hand. And of course, there was no way to determine voter intent. And the voter actually called us after seeing that in the paper. How is that not what you put the word? No, it's the opposite. So it's like, so you've got three candidates. And they all connect over to one of the arrows. So it's interpreted and over. But we actually had a voter call. So I thought all of the paper and said, no, I did that intentionally. I wanted to divide my vote one-third for each of the candidates. I'll see you in the room. That was good. You know, I didn't know how you'd do it. Are you on the position? No, I'm present.  Thank you. I'll be right back. I hear you. I'll be right back. We're trying to figure out what you're going to do. And so I guess we're going to answer all that. But no, it's bad. Don't put it that way. No, no, I totally agree. It's a policy. I was going to say, it's fine. I'm going to serve as a candidate. Right? Just give me one. Yeah, OK. Well, version one. Yeah. OK, go. You can put that out. They can put the job out. They can put that out. They can put that out. They can put that out. They can nail it up. They can put it in the library system. Yeah, and like, God bless them. But, you know, perhaps, I'm just glad you're here. Right? Finally, I'm going to go. I am still up. Although I am skeptical. I'm not having to do the main thing. I could have both. I'm glad you got it. But next question. So, class, yeah, this is great.  Suggesting your board. Right. Positive optics. Hard, hard. Hard. And you're already getting positive optics. Yes. And there's always people who touch out there, whatever it is. Yeah, yeah. And it's just, it's just a fascinating environment. You're in an environment where, you know, the President of the United States comes out and makes a comment about the behavior of people and becomes new for two weeks. And there's nothing factual based on it. It's just, it's just a process of development. Yeah. So, the, it's there. That's what we mean by, you know, it's there. Sure. What's the potential for the positive optics? I think it's far away. I agree. I agree. Did you, have you been a DEF CON yet? The voting privilege? Yes.  We sent the team the last two, the last two. Okay, great. Neal was there last year? Yep. Please come this year and bring what you can, talk about what you can. Okay. And trade it up. And there's media. Yeah, I don't know. Great stuff. Ford there, and they would love this, I think. Yep. And it'd be like changing the story if they'd done for the last two years. Yeah. And we're doing third party. We're, we're, even before we go to certification, we're doing third party impacting. I mean, too. Great. Yeah. Yeah. And, but. Of course. Thanks so much. Thank you. Thank you. Come here. I will say my honor.  I'm a local resident and I just want to say that you are fantastic on social media. You're really responsive. I think I had a polling place issue and I tweeted you and you were like immediate. Like, you just don't honor it. Good to hear it. Thank you. That's good job. Thank you. Hi, so I wanted to talk a little bit about like my precinct reporting because I, I'm coming from a project we presented on Thursday called Open Precinct. And our goal is to. Performance based back here. So we're, we're, we're trying to make a national database of voting precincts with elections like matched it by 2021. So that kind of redistricted chart happens. Everyone has, every citizen has a perfect quality. Yeah. And you can use a strong map of the citizen. So you can sort of commission stuff like California. You can show the map directly. We're going to try to make tools. So basically we're at the very early stages of development right now and we're trying to think about how we're based in the journey. I grew up in California but LA is so different from what I grew up. So I just want to know. There's base precincts and then there's election precincts. Yeah. I've heard a little bit about LA precincts. I don't really understand it. So I just want to make sure that what the tool that we're going to produce is going to. Right up to that. Oh, okay. But do me a favor. Send me a email. Okay. We can, if you are interested, we'd love to bring you in and we'll just talk about tracking precincts in the university like across the nation. So. Yeah. So we're developing a tool called Open Precincts. It's going to try to have a national database of voting precincts with data. And I just want to make sure that like we're going to, I've heard that LA precincts are different in every single election and not. Well, that's what I say. There's differences. Election precincts and then there's base precincts. Okay. And they can show you that. And we've done a lot of work on that with the Google Voter Automation Project. And a lot of people see this text for polling in the nation or something like that. So I think we've got some good stuff we can do. Yeah. Yeah. Great. Great project. Yeah. Yeah. This is nothing. Thank you. Great. We'll thank you. Oh my God. Thank you. Yeah.  All right. Yeah. Yeah. Yeah. Yeah. Yeah. What is all this? Yeah, I was scared. I was scared. I toujours, I was scared. But it was great. Yeah, I was scared. Thank you for being here. Which are you? No, that's my mic. Did you get your last one? Yeah. Put that in my back pocket right? Yeah. Put it wherever you want this stuff to go. Okay. That's reasonable. And then you want me to repeat the question? Yeah. Okay. Okay. Okay. That's reasonable. And then you want me to repeat the question? Yeah. Okay. I'll be able to hear it on that thing there. How much time do you have for questions? Whatever I finish. Have you ever talked about this? We don't have a laser pointer, do we? I'll intro myself there. Wait another two minutes. All right. I'm going to start now. Hi, everyone. Thank you very much for coming. My name is Peter Goldborough. I'm a software engineer at Facebook in Seattle. I've already got more vitamin D in the last 24 hours here in Pasadena than the last six months. So it's been a good trip so far. I've been at Facebook for about one and a half years now. And the first year or so, I worked on a project in a team called PyTorch. PyTorch is a team under Facebook AI research, which is Facebook sort of academic, out in the open, blue skies, artificial intelligence research division. And I'm going to give you a bit of a whirlwind tour about what PyTorch is. I don't expect any prerequisite knowledge about machine learning or about PyTorch. If you can read Python, that would be useful, but that's probably just reading English. So I just want to give you an idea of what PyTorch is. I don't expect any prerequisite knowledge about machine learning or about PyTorch. If you can read Python, that would be useful, but that's probably just reading English. So I just want to give you an idea of what you can use PyTorch for and what other people use PyTorch for. So what is PyTorch in a couple of words? PyTorch is an open source machine learning framework for both flexible research as well as now high performance production environments. That's exposed both in Python and C++ and now also more recently other languages too. And so it's just to focus on each of these points a little bit. PyTorch is open source. It's on GitHub. You can go on github.com. PyTorch, it's always been open source and it's actually based on a very long history of open source framework. And it's used for machine learning. So that can be any kind of scientific computing, neural network, but really anything that involves matrices and predicts something. It was originally focused on being very flexible for research, so enabling researchers to try out the wildest ideas. But now in more recent releases we've also focused on supporting researchers or engineers taking their research and deploying it to Facebook scale or start-up scale or any scale production environment. I'm going to talk about that too. And as the name implies, PyTorch was originally written in Python and exposed in Python, but we also have a very big C++ front-end which I actually wrote so you can ask me about that more. And now we also have a couple other language frameworks built on top of the C++ front-end. A little bit of history about PyTorch. So today we have PyTorch 1.0 that was our biggest release a couple months ago, which focused really on the production environment. But PyTorch is actually based on another much more venerable framework called JustTorch, which started out in 2002, which was actually quite a while ago. It started out as a scientific computing framework. Back then deep learning was merely a thing, but it was intended for any kind of scientific computing at the time support vector machines and other machine learning techniques were more interesting. And it was originally written in C++ and then the author called Ronan Kullabere, who also works at Facebook at Research, was always looking for a better and better front-end than JustT++. So at one point he released Torch 4, which was exposed in Objective-C. That sounds like a bad idea anyway, so he switched to Lua with Torch 7. And Torch 7, also called Lua Torch, was the last really popular release of the Torch framework and popular up until a couple years ago. And then in 2016 an intern at Facebook had an intern project of taking the underlying C++ back-end of Torch and building a Python front-end on top of it, and that's how PyTorch was born. But what's even more interesting is that Torch has an even longer history, so the first Torch ever was actually released about 300,000 years ago by the first Neanderthals. And it's funny, but it's also not funny because Neanderthals are still more advanced than NEAI we can build even with PyTorch today. So I want to give you a bit of an idea of what you can build with PyTorch because you might not be familiar with machine learning that much. So one example use case is machine translation. So here's Google Translate, which probably doesn't use PyTorch, but for example, Facebook's machine translation, which runs about 2 billion translations on the Facebook platform every day, uses PyTorch, and the source code for the models that use that is on github.com. PyTorch.translate. Another more researchy topic is any kind of gamie eyes. You might have heard about AlphaGo a couple years ago, which was released by Google DeepMind and Facebook AI Research actually released an open source version of AlphaGo implemented in PyTorch. The original Google version was not open source and researchers at Facebook AI Research re-implemented and open sourced it using PyTorch. And another example is really any kind of computer version application. So this here is neural style transfer, which takes some existing image, usually a painting, and applies the stylistic features of that to another picture. If you go to the Facebook booth in the other building, you can actually see a lot. And a very brief idea of what using PyTorch looks like. So this is Python here. If you were to write your own model in Python, you would define your neural network, which uses other smaller neural networks built into the core library. You would create optimizers, and optimizer is something that uses the feedback you get from the training process and updates the parameters of the neural network. You would have a data loader. PyTorch, which you give some data set, some image set, or text data set, or speech data set, and load that in parallel for you. And then you would have a training loop, which loops over the data set, in this case 10 times. It gets predictions from the model for your data, computes a loss between the predictions and the labels, and then updates the parameters of the network based on the feedback from that. So that's what a typical PyTorch program would look like in about 30 lines of code. And I first want to talk about how PyTorch is useful for research, and later on about how it's useful for production, since research probably takes up about 95% of the time of any machine learning pipeline. And the first two reasons are eager execution and dynamic neural networks, and I'm going to talk about what that means exactly. And to get an idea of why eager execution, which is really PyTorch's core sort of feature, is so useful, I'm going to talk about the opposite first, which is more popular in other frameworks like static execution. So in static execution, if you write a program, for example here, c equals a plus b, then the variables a and b would not actually represent numbers, they would represent some kind of variable object that doesn't have any concrete value at that time. And when you write a plus b, you're not actually getting the value of a plus b, instead you're constructing a computation graph, which represents the operation a plus b. And then later on, you would feed values into the graph, for example, where the graph gets evaluated, and then you get the results. So as you can see, it's a bit complicated because there's this two-step process, you're first creating a graph, kind of a program inside the program, and then you're evaluating it. And that's static execution. Now with eager execution, it's actually a lot simpler, and the reason why I put static first is because eager doesn't really need any explaining, it's just a program. So in PyTorch, when you write a machine learning two and three, when you run a plus b, you get an actual value, there's no graph involved. So this is a lot easier and a lot more flexible. So in a couple words, what this means is that in both cases, you have a program, and in the eager mode, you go from the program directly to the results, whereas in the static mode, you go from the program to a graph, and from the graph, once you execute it, to a result. And the reason why you would want to do that in the first place is that, allegedly, you can take the graph, if you use some operations or reduce the amount of memory that certain operations take, you can even take the graph, put it into and then make one run in one machine, the other in another machine, and hopefully have the results faster. But it's just a lot more complicated conceptually. So in a couple words, in eager mode, the benefits are that you have no boundaries on your flexibility. If you want to download something from the internet in between your neural network model, you can do that, you can't do that in a graph. Now, here, you can just import PDB, PDB.setTrace within your model and see all the values live, which you couldn't do in the graph framework unless the framework itself provides some kind of debugging utility, which they often don't. And there's no expensive compilation process, the code just runs immediately. Whereas in the static mode, the benefits are that you can optimize it, which is often interesting for deployment, which is why I'm going to talk about PyTorch's static mode later on. And it's also easier to deploy. And the reason why it's easier to deploy is that if you need to update your model, in the eager case, your model is your code, so you would have to redeploy your code, your binary. Whereas in the static mode, you can actually sterilize your graph and then to update to your model just means sending your new model the new sterilized format on the wire. And then your server, whatever, can just load the model without any actual coaching. It can just load the model and execute the new model without any new code being deployed. That's why it's a bit easier to deploy. Another reason why PyTorch is very useful for research is that it supports various different hardware platforms. And here's a selection of the different categories of hardware platforms that PyTorch can run on. On the left is the usual CPU, which is really too slow for most research today. The CPU gets about one to three tear-ups per second. That's trillions of operations per second, running things like matrix multiplies, in this case here the infill C on five. And then in the middle with the GPU, which is really the device that sparked the most recent machine learning revolution because people found that running machine learning on GPUs, which is made really for trillions of parallel point-wise operations, is actually a lot, lot faster. So as you can see, a GPU can get between 10 and 20 tear-ups, which is about 10x what you can get with a CPU. And shown here is the NVIDIA Tesla V100, which is NVIDIA's most recent version. And even though the original revolution was kind of based on just everyday graphics cards used for gaming, more recent versions like this graphics card actually have explicit units within it, optimized for matrix multiplies found in machine learning research. So these are called tensor cores and apparently have up to 100 tear-ups per second, but that's very, it's usually limited by memory bandwidth and hard to achieve. But then the third category, where it gets very, very exciting and very specialized are ASICs, which are application-specific integrated circuits. So these are chips built specifically for particular machine learning use cases. So in this case here, this is Google's TPU, which it released about two years ago, which is really made for just getting as many flops out as possible, so this achieves between 40 and 100 tear-ups per second. At the same time, it's very limited. It can only run computer vision models that have a lot of matrix multiplies that don't do a lot of control flow or anything fancy like that. So it's very specific, but if that's all you need, they can get the most operations per second as possible. And another example here is the big basin server. So this is a typical server you would use for machine learning research. This is actually Facebook server, but it's also part of the open compute project, which means you can actually find the hardware plants for this box online and build it yourself in your garage. So this one has eight of these, NVIDIA V100 GPUs and it gets about eight times the 40 tear-ups per second, so it's a really beefy box and we have a couple of those, many of those in our data centers. Another exciting thing about PyTorch is that it's optimized for distributed training. So most machine learning models today don't fit onto a single GPU after a very short amount of time, so PyTorch has great support for running distributed training, and distributed training can mean running a single model on many GPUs on a single box, as well as many boxes and many GPUs on each box. And to give you an idea of just how simple that would be, if I have some model instantiated, the model, I can just pass that to the Torch data parallel class, which will actually take my model and automatically distribute any input they give it onto, in this case, GPU 0, 1, and 2. And if you have multi-machine setup before you have many different devices communicating over TCP, RPC, PyTorch also has support for that. It's a little bit more complicated, but still only requires a couple lines of code to set up the server on each machine. And then when you evaluate the model, you just pass it to your inputs, and PyTorch will automatically scatter it to each machine, run the inputs, and then gather everything back, and also update all the parameters appropriately. And to give you an idea of just how much of a speedup you can get, here's the training performance on a ResNet 101, which is a very popular computer vision model, running in distributed mode with PyTorch. So the green box is the ideal speedup you would get from scaling from one GPU up to, in this case, 8 GPUs. And you can see the results for running PyTorch in distributed mode using 100 gigabit TCP connections in the blue box and 100 gigabit infinity band connections in the orange box. And as you can see, PyTorch gets really quite close to ideal scale-up. And if you distribute that to even more machines, you get even better performance. For example, one popular training set in computer vision is called ImageNet, and it used to take between 20 hours and even two weeks to train a single model on ImageNet. And about one and a half years ago, Facebook had a research release paper on training ImageNet in one hour, where they basically scaled up the training on 256 machines with PyTorch running many GPUs on each machine and reduced the training time from originally a couple of weeks then a couple of days to just one hour. Of course, after that, more people published papers on training ImageNet in 19 minutes, but Facebook did some pioneering work there. And I think another important thing about PyTorch is that it's always valued simplicity over complexity. And this sort of follows the then of Python, so PyTorch was originally written in, or originally exposed in Python and still is exposed in Python. And the then of Python sort of tells you too when you have multiple options for doing something, you should always pick the simplest one. And PyTorch really does that. And I think its API is incredibly friendly to users. Even people who just started programming Python can oftentimes pick up PyTorch, because it's very native. It feels like Python, it feels like NumPy, and it's very easy to pick up. And another reason why I think PyTorch keeps this simplicity as a high priority is that it keeps this API across different languages. So PyTorch is exposed in a number of different languages, and they're all built on top of a C++ back-end. So this C++ back-end provides some basic things like tensors or matrices, hardware support for GPUs or ASICs, automatic differentiation, which is the technique used to update parameters based on feedback from the training. And based on the C++ back-end, there is the Python front-end, which has always been available. And then since a couple months ago, also a C++ front-end, which is really quite equivalent in its API. A couple of community users have actually contributed more APIs on top of the C++ front-end. So we have an OCaml API now, a Rust API, a Haskell API, and a couple other languages too. And while the Python front-end and C++ front-end are from the core team itself, these other languages have actually been added on top by community users and contributed to the PyTorch project. And to give you an idea of how this simplicity is kept across APIs, on the left you see PyTorch in C++ and on the right you see PyTorch in Python. And as you can see, there's really not that much difference. So all people would expect the C++ API to be like three times the amount of code, but I think we've kept the APIs quite similar. And the same is true for the actual training loop.  The for loop looks a bit different, but otherwise the lines of code are almost the same. And I think that makes people's lives easier, especially if they have to use C++ because of performance constraints or reasons like that. Now to the reason why I think PyTorch is really useful for production, there's two reasons I think so. The first is that we have recently gotten great support for bare metal deployment, which means deploying PyTorch models in resource constrained environments or basically not Python. And then there's also a couple of cloud solutions which I think could be interesting if you want to run PyTorch in production without actually getting your hands dirty on writing the code yourself or deploying the model yourself. Bare metal deployment. If we go back to this original slide I had on the pros and cons of eager versus static mode, I mentioned that all the eager benefits are sort of great for research, so you want no boundaries on your flexibility, you just want to reduce the latency between the idea in your head and the code in your editor. You want debugging, you want to see values live, and you don't want to wait very long for your program to run. At the same time for production, we actually care a lot more about these other things. PyTorch would optimize, so we wanted to run faster than not faster, and we also wanted to be easy to deploy. And so for this reason, we've recently added on top of all the benefits that PyTorch already provided, also benefits of the static mode. And so right now PyTorch or before this release, PyTorch were just Python programs which were native to Python, Pythonic, intuitive, debuggable, hackable, you could use any other library, like NumPy or anything you want to. It needed Python to run, and it was difficult to optimize. So what we did instead is to just label PyTorch as it was eager PyTorch or PyTorch eager mode, and now we've added to PyTorch eager mode also PyTorch script mode. And the idea of PyTorch script mode is that we allow the user to annotate parts of your Python program in such a way that we can take that Python program, analyze it, compile it, optimize it, and turn it into a graph, and you change any of your code. And once you have that graph, you can serialize it to disk, you can deploy it, you can run it without Python, as I'll show, and you can also optimize it further to run an optimized hardware and other hardware platforms. And the way this looks is actually very simple. So this was the existing PyTorch model. And if you now wanted to turn it into a graph, all you would do is, instead of inheriting it from NM module, inherited it from script module to a method you want us to compile with Torch script method. Of course, the one catch is that we can't compile everything, so we can't compile every possible Python program, and the important part here is really an optimizable subset of Python. What this means is that we still allow you to use control flow, you can use if clauses and while loops, you just can't do crazy things like you can't download an image from the internet inside your Python program. Right now, we also don't have support for calling it to other libraries yet we also don't have some control over that library, but really 99% of PyTorch models can actually just be annotated and turn into a graph already and also be used for production environments. And then once you have your graph, you can sterilize it to disk here with the net.pt and then in your C++ server, we have a very small C++ library which you can link into your server. You can load it and then have some inputs, for example, you could have the image coming in into your web server, you turn it into an array of numbers and then execute your model without any Python dependency at all and then do whatever you want with your model. And so you can still keep all of your research in Python where your data scientists are fast and have all the flexibility of the Python ecosystem, but you can then run it without any Python dependency at whatever scale you need. Now, a couple of cloud solutions that I want you for now. I really got support by all the major cloud providers to have some kind of PyTorch integration. So Amazon SageMaker, which is Amazon AWS's kind of high-level API for use machine learning has PyTorch integration. Google Cloud also has PyTorch support. So for example, one offering that Google Cloud has is Google Colab. So this is a service where you get a Jupyter Notebook for free running on, I believe, two GPUs and already has PyTorch and also TensorFlow but also PyTorch pre-installed. So that's absolutely no cost at all. You can just get a Jupyter Notebook with two GPUs running on Google Cloud and import PyTorch and run your models on that. And Microsoft Azure, similarly, also has support for PyTorch. And here's an example of what using Amazon SageMaker with PyTorch what it looks like. It's basically a very, very simple API where you give it some Python file, you tell it what kind of Amazon instance and pass it some S3 bucket to your data and it deploys instances for you, runs PyTorch trains the model and gives you predictions from that. So that's actually a very nice API. And as I mentioned, PyTorch is quite popular. So this is about 0.01% of PyTorch users but a couple I throw up here. So Media is a big user. UberAiLabs, Twitter, and really has in terms of other industrial partners. We have just as many academic partners, for example, Stanford, CMU, Oxford, these are just really a couple off the top of my head. But really all across the world people are using PyTorch and that's both in academia and industry and really just people at home trying out machine learning as well. And if that sounded interesting there's a couple resources I would recommend. So one is we have actually a Udacity course that's free called Introduce to Deep Learning with PyTorch that's developed with Facebook AI and that sort of teaches you how to train machine learning models without any prior experience using PyTorch. And another good resource is Fast.ai. Fast.ai is a course meant for existing coders who want to basically learn how to do machine learning with a little bit of Python background. And it's also very friendly course. It used to be based on another framework but they recently switched to PyTorch as well or recently I think like in the last 6 to 12 months. And it has very great resources as well for learning both PyTorch and machine learning in general. And that's actually all I got. I'm a bit fast but I guess I give this about 36 minutes for questions. And that's also I would also recommend PyTorch.org is a great resource. We have tons of tutorials, tons of examples. But yeah I think we're going to use this time to if you have any questions about machine learning frameworks or PyTorch or Facebook.ai or Facebook in general if you have to take anything. So let me pass the mic around but before we get around to that let's thank our speaker for giving us a very well optimized speedy talk. He seems to be quite good at that sort of capability. Does anybody have any questions? Like just blue skies research but I think the requirements for joining PyTorch were mostly like have you worked with C++ before? Have you have some knowledge about compilers that was kind of useful? And generally yeah I mean I knew what machine learning was as well. So I think those three things yeah. I have an oil on the fire question. How do you compare your use of like PyTorch versus say TensorFlow? Do you see them fulfilling different use cases? Do you see them as competing technologies? Yeah for a long time the main difference was that PyTorch was really kind of meant for research because it was sort of flexible and it has this eager mode whereas TensorFlow was always less usable and people didn't really like using it but it has the benefit that you could deploy it very easily. TensorFlow even has a thing where they make a web server for you out of your model immediately. So there used to be a bit of a distinction but now with our latest release we added this whole graph mode on top. We're basically giving our existing users the opportunity to also use PyTorch in production which really closes the gaps of TensorFlow a lot. At the same time it's interesting because TensorFlow for about a year now has been working on TensorFlow eager which is basically an eager version of TensorFlow. So really we're just seeing both frameworks converging to the same thing with some different sort of flavors on top. But I think now we have pretty good parity between PyTorch and TensorFlow and we're going to go through some of the things that we're going to do before we go through a researcher production. I think about a year ago it would have made sense to use MXNet. Amazon supported it and still is supporting it. They also have a PyTorch like API. I wouldn't know why I would recommend it today honestly between PyTorch and TensorFlow today. I think it was a great control of it. It was like super open source things that was kind of a nice benefit. It was developed by some random guys out in the open. But I think today Amazon is also ramping down its use of it. Even so another framework was CNTK which is used by Microsoft but Microsoft actually three months ago decided to use it. What we usually say for Java is that you we have a JNI interface to do the model loading and execution. We don't have an interface to do the training not like DL4J or something but you can run your models on Android and stuff like that. So this is a bit more in line in comparing the two. Is there something like TensorBoard for PyTorch these days? We've made that work and Google has been helping us with that. So TensorBoard has turned into a standalone tool and we've made PyTorch work with it as well. So yeah. Let me give you this so that we can record it. Sure. So you talked a little bit about the static mode and making it more friendly for deployment. Is there like, again I'm going to make a reference to TensorFlow TensorFlow has Tensor serving there's also NVIDIA has TensorRT is there like a deployment story that's native or I know there's like some onyx support so what does that look like? Yeah. So we don't have anything similar to like a TensorFlow serving. Basically what our like deployment story is right now is this chip mode. So you take your PyTorch model you convert it into your graph and in between you could have an organization. So also once you have that graph you can always do more things with it. So that graph can be converted to onyx for example and then there is a hardware platform for example like the TPU is powered by XLA which is the compiler for the TPU and XLA is getting support for onyx so basically very soon you'll have this whole pipeline where you can take your PyTorch model turn it into this graph using the script mode and then optimize it further to run on TPU and I imagine that NVIDIA is also working on support for TensorFlow and PyTorch. Yeah. So I'm not very like machine learning is a abstract thing for me so I'm slowly I'm a programmer so I understand I can read English Python scripts but so at what point does math become like oh you need like so much knowledge of math to work with machine learning at what point is that a factor I think it becomes a factor at the point where you want to do your own research so if you're exclusively interested in using machine learning or trying out different models doing something with machine learning like building an application you don't really have to touch the math at all I think the only point where you would be touching the math is if you wanted to implement something that's state-of-the-art that doesn't have an existing like pre-canned API or if you wanted to develop something on your own so I think for example I recently built an app that you had to use some computer vision and I just used Google Cloud vision for that I didn't like need to do any math at the same time if you wanted to implement a paper that just came out like at a conference two months ago there's probably no like nice API for that there's also probably no pre-chain model for it so then you would probably have to read the paper understand it to implement it and that's probably where you would start doing math and especially if you don't want to like build your own research on top of that it also helps with debugging like if your thing starts to go sideways and you don't understand why that is being able to debug into it and say like oh this equation requires that I do this like that that helps yeah I guess that's also the thing about like I mean like you would need like advanced calculus knowledge to like read papers and implement them but then even to like do some basic like machine learning models it's good to know like what a tensor is and like understanding some operations between them having like a basic like friendly approach towards math I think is helpful but for just high level use cases I think it's not required so if you're like implementing a new model and let's say the data type is video like you have to spend a lot of time like making this thing that can take video file data like frame by frame per say let's say yeah or if you're doing audio you have to create something that it takes in audio and covers individual like data frames is there like a large sort of I guess ecosystem of like basically user contributions for different things like that that makes it you don't you don't have to reinvent the wheel basically each time with like different stuff like that yeah I think I mean but I think there's about like a hundred thousand repositories on GitHub that have its name and like a lot of those are like data loaders for specific data sets so in the actual PyTorch organization on GitHub we have PyTorch audio which is which deals with like taking audio data and turning that into a nice data set for you we have PyTorch vision which is for everything like vision related there's speech recognition tool sets tool kits as well so I think like today you wouldn't have to implement anything like that for whether it's speech, video, text and basically all of those data forms people have done research on and have built nice APIs for that for PyTorch today so I don't think you would have to reinvent the wheel it's really about picking the right project yeah another question Hi so I went through the Kubeflow thing they had here on Thursday and I was looking at your at your photo of like the different the CPU, GPU and all that for training especially how do you know in advance how much of of resources what's the right processing before you go ahead and do it especially I could think either you may not have enough resources like you're doing it on-premise sort of thing but if you're in a cloud you may get either a surprising build or I think I would if you're like training a particular model I would probably look at the paper for that model and in our experiments we train our model on like so and so many devices for so much time that would probably give you a good idea otherwise I think eventually you would get a feeling for how much compute you need for a particular kind of model whether it's like a big computer vision model you would probably need maybe like 8 GPUs or something also I think 8 GPUs would probably get you started with almost any machine model today and then if you wanted to make it even faster you would have to scale it up or scale it down if it's too expensive but I think basing it on what previous people have tried in terms of compute is probably the best idea Do you have any other any graphic providers that are your partners besides NVIDIA? Yeah, so we've been working with AMD for a long time too AMD has an API similar to CUDA where they basically you can take an existing CUDA program and make it run on AMD basically and we've been working together with AMD to also make PyTorch run on AMD I think it's sort of an alpha support so it works it's not as well supported as CUDA but we've been working hard with NVIDIA to get that working and I think that's the only major GPU provider other than NVIDIA today So yeah, AMD Any other questions? Yeah? You got 25 minutes of your livestock Thank you