Okay, welcome everyone. I'm dead addict.  Hey there Four four one eight it might be coming. All right So I'm dead addict It's interesting The the audience for this obviously this is a hacker convention a hacker conference and Most of my rent is gonna be talking to vendors And people that write this buggy software and want to squash the speech It might not seem like an obvious audience because you're gonna find the bugs and humiliate the vendors But it's it's my proposition that we're all gonna wear many hats and I know I have in the past. I Do consider myself a hacker. I have worked for many many vendors with many many security issues I've been a customer of these vendors and I've also been involved with PR and in media activities So I think it's inevitable that most of you will have two or three of these hats. So Keep this in mind. If you're not a vendor currently, there's a good chance. I'll be working for one at some point in your career and what I would hope to do is to convince you to take away some of these lessons and Convince people in the organizations you're going to be working for or are working for to do the right thing I Don't want to talk about these vendors as unified companies are not single entities just as governments aren't single entities There's politics within these organizations. There's people within these organizations that are trying to do the right thing and even when they do the wrong thing there there's employees and people in these organizations that are really unhappy about that so To do the right thing there's going to be internal battles And I hope you fight the internal battles and fight your company's politics to to enable you to do the right thing I Can talk about my qualifications. I've been speaking for a while. I've been involved with Defcon for a long time I've worked for large multinational organizations at the end of the day. All this is irrelevant It doesn't matter what I've done in the past.  So don't listen to them either because of who they are By the way, I plagiarized that last one if anyone can recognize the general gist of that You know who I plagiarized that from But it was a long time ago.  You are finding the bugs. You're the vendors. You are creating the bugs There's some government folks here Probably not legislature, but there's people involved government has a lot of rules government is actually a customer of these bugs and I'll give an example at the end of Michael Lynn's talk at Cisco gate some time ago. There are a lot of government agents that were very unhappy Michael Lynn Michael Lynn was very nervous about that But they were unhappy at Cisco. It turns out as customers of the software They were unhappy that they were not informed about it and that they were vulnerable and Cisco knew about it I'm not going to talk too much about Cisco I'm gonna bring up a bunch of examples of various issues that have happened here I won't bring up Cisco too much because the darkest of tangents is speaking on that matter in depth in a couple of Hours and there'll be a fascinating speech. I look forward to attending it myself And I generally don't attend speeches. I recommend you go to that speech Also, I very much recommend that you go to the speech two hours ago. That was an excellent speech That was Jennifer Granik. She gave legal advice sort of She talked about the legal environment that exists for security researchers At some point her slides will be available the talk will be available. I recommend everyone Understand the legal environment that exists for security researchers One thing that Granik said in her Q&A afterwards that I thought was remarkable I I asked her the question couldn't most of the speeches that have occurred at both black hat and Defconn Couldn't the researchers have been threatened and the way they have in some of these dramatic cases Isn't all of this stuff we're doing on the gray area where teams of lawyers could go after us Jennifer said yes. Yes, we could all be in trouble for all of this activity all of this research all of this passionate discovery of issues and She said it's a credit to the entire industry that more of these incidents haven't happened So I'm gonna talk about some bad things that then vendors have done But these are dramatic issues that the when vendors mess up it becomes news It becomes newsworthy and it gets all of our attention What doesn't get everyone's attention is all the bugs that the vendors deal with appropriately and most of the time They do deal with these appropriately and this is not newsworthy So just think of all the speeches at all the Defconn's have occurred from here in the past and at black hat and realize if the vendors were truly the bad guys they would These conferences wouldn't exist at all and they would be asserting their dubious legal rights to squash all of our speech continually so Most of my speech like I said will be focused at the vendors mainly because they're Their decisions to attempt to squash the the speech of the researchers is the cause of the problems There's there's some other problems too. Sometimes the hackers that do this research aren't doing a communicating effectively their case If they were actual bad guys, they wouldn't be up here on stage discussing the vulnerabilities. They would be selling them to Russia or Hostile governments or our government Finally, I'm going to talk to a little bit to the media and I'm going to try to encourage the media to report some of the success stories it's interesting the success stories are Not discussed and there's a couple of companies I know of and have inside scoops about them dealing with these issues very effectively and Doing the right thing and not squashing the research and going through the internal battles legally and Telling their companies look as a security person in my company if you sick the lawyers on this problem I'm going to walk away and have nothing to do with you at all Unfortunately for some reason when companies do the right thing. They want to be very quiet about it It turns out that companies want the to have the option to pursue these dubious legal remedies in the future and Essentially the lawyers say well even though we haven't won this battle and even though the security researcher in our company said Let's not go after this guy. We still want to be able to go after people in the future So don't discuss the fact that we're playing nice I would hope that that that changes Why is this important? Well? at this point in time the critical infrastructure of the globe is written in software and if we allow Research to be squashed and not be publicly discussed the research is not going to go away the true truly bad guys are going to be pursuing this research and The public safety will be affected The quality of the software and the quality of the the security industry depends on open discourse I Want to I don't want to beat up on the vendors To say they're they're doing the bad thing for ethical and moral reasons Although in many of these cases that I believe they're acting unethically and immorally But what instead I want to do instead of just beating up on the vendors is I want to convince them that this isn't their best interests Their best interest is to have a positive relationship with security researchers to encourage security researchers to come to them with issues And to essentially have some harmony with the hacker and security community Not for altruistic purposes, but because it's good for their product. It's good for their customers and It's good for their reputation Yeah, also I want these conferences to continue as I mentioned before if these organizations continue to assert dubious legal rights and and threaten us all All these conferences can go away and that wouldn't be any fun for any of us so When I talk about dropping the zero day or when I talk about picking up the zero day It's largely not not exactly zero day. I'm talking about There's no announcement of zero day if you're dropping a zero day if you're if you just release something It's largely too late for the corporations to do anything about it so That's not what I'm talking about. I'm talking about announcing You're going to release research, which is a very legitimate thing to do, you know you Professor Felton was going to release some research at a academic conference and he announced he was going to do so before announcing your research before you publish it is a very legitimate thing to do But essentially this is This is about not asking the vendor's permission beforehand and and what happens at that point It's interesting too in the HID case very much confused me and blew me away And so many of these cases are the thing that bewilders me is how much it's not in the vendor's self-interest So I'm going to refer to a number of cases here where you can do a lot of research to get the background on them I don't want to go into detailed case studies about them, but HID makes proximity badges and There was going to be some demonstrations of how to clone these proximity badges and Iowactive the company that did this research Was concerned because they lived in or they worked in a building where there was critical infrastructure housed They realized it was protected by something that was not secure What confused me about this was twofold One this research was already done people had made these badge copiers the academic research was out there Schematics were already available. I respect Very much the people at Iowactive the work they do there's some brilliant people there But as far as I could determine it wasn't entirely groundbreaking research So they the HID attempted and succeeded in stopping a talk of information that was already known The second thing that bewildered me about this is HID as a vendor had solutions to the problem All right the the the technology that was going to be released. They had more secure versions that would allow better access control and Their marketing people should have stepped up to the plate and realized this is a better way to upgrade all their user base released at critical infrastructure and and they didn't do so so Because of my modesty, I'm going to tell everyone what to do And I think like I said I'm not going to tell everyone what to do because it's the right thing to do Although I believe it is I'm going to I'm going to try to tell everyone What the right things do that's in their own self-interest and it's in the interest of consumers of vendors of hackers of everyone Except some offensive folks in the government. I'm sure you're you won't be spotted here and Corporate lawyers whose billable hours Make them a lot of money harassing security researchers The problem the problem with these threats with these legal threats is it's chilling to free speech And this is of great concern to me if we don't have this discussions of Problems and the products we all use then we won't be able to make informed decisions on which products to buy Which products to purchase as customers? We won't be able to pressure the vendors to fix their products and Again and again again the companies after all this happened and try to backtrack about what their actual stance was We weren't trying to squash this research HID when they sent their threatening letters saying they would use every legal remedy at the available Later said no no no we didn't say don't do the presentation But yeah, I didn't quite purse the words That way and looking at their their actual threats. It seemed to me that indeed They succeeded in chilling the speech the fact is When the companies make that when the vendors make the poor decisions when the Vendors decide to be confrontational They they are squashing a free speech and I believe they're squashing our First Amendment rights Jennifer Granik and her talk a couple of hours ago talked about a litany of tools that could be used against security researchers And there's a lot of law involved Like I said, I'm gonna try not to give anyone legal advice whatsoever And Jennifer actually didn't give anyone legal advice, but she was the expert so definitely listen to her it It doesn't matter the the the one of the other problematic things about these legal threats that the these organizations pursue against researchers is Even if the researcher would prevail in a court of law Even if the law at the end of the day would be on their side It it doesn't matter because the defense costs and and IO activists an excellent excellent example They're a small company just to defend themselves and to litigate the matter which Many many people believe they would have prevailed in It would have possibly destroyed their company And if they lost They certainly it would have been demolished as a small company and a lot of the most interesting research That's not being done independently is being done by small firms small security firms such as IO active also These threats Don't help the security posture of the industry lawyers Can't fix bugs lawyers do not make a safer I guess I would argue some of the civil libertarians that are here to do make a safer Jennifer Granik the EFF There's some good folks here. I think everyone loses when we're silenced and hopefully we can Create an approach and fight fight these battles internally to to prevent everyone from doing the wrong thing As I mentioned before I believe this is a public safety issue and the previous I Granik mentioned in her speech The unsafe at any speed would not be allowed if the car manufacturers were to say well You can't discuss this vulnerability in this danger in our cars People would die because of it. That's sort of a dramatic thing to say people would die but The software in question and the software that runs our world is is running our critical infrastructure hospitals are running this code public safety institutions are running this code The most bewildering thing about this is that when companies do attempt to squash research. It doesn't work. They fail miserably Many of these talks the IO active one for example, I don't think that would have generated much press With all due respect to IO active. I don't think it was that Dramatic or groundbreaking of work. It was only when the legal threats were created that suddenly those schematics Schematics that were online for a long time were suddenly accessed and referenced and talked about and people suddenly had a lot of interest in these products and and these vulnerabilities and people that Couldn't understand Michael Lynn's talk We're downloading and mirroring this all over the globe and that wouldn't have happened if there weren't legal threats and The internet does really a wonderful job at distributing information when when there's threats of censorship And I don't think the PR folks at the organizations that have made these poor decisions understand this I don't think the lawyers understand this. It's not good business for anyone and I I'm I'm honestly baffled that people still continue to attempt to squash this research Even even if they could squash it Even if they're the entire public did not realize these vulnerabilities even if they were able to prevent it from making the Wall Street Journal and whatnot There's no reason not to believe this information still wouldn't be disseminated in the underground In which case the the customers would be Would lose even more But by the way, I like the idea of using very bright people and well respected people to make my points I I think that's a lovely method it's so hard in any relationship to to to build trust and Many of these organizations depend on independent security researchers from con to contact them and to point out issues that occur and I think the the month of Apple bugs is an excellent example of what happens when a Vendor has poor relationships with a security community the security community instead of going to the vendor now We'll very often say you know what I'm just going to release this and it doesn't really matter Working with the vendor is is is going to be counterproductive. I mentioned some of these Adobe versus lcom soft it wasn't necessarily about Squashed research that was being presented, but it certainly was an attempt by a corporation to to stop research Being published into their work. There's a very interesting case The SDMI versus professor felt was an excellent example of not only a so-called hackers doing this research But very well-esteemed respected academics doing this research and still feeling As if their research was being squashed and successfully so man, oh So a number of years ago, I would say seven eight years ago Microsoft had an incredibly poor reputation In regards to not only their security profile on their security stance Their security methodology, but also the relationship with the community And Microsoft in the last five years while their software isn't necessarily secure They are surprisingly a leader in secure development life cycles With community relationships with hackers And they haven't they haven't had any of these incidences that I'm going to describe They've been doing the right thing. I'm not endorsing their software But I'm just saying if Microsoft gets it right, why can't why can't everyone else? It's it's possible that there are actual tools that can be used to Successfully to squash this research It's possible that the law is on on the side of the vendors in many of these cases and the lawyers and the internal Discussions that these organizations have present these options these business options To to to the vendors to decide whether or not they were going to attack the the research or not But they're not attacking the right people the people that are going public and publicly disclosing these issues They're not the enemy the the enemy are the people that are selling these exploits to people that will commercially profit from them And selling them to to governments that will use them for offensive purposes and whatnot Vendors do appreciate the research that independent hackers do They come to these conferences they send their folks to them they sit in the rooms They take notes they talk to them afterwards. I've seen this again and again But they seem to expect that the research should be done on their own terms They're very happy to get free work, but the fact is you can't get free work from people And then expect everything to be on your own terms. I'm not going to discuss In-depth the disclosure to date that's been What appropriate disclosure is that's an ongoing discussion that that's kind of interesting, but it's been happening for half a dozen years or more But I would like to point out that when someone contacts an organization And says hey, I have I have an exploit. I I found some problem in your software It's it's really inappropriate for the organization to expect the hacker to start doing full-time volunteer work for the large multinational You can have it one way or the other Again, the customers don't like this the customers don't like to be vulnerable They don't care how the problem was found They know that lawyers are not going to solve their problems or as customers will they won't protect them they need the bugs fixed and I think it looks really bad as an organization When essentially you panic and I I think I think you can characterize some of these litigious attacks on researchers as corporate panic and as a customer When I see my vendor panic and that doesn't instill confidence in me at all. What instills confidence in me is The vendors saying oh, yes, that was an excellent issue. Yeah, we're looking into that. Here's our response Here's immediately. We're gonna tell you how to mitigate the threat. It's about threat Here's how to mitigate it. We're not worried about it at all. We're gonna get a patch very quickly And it's all good. That's a response. I think as a customer that I can have some confidence in When when a vendor says oh my god, what did they release? We're all in trouble That terrifies me as a customer that's not good Also, man, so Of all the people the vendors could piss off in the world a Conference full of hackers is probably the wrong decision When you antagonize and attack a researcher, maybe you can get them to shut up Maybe that researchers is affiliated with a company that doesn't want to go under because of legal threats The problem is there's suddenly an audience of five thousand hackers that may have some free time on their hand and not Be afraid of the litigation. Maybe they live in another country Maybe they're not affiliated with the company. Maybe they're 16 years old and still can reproduce the bug You don't you don't want to upset us. It's it's not wise And and if you do upset us, we're not gonna give you that free research We're not gonna help you out. We're not going to work with you in a constructive manner If they were the bad guys if we were the bad guys We would never tell anyone and certainly that's what black hats do Black hats find these holes. They find these security issues and then they keep them in their private arsenal The private arsenal becomes useless once you release the bug to the public It's counterproductive for doing bad things when everyone knows about this. Ah excellent This tastes better than water so a Company will often say internally what are our options and Unfortunately, this is not the conversation. They often have with their PR folks or their security folks They'll they'll have this conversation with their lawyers The lawyers know how to litigate that's their job. They know how to create billable hours Their job is not to create a positive PR stance their job is not to fix software Their job is not to look at the larger ramifications of how the reputation will be impacted If you come to a problem with a lawyer to a lawyer and say how can I stop this or what are my legal options? All they're going to give you is legal options So by the mere fact of engaging Within a corporation as this disclosure process is occurring the mere act of engaging the lawyers the lawyers are going to give you legal remedies and They might find a large arsenal of Possible issues bogus patent claims Claiming reverse engineering is illegal claiming that eulahs are binding even though the software was sold third-party or was a click-through They might find many novel legal theories that are untested and inappropriate, but that's all they're going to do Yeah, this is the big one Well, one of the big ones It if you want to appear on the front page of the Wall Street Journal and a negative light You will threaten a security researcher if you want to appear in the Associated Press And in negative light and have your bugs disclosed You will threaten a security researcher The press doesn't want stories about people getting together and getting along and everything working out fine and and a Smooth process between hackers and companies. It's generally not a story I mean it might be a good in-depth story and I hope some reporters take up this challenge and I hope Some of the companies are willing to share some of their stories of working with hackers because it happens on a daily basis But the press is event-oriented So talking about good things that happen is not really an event A large legal threat against a small person. That's a story That's a juicy story that a non-technical person can be interested in that's a story that will sell newspapers That's a story. They'll get picked up by the AP and then it's not good for the vendor. The thing is no matter how bad the bug is That's not the story the still the story is this oppression of the bug The story is of the company panicking even even if it is a serious issue even if there is a large problem If the vendor is not panicking There really isn't good press there. Oh, yeah again When you attempt to squash these stories suddenly the entire world is aware of the specific bug you want to squash and What's that? Yeah, like I said see dark tangents talk for more discussion of Cisco Now so I talked a lot about what not to do as an organization And why this is not the appropriate response for vendors why not to be confrontational What you need to do is you need to let cooler heads prevail you need to eat even if it was an inappropriate Disclosure even if the security researcher was not savvy in how they handled it Even if they were perhaps inflammatory in their rhetoric and said bad things about you as a company Confronting them in a negative way is not going to help this the situation at all Engaging the researcher is going to help the situation Finding out more about the bug finding out more about these issues And and as you're interacting with these researchers Just remember that your threats will be published your legal threats are pieces of news Remember every email that you send out to this person particularly if you have a confrontational Approach the situation very well might appear in the press and and I think that might be appropriate Microsoft has learned and other companies have learned If you just look at the sponsors of the black hat conference If you look at all the corporate parties which tend not to be that good quite honestly, but they try so so I appreciate that All these companies realize that establishing a relationship with a computer security community is advantageous to them The people that are finding bugs in their software could be future employees the people researching their their their software Could be customers of theirs in the future And and there's a lot a lot of corporations That have realized having a successful relationship with a security community is a positive thing Why while I didn't hear good things about the Cisco party the sheer black hat the fact is they were Sponsored black hat and they they did throw party and and they tried And I think that's that's a positive step And I think a lot of these companies that have made mistakes in the past can rectify them Can try to have a better stance? It takes time to rebuild these trust issues After you've stumbled people will distrust you for a while, but you know, it's interesting looking at the Microsoft story The perception of the security community regarding Microsoft's stance on security is much larger now than the general public as a rule The general public as a whole still sees the bugs that are being created. They're still nervous about them They remember when secure then might when Microsoft didn't have any security stance but as a whole the security community realizes that there's a commitment to security and That commitment to security is what everyone wants the government wants this because their critical infrastructure is running on this The customers want this The hackers want this the hackers want these bugs fixed. We're running this software The the software exploiting we're also using The reason why we scream up and down scream loudly and jump up and down regarding these bugs is because we want the world to be a safer place Bruce is very quotable I might not run out of power yet. Excellent. Thank you So I think it's important We'll wait till we see some scary blinking lights before I find the plug for that I Found issues as a customer and many vent software vendors when they're alerted to these security issues by their customers are very They'll fix it very quickly The customers you need to be savvy about the security stance of the the companies you're buying from and Having a bad relationship with the security community. It is not the only problem I think that's indicative of a larger problem that your entire security stance and your attitude towards security is Is not a good one and I think customers need to look at some of these incidents. Ah I was wrong. I found a plug that's in the podium that No, no, it's the right cable I think I'm good to go now We'll see in a second Okay, let's see if I can continue without my haphazard poorly written slides. I think I'll be able to Maybe a few less glorious Schneier quotes As customers it's your responsibility to make intelligent purchasing decisions and While some vendors will not have a good security stance Some vendors will have a better stance and I think these are the ones you want to look to and you're buying decisions I think it's terribly important as customers that you give feedback to the vendors The vendors like to hear from their customers and will largely do what they're told If you talk to to the marketing people if you talk to the people particularly after you bought licenses You know if you bring up concerns and if you say this is a deciding factor in my purchasing decisions they listen to that their company depends on customers and Flexing your power as a customer is an excellent way to help companies and vendors better increase their security stance and arguably it's your responsibility as customers to Complain loud enough to make these changes That was one reason why Microsoft finally drank the security Kool-Aid is they got enough negative feedback from all of their customers That said look no matter how usable your software is it's so insecure. We're not going to use it anymore. We can't and You know it was a long process for such a large organization to try to improve their Security stance and their security development life cycle. Yeah question. Oh Okay Very good So yeah, you have an impact and if as a customer you're dealing with a organization and a vendor that doesn't make Good security decisions. It's sort of your fault as a customer. Don't be the victim Don't be victimized by the people you buy your software from and your your voice in the market can and will make a difference So I'm gonna move on to Talk a little bit about hackers. Ah 10 minutes and what at what what y'all should do now In some of these cases like like I mentioned it some disclosure is arguably inappropriate or done in a hostile manner The fact is if you're releasing this publicly, right if you're releasing these vulnerabilities publicly I say you're not the bad guy because you would be doing other things with this vulnerability You'd be keeping it private you'd be selling it to people with nefarious ends and means So by the mere fact of publicly disclosing and shouting. Hey, there's a problem here You are the good guy now now that you're the good guy You should probably be careful of your rhetoric You know, I told the vendors to be careful all the emails they they write might be used against them Well, also all the public statements that you as a researcher make Think about how the press will react to that think about the tone of your language Write your your descriptions of these issues as if you're trying to help the world because I posit you are trying to help the world and I think Particularly when the press get involved in these stories with confrontations when the hackers were doing the right things say Inflammatory things They don't look good and they should and you should and and being really careful Ask some advice before sending emails out ask your friends about the language you use This will impact the public perception of you and that's really important It's important that we are recognized as helping Global security because we are as publicly disclosing these vulnerabilities Oh Seek legal advice when you get scared now I'm gonna quote Gail Thackeray who is a prosecutor involved with Operation Sun Devil Which happened some time ago? Gail Thackeray once said to me drug dealers give themselves better legal advice than hackers do so When you're when you're curious or concerned about what the legal impact of something is do not ask your hacker friends They will not help you at all The the advice to trust when you when you go to your hacker friends the only advice I would say to trust You know what should I do? I'm scared about some legal repercussions of something the answer you trust is Call Jennifer Granik Now she's not very scalable. So thankfully the EFF exists and even more. Thankfully. She's going to work for the EFF So she can work full-time Protecting our civil liberties our electronic civil liberties but if you're concerned See a lawyer and see a good lawyer and the EFF May not oh don't count on them now someone Just a few minutes ago complained that the that the EFF was not representing every hacker that ever got in trouble Well, I don't think they should If you go hacksore the Gibson and then you get caught and then you run to the EFF and say help help I'm being prosecuted you broke the law. There's what's your defense exactly now one in the organization comes and Exercises legal threats of dubious value towards you. That's a that's a worthy case and at the very least It's entirely likely they'll give you some advice not necessarily represent you but but give you some advice But certainly don't count on any of these organizations It's not their responsibility to defend all of us and and all of the legal adventures. We might get into Finally just have a couple minutes left here Like to address the press I mentioned before that controversy sells These things are event-oriented right when a big event occurs that becomes the story It would be nice Well a I would like you to realize that even if the hackers are not well-versed in Speaking in a manner that makes themselves look good, right? The corporation has an army of PR folks They they train their folks on how to communicate successfully just because the hacker does not necessarily speak Successfully and communicate successfully keep in mind the fact that they're Public about their disclosure the fact that they're not selling it to the bad guys They're not keeping in their private arsenal that makes them good guys now some inflammatory rhetoric that makes them inept good guys But they're still good guys so keep this in mind as you're covering some of these controversies also I would greatly appreciate it if the press out there would cover some of the stories of the good guys some of these You know for every speech that occurs at DEF CON You know 99% of them don't have legal action taken against them and often times There's panic within organizations. I've been involved with some of this panic. I've been sitting in the front row as a as a vendor Scribbling taking notes furiously and paging people and whatnot But we did the right thing But it wasn't a story. It was a non-event because there was no conflict if you can do some investigation Investigation and find some of these cooperation stories I think it it makes for interesting reading and I think it might help set an example for other customers and other vendors and everyone else Also, I don't know To yeah, I would like to speak to the government I would like to tell them to help out in these situations in the case of Michael in the Cisco case They were very supportive, but it was largely moral support Unfortunately the legal tools that were being levied against many of these researchers in many of these cases were written up by the legislature so Contact your legislature when you contact your congressmen your senators when you see laws They're being drafted up in regards to the cyber arena And cyber security in particular and voice your concerns as constituents and and don't just put it Don't frame these letters in the light of this is wrong because it's wrong Frame in the right of the the light of it. It will harm our infrastructure We're concerned as customers The overall quality of software will not improve if this legislation is is enacted Ultimately, we're supposed to be a democracy. So I would hope that participating in democracy can help impact positive change and revoking some of these dubious legal methods used to squash some of this research I'm going to Spend some time in the Q&A room. I have three minutes left. So I'll take questions and be open for discussions next door In room four and the Q&A room for also during the conference feel free to come up to me and Buy me a drink and ask me more questions. Oh Finally one one other message a little bit off topic Oh, if anyone my email address is D addict D. A. D. D. ICT at gmail.com also a Large number of us on Sunday night are going to the pen and teller show at the Rio So I thought it'd be a lot of fun if half their audience was flashing flashy badges So if you want to join us there, that'd be a lot of fun. So thanks much


Hello, everyone. Thanks for joining this session. My name is Yi Liu from Intel Corporation. Today, I'm going to share the passive management in KVM together with my colleague, Jacob Pan. This is the agenda for the session. I would do a passive recap first, and then review the passive usage in shared virtual addressing and the Intel Scalable LV. After that, Jacob would introduce more details on the passive management from the software side. Passive stands for Process Address Space ID. With the introduction of it, DMA Remapping happens at the request ID and passive granularity. To achieve such isolation granularity, platform vendors should also support IMU Passive Table. It is a part-device table by hardware design and its storage in virtualization environment differs across vendors. Like inter-virtualization technology for directed I.O., it is maintained by host and IMU lessed translation. But for ARM System Memory Management Unit version 3 and MD-IMU, it is maintained by guest and lessed translation. This difference results in different ways to set up IMU lessed translation for guest. One is to bind guest passive to host one by one. Another is to bind the whole guest passive table to host. Okay, let's see the passive usage in shared virtual addressing. The diagram in the left shows the steps to set up SVU usage in native. Application will issue bind process request, which goes into device driver, and then will close into IMU driver. IMU driver will allocate a passive and bind the passive with the CPU page table of the current process by creating a passive entry in the passive table. Then passive is programmed to hardware device. After that, device is able to access the process virtual address space with the passive. When it comes to virtualization, guest follows the same steps to set up SVU. However, hypervisor needs to travel guest's specific operations in order to set up lessed translation for SVU. For example, inter-virtualization technology for directed IO requires to travel guest passive location and the passive cache flash to set up IMU lessed translation for each passive in host site. For Windows, which allows guest to maintain guest passive table and lessed translation, it needs to trap the guest passive table initialization to bind the whole guest passive table to host. After set up guest SVU, device would be able to access the virtual address space of the guest application with the passive. Passive is also the foundation of inter-scaled IOV. Each available device interface will be associated with a host passive. This association happens when its parent device is attached to a domain in auxiliary manner. IMU driver will allocate default passive for exterior domains. With the default passive, available device interface can access to the virtual machines guest physical address space. As of now, passive is no longer just a tagging process address space. It can also tag guest physical address space. So it's actually IO address space ID instead of just a process address space ID. This is also the base of passive management in software. Since SVA and SIOV are both based on passive, can they coexist? The answer is yes, since they are orthogonal IO technologies. For example, we can set up guest SVA based on assigned ADIs. Guest just needs to follow the normal steps for SVA set up. However, there is still a difference on the passive program between physical functions and ADIs. For physical functions, passes from guest are programmed to hardware directly. While for ADIs, passive program should be mediated by a host, which means passive from guest should be converted to host passive first and then programmed to hardware. My colleague Hao just talked about interest in command for this guest passive to host passive translation. While it comes to the example in the diagram, if VM0 programs guest passive to device A to VM0 for device A, it will get guest passive while for VM0, it will get a host passive. Considering the host support for net translation and the IO page request from device, host will have both guest passive and host passive simultaneously. This may have potential conflict, so we need to get a proper passive management in host side. This would be introduced by Jacob. Hi Jacob, I think you can take over it now. Thanks. Hi, thanks Yi for the introduction. Again, my name is Jacob Pan. I will continue to talk about IO ACID and passive management. This is a generic library we introduced since kernel version 505. But ever since we have been continuously improving it to meet guest SVM IOVA use cases. So in the next few slides, we'll touch upon four aspects of the passive management. The first one is guest host passive mapping. And then because IO ACID is a system-wide resource, we'll talk about partitioning and namespaces with support. Also, IO ACID is just not a single simple number. It has multiple users with hardware contacts associated with it. So we'll talk about how to synchronize the IO ACID states when things change. In the end, we'll walk through a typical life cycle of IO ACID in the normal flow. Well, we probably don't have time to talk about exception cases. Now we move on to guest and host passive mapping. So far, there are two main approaches to support guest passive. The first one being shadowed guest passive table. This is used by VDD scalable mode. The second is to bind guest passive table. This is used by ARMS as MMU version 3. So now looking from IO ACID point of view, what are requirements because IO ACID is a generic infrastructure. So if you look at the shadowed approach, it requires a guest host passive translation because guest host passive may not be equal to the host passive. It also requires every single guest passive as a host passive backing. Because on scalable mode, we support so-called shared work queue, meaning a single work queue can be assigned to multiple VMs. Therefore, in order to identify two DMA streams with passive, the passive value must be unique. That's why the passive value, the passive has to be system-wide. So there's also a little caveat when it comes down to PF assignment. When you assign a PF to the guest, the guest can directly program a passive onto the physical device, which is not mediated. This is fine when you don't have any other devices that allocate passives from the host. But when it comes to a mix with assignable device interfaces, which is mediated device, and it allocates a passive from the host, so they may create a conflict. Therefore, some sort of enforcement must be done to prevent this conflict. If you look at a second approach, bind as passive table, it's much simpler because the guest simply owns the passive table, and the host simply don't care. In terms of requirements for IOACID, number one is a super set. So IOACID is a limited system-wide resource. In the PCIe spec, passive is 20 bits. So we must partition them into groups in order to support multiple users. In this example, we have three IOACID groups. We call it IOC sets. Set zero is used for host usage, such as native SVM or native IOVA. And IOACID sets one and two were given to the two VMs we have here. In terms of namespaces, we do not support multiple namespaces. Each native environment has only one single namespace for the IOACID. So in this example, the guest pass, the VM1 could have IOACID number one, and VM2 can also have IOACID number one. But the backing IOACIDs are different. The host is 101 and 102 respectively. They must be unique to identify the DMA streams that match different page tables. So in terms of sizes, we illustrated here the VM2 has a smaller quota for the IOACID set because the system administrator may give less resource to VM2 or to VM1. They could be depending on how many assigned devices they have or other use cases. So IOACID is not a simple number. On a real system, it has many users and each user may have hardware context associated with it. In this particular example we'll use Intel's VTT scalable IOVA platform. On this platform, we could have five potential users of the IOACID. So for CPU it has a PASID MSR. This is used for in-Q command. They must be propagated or set up before in-Q command can be used. The VFIO is the pure software construct. However hypervisor must use VFIO to communicate with the kernel for allocation, free bind-on-bind of the PASID. IOMMU of course stores the PASID table and the context. It performs the actual bind-on-bind of the guest page table and the PASID. The KPM maintains a PASID translation table that performs the guest host PASID translation. Device driver such as media device they program the actual PASID onto the device in order to generate DMA request with PASID stream. So in order to synchronize all these users when PASID state changes such as being unbind we must get some sort of a notification. Here we proposed a IOACID notification chain for each VM or each IOACID set. For example when the PASID is unbind KVM will receive a BI notification event and then it will tear down its entry in the PASID translation table. There is also notification priorities and other things we will talk about in the upcoming slide. So now let's talk about PASID slide hopefully it's a little more interesting than a bug slide. On a typical page it consists of five steps. The first is initialization. This is done on a per VM basis. But on a per PASID base it really has just four steps allocation of the PASID bind with a page table guest base table and unbind and free. So DMA can start after step three. So now let's walk through this example. This is a slide in the VTD skittable mode as example. Here it's so starting with a VM issues allocation of the PASID. Once the PASID is allocated it can be passed down to the VFIO interface through the bind on bind IOACID this gets propagated to the IOMMU driver to perform as the bind. They basically set up the PASID in the PASID table where the nested translation is enabled. Once the PASID is bound to a guest page table IOMU driver can notify the rest of the users of the status change. So the PASID is ready to go for all the users such as KVM so CPU can issue into command or device driver can start doing DMA on that PASID. So the notification can reach the other users such as KVM and device drivers. Of course during the initialization time KVM and device drivers must register notification. Those are per VM or process and then basis. We have a mechanism to identify the common token between users of the same process. So the tear down is just a reverse process. Initiate it from the VM to do unbind and pass on the IOMU driver performed unbind and then notify the KVM or device driver. For example when KVM gets notified it will clear its PASID table entry for that particular PASID and we also implement the web counting mechanism to make sure the PASID life cycles are clearly aligned meaning not until the last user drops references of the PASID the PASID will not be returned to the pool for reallocation. So now the status update so our team has been working on scalable IOV enabling shared virtual memory for the past 2-3 years. So we have made a lot of progress in IOMU driver VTT driver and the user APIs. So here we listed a couple of related features related to PASID IOACID core enhancement and VFIO interfaces. We also have some opens we want to discuss in this forum. The first one is should IOACI allocation exclusively being done so VFIO as we have today or we can have a standalone user API. It has many implications for complexity and life cycle management. Also how can a user manage IOACI quota from a system admin perspective. So we have done some research on using our limit or C group but those seems to be too heavy handed for a simple PASID allocation quota management. So finally let's summarize what we covered today. So we talked about the MA request with PASID with remapping this is done at the request ID and PASID granularity. PASID in Linux is managed by IOACID core. We don't support multiple namespace for PASID on the host. The cases also has its own PASID namespace. PASID could have multiple users each with hardware context association. So we must not synchronize them during setup and tear down. So we use notification and ref counting to manage the life cycles. I want to thank you for your attention and IOACI are ready for questions and more discussions on the two opens we have. Thank you so much.


My life be like Okay, you can come on, I ain't doing it, woohoo! Hold on, I'm trying to saw, I see your eyes. Here we go, here we go, here we go, here we go!  Damn! Let's get it. Keep showing off, focus, Greggs. Keep showing off, show my focus, Greggs. Come on, come on, show my focus, Greggs. Come on, come on. Nah, he don't want it. He don't want it. No, no, no, no! Why? Call the switch, bro. Call the switch, Davis. Well, iPod brings the ball up the court. Across the half court, Davis and agent in the corner. One of the defenders flops. iPod just burning clock here. Drives right, hits the... Oh! Let's go! Ha! Who would have thought, man? We're playing the devs. So, I'm playing on the park. It's like 3 a.m. and I get some notifications in my ear. I hear the devs are playing and they hopped on the cages. So, I came to follow them because I don't have shit else to do, alright? 2K day and night right now. So, let's see how this goes. Scott O'Gallagher. Mr. Stoff, who does the ratings. And I don't know who the third person is, but he is the dev. I'm not losing the people with numbers in their name, bro. Alright, show me. Listen, they have the Discord tech, man. They don't play... They made the games, so they should know more than anybody. Yo, what the heck? Yeah, Davis! Let's go! For three. Yeah! Wait, do you guys want to shoot threes? Or do you guys want to high-five? Which one are we doing, man? Hey, we just grabbed them and showed them how broken it is, dude. Oh, my gosh. Let's go.  That's a brick. Let me show you something, buddy. You just left your player out there, bro. That was your strategy, my guy. They know too! Look at this player! High-five for three. Yo, I feel like three-point shooting is easier. I made a pretty bad release earlier. It's funny that there's spam experience. Spam, spam, spam, spam. Hey, okay, okay, okay! He's dialing on us. Interesting, that is interesting. Okay, you can come on!  Here we go, here we go. Here we go, here we go! We need to stop, we need to stop, we need to stop. Do you guys want to try hard or you guys want to go for clips? Which one are we doing? Try hard clip. Try hard clip, this guy said both.   Yo, guard your man, guard your man. Guard up, guard up, guard up. Clamps, clamps, clamps.  A three!  Let's do it! Keep shot on our focus squares, keep shot on our focus squares. Come on, come on, show my focus squares. Nah, he don't want it, he don't want it. Why, call the switch, bro, call the switch, Davis. Give me that. What you got, Davis? Oh, I got you. Nah, that ain't it, yo, that ain't it. Pass out, pass out, pass out, easy dog, easy dog. Yo, they're actually pretty, I can't tell because I think the sliders are easier, but they're pretty decent, man. Get the man in the corner. Hit that square button, hit that square button, bro. Keep spamming square, yo. Give me that.  What? I didn't know you could miss on this game mode, man. I missed. Hold on, though, where you trying? Yeah, there you go. Clamps, clamps, clamps. Yeah, I couldn't stop him. Yo, how you cold? Okay, wait. I don't know, he's spamming scrims. Oh, snap! Yo! Hey, good shots, good shots, good shots, good shots, good shots. There you go. I'm just scratching the video. They're doing that, bro. They're doing that to me, bro. They're doing that to me, all right? Yo, where's it? Watch the three. What if they're messing with the sliders so they can win? Oh, what are you doing? He ain't going to hit, bro. He ain't got agent range. Watch the pass out, watch the pass out. All right, get the top of the key, get the top of the key. Give me that!  I'm here, I'm here. Let's go.  Oh, pass out, pass out, pass out. No, no, no. Shoot that, shoot that, shoot that. You got it. Pass, pass it. You're freezing cold. He's hitting that. That's a pass, yo. Oh, okay. Oh my God. Get that square one time. Yo, man, why is he opening? Whose man is that? I need to find out. Davis. Davis. Davis. Jesus, man. Davis! Yo, not only are you freezing, you're giving up buckets. You didn't shoot them, man. As soon as you got them, you didn't shoot them.  I got kept to shoot. All right. There we go. There we go. Yo, that's gay, bro. Beautiful. Yo, we're about to lose. Oh, man. Well, he got his lock down. Watch it. Man, he blocked me. Kick out. Kick out. You need a sweep. Bro, kick. Oh my God. All right, Paul, you keep shooting from the half-court with that sharp slasher. Oh my God. I thought he came on here to troll, bro. He's out.  Yeah, he was hitting it literally all day today, except now. Get it, get it, get it.  We need two. Me, me, me, me, me.  One stop, one stop. One stop, one stop, one stop. One stop, one stop. Let's go. I got you. Yo, try hard. Try hard. Yeah, yeah, yeah. Trust me, I'm trying. Who's that? Oh, that's gay, man, bro. Okay, I think my guy hit that last one. I got to be honest with you. It was my fault, guys. It was apologies. I was going for the steal. I got carried away. I don't know what happened. Hey, hey, Gigi, you guys. Yo, first Scott pulled up to our faces and gave us some eye contact, then he cut. He said, I don't want to catch no heat. He left the building, man. IP. Damn. Why should they pay you for missing all those shots? Just be honest. Thanks. Can you imagine? Block. Oh, what? Wow. I got you.  Someone tells me he's not compatible. Okay. Oh, snap. Did you get a contact dog? He shot it.  That was not that exciting. He could have upped it to somebody. I saw it. It just wasn't that exciting. You just hit it. He went up for the dunk. Did he jump over you? That was not exciting. That was not exciting. That was not exciting. That was not exciting. That was not exciting. That was not exciting. That was not exciting. We were trying to do alluves off the wall and I thought, wait, we'll watch this three though. Oh, that was the one. We have like grade school children here just trying to figure out how to put the ball in the basket. I put it for three again. Let's go. Round of four, slow clap. I want to learn how to kiss the alley. What? You'll get the ball. I Don't know what's happening no more bro. I got a block Sorry guys Bad grime, bro, it's the best spot to do it the sliders are easy And I Not only did I accidentally do I yo Davis. I promise. I'm not rolling it just happened by accident.  Oh Snap Even what I wanted to do Yo, he got it. Let me get it back.  I don't want to hit that. Yes. All right. Well, I thought brings the ball up the court Across the half court Davis and agent in the corner one of the defenders flops iPod just burning clock here drives, right? Hey, what I tell you what I tell you hold on hold on wait for it wait for it wait for the cream cake Did you get there buddy? Oh, you want to measure their grass real quickly I don't have a ruler. Do you have one? Let's go buy a ruler and then measure the grass. What do you think? All right, ladies and gentlemen, I just wanted to take on the devs.  Yeah, you know how the excuse is going on. Okay Let me pull out the book of excuses First of all, we're playing in the cage doesn't count The latency was kind of high there's a lot of people in the crowd


What's up everybody, Jason here for JaspysCaseBase.com. This is a 2022 Canadian One football 10 box break, Pages two R&B for just the Steelers, Falcons and Chiefs. Now again, this doesn't unlock the break, we still have to do one more R&B randomizer for this break here. And once we sell this one out, then we can unlock the break. But in the meantime, we're just gonna do the randomizer here for the R&B for the Steelers, Falcons, Chiefs. Now these customers that are in this break got a piece of these three teams via a serial number break, round number block. And of course, if any of those teams do get a hit in the break, they'll be distributed via a serial number break. We're gonna go by the first number on the left side of the serial number dash. So for example, of course, if you get a randomized block three and we hit a 13 out of 99 serial number break card for the Steelers, Falcons or Chiefs, it'll go to block three. Another example would be three out of 25, 43 out of 49, 93 out of 99 and so on. As long as the three is the first number on the left side. Now of course, if we do get any redemption cards, they will go to automatically spot zero. And even if it is a one on one redemption, it's still gonna go to spot zero. Now there is a super short print in this product now called Surge. If we do hit one and it's one of these teams that get it, we'll randomize that to the customers in the break. So it'll go to name at number one, since it's non numbered. But everything else should be numbered. Here's the dash roll, here's the customer names. Here's the numbers. Let's click roll. Roll, it's a one and a three four times. Four, one, two, three, four. Four times, one, three, four, four. Zach at the top down to Robert. Boom. And four times, four times, one, two, three, four. Four times, four, four, zero down to four. So again, any of our redemptions for Zach, if there is any for these three teams, all the way down to Robert in spot four. So Zach, you have zero, Josh with nine, Eric with two and five. And then Zach, any one of ones for these three teams would be yours, as well as seven. Josh, you have eight, Jose with six, Eric with three, and then Robert with spot four. So there you go guys. Like I said, obviously it doesn't unlock the break, we just gotta sell out the next R&B, which we can tonight if you guys want to. You got 14 teams in this R&B, and all the teams are in here and some pretty nice teams as well. So for the price going to 200, remember all these teams combined cost over 2,000, so we just divided it by 10, we split them up in a serial number break, and there you go guys. So appreciate it, folks, jazbeasecasebrakes.com.


Okay, so Let's start Welcome everybody. Thank you for attending this session. It's a pleasure to be here in person in Prague with you After such a long time to know known faces and to discover new people so just a little background about me before to start so I Used to be a civil engineer working in a big international telecommunication companies where I gained some project manager experience And then I started in parallel in 2006 software development and we discovered repel with my friend Nicola on the picture also Back in 2007. I was then freelance backend developer and one of the best thing that happened in my professional career was to to be fired actually and to To gain full-time to to do what I love meaning software developer and Drupal And our first Drupal con was in 2013 in Amsterdam and we also discovered the community and we really liked it So that was a chance and We then started our own company, which is called web stands. We are located in Belgium in most in the south part of Belgium So we are now growing a fantastic team of 22 people This also means that since the last two or three years had less time to do real backend development But I had to focus on the management of the company and Because we like the community it also means that we like to contribute and to give back And this is also the philosophy of this talk today to share some some experience building ambitious Drupal project over the last few years So the title for this torque is mine the last 20% Because why we were building this ambitious website We come to the same situation over and over that the first 80% of the project are running quite smoothly Seamlessly we see good progress We are really convinced that we reach everything on time We even accept some change request from the customer and then all of a sudden We don't really know what happens, but to cover the last 20% Takes more time than expected and we see some overruns in budget or in timing So may I first ask in the audience? Who is familiar with this situation? Can you please raise your hands? Okay, so Squad reassuring so we also wanted to check if this was something inherent to our company to Drupal or to others So we also did a small survey in our network and we collected about 20 answers So the people who answered the survey they have about 70 years of experience in software development They have profiled like CEO CTO project manager funders lead dev bis dev 80% of them they are doing Drupal and For the people who are not doing Drupal. They're doing either WordPress angular node. Yes, Amber Java, etc So we have small companies and bigger companies when they have a 42 employees And the first trend is that already more than 90% are familiar with this situation But the good news is that two-thirds of the people they are not surprised anymore. So if you have experience you get used to it So today I would like to go over five different causes for hoverence and for each of this I would like to explain a bit what we're talking about to give some hints or tips from our experience and then to enrich the topic with Advices from the survey. So the first group of causes is the misalignment between the customer and the agency And you won't be surprised that everything is communication and setting the right expectations in the beginning It can be a problem if your customer is also not involved Completely during the project For instance, we see sometimes customer that are not available until the end of the project when they see some concrete deliverers That are ready to be tested And this can jeopardize your project. Also, of course, if they come with late approvers late provision of content This becomes more difficult to finish of course in time and if you budget so What we are doing to cope with that is we are trying to be very clear from the beginning of the project Explaining what will be the different phases of a digital project. So you may know this picture from the internet Not all of our customer are really always digitally mature. So we they have to learn also during the project and You have to tell them what you expect from them at which fades what kind of content When do they need to be more involved or less involved? So everything is communication also to align of course, we're doing kickoff meetings But it's also very good to to keep the alignments during the project. So we organize steering committee meetings every three or four weeks Eventually if you know that the customer won't be available during The project but they have more time at the end of the project It's not a bad idea to also foresee more time and resources at the very end of the project to to cope with this feedback So from the survey we asked the people do you think there is a first time effect? So that it's more difficult when you start working with a project With your customer for the first time and the answer is clearly Yes, more than 90% of the people do believe that there is a first time effect and it goes better with time So that's a good news But maybe the lesson from this is maybe that we should try to limit the scope of the first project with customers So that we learn to know each other better Also some other advice we collected from the survey We should not underestimate the time that is needed by the customer to to provide inputs even if these are Simple inputs that you are used to to get maybe that's not so easy for the customers We should clearly agree on the definition of done. What does that mean? Are we aligned on what you're expecting? We should invest in prototyping so we use clearly we use figma for this But there are also other tools It's very good to to show it concretely what it will look like and also another advice is to deliver Something on paper, which is very clear on your assumption of what you're going to do For the second group of causes, this is about specification and you can have incomplete scope Never details specification or how do you handle change management? So When we go over over detail specification Well, I think you've all seen that we receive another request for proposal and it looks more like a Christmas wishlist than really Explanation about the business needs of the customer. So there it's very important to To discuss and to align with the customer to tell him that we are doing tailor-made Software development and we are not delivering industrial products So they have to understand the impact of each and every request that is in their document And we have to to manage that and to answer Also, some of the customers sometimes they just try even to to detail too much up to the technical solution That you have to implement and that's also an issue when you have to to tell him What you need is more of the business needs and not what you are supposed to do That's more the whole of the solution architect When the scope is incomplete what we try to do So it can be incomplete but because the customer just don't know what he has to specify So what we try to do is to allocate a bit more time for analysis at the beginning of the project in this case This presentation should have been called mind the first 10 person because in the end everything has to be clearly specified from the from the start Finally we have to what we propose also is to learn from project to project and For instance, there are some features that are common and are coming back in every projects What's what you're doing is we are going trying to build some templates for specification Asking the same questions to the customer from one project to the other. So for instance for forms We may ask them what are the fees that are required should the answers be sent to third-party services Do we have to go to a thank you page or to send an email things like that? You can save a lot of time doing this So from the survey people also notice that the business requirements Sometimes expressed differently when the user really start testing So you have to cope to that and be able to to change and to adapt also People remind us that we are doing estimates. We are in a competitive market and estimates are difficult so what we Encourage you to do is to do the job after the project is finished to compare your initial estimates with the time that you have really spent You are not always doing this, but it's on my point of view That's the only way to learn and to do better estimates in the future Also, that's not easy of course, but they advise us to Not come back on the decision on scope. We thought having a formal change request That's not an easy one, but we should sometimes try to be more strict on that Okay, so the third group of causes is the pressure So at the end of the project we often see pressure from the customer or the different stakeholders You can also be internal pressure for instance from managers or even from colleagues and What we tend to do is to allocate more resources at the end of the project Which also means that you burn your budget faster than expected So to cope with that what we have done internally we have built Dynamic dashboards that are connected directly to the time sheets of the developers So that we can see directly in real time the the budget that is spent Of course, it requires that developers are completing the typesheets and the level of advancement with quite some accuracy But this gives the ability to project manager to give early visibility to the customer But the status and the the progress of the project Also what you're doing is To ease the the final steps of the project We used to build some checklist for the goal life the usual suspects the things you have to do always for each and every project For instance a CO checklist infrastructure or configuration of the website So we can go a bit faster and be sure we don't forget any of these details finally We also advise do not an easy one, but don't give into the customer pressure Someone told me a spoiled customer is the last customer So in the end it's true if you always say yes to each and every demand Sooner or later you will lose the customer because you will come to some point where you you cannot say yes anymore so from the survey people also Tell you that you should Evaluate the risk in real time and be transparent with your end customer about that and and share the risk. That's very important You should accept uncertainty and imperfection and we completely agree with that. We are not perfect. So we should We should tell that to the customer from the start that we are not perfect and that there will be difficulties But instead of being perfect We are flexible and we are responsive and we are ready to to be really there when we need it to For as they said for instance to response to drifts and to the implication Finally, we try also to learn and to optimize for the future and that also what what's we are doing now The fourth group is the details and the finishing touches so That's one of the problem that some of the things may may look very simple But in the end for instance if I take the example of just user registration or login There are many different ways to do that and if you go into the details you can spend quite some time There are a lot of Exceptional special cases and that that takes time. They are unpredictable and this is often difficult to anticipate So you should try to give more visibility with the details to the customer and to do that Of course, we try to use some tools for the prototyping like Figma Another advice would be to go for the MVP. I'm sure everybody does that but MVP should not be 100% of the customer budget An advice would be to speed the budget and maybe only have 80% of the budget for an MVP and be sure that you will have some More budget for second phase or third phase and that will help Finally I should have some testing strategy and quality assurance manager. We still don't have that internally the quality assurance manager, but I think it's a it's a must to To to optimize the the last 20% So from the survey it's interesting remarks that says, okay, we also use tool for prototyping This they are very nice But they also hide the complexity of what you're doing and from the lower layers and so people tend to simplify and So that's that's very easy and you should try to explain the customer where the complexity lies another interesting remarks is that Okay, it is just normal that we spend so much time and resources to perform the last 20% But it's just perception that it's an overrun In the end it isn't The quote from someone that I really liked I wanted to share the devil is in details Okay, the last one never ending sprints so don't know about you but Sometimes it comes difficult to finish the early sprints for many good reasons can be lack of resources lack of people dependencies with other sprints or your Customers not really with the content or they cannot validate it and so this just jeopardize the the next sprints and the The end of the project So if you have any idea on how you can help with that and just use the app and put your comments and we'll discuss that in a minute What I would say is maybe this scope some of the features from the sprint if you cannot do it Of course the definition of the print before and is very important try to avoid dependencies with things that are not ready Hope won't be ready in the sprint the last one is really To avoid testing too early if it's not finished if the quality is not there Don't try to spend time for testing because you will pay it in the end and you will lose too much time and too much budget Okay, the last one from the survey we asked the different companies how much percentage of The rules they allocate to the different jobs and you can see the result here So 70% on the average for the project management analysis a bit more than 10% and They have quality assurance 6% and test 6-7% So I would like to thank already the different people who have answered the survey some of the logos are there And also the people who didn't want to have the logo on slide We'd like to thank my colleague Joffrey for the support with the slides and the project managers For the discussion and the inspiration about this talk. That's it. Thank you everybody so if you have question you can come to the mic or raise your hand and I'm also available tomorrow during the contribution or after this session if you want to discuss it further Don't forget to fill in the survey of course anybody Thank you for presentation just about the last slide the question you combine together the front end and back end depth Time like it will be nice if you split to see how close each of them Okay, just Do you have this information? No, you can share. I don't have it, but next time we'll split it Thank you. Don't be shy Do you work mostly with time and material or fixed cost projects mostly time and material and All this topic are related as well like or what are the expectations in the beginning of The customer in terms of time and materials. Do they have this? limitation of the budget in mind or you go as you go and see what exactly is Deployed and how it should be changed. No, it's very difficult I think they have the global budget in mind and so they don't really look up at the different phases and you know, so That's important to to explain to the customer the rules of the game from the beginning and I think If you see that they want to spend too much time into the details and you don't have the global objective in mind It's your role. It's your duty to inform the customer and to tell him look if you are doing this We won't be able to do this other test that is maybe more important to your business Do you try to clarify it that like the top limit of the budget they say you have That's the question they don't like to answer of course So yeah, if you that's maybe sometimes difficult with the first Project and it's come easier with the second or third project And so that's why also encourage you to not put the full budget of the customer in the first project and maybe Try to work together to get to know each other and when you have confidence it becomes easier after impression. Thank you so one of the things I struggle with is convincing Customers that it's important to pay Money for project management or scrum master. How do you what has been your observation and how do you deal with that? Well, I think we have the same observation when we send our time sheets and they always look at the time or the ratio of the project management against the development and Yes, we meet the same difficulties, but we try to insist also what kind of mistakes that we did was to timesheet Project management and testing other the same umbrella because they were this was done by the same people So we tried to split that and to be really clear on what time is spent on functional analysis What I miss by the project management on testing so they understand a bit better why it takes so much time Okay. Thank you everyone