Thanks very much. Can you hear me? Okay. Yeah good. So I'm the idea for this talk is to look at some trends that are changing the way we test software and I'm gonna make a couple of Kind of predictions what's going to happen over the next five to ten years in terms of automated testing and How teams need to prepare for that most of the things that I'll talk about You can do even today if you want to get your hands dirty some of them are not still that easy, but Most of them, you know if you find any of these ideas interesting You will be able to do even today, but I think over the next five to ten years They will become much cheaper much better and much easier one of the things that's kind of happening in the industry now is Stuff is kind of getting more and more fragmented And that's going to create a huge challenge for testing over the last Five years in particular. I think we've had it relatively easy in terms of testing because most browsers are kind of now cross compatible and everything seems to be kind of working relatively okay and hmm, I Think what's going to happen over the next five to ten years is the number of platforms is going to explode People who are doing Android testing now can complain You know for a good reason, but what's going to happen over the next? five years is that's going to be multiplied by a hundred times and There's a couple reasons for that first of all there's a prediction from ABI research that says that there's going to be 40 billion devices connected to the Internet by 2020 now Some of those devices will be reasonably good computers some of those devices will be you know silly iot experiments, but more and more will get the unexpected stuff for example, I'm working on a collaboration tool now and We try to test it in as many browsers as we can and We got a bug report a couple of months ago that it doesn't work on a Samsung fridge just kind of Slightly ridiculous. Why why on earth would use a collaboration tool in a Samsung fridge, but you know you can so people will report bugs in it and I Think kind of the estimate for the Internet of things as a market according to IDC is that it's going to be worth 7.1 trillion dollars by 2020 so I think we'll see a lot of investment in new devices new ways of communicating and we're already seeing stuff like Amazon Echo and Google what I call it that doesn't have a screen anymore and just as a voice control and things like that So the the number of devices the number of ways we integrate with things is exploding the other thing that's happening is cloud is kind of happening everywhere now and IDC estimates that by the end of 2017 65% of the companies worldwide will be on some kind of half private half cloud hybrid and That's a trend that's continuing and one of those things It's becoming fantastically easy with the cloud now is to Provision infrastructure and make kind of testing cheaper ten years ago the biggest problem most teams I worked with had was having a Relevant production like copy of the infrastructure for testing well now if you're on the cloud that becomes trivial easy But some things become almost impossible because you do not own the infrastructure anymore so Because kind of you no longer in control of the infrastructure any ideas around integration testing become a lot more complicated because you can't really cause I Mean you know apart from launching a nuclear attack on an Amazon data center You can't really close the infrastructure to properly go down and Any kind of testing around infrastructure becomes a lot more complicated So those are some of the trends that I think are becoming really really interesting there that are going to both open up opportunities and cause some serious problems for how we doing testing and make us rethink how we doing testing and The first real opportunity that I see there is that we'll be able to start changing the balance of What's considered expected versus unexpected most teams today? They have this division of kind of expected and unexpected where there's a bit of unit testing component testing and some kind of automated testing for the expected stuff and then there's a whole set of exploratory testing to catch unexpected problems and What people typically do with exploratory testing is they go through a bunch of heuristics looking at different formats looking at different kind of Boundary values and things that then try and see what what the system is going to do and This is kind of in my view a bit silly because most of what people say that talk They're doing when they're doing exploratory testing Is not really looking for unexpected stuff. They're applying heuristics for stuff that they can predict and they can expect For example, there was this a big thing two years ago where if you typed HTTP colon Without anything else into the Skype client Skype crashed so badly had to uninstall it and then reinstall it Restarting Skype didn't work restarting the machine didn't work now I'm Kind of we can say that's kind of unexpected and should have been caught in exploratory testing But that's silly. It's just kind of a URL that's invalid Kind of why on earth would we not test for invalid URLs? I and I think kind of the big reason why most teams consider things like that as kind of unexpected Is that it just costs too much to go through all those cases? It's just economics. So automating for all those cases because there's five trillion ways a URL can be invalid Automating all that is is just would cost too much and the maintenance would cost too much But it's not unexpected. It's something that we need to start treating as expected during testing and This is where a couple of things that kind of are emerging now. I think are going to make testing like that Significantly cheaper and easier to automate We've known about mutation testing for at least 15-20 years mutation testing is where We apply algorithms to modify inputs and kind of figure out what's going to happen Whether something is going to break and mutation testing exists in the huskill community as quick test and there was I Think it was it wasn't subotero something like something like that But it was kind of doing mutation testing in Java So we've had tools for mutation testing already, but kind of it's just too expensive to run It's too expensive to maintain. That's why people don't do that but calling it unexpected is kind of a bit weird because there's for example a Database on github published by max wolf of 65 kilobytes of typical problematic strings It's available on github. We can kind of just use that and run through those things And it has all these kind of weird edge case combinations I published a tool for chrome called bug magnet that on right click makes a ton of these typical Problematic unexpected cases available so you can just try how a field works with the valid URLs invalid URLs and a ton of other things So we have heuristics for this we have databases of this stuff It's just we're not using that for automation because it's too expensive now. What's really interested Interesting is if you look at the trends happening in the cloud, for example, Amazon has this device farm Where they will run your tests on hundreds of real devices in parallel and just report it in two minutes and That costs almost nothing so all of the sudden It doesn't cost too much to kind of run these tests on you know all the inputs fields in your application Because Amazon can do that at scale and they do that by automating through the accessibility API's on all these devices But there's another really interesting project that Jason Huggins is working on called topster bot topster bot is a physical tapping device that you can automate Using a kind of a stylus and the first version of topster bot was developed using The technology they use for 3d printers to kind of move the pen around and it was this whole I mean on github You can download the 3d print schematics to print your own topster bot and That means that we can now start automating the testing of phones and kind of devices and things like that and and all those weird Samsung screens that explode if you leave them charging In a kind of not just an automated way through this accessibility API that kind of you know has to simulate a lot of things but really tapping and At the moment there's there's no device farm that does this stuff But given where Amazon is going with kind of device farms for everything I can imagine that in a couple of years You know somebody's going to start offering a device farm of topster bots running on Real devices on lots of these weird things where you will be able to send a Database of you know weird strings that are completely predictable To see if any of the fields in your application given this weird string does something unexpected and kind of There are farms of applications now that are really really interesting for this for example browser stack allows you to Test how your app works and see how your app works in over a thousand combinations of operating systems and browsers source labs That's kind of done by the people who've done selenium allows you to automate over 800 combinations of browsers and operating systems using selenium and Kind of that's already here Now given those capabilities. I mean this If we were going to run tests on all our fields for all the data points and everything would kind of yeah cost a bit, but Is manageable now and again combining Amazon's kind of hunger for killing or everybody else by creating cloud farms and Devices that are being able to tap and kind of applications like this What I think is going to happen over the next five years is going to be amazing is we're going to get this combination of cloud device farms browser farms and kind of databases of testing heuristics that Will allow us to Not have to kind of say that these things are unexpected anymore because You know there are databases of this stuff. So it's you know 2016 somebody typing a Apostrophe in a name or a kind of incomplete URL is you know, it's irresponsible to call that unexpected now And I think What that's going to do is allow us to finally do automated UI mutation testing the way that can liberate humans to do Exploratory testing for real unexpected stuff. So My prediction is that well a lot of what we do with exploratory testing today is just going to be automated because that's kind of Predictable it's no longer kind of unexpected now Kind of with that in mind, we still will have a ton of stuff That's completely unpredictable because humans are unpredictable. I'm not saying this will replace testers but it will give testers more time to do important stuff and this is where When we start looking at humans being unpredictable the big question is you know Is one person testing something using heuristics Really better than five million random monkeys tapping stuff on the keyboard For some stuff absolutely, but for some stuff it will be better to get you know five million random people to kind of try and use something and This is where we can start detecting completely unexpected things because humans are unpredictable. I mean We had cases for example where people would misuse our system in weird and wonderful ways using Google Translate that will break everything and And You know It was impossible for me to predict this until we started working with real users and figure out figuring out how they use the system So our testers were kind of not predicting those cases now One one thing that's really interesting that is happening in the industries There are more and more ways of kind of organizing groups of people we're getting these kind of crowdfunding campaigns We're getting crowds Management websites and one of the things that Amazon is doing is this called mechanical Turk thing where you can hire 10,000 people to do a very simple task for almost no money and For example, if you're trying to figure out well, you know, will 10,000 people Smoke testing this stuff break it in some weird and wonderful ways. That is now accessible it's reasonably cheap to do that and I think kind of What that's going to allow us to do is allow smart testers to really focus on real proper exploratory stuff where kind of random human stuff we can farm out to random humans and predictable databases we can farm out to mutation testing and The big problem why we can't use this stuff now is Management of information because if I I can I can organize a smoke test with 10,000 people for almost no money But I cannot go through the feedback in any reasonable time. I Will see that the system broke but it's impossible to kind of organize that information easily or and it's impossible to Kind of figure out what actually happened where because we lack management tools for that and we like organization tools to kind of do testing sessions here now There are already kind of some Companies like user testing that are kind of you can pay them to bring relevant users on your website And they will film a session they will show you sessions and things like it So that's not going to kind of scale to the level of Amazon Turk, but we you know, there are some emerging testing management Crowd management said tools emerging So what I would really love to see over the next five to ten years is kind of the marriage of these two things Where we get better crowd management tools So smart testers can direct 10,000 people to do something and get a nice summary in five minutes How many of those 10,000 people were allowed, you know able to complete the workflow How many got confused how many broke the system? How many did something weird? So I think kind of that my second prediction over the next five to ten years is we'll start seeing these automated crowd source organization tools That will help us get humans real humans to use our stuff on real devices Not automated browser farms that are kind of good for predictable stuff But this is really random unpredictable stuff where kind of we'll be able to do that with a statistically significant level of testing Doing a you know a random test with three people. Okay, you know, they might break the system But if everything works does that really tell you anything but doing a random test with 10,000 people or 20,000 people is going to become really really interesting and I think what what we'll start seeing there is Not kind of the replacement for smart testing because it's impossible to do any smart testing here But what I think we'll start seeing here is smoke testing as a service Like, you know, if we send this and we send this to people in Thailand and in Finland and in Germany and in Czech Republic and in Brazil and they start entering their real names and their real addresses and stuff like will something weird break You know, if they use it with Google translate or some kind of smoke testing stuff is going to become a lot easier to do With these crowd management tools and I think kind of from a usability testing perspective this can give us automated focus groups lots of companies now make Bold decisions about their products by doing focus groups with 10 or 15 people That's not statistically relevant. You might have picked the wrong 10 people and then made a stupid decision for your product But imagine that you can do an automated focus group for 20,000 people and get statistically relevant results That's going to kind of revolution is how we do kind of usability testing how we do product insight and things like that so Um, the other really interesting kind of thing that is then left there is How do we help humans make better decisions given all these management tools given kind of a explosion of devices given these better test automation opportunities that are now cheaper and kind of that's what I think the second big opportunity of The the changes that are happening in the industry now is really important is kind of assisting humans in making testing decisions now Predictable stuff on the side is always going to be unpredictable stuff. There's always going to be stuff that we can only explore and kind of look at and one of the really interesting Potentials that happened over the last kind of 10 years in the testing community is approval style testing Approval style testing never really took off because it's too expensive to manage and approval style testing is Kind of as opposed to the other stuff the type of test automation where you predict the results and you compare Whether the system does what you predicted approval style testing is you have a baseline of what the system does Then you run your test and then capture what the system did and kind of approve that the change was not the first tool I've heard about this was called text test where it was really smart comparing log files and text test is incredibly good for Doing very very complex changes to big systems where you can't really predict everything that's going to happen So you tweak this thing here you run the process and then it compares log files And it's really smart kind of to discard timestamps to look at important things and just give you the difference and say well This is what the difference in the process It's kind of this what you expected or not and then if it is you can use the new processes the baseline So that can help significantly, but you know automate legacy systems complicated systems unpredictable system systems with too many moving parts, but it's too expensive to manage and Looking at what's happening in the industry now with the number of devices number of platforms number or you know the Deploying on the cloud where you no longer control the infrastructure or kind of all these other things We will have start seeing unpredictable stuff more and more and approval style testing is going to become a lot more important And this is where there's a couple of really interesting ideas that we merged recently I I kind of tend to waste months of my life playing video games when I can and One of the things that's really really exciting In the video gaming space now is this game called no man's sky No man's sky is a it came out a couple months ago but people have been writing about it for years because They are it's a space exploration game where they're doing a mathematically generated universe of 18 quintillion stars now I have no idea how much 18 quintillion is but it sounds kind of huge and each of those stars Has planets orbiting around it and each of those planets is supposed to be playable for months and months and months and This is kind of completely mathematically generated now That the what they what they did to create some sense of realistic environment there is they modeled Kind of earth's animals and earth's buildings and things like that and then they did some kind of mutation math So for a new planet, they would take a giraffe They would kind of shorten the neck they would change the colors or they would take a lion and they would kind of modified Somehow they would take some buildings and extend them shrink them So it looks kind of realistic now the big problem with something like this is How do you test this stuff? It's kind of there's no expected result and even if there is there's too many expected results for any human to kind Of reasonably test in any given amount of time and again, there is no right or wrong here. They're doing kind of mutations mathematically generated. So the way they're testing this is Amazing the way the testing this is pretty much the way how NASA explore space What they've done is they've built software probes that fly around this universe and film what they are seeing and in the developer room they have big screens that are kind of Showing a video of what these probes are seeing and then at some point when somebody sees something That's kind of weird. They stop they pause they rewind this well This doesn't really look good then they change kind of the math a bit They run it again They see how it looks and kind of they can approve or or deny the change and I think kind of this whole idea of Running probes through the system and filming what the probes do Okay for a video game you can physically film what you know they're doing and showing but kind of an idea that way something is going to kind of fly through our software in some way and Record what it's seeing or record what it's doing and then show it to us and then say well does this look good or not? I think that's going to revolutionize the whole kind of unexpected testing and help humans make better decisions but for that we need to be able to kind of very quickly see a comparison of the old to the new and There are some really interesting tools emerging in this space Five or six years ago BBC wrote this tool called rate Which was able to compare screenshots of a website and just show the differences Just you know help me make a decision. Is this what I expected or not? So we changed some CSS Here's a very quick diff visually and yes, I expected this or let's go back and This was kind of the early Visual kind of comparison thing helping people to make better decisions but then kind of tools like ZB a visual review emerged where there was some workflow built-in So not just kind of you know take screenshots manually and compare them But hey, this is the five screenshots We want do you approve or do you reject this and you can see kind of that integrated in the build system and for a while there was a tool called domreactor that He's now deprecated But I would love to see somebody kind of pick up something like this Which was really good comparing how a site looks in two different browsers and just kind of visually highlighting differences again just helping humans make decisions faster approving or rejecting this and Kind of now we're seeing kind of some really interesting tools like up little's up little's is this hole is a management tool that you can use up you more selenium or a bunch of other automation tools even I think QTP and then it kind of records the session automatically for you and is really good presenting the differences in the video So you can make a good decision whether this thing kind of whether the change is good or not without having to predict everything They changed up front So I think these things are still kind of emerging and it's very very early for these tools to be kind of really really effective But what I would love to see is kind of more ways of doing automated probes through the software Text test does log files up little's does user interfaces But I think this is a really interesting perspective if you start looking at really complex systems that are deployed on Complex infrastructures internet of things things that are kind of you know unknown and unexpected where I think what we're going to start Doing is a community a lot more is designing probes that kind of fly through our software record What's going on and then have these management tools present the differences so we can say well Yes, kind of that's the change I expected or hey, this is a bit of an unexpected change But let's talk to a business users whether this is okay or not and be able to manage those large processes like that So I think what we're going to start seeing is approval style testing is going to become a lot cheaper and a lot easier and kind of there's of course some other stuff when we talk about user interfaces that are more predictable than kind of let's you know where everything that changes and There's a whole class of new tools emerging for user interface testing now That's really interesting one of the biggest problems for UX user interface testing is layouts where kind of we can do a lot of functionality testing but even the kind of you know best predictable functional tests now can't tell you that something is kind of 10 pixels over something else and it looks really ugly and that's why we need humans to look at it and We're getting new tools now to be able to do Automated testing of layouts for example Jim Shaw wrote this tool called kihote, which is a unit testing tool for layouts for CSS and You can kind of use something like well I expect the top edge of Navbar to be you know at this point and It's not going to do this on a computational thing It's actually going to look at the screen and figure out where it is where it should be and say well It's kind of a bit lower than expected It's a bit higher than expected and you can use tools like karma to run an automated test like this in 20 browsers and 20 devices and very quickly get a report. It's amazing stuff It's kind of something that you know, we've never been able to do before there are tools like the Galen framework that does Kind of automated testing of layouts in a kind of given when then style and allows people to describe layout In a textual way and then compare this across different devices different platforms but the big problem with all that is kind of the people who are most concerned about layouts and for the most relevant to say something about layouts are graphic designers or UX people and They are not the ones doing the testing Because all this stuff is still very technical Developer has to automate the test or a tester has to automate the test and then there's all kind of roundtrip and bottleneck feeding back to UX people or designers and That's what I think there's another interesting trend emerging and We're getting much much better prototyping design tools The designers are now using to almost build the prototype up without any development on their own One really interesting up like that is a pop-up pop-up allows people to take stickies and draw screens buttons labels and things like that and then kind of Take photos of that and very quickly kind of assemble a workflow saying oh, this is a button when I click this button It goes to this screen or this is kind of a list and things like that and that's mostly used for rapid prototyping, but kind of What I would really love to see is kind of pop up plus one of those technical testing frameworks now We have technical capabilities for automatically testing layouts. We have technical capabilities for doing visual prototyping When those two things get merged What I would love to be able to do is kind of create a completely new language here That allows designers to describe visual tests by drawing something and saying ten pixels here five pixels there And I think we're not that far off technically I think this is going to happen over the next five years if you're looking for a startup idea I think this would be an amazing thing to do Because that would allow us to shorten the feedback cycle from kind of layout testing quite significantly And that would be absolutely amazing so the next opportunity that I think kind of We have over the next five to ten years is dealing with things that are kind of impossible to predict Humans are unpredictable Google changes stuff all the time without telling anybody Competitors do stuff that kind of we can't really predict or we depend on a third-party API that changes weirdly. I was changing between two flights six months ago in Paris and the Wi-Fi there was absolutely horrible and Google Decided to kind of break their API for real-time collaboration and kind of our entire system went down So I had to fix it over a stupid Wi-Fi network Panicly debugging stuff and I think kind of things like that are very difficult to deal with now about We tend to leave them off and you know test them in production, which I kind of is okay But it's not necessarily the best thing to do Facebook is famous for Releasing their stuff to 1% of users and then kind of there's no problems then expanding that a bit further Expanding that a bit further than testing in production lots of companies are kind of starting to do that now and We're now I think Starting to accept testing in production as a viable strategy But it kind of costs too much to organize It's not that easy to do and there's lots and lots of companies where they're not you know regular regulatory And compliance problems prevent people from doing that I think what's going to happen over the next five to ten years is we're gonna get a lot of help around Kind of dealing with things that are impossible to predict primarily because we're getting better tools for operational awareness We're getting better tools for how we can manage stuff and what we can manage. There was a Really interesting case study of Kind of things that are impossible to predict or whether they're actually impossible to predict or not Happened five or six years ago when Heathrow launched Terminal five in London and The first couple of days after launch it was complete chaos. They lost about 1500 bugs or misplaced them and They had to kind of Ship all those bugs to a completely different airport to be sorted and returned people got delayed Some bugs were kind of delayed for two weeks or so. It was complete chaos and At the same time the terminal three in Beijing kind of opened and it was flawless everything worked perfectly and Mary that sitting over there wrote a fantastic article about that a couple of years ago how Beijing Terminal launched basically flawlessly because they organized 8,000 people to try it out and They got people to come in and some people were I Assume, you know, they're told to be terrorists some people were told to be lost Some people are told to be busy and kind of they tested the terminal before running it and you know Of course, you know, you can say that you can do things like that easily when you know You have the Chinese army and everything is free but Actually You know Britain being Britain they had a parliamentary inquiry after kind of the disaster in terminal five and it turned out the terminal five was Tested by 16,000 volunteers They did 66 simulations there Before launching the terminal they were just looking for the wrong stuff so All those tests for unexpected didn't help them because you know, they never really tested what's going to happen when they bring so much luggage so And that's why I think kind of this whole problem of dealing with things that are impossible to predict is really really interesting and You can automate as much as we want but unexpected things will still happen And what we see now is a lot more push to kind of monitoring on the client with The kind of mobile devices becoming more and more fragmented with browsers becoming more and more fragmented And and kind of deployment on the cloud magical infrastructures You can do as much testing as you want on your kind of environment, but the app no longer runs on your hardware the upper and somewhere else and That introduces some really interesting kind of potential problems and constraints. We are getting constant problems from people installing stupid out blocking software and misconfiguring it and We're getting kind of lots of weird problems from people working on Browsers that I never knew existed and and things like that And that's why kind of I think monitoring on the client is going to become really really important and adding all that data up so One of the really interesting tools that came out recently is hot jar hot jar is kind of Analytics on steroids it does heat maps. It does Workflows it does all sorts of weird Analytics and predictions on the data That you're getting to kind of pinpoint where the problems are going to be and there's lots of libraries coming out like truck J s that is doing error analytics on the client and kind of Throwing away all those outlier edge cases that are you know Somebody trying something out on a completely weird browser on a completely weird device But then aggregating that data to give us statistically relevant information about well, you might have a problem here You need to look at it. You might you know start investigating this stuff so what I would really love to see is a combination of kind of tools like this with cloud deployment now where Maybe we can you know deploy this using mechanical Turk to 20,000 people Different countries different platforms different tools and use Tools like this to monitor what's going to happen and how it's going to work and then get statistically relevant results out of this So that's kind of I think what we're going to start having over the next five to ten is is much much better tools for behavior changes in production and Looking at well, you know are people doing something unexpected with this size is this exploding in weird and wonderful ways Are the workflows working as we expect and I think that's where we're going to start seeing much more interesting results and kind of Potentially integrating that with continuous integration getting reports like this Google is famous for this episode called 40 shades of blue where the Engineers and and the head of design had a big fight around the kind of changing the colors of the hyperlinks on the home page and This is a really interesting story because it's well documented for both sides You can read the blog post of the guy who kind of designed the stuff he was the head of design at Google then and He wrote a blog post why you cannot do good design at Google anymore And then there's a whole series of posts around kind of How they tested the stuff and what they've done so he wanted to kind of change the color on the home page to something that you know is is more beautiful and better and Whatever and they were asking why you know this change needs to happen and his idea was that This color is the you know He wanted to introduce some new blue color that is much more noticeable to a human eye and people will click more on the add links and What the engineers have done that night is they've deployed that color but also 40 other shades of 39 other shades of blue and They've proven that his color was nowhere close to kind of being better than the Current one and there was a article from the Guardian couple of years later where They were saying at a conference in the UK one of the kind of business people who was working on the Google home page said that The money difference was something like 250 million dollars Expanded to a whole year and the whole Google population. So stuff like that is kind of you know unexpected This guy was I'm sure a very qualified designer He rose to the ranks to be the head of design at Google. He knew the color theory He knew all those things but there was something unexpected that he couldn't predict as a result He quit and I think he's now working for Twitter where they have amazing shades of blue, but they can't make money where kind of You know Googles making a ton of money because they know how to do this kind of testing and I think with cloud with kind of these better tools This is going to become possible for almost everybody to do and we have those tests not just on production But integrated into a continuous build so kind of the last opportunity that I think is happening is kind of big data Is now everywhere and we're getting better systems that can manage big data. We're getting much much Better ways of analyzing this stuff and especially when we do sort of client side analytics and collecting data on the client that's a ton of Information that somebody needs to process if we're doing crowd-based crowdsource based testing that's a ton of information we need to process and There's going to be some really weird stuff to kind of of course will do dashboards and heat maps that's already emerging but One of the things that's really emerging now as well is It's kind of hall Discipline of machine learning The first time when it really hit me that this is kind of incredible was when I read an article in in I think 2012 About target the US supermarket chain where they were sending pregnancy coupons to this teenage girl And then her father went into the shop and started yelling at the manager wide You know at the sending pregnancy coupons to to baby coupons to this girl and a week later It turns out she actually was pregnant so the supermarket based on her buying patterns was able to kind of discover that she was pregnant before she knew she was pregnant and You know, there's no algorithm there. It's kind of machine learning. It's machine concluding stuff on on on patterns And I think kind of you know, we're given that stuff Imagine possibilities for kind of Testing there if we can collect all these data analytics on these weird Devices and things like that. It's not just creating hash maps organizing that information. It's kind of Putting the machine to learn stuff from that and predict where the problems are going to happen and This is where kind of Machine learning was for a long time Too expensive and in the domain of kind of companies who have too much money and too many kind of really smart people But it's now becoming available for kind of almost anybody Amazon has launched these kind of monster monster monster machines with kind of CPUs that I don't know thousands of kind of virtual CPUs that you can process easily and Kind of you can rent the trailer cheaply So Google open source tensorflow that's their machine learning toolkit and you can kind of benefit from you know This is what they use to power Google brain and to kind of predict odds and things like Microsoft of course being a quick follower Release their kind of distributed machine learning toolkit as well So stuff where kind of really smart people have developed We cannot just take and it's open source and we can run it on Amazon cloud and run it on the big data We call it to learn from this and the opportunities are really amazing MIT did this experiment a couple of months ago where they ran it on Ren machine learning on the github so source code of lots of projects of github and Try to figure out how people fix the most common bugs And there's this brilliant paper where they were able to kind of run a machine learning toolkit to Automatically detect kind of some very common types of bugs and to fix it to start submitting automated pull requests and This is why I think is really amazing where we can have machine learning trying to kind of analyze our development patterns our deployment patterns and kind of start predicting where problems are going to happen so Kind of I'm kind of almost Expecting that in you know couple of years will have this kind of big data risk threat modeling where as soon as you start Developing something is going to say well, you know you might want to investigate this component here I'm not entirely sure what to look for but you know statistically. There's a problem there So try and kind of you know investigate that a bit and that's where exploratory testing I think is going to move and Kind of I wouldn't be too surprised if we see kind of something like this happening in a couple of years where as We develop this you know an assistant or something like that proposing where to go and test so that's kind of those are my predictions for the next five to ten years and Most of these tools are already available if you want to kind of Manually combine them and experiment with those things you can You know immediately benefit from that stuff. I hope of kind of Giving you at least some new ideas how to start approaching testing. Thank you very much So I know over and a bit. I don't know if we have time for questions or not Okay Two minutes for questions. Okay Does anybody have any questions or is it too early for that? So the question is what are the top barriers to mature testing to get this level I I guess time Kind of people seem to be too busy all all you know doing stuff So, you know these are all new techniques new technologies We need to kind of I guess, you know invest in exploring them Although, you know all these tools are available even today So they're not that popular yet because I assume, you know people are too busy Doing exploratory testing on stuff that's completely predictable to start looking into this stuff So it's kind of a chicken and egg problem. I Don't know. It's it's probably contextual for every company Good. Well, thank you very much for spending an hour of your life with me. I hope this was useful. Thank you


to detach in our mind from those false thoughts from the attack thoughts and we'll be able to see it just commonly see the false as false but that's one of the stages kind of that we go to once it starts coming up and you start working with this place it's like there's a real tendency to want to project or be the unhealed healer Jesus calls it the unhealed healer you don't want to go around and get moving without having healing ourselves and it's you know just to watch ourselves do this you know and go through it it's really something I had him as a therapist or anything else here and I wasn't there was a man here in Jackson a while back that talked about there is no compromise and that's the part that I want to do at the family level I mean how else is a 15 year old and a 12 year old going to know things if I don't tell him if I don't compromise this course and make sure I bring it out and I keep referring back to to his thing it's just plain no compromise and in the family level I have the that's the hardest part for me to work at you know with my children in my life we just when we were up at Travis City we just went through that pretty deeply because as well a lot of the issues were coming up of how as a parent you know I need to just that limits when we had actually repeated again last night where a gentleman was saying you know I have a son and he's got a duty to clean the bathroom and he doesn't I suggest I remind it's just filthy I look at the bathroom I start to get angry you know just describing this whole dynamic and what what we've kind of been going into is once again it's these self concepts that the parent is another role and there's lots of unconscious stuff down there about good mom and good dad good moms do it good dads do it this way we have these these unconscious beliefs that are down there and what happens is the basic situation is is kind of like with parents and children a lot of times there's a there can be a big control struggle not just with parents and kids it can be in relationships all kinds of relationships but but there can be a big control struggle where it seems to be a battle of egos um where the world is kind of teaching you've got to help tame and train this ego to at least adapt and become somewhat normal to function you know they have to you have to learn that there's things that you have to do in life and you can always tell when you're sliding into ego is that it's the control feeling and the tone of voice when the demanding and the commanding starts coming out you know I think of the the Course in the Holy and Jesus saying that the Holy Spirit never commands and never demands He always reminds and suggests well right away the I'm right away the students will come to right with certain situations or with certain children like wait a minute I've reminded and suggested to a certain point then I've got a really guy I've got a bear down here you know and this and that but really it's always our own lesson I mean the children are those great mirrors again you know that whenever we start feeling we're off the beacon whenever we start feeling controlling or angry it's kind of like if you can pull back to a level of spirit or a mind level it's kind of like what's the deal we're equals here you know did you forget there's just these roles now you know they're but you see how wait a minute does this is the role I am the dead you are the son or you are the daughter you know as soon as the mind clicks into I am the role you know then the control comes in because there aren't any roles in heaven it's clicking into ego because okay okay because we're so long into that whole that role that those expectations take over and we have expectations then of our brother it's kind of like I'm in this role and you're in this role and we're in this role same together that isn't I mean it's an illusion it may be an illusion but it's also a fact well of course this thing is basically it's an interpretation that the only fact is is that of that Christ or oneness but within the dream world these identities these soft concepts that we make up include a lot of roles plus also I'm sorry I don't mean to jump into the thought that came to me was that we also have perceptions about what that means as far as our roles and as a parent or sometimes that's an issue that that's hard to let go of when you're in that when you get into that role it doesn't need to do this you need to do that in the ultimate sense the course is saying that the the real authority problem it talks about like on page 43 of course the authority problem is with God coming down to can I create myself or was I created you know and what happens is is when I get into that I can create myself that's where the self-concept comes in knowing can I but I have and part of it is I chose to go into whatever to be a parent don't education take this business job this and that so you can see where all these lessons that come up really are ways to start us to loosen our identification with these roles and because all of our expectations that we place on our brothers that seem to be violated by their behaviors or really just expectations we're reading meaning on to the situation from our own role and being identified as a male being identified you know even as a course of miracles if I identify myself as a course of miracles teacher and people and I perceive somebody as trying to weaken my position then here it goes again I got another concept or construct that I have to let go of because I'm still as long as there's a defensiveness or or fear or anything that's uncomfortable within us we're still clinging to some kind of role or some kind of a concept that we leave as more valuable than the truth that's really interesting because I know time started with it but both of us has even I'm only like 40 lessons into the course but it's like you had to wear two different hats I mean you wore this the course of miracles hat and you applied it to your life but when it came to your kids you had to put on a different hat because they have certain rules and and regulations they need to follow in order to get where they need where we think they need to get to and I mean we've talked about it in our group meetings you know when do you apply the whole course to your whole life or do you apply it to you know when are you supposed to be trying to become and it was it was really difficult trying to cross over that fine line of one year it's okay to do it and when it's not so it's better just to not compromise it don't compromise it at all and work it work it you know in our whole family life just this thing about judgment that Jesus says in the course that you believe that without the ego all would be chaos but he says I assure you without the ego all would be love see when the mind had these dark beliefs in it and there was all this seeming horror and chaos when it seemed to buy into the separation that all this judgment is is a way to try to bring some control and order into the chaos and this is addressed if you want to go back to the book it's addressed at page 273 in the in the text where he's talking about ordering of thoughts where he says but you've literally limited yourself by all these judgments and orderings that it takes a lot of trust to let go of like you're talking about these ideas and beliefs about letting go of these constraints and restrictions and judgments about my children because an underlying belief is if I let go of that they're gonna be all hell's in a break list you know and they're gonna grow up and what a reflection on us to have wild children you know you see how it goes ego will just run with that and it does take a lot of trust to generalize this to your to your family situation if I'm going to insist in a classroom of you know 30 adolescents that that they have something to complete and then I expect it completed it's like watching your emotions and reactions that's how we can tell whether we're listening to the voice yeah and the thing about it is that the course isn't a path to saying yes yes yes I mean there is a part of an absence from Felicity where and it's been talked about before like let your yay be yay and let your nay be nay but but feel real clear that if that there's still ego centricity and codependency involved if we associate following the Holy Spirit with saying yes to everything and everyone I mean you can see how you can you just totally give away your sense of integrity but that the spirits in there and the spirit literally if we if we just stay with our emotions and we keep surrendering that will tell us when to to say yes when to say no when to maybe put down a guideline that could be helpful you know we can put down guidelines but but it's that insistent and demanding and commanding and getting upset when they don't seem to follow in everything that the whole lesson for us is always to be extending the life and extending the peace and it really does get at our own beliefs and expectations and roles I mean I see where that's the thing that's like always the lesson what do I value in my mind what am I holding on to it's more important than than seeing the Christ in this person or child and that's it that's the thought I just had was when you said seeing it and I can think of certain conflicts that I've had with with my children and the thought came to me was if I if it gets to a conflict then it's obviously something I'm not willing to see that's left to be seen so I haven't given it enough time for that for for what it is that I need to see in this situation so I said so I lost patience and say I'm not ready I'm not willing to go that far let's just do this let's get into the conflict of it and and it's when I find that I just if it's not there yet then I just wait and and then I can I get the answer so to speak and then you see it so it's a matter of maybe it's just like that taking that extra moment to wait for the to see it that's like that so it is so in the situation maybe where in a classroom situation it's not necessarily to let it go it's what do I need to what do I need to see in the situation what is what is it in what do I need to hear about what it is I need to do next and like you said which I thought was great but is this is something that I can say yay about or is this something I can say nay about it'd be a piece of it right either way yeah I think teachers are a really hard time because society puts some role expectations on the teacher and what they're supposed to get out of each student you know in order to go from grade to grade I mean like time going in fourth grade just can't go in and say well let's learn what we're going to learn and hope we all do well you know I mean there's certain standards that every kid has to do every year you know and he's got certain things he has to make them produce in order for them to get to the next year so you can't just have that oh free well kind of attitude that you can in other in other jobs or other situations it it still starts to be more of one of those sneaky ego ploys that that that certain situations or certain people have it more difficult or easier and really what it does is it flies in the face of the first principle of miracles which is that there's no order of difficulty miracles and that's the hardest one kind of like Jesus at some point of course said if you could get the first one you've got the whole course you don't even need the other 49 you know just this is a very basic one but but that's the basically ego strategy towards all kinds of conflicts is it basically always says if something were different in the world or if the situation was different or you know it always is trying to change something on the screen to bring about the peace and and the mind is so resistant to just saying oh it just takes a change in my perception in the moment but that's the one thing that the mind in the deceit state is so resistant to and in the end that's what the atonement is to start to see there's nothing outside of me that can give me peace or take away my peace to do that though you literally have to um I would say just transcend the roles they were asking one guy asked one of these channel entities I don't know if it was um Samuel or uh there's different ones but they kind of the question posed to the gentleman was um it was kind of like well how how can I be enlightened and still be in the corporate world you know and the answer came back you won't kind of like you know that the deeper we go and the more we we literally transcend the concepts that will be used in ways that we can't even fathom and foresee but that but what the ego does is it takes the current seeming concepts and situation and just kind of projects them out you know into the future so I see it as a path that's just being present in the moment whatever it's the situation is that you seem to be in and and using those opportunities and then not getting too concerned about about the next steps or trying to figure out how is this ever going to work am I ever going to be a teacher and be enlightened or am I going to have kids they used to ask Tars saying can you become enlightened and have children you know it keeps coming back to me back to the compromise I actually have less trouble in the classroom than I do at home because I take it back home and say you guys aren't going to be like those but Dan hit me with that compromise and then as I've been you know see I wanted I just want you know it's a another word for you hear you say justification or you know the defense or whatever um as I've kept going on here and then hearing you know like I say with these all these different ones sales put it in our group really well I've got to hear it and I got hearing in my voice or your voice before I get you know


Hello everybody! Watch this video to see how you can clean up your hard disk and registry from junk files in Windows 10, 8 or 7 with free programs such as Ccleaner, Windows Tat, and Visipix. These utilities will help you to correct registry errors, find the files, take up your free space and delete copies of documents, images, and other files. In the previous video, we showed how to clean the system drive manually, but it is a bit of hard work, which is not always effective. Clean up your disk and eliminate registry errors with this free program, Ccleaner. First of all, download and install the program. It's a simple step that takes little time. You will find the download link in the description of this video. Start the program. In the Cleaner tab, you can check boxes next to the files you want to delete. I will leave default settings as they are. Click Analyze and wait for some time. Then click Run Cleaner. Ready? To register a tab, click on Scan for Issues, wait for some time, and then click Fix Selected Issues. Ready? You have cleaned the disk from junk files and fixed registry errors. The operating system will work quicker and smoother now. Note, don't worry, all important files of the operating system and your personal data will be intact. Find the files that take up most of disk space with a free program Windows Tat. First of all, download and install the program. It's a simple step that takes little time. You will find the download link in the description of this video. So you start the program. Disk selection window appears and I select the system drive C. Okay. Wait, the scanning will take some time. Ready? The application has scanned disk C and showed which folders, file types and files take up most disk space. Make sure that you're not going to delete any important system files. It's safer to delete personal files only. If a folder of some program in program files takes up much space, you can delete it. Windows Tat will show you. How much space it takes, even if you can't see it in programs and features. Find and delete duplicate files with a free program Visifix. First of all, download and install the program. It's a simple step that takes little time. You will find the download link in the description of this video. So you start the program. You should select the disk to search for duplicate files and add it to the list for analysis. In my case, it is disk C. Add it. Click on the start button. No wait. It can take some time. Ready? As we can see, the program found several duplicates taking up quite a lot of space. Select the unnecessary copies of this file and delete them. Ready? Now you can see that I've managed to free up enough space in disk C. Make sure that you are not going to delete any important system files. It's safer to delete personal files only. Thanks for watching this video. If you liked it, click the like button below and subscribe to our channel to see more. Don't forget to post your comments.


When I was 14, my classmates and I got to talk to Andy Thomas, an Australian-born astronaut. While he wasn't in space at the time, it was still an experience that shaped the rest of my education. Just last week, students from the United Kingdom, Russia and Italy got the chance to talk to astronauts on board the space station as part of ARRES, Amateur Radio on the International Space Station. And this is your space pod for March 2nd, 2016. ARRES aims to get kids excited about STEM, science, technology, engineering and mathematics by letting them ask questions to real astronauts in space. It also gives them hands-on experience in radio science and the program is available to any educational institution around the globe. In fact, later this month, the ARRES program will initiate their 1000th Radio Educational Contact at the International Space Station. If someone wants to be an astronaut when they grow up, what should they be doing now as a kid to prepare over? Radio contacts can either be direct, where the school or organization uses their own radio equipment, but this can be a little bit expensive and require some expertise. And the other option is to use a tele-bridge contact, where the school uses a telephone to contact a ground ARRES station, and this ground station relays that telephone conversation up to the space station. There are 12 tele-bridge stations around the world, including one in South Australia, run by Mr Tony Hutchinson. One SS, you know, one SS. This is VK5, Azulu, Alfie, India, VK5, ZAI in Kingston, South Australia. I got the chance to visit Tony's station, and upon driving up to where it looks like any other regular regional Australian household, he looks up for the huge antenna sticking out of his roof and the racks of radio equipment that are all scattered around his lounge room. These tele-bridge stations can also be used as a backup communication system for the space station. We had a small emergency on the docking of the Progress Supplier Rocket. They obviously rang up and said, can you go and stand by? That's amazing, isn't it, that NASA's own antennas failed and you were called in to rescue communications? Yes. And as a way for station crew members to contact their family while on station. In fact, when Andy Thomas was aboard the Russian space station Mir, Tony Hutchinson was able to connect Andy with his family so they could have conversations every time the Mir space station passed over Australia. So, how can your school get involved? It's as simple as deciding whether you want a direct or a tele-bridge contact, filling out an application form, waiting for a scheduled time that suits both your school, the tele-bridge contact, if you're using one, and the crew members onboard the space station. Scheduling is the biggest hurdle to obtaining a radio contact and can lead to wait times of about 12 months between filling in an application form and having your students talk to the station. Most radio contacts last from about 8 to 10 minutes and your students can ask as many questions as they can fit into that time. Now, ARIS is designed for educational institutions, but that doesn't mean that the general public can't get involved. Upcoming contact details are posted to the ARIS website and if a contact is scheduled while the International Space Station is passing over your head, you can use radio equipment to tune into the conversation by listening to the frequency 145.8 MHz. Now, you might need some specialised radio equipment to do this, so this would be the perfect time to head to your local amateur radio club and get involved. And although rare, sometimes station crew members get the chance to conduct unscheduled radio contacts with amateur radio enthusiasts all around the world. Thank you for watching and I hope you learned a little bit more about amateur radio on the space station than you knew before. And if you liked this video, feel free to share it on social media and subscribe to our YouTube channel at youtube.com slash tmro for more space pods like this one. I'd like to thank Tony Hutchinson for letting me come and see his ARIS ground station and I'd love to give a huge shout out to all the patrons of tomorrow whom without this space pod would not be possible. Thank you for all your contributions and all the help you've given us in crowdfunding this show. If you'd like to know more about crowdfunding space pods, head over to patreon.com slash space pod. Before I finish, I'd like to welcome Scott Kelly, Mikhail Konyenko and Serge Volkov back down to Earth and wish them a speedy re-adaptation to Earth's gravity. My name is Lisa Stojanowski and until next time, keep on discovering.


Welcome to Toffy TV, it is the Everton News Daily. Callo Antio Lottie has given his pre-match press conference ahead of Everton's trip to the Hawthorne tomorrow to face West Brom and the Albion. Antio Lottie again asked about Everton's top four aspirations but said that it was just important that Everton won the game. If Everton did win the game he would go into the top four of course ahead of Liverpool and Chelsea. If that game ends in a draw then Everton would still be in the top four. But Antio Lottie said he wants to experience it, wants the players to feel what it's like to touch that top four. But they'd accept that it would be a difficult game because West Brom are fighting for their lives in the Premier League of course. He was asked about the fitness of players. He confirmed that Fabian Delph, Yeri Mina and John Filiig Ibaman were definitely all out. And was asked about the fitness of Seamus Coleman, Hammers, Rodriguez, Tom Davies and Robin Olson and said that they were unlikely to be involved. There was a training session of Finch Farn this afternoon and a couple of them would be assessed but they were unlikely to be involved. So it's going to be interesting to see what Everton do tomorrow because we've got a few injuries at the minutes of the bench which was very, very young the other night as Everton beat Southampton. So again it looks like that could be the same kind of squad that we see tomorrow at West Brom. It was also asked about Alan and whether he could play given he's only just come back and said he would be monitored again today and then ahead of the game. So it's going to be interesting with Alan because obviously he's been out for nearly three months and then he come back and played the full 90 the other night so he won't want to take too many risks with him but if his fitness has shown up as he's okay then he may well start as well again. A few days after this one, a few days off after this one ahead of Everton's trip to Chelsea next Monday so down to the bare bones but hopefully we'll still have enough to get the results tomorrow. Everton today again linked with making Josh King's move permanent in the summer. King joined Everton in January on a short term deal till the summer and apparently according to the report Everton are wanting to make his deal permanent. Probably looking at a two year deal with an option for a third but King's had he done anything at the moment as he's been trying to get his fitness up. We've only seen him in fleeting appearances, you know maybe he's one who might start tomorrow at West Brom because of the thinness of Everton's side at the minute so yeah I suppose if you get him for nothing in the summer it'll save Everton one problem because obviously if he goes we're going to need to bring another sliker in anyway aren't we so I guess it kind of makes sense. Moise King has tested positive for COVID-19 while obviously on loan of Paris Saint-Germain he will now miss a couple of day games and on top of that the French press again linking him with a permanent move from Everton through PSG in the summer. He's had an excellent season so far and still scored goals when Potatino come in. Leonardo spoke about him, said they're very interested in making the deal permanent but it depends on the price but the French press seem to think that PSG will want to keep him so you know I've seen yesterday that there was talk of another loan deal for him for next season but I can't see Everton really wanting that I think they'll want to if he's not going to come back and be part of Carlo Ancelotti squad which is still you know a chance he may do. I think if Everton are to allow him to leave they'll want him to do it permanently so that they can reinvest that money into the squad next season and finally the loans a little bit of a loan round up last night, Benny Beningamy played the full 90 minutes for Derby County they were beaten four nil a card of but according to reports Beningamy was very good. Anthony Gordon played an awet as Preston lost 2-1 away at Millwall and Jared Branthwaite was substituted for Blackburn but they were beaten 1-0 at Redding, that is if for the news daily make sure you subscribe give the video a thumbs up and if you want more videos including daily live videos join us on Patreon. See you later.


Hello, this is video on how to calculate binomial probabilities. A pharmaceutical company receives a large shipment of pain pills and uses this acceptance sampling plan. They randomly select and test 24 pills, then they accept the whole batch if there are at most one that doesn't meet the required specifications. So if a particular shipment of thousands of pain pills actually has a 3% rate of defects, what is the probability the whole shipment will be accepted? So you're doing probability that the whole shipment is accepted. Well, let's think about what it means for a shipment to be accepted. It's accepted if there are at most one pills that are defective or doesn't meet the required specifications. So we'll just say defective here. So I'm looking at the behavior of a pill being defective, meaning it doesn't meet the required specifications. So the probability of a success, in this case the pill being defective, that's the behavior we're looking at is P equals 0.03. So at most one, that means either zero or one pill is defective. If that happens, then we will accept the shipment. If there's two, three, four, and so forth, then the shipment's going to be rejected. All right, so in Google Sheets you need your number of trials, number of pills you're going through would be 24. Probability of a success or P is equal to 0.03 and lower bound, upper bound. You're looking at either zero or one pills being defective. So lower bound zero, upper bounds one. Google Sheets, 24 is the number of trials, 0.03 is the probability of a success, zero and one are your lower and upper bound. The answer is 0.8388, 0.8388. So that's the probability that a shipment of pills will actually be accepted in this case.


fierce supernova review. Hey, what's up guys, it's Josh here today I want to do a little review on this fierce supernova by the house of duo fragrances. Now this is one that I reviewed last year I really liked I was just kind of going through some of my favorites of all time spraying it up and I actually really really enjoyed this one so I decided to really take it on a good amount of wears almost done or like a little over halfway on this one really really enjoyed it just kind of been spraying it as almost my everyday scent along with Azora Watts and I'm trying that one out again as well. This is a copy of Abercrombie and Fitch fierce the vintage batch and then also Elysium which is a very very famous fresh fragrance that is very similar to Aventus mostly everybody knows about it but that's what this fragrance is supposed to copy and then they had the notes listed on duo's website but for these kinds of fragrances they put all the notes of both fragrances into one so it's this huge list of fragrance notes. I actually did it for drowning in Blue Dee's Sauvage fierce and it took me forever to do so I will put the notes on screen but it combines two very very famous fresh scents and both scents were very good on their own both scents were very clean and they smell very very good together I'm surprised how good they smell but they do smell fantastic here with fierce supernova so anyways guys let's jump into the pros of this fragrance the pros is that it jumps into those classic fierce vibes now I absolutely love vintage fierce it's one of my favorite fragrances of all time if not my favorite fragrance of all time so immediately spraying this you do get little bits of that classic fierce in here and it just is so pleasant and so good as somebody who craves fierce and loves to wear fierce all the time it satisfies enough of my fierce craving while also changing up the scent which I do really appreciate it is a very very clean scent it is a very very robust scent and it is very very likeable this is definitely one that I have been complimented on I've actually been complimented quite a bit recently and this is definitely one of those ones that has made that list and I've just loved wearing it every time I bring this around people whether I forget that I sprayed it or not it just seems to be likeable by everybody it's just such a easy to wear very enjoyable clean fresh scent and then it is also while being super likable and familiar it is also very different you know just enough difference where I just really really enjoy wearing it because it smells so good but it's also so different yet so clean and all the good stuff about fierce and Elysium come out here it just smells so so good some of the mixed aspects I have here is the longevity and projection now if you watched my original review of this fragrance I did note that it was much lower projection than vintage fierce and fierce attire which are two other duo releases that both smell very good if your fierce had definitely check out their vintage fierce and their fierce attire because it is fantastic I did say and then I still do agree that this has a lighter projection than those two Elysium is known for having very short projection kind of newer batches of Aventus are the same way Elysium and Aventus I think are supposed to be somewhat similar so it kind of makes sense that if you make fierce which is a longer lasting fragrance with Elysium which is a lower projecting fragrance that it would be a little bit lower but I was actually surprised how satisfactory this is on two three four sprays it does have a decent cloud you do get that vibe of yeah when I want to definitely be known but I don't want to choke people out and you kind of get that with this here so although it isn't the best projector although fierce does project more I do find that this was much more satisfying projection than I had the first time around all right cons of this fragrance guys I really don't have too many the only ones I could think of is is if you don't like fierce and you don't like Elysium you probably won't like this fragrance certain people just don't like those DNAs and this one isn't going to be for you certain fragrances they can kind of hide bad smells like if you have some really smelly body odor or you're just really dirty there's certain fragrances that you could just spray up on and it's really going to hide that odor very well this fragrance I don't really think so much this is something I would want to wear when I'm 100 clean and fresh and don't have any other odors going on of any kind it's just one of those ones where you want to wear 100 clean I think most people would agree that you're supposed to wear colognes when you're clean anyway you're not supposed to just wear them while you're dirty but there are some other fragrances that are better than that than this one I would just want to wear it while it was in a clean situation and then another con I was just saying that in my mix is that it does not project as long as the og fierce but again on this time around I was really pleasantly surprised and I really do enjoy this one and I stand by my initial statement that I made in my first video that if someone provided me just an unlimited supply of this and I had to only wear this I could totally do it I just really enjoy this fragrance something holds me back from saying this is the best fragrance ever something holds me back from saying this is better than fierce but it's almost like it's better but I am like I can't say that it is and I'm like on the fence but I just really really enjoy it and I think maybe if this had the projection you know this would be in talks of being better than fierce but I can never turn my back on my baby that is fierce I just absolutely love it it's just my favorite so far but this comes pretty damn close and I really really do enjoy fierce supernova such a great fragrance I've enjoyed it on my first reviewing and I enjoyed it now I could really wear this non-stop without a problem anyways guys we're on the road to 50,000 subscribers and I couldn't do without any of you guys help you guys are the best I'm having a great day out here hopefully I'm having a great day at home see you all in the next video peace


Hello, everyone. My name is Shinno Iwami. I would like to talk to you about methodology to grasp OSS as open source program offices. First of all, I have a notification about intellectual properties. Although I am working as an expert in NEC cooperation, I give this presentation in a private position because this presentation includes the analysis before I joined NEC cooperation. However, I have more than 20 years experience as an engineer and I have addressed data science for technology management in the latter half of my career during the current trend of evidence-based policymaking. Two, I believe that I can give you some hints about your open source program office. The figures in this slide were performed to overview the field of robotics and to discover the emerging technologies using databases of academic papers. In the upper figure, the analysis made a citation network and extracted field-search as mobile robot, manipulator, and medical robot. In the lower figure, the analysis discovered the field of perovskite solar cell as a rapid emerging technology. This presentation has four parts. The first section is introduction. In that section, I will talk about open source program offices and the scope and the framework of analysis in this presentation. The next three sections will introduce methodologies and results of analysis. Open source program office has several roles which are classified into outside, inside, and both. For outside, contributions to the OSS committee are required. What insights open source program office promotes strategy and utilization of OSS, fostering open source culture. Additionally, it has role to communicate between outside and inside and it maintains compliance related to OSS. The scope of this presentation is to provide methodologies for making a strategy. Especially, I use quantitative analysis that is called data science with network analysis, machine learning, and natural language processing. The merit of data science to obtain established evidence with automatic operations. However, data science requires a large number of data, so it takes time before the data has been stored. For discovering teeny sign, similar to intuition, qualitative analysis sit as interview and observation has an advantage. The merit of the qualitative analysis is the increase of manual work. Next, I explained the framework for analysis in this presentation. For making a strategy of OSS, we often want to grasp the latest situation of society, technology, OSS, competitors, and our belonging organization. I proposed the framework to seek about society, technology, competitors. This comes from the basic about basic and simple analysis, 3C analysis. The 3C analysis is to consider business from three positions, customer, competitors, and company. Regarding OSS, the concept of OSS community does not match with any of the three positions. OSS community is like a society, but OSS community is having participants from competitors and company. This presentation proposed the framework that OSS community is added to the conventional three positions. From this slide, I introduced three methodologies. The first analysis is about the strength of society and technology. In order to make a decision which field my organization works on, this analysis gives an overview of OSS and methodology. For this analysis, I explained data source methodology, results, and findings. In the methodology, I often use TFIDF and cosine similarity for other analysis. This slide shows the steps of this methodology. At the first step, videos are selected. In this analysis, at WWDC 2021, Google IO 2021, Microsoft build from 2018 to 2021, and SES 2021 are used. The reason of several years for Microsoft is that I can discover videos for several years. In addition, keynotes of SES include a keynote of Microsoft, so I expect to compare keynotes of the same company between different events. At the second step, the selected videos are played, and the text data of the sound are retrieved via a function of sound recognition. In this analysis, Google Docs was used. When you perform this analysis, you had better to use text data of transcription if you can get it. Because analysis will become more accurate. However, using a sound recognition has a merit that we can get text data from every video and sound format. Now, virtual events increase, and we can access more events easily. So the opportunity to get more text data are increasing. At the third step, knowns were extracted with natural language toolkit for English sentences. In Python programming, natural language toolkit is called NLTK shortly. When you extract words, Lentmizer is better than Stemmer for making words original. The step one, two, and three are performed for each keyword, keynotes. Then the steps four and five are performed once comparing between keynotes. At the fourth step, knowns are scored with TFIDF. At the fifth step, cosine similarity are computed, and the results are drawn as a feed map. Next, I explain about TFIDF and cosine similarity after extracting knowns. If you collaborate with data scientists, please remember only the expertise and the purpose. Data scientists must know, well, the tips to use these technologies, and you will just ask data scientists to do so. TFIDF computer score to identify feature words comparing between several documents. TFIDF gives us high score to what does appear frequently in one document. Meanwhile, TFIDF gives low score to words that appear frequently in all documents. For example, frequently words such as I, mine, me, the, of, are given a low score, and they are called stop words. For implementing TFIDF, the Python module scikit-lan is available. The scikit-lan has a setting that it can remove the stop words before completing TFIDF. Cosine similarity is a score of similarity between documents. On this slide, I write down sentence x and y, but we can apply similarity for documents thinking one document as one sentence. In addition to cosine similarity, there are several similarities such as Jackard index, Dice coefficient, Simpson coefficient, and others. From my experience, cosine similarity is one of the best similarities. The Python module scikit-lan also includes the function of cosine similarity. These are resolved by computing cosine similarity after computing TFIDF. These colorful figures are heat mapped that show the result of cosine similarity. Apart from this heat map, the analysis output, the list of known, ordered by scores of TFIDF. In the previous slide, I indicate cosine similarity that uses zero or one for each word. The left heat map of cosine similarity is computed with simple counting of frequency instead of zero or one in the previous slide. The right heat map of cosine similarity is computed with TFIDF instead of zero or one. On the heat map, let me high similarity and blue means low similarity. One of the list of features was by TFIDF. AI software development, cloud, mobile, and cybersecurity were constructed as frequent topics. As outstanding topics, drone, game, and sports were extracted from Verizon Kino. There is the world of Starbucks from Microsoft. And it means digitalization of conventional non-IT equipment. Investigating which words related to a line of low similarity on the heat map. We can know regular topics. The upper line indicates Microsoft 2020 that has unusual topics about COVID-19. The lower line indicates GM that has different topics from the other organizations. GM referred to electric vehicle and battery. Investigating which words related to a line of high similarity on the heat map. We can know organizations having similar topics. Microsoft spoke similar topics except the year 2020. Best Buy and Walmart spoke similar topics naturally because they are belonging to the same industry. In summary, this methodology can extract technological trends and changes of technological trends from keynotes of worldwide events. Again, the extracted topics for trends of society and technology in this analysis are AI software development, cloud, mobile, cybersecurity, drone, game, sports, digitalization, medical ID, electric vehicle. The second analysis are about the key groups of competitors. In order to make a decision whether my organization becomes a leader, follows a leader, or withdraw, this analysis gives a methodology to know competitors and their alliance by topics about OSS. For this analysis, I explained theoretical background methodology, data source, result, and findings. Before introducing the methodology, I explained the theoretical background. Regarding forms of sellers in a market, there are four forms in the economics, perfect competition, monopoly competition, oligopoly and monopoly. For monopoly, some countries prepare antitrust law. In the forms of oligopoly and monopolistic competition, I have heard that business tends to be stable in a class of technology management. For example, I consider that operation systems of computer are provided by two companies, Microsoft and Apple. Moreover, operation systems of mobile phone are provided by two countries, Apple and Google. As an ongoing example, strong seller of cloud computing as three, Amazon, Microsoft and Google. In addition to four forms, there is a form of excessive competition which is caused by too many sellers or selling too cheap. In this class, the market of Japanese home appliance around 2010s was said to be under excessive competition. There were mainly eight makers, Hitachi, Panasonic, Sony, Toshiba, Fujitsu, Mitsubishi, Sharp and NEC. Now, several companies withdraw the market and the other companies joined. So there are still too many companies. In short, for making a decision to join, collaborate or withdraw, we must grasp the situation of key players in the market. Next, I explain about the methodology. Data were retrieved from Microsoft Academic Knowledge. The first step is that data are retrieved via Academic Knowledge API and the formative data are inserted into the local database. The second step is that the number of paper, citations and co-authoring relations for each organization are counted. The third step is that the figures of bar graphs of each numbers and co-authoring networks are drawn by year. The fourth step is that the figures are organized in order to find insight. In the co-authoring networks, the top five organizations and their top four co-authoring organizations are drawn. Namely, co-authoring networks have at most 25 organizations in a figure. For this analysis, the main selected topic is blockchain as a topic deeply related to OSS. So, parent topics and child topics of blockchain or Microsoft Academic Knowledge are also analyzed. Additionally, open source software, open source hardware, open innovation, and open data were added to the analyzed topic. This slide shows the characteristic result. Please pay attention to the gray scale co-authoring figures on the lower half of this slide. Here, the co-authoring relation means a type of good partner. Because each figure shows several stages from decentralization to oligopoly. As a model case before the analysis about OSS related topics, the case of AI shows the growth from decentralization to oligopoly. As shown in figures of co-authoring networks on the blue arrow, applying the blockchain for this timeline, the growth and current status of blockchain are revealed. IBM and Alibaba Group became leaders about blockchain. In the same way, open source hardware, open source software, open innovation, and open data were plotted on the timeline about AI. These topics are too conceptual, so there is no sign of oligopoly. To observe the growth of partnership by co-authoring networks, one requirement is to be specific technology. Then, these figures show about all analyzed topics. From the horizontal axis, the growth to oligopoly is related to data size. That is number of records retrieved from Microsoft to academic knowledge. That is the number of results when you search a topic on the Microsoft academic knowledge. On the previous slide, one requirement to growth to oligopoly is to be specific technology. However, from the left figure to specific technology such as hyper-region, Ethereum and Bitcoin does not show growth to oligopoly. In this presentation, it is difficult to check the organization for all topics. The usage of this methodology will be to check the organization after identifying the state of your interesting technology. In summary, this methodology can extract from the error of decentralization to the error of oligopoly about specific technologies. The third analysis is about portfolio management by countries. In order to discover markets related to my own technology, this analysis gives a methodology to repair strengths and weaknesses. The analysis about security is indicated as an example. Thus, I introduce the analysis as a methodology. For this analysis, I explain data source methodology and result. Next, I explain about the methodology. Data were retrieved with the query of security from Web of Science provided by Clarivate Analytics. The first step is retrieving bibliographic data from database. The second step is that the citation network is composed. The third step is getting the monkey's mind component graph. The fourth step is clustering by a new method. The fifth step is automatic extraction of future words with TFIDF. Manual identification of topics and plotting positions by topics and companies. This figure and table show small fields in a large field of security. After clustering a citation network, one group is colored with one color. A line means an edge which indicates a relation. Topics were identified manually. Security includes other than security. The latest largest group colored by orange has a topic of IoT and cloud. The second largest group colored by light blue is about food science. The slightly far distance between orange and light blue imply that the relation between both topics are weak. However, based on the digitalization as agricultural IT, food science must have an opportunity to collaborate with cybersecurity. The third largest group colored by yellow is about energy, energy security, which is deeply related to cybersecurity. The clustering tool provides number of countries and average years when user events happen. Using these data, each topic can be plotted on the chart of portfolio management like this slide. Technologies on the positions of stars are emerging and earning money, and technologies on the position of cash cow, are derived technology and earning money. Technologies on the right half of the chart will be strengths. Results can be plotted like this slide. It is considered that the differences between strengths and weaknesses indicate opportunity of business. The United States is strong in food security, so the United States has an opportunity to export something about food security. Lucia is strong in security game, so Lucia has an opportunity to export something about security game. That Japan is strong in energy security, so Japan will have an opportunity to export something about energy security. This analysis was performed before I joined to OSS related team, but I think the methodology can be applied for OSS. In summary, this methodology can extract technological strengths and weaknesses and find market candidates. Lastly, in this presentation, one analysis about trends of society and technology and two analysis about competitors. In the future, I will continue to develop other analysis and present them to the extent avoiding conflict of interest. Thank you for your listening.