In 1987 Square released the first Final Fantasy game. It sold well and gave the series a reputation for excellent graphics, but playing it as a kid, I recall thinking that it was kinda boring. As I got older, I ended up really liking RPGs, but always in spite of their gameplay and not because of it. I really liked Baldur's Gate, but when I watched YouTube's series Retrospectives, it always puzzles me when people look back fondly on turn-based combat because it was tactical. In my experience, there's not that much tactics in a turn-based CRPG. They either rely almost entirely on stats or very simple rock-paper-scissors systems. I recently watched a video on the Dragon Age series and had argued that Dragon Age 1 had interesting tactical combat. My opinion is that the combat in Dragon Age has always been a tedious slog, which is the same feeling I get playing Persona. And while I appreciate that others may enjoy that kind of combat, I think it's now pretty obvious that the average player finds turn-based combat really boring. The last two Final Fantasy games were 15 and the first part of the Final Fantasy 7 remake. Both games took tiny baby steps toward an action combat system, but also tried to keep the tactics feel of the turn-based games allowing you to kinda control your party. Both of those games have enough spectacle to make the combat fun to look at, but both of them also try to do two things and as a result, do them both pretty badly. Neither of them is a good action game, but they're also not very good turn-based RPGs. By trying to thread the needle between the old and the new, they do neither system very well. A couple weeks ago, Final Fantasy 16 released with a ton of hype, with much of that hype centering on it being the first Final Fantasy game with a full-on action combat system. Well, the reviews are in, and it's sitting at like a 90 on Metacritic with people praising its world, story, and that new combat system. It's the kind of game that seems destined to be talked about as a game of the year contender. Well, I finished it last week, and I noticed my shock some of you, but I'm actually a little bit conflicted on the game. Final Fantasy 16 has a lot to love, and in its best moments it is one of the better games of the generation, but it's also a very inconsistent game that has a lot to criticize, as well as celebrate. So today, let's take a look at this new action-focused Final Fantasy game, and talk about all the things it does really well, and all the things it does not so well.  Final Fantasy 16 after the logo. Fantasy May Cry The general consensus seems to be that FF16's combat is a massive improvement over previous games. And while this is absolutely true for me as well, I have a suspicion that it's not true for more than a few people who are long-time fans of the series. For all of its history, Final Fantasy has been a game that was all about party control and turn-based combat. Although I've only played the games from the 80s and early 90s, probably the first four or five, as well as Final Fantasy 7 and 15, all of those games still revolve around controlling your party members in some way. And this system has always been more than just a mechanical system. One of the compelling parts of old CRPGs is that controlling your party in combat creates a connection to your companions. By making FF16 an action game where you do not control your companions beyond using your dog as a combo finisher, the developers risked the attachment you feel to your party members. The combat in Final Fantasy 15 was a pretty significant overhaul that had some action elements, but because the combat was very, very simple and you constantly cycle through your party's abilities, it managed to retain much of the classic CRPG companion feel. FF16 took a bold step by creating a straight-up action game.  It's very, very rare that you have more companions than that. And the times it happens are very brief. On top of this, you have zero control over any companion other than Torgul. And even with him, the combat's so fast-paced that the only time you use him is as the fifth hit in a standard combo or to occasionally have him heal you. I thought it's just important to touch on the fact that this is a tremendous departure from the series and probably has at least some long-time fans feeling kind of conflicted. So the game's now an action game. And a bunch of media talked about the fact that a former DMC developer helped with the system. And when you read reviews of the game, the combat system gets pretty universal praise. But I think all this praise fails to talk about a few issues I think the combat system has. At its very best, the combat in this game is really quite excellent and it's very rarely terrible. But it does have issues. So let's go over how it all works. The core of the combat in this new game is a four-hit sword combo with a magic finisher. When you add in the dog attack, this means you will press X, X, X, X, Y up on the d-pad. This would get extremely boring very quickly. But the combat in the game is greatly improved by the mobility you have as a player. Most fights are horde fights. So you'll cycle through that core combo and then use the special abilities you get and then like dash to another enemy. You start with a long-range gap closer that works with every different icon set by pressing X and square. Once you unlock three classes, the combat starts to feel like an action version of an MMO almost. And what I mean by that is you're basically running a rotation most of the time. By under the game, you're running through that regular combo, cycling through six different cooldown abilities and then repeating. At its best, that combat is really good. It feels great to move. You have a ton of mobility. There's at least some skill involved in actually properly pulling off the combos and balancing the cooldowns is very fun. But I find it interesting that they didn't implement one of the most important systems of a DMC game. There's no combat score.  I understand that implementing a combat score might make it feel a bit too much like DMC, but there is so much combat in this game and the encounters are extremely similar to a DMC game. You travel a corridor until an arena is locked down and then you defeat waves of enemies until you can move along. I honestly think having a scoring system that gave more or less XP depending on how stylish you were would have improved this game's combat quite a bit, but that's just a small nitpick. The actual serious issue with the combat has nothing to do with game feel, which is fantastic, or abilities which are great, or even challenge which is acceptable if not perhaps a bit too easy. The problem with Final Fantasy 16's combat is entirely down to enemy design and variety. Here's what I realized pretty early on in Final Fantasy 16. If an enemy with a yellow health bar is on screen, the combat is going to be fantastic. If there are no enemies with a yellow health bar on screen, the combat is going to be pretty boring. And it's important to note again that the core of the combat feels really good, so even normal fights against a bunch of regular mobs feel good, but you're so insanely strong and the enemies all stagger on every hit, so pretty soon you'll find that every single fight just plays out the same, like this. You'll lunge to an enemy and then just lock them down with the regular combo, jogging them up until they die, and then dash to the next one and repeat.  And wailing on them feels good, but for how long? Every single encounter with the normal enemies plays out exactly the same. And I find this really odd.  You've got these really awesome elemental skills, but the elements don't actually matter. No enemies have weaknesses to one element or another. No enemy needs to be killed with fire. No other one needs to be stopped with ice. As a result, the skills are entirely interchangeable. They're really just flair, and this is a major flaw in the combat. These red bar mob fights would be so much better if you had to take into account which element you were using against them. If when you were fighting kind of the robot enemies, you had to use electricity to finish them, or if you had a mix of robots and animals and had a cycle between electricity and fire, it would greatly improve the combat and really help with the boredom of these encounters. And that lack of crowd control is also another big problem. 95% of the game's enemies will just take turns and let you kill them one at a time. The other 5% of the time, you'll have one major enemy and one minion. I find it so hard to understand why they didn't code the AI to attack you in groups. I've got all of these amazing tools that lock down multiple enemies and knock them down, and instead I'm just slamming them to death one after another. Only a tiny handful of enemies feature ranged attacks, and those enemies also very rarely attack while you're already engaged with an enemy. Eventually, you do encounter enemies who heal or buff their buddies, but because of the way the AI works, this adds nothing. Right at the start, you just instantly nuke those dudes down while their friends watch. And that's a real shame. Because when you're fighting an enemy with a yellow health bar who cannot be spammed to death by stun locking them down, the combat proves to be really extremely well done. Most boss fights, with the exception of a couple that we'll talk about next, are when this game is at its best. It requires precision dodging, careful management of your cooldowns, effectively using a rotation, some strategy about which skills to use and what order to use them in. It's so great that I never once passed by the opportunity to fight a mini boss or boss. But by about 15 hours in, I was always riding past any enemies who are normal mobs because it's so boring and easy that it feels like a massive waste of time. And this is one of those situations where the normal ways that games do difficulty simply doesn't matter. You could triple the amount of damage that normal enemies do and it would not change a thing because the issue isn't damage values, but rather how the enemies are designed. If an enemy does 10 times the damage it normally did, but allows me to stagger them to death and no one else will attack me while I'm doing that, it doesn't matter what kind of damage values they have or how much health they have. Then there's the enemy variety itself. If you made a list, you'd probably find that FF16 has a pretty good variety of enemies, but that number is kind of misleading because in reality, there are practically speaking only a handful of enemies. There are the yellow bar enemies that require you to learn their movesets and use your skills, and then there are red bar enemies that you stun lock to death. Enemy movesets do not matter if enemies never get to use their moves. Red health bar enemies stagger on every single hit. They can all be juggled or grapple hooked back to you. So no matter what they look like, it doesn't matter. A knight with a sword or a flower that shoots poison or a lizard or an orc may all be different models, but they all behave exactly the same and you deal with them all exactly the same. Just these two things. The fact that all normal enemies stagger on every hit and the fact that no enemies will attack you at the same time makes the normal combat encounters really boring because they all play out exactly the same, which is a problem because they're seriously like 20 hours of killing these normal enemies.  It's such a massive issue that I find it very odd that the combat has been so praised. Yes, at its core, the combat feels good and against tougher enemies and bosses, it proves to be fantastic. But for the vast majority of the game, you're not fighting those tougher enemies or bosses. Whenever you're fighting the fodder enemies, which is most of the game, the design falls apart a little. For the first 10 hours, this doesn't matter because the game feels really good in your hands and looks and sounds so flashy. But eventually, the shallowness of the regular combat starts to wear thin. And I found myself thinking about how much better this game could have been with just a few small tweaks. Make some enemies only stagger after three hits. Make others only stagger after a magic burst. Make some enemies only stagger with ranged attacks. Make some enemies that attack in packs so you're forced to use your AoE attacks. There's simply not enough variety in enemy AI here. And as a result, you almost never actually need to use any of your skills unless an enemy with a stagger bar appears. A huge missed opportunity. The boss fights are where this game is at its best and almost all of them are between very good and great. Because bosses with yellow health bars can't be stun locked, you actually have to be strategic about which skills you use and when you use them. You have to learn their movesets. You'll need to actually heal. The game's still relatively easy to be sure. I think I died like twice in the entire game. But a game doesn't need to be hard to be engaging. I hardly ever die when I play Sekiro now and that game is still engaging. In fact, I'm playing it again now. The boss fights in Final Fantasy 16 have you paying attention to patterns, balancing cooldowns, keeping your combos timed correctly and managing your health through Torgul, potions, skills and limit breaks that allow you to get your health back. It's more than good enough to keep you interested even though you are very rarely going to die. The times when you transform into Ifrit and do the Kaiju battle boss sections are amazingly well animated and add another layer to the combat. Early in the game, these sections are basically glorified QTEs but eventually this opens up and becomes an entirely new combat system that looks and plays unbelievably well. The production value in these sections is so high and the encounters are so memorable that these parts alone would make the game worth playing. It's basically like playing a cutscene and for that alone the developers deserve a ton of credit. In fact, the boss fights are so good that there's only two in the entire game that annoyed me which is quite amazing because I'm annoyed very easily. Unfortunately, however, those two are the final two main bosses of the game. Odin and Ultima both are visually stunning fights and they both have very good Kaiju sections that are the best part of the fights. Sadly though, both fights suffer massively from A being a total visual clusterfuck of particle effects and animations so that I have no idea what is happening at any time. I mean seriously, look at this. How am I supposed to read what the hell is happening here?  It is impossible to actually see what's happening which means you're basically button mashing and just tanking damage because it's such a mess.  I can think of like three enemies in the history of games who teleport who I do not hate. The Lothric Princess in Dark Souls 3, Minotaurs from Destiny and that's actually probably it. So two, a classic example is the Taken Captains from Destiny 2. They're just horrendous awful trash. What am I supposed to do with this? What am I supposed to do with the teleporting arm monsters from Elden Ring? What am I supposed to do with these two bosses?  I hate it. Who thinks it's fun to hit something twice and then I have to spin my camera and chase after a coward? If it's still at the fight last longer then just give the boss health I guess? Give it more health? I don't understand. These bosses already have preposterous amounts of health as it is. These are like 15 minute fights man. I just hate it so much and by the end of the game I was literally saying to my wife like alright I'm good now.  Like why does it have to go on so long?   Overall the combat is simultaneously the best action combat the series has ever had but also flawed in a bunch of ways. Let's move on to the other part of the game that's excellent at times while also being massively flawed. Story and quests. Okay so as usual when I criticize something that everyone else thinks is the greatest thing in the world let me preface this by saying yes I agree Final Fantasy XVI story is quite good and as far as epic AAA video game stories go this one is good but the gap between bad and good is just as big as the gap between good and great. FF16 has a good story but it's also a story that could have been great and instead never quite reaches the heights it could have for a variety of reasons from plot to pacing to the core focus of the story. Before we go any further I want to point out that if you have read the song and Ice and Fire novels by George RR Martin there are a positively distracting amount of similarities here. I had seen all the press that the new story feels like Game of Thrones. I took that to mean it would be a story that focused heavily on politics. What I did not expect was that it would literally constantly reference the books in ways big and small. The main character is from a super honorable house that's destroyed through political machinations and his father dies tragically. He happens to have a dire wolf that was discovered as a pup when they were exploring the North. He himself is hated by his mother for no particular reason just like the books. There's a very clear Cersei character. They literally say things like little and less. It's not some terrible thing but if you've read the books as many times as I have it is distracting and weird. I just wanted to get that in there. I just find it kind of odd. Anyway, there's two main reasons FF16 story is only good and not great despite all the love it's getting. And the first of those two things is pacing. Think of the best narrative games and you will find yourself thinking of games with tight pacing. Mess Effect is a similar style RPG but it pushes the plot forward with almost every single mission. And during the side quests that don't directly push the plot forward the constant companion dialogue goes a long way to building character death. The Last of Us 1 and 2 are both paced almost exactly like films with story and plot beats perfectly spaced to build and release tension. FF16 has wonderful characters with a fantastic world and even a bunch of pretty good plot points but the overall story of the game is badly mauled by its terrible pacing which is also kind of related to its progression system which is bad. This is most apparent in the appallingly horrendously terrible side quests and the bigger issue with those quests isn't that they're all terrible. The real problem is that they are not all terrible. You see if the side quests were all total garbage then I could have quickly realized they were just chaff and ignored them all. But sadly they're only about 85% total garbage with 10% really good character stuff in world building and 5% important progression quests. Luckily the 5% progression is labeled with a different icon. If the icon is a plus simple you know you have to do that quest but the rest gives you no indication whether it's going to be a story list pointless fetch quest or a barely contextualized combat quest or a compelling story that is important to the world and characters. As a result I wasted hours doing total garbage quests that were a complete waste of time in every way. I wish I had actually stopped doing them early on because the first 10 are so terrible that I was about to stop bothering them at all. But early in the second act I made the mistake of doing one that was actually very good which meant I now could not ignore them. FF16 is already a very wrong game with a tremendous amount of backtracking across maps and until you unlock the chocobo like 10 hours in you have to run across those damn maps. It's annoying enough running across a map for a quest that's interesting but it is just awful to run across the map to find four flowers or get a note that a dog stole or go find a lost merchant or any number of the other garbage quests. FF16 has an extremely long main quest that's full of great characters and super important plot points. So it confuses me that there are all these garbage quests when they're just not needed.  There were two obvious solutions to this issue. First, you could just look at all the quests the writer's brought to your desk and if they are trash you can just say no. Like you could just not put those ones in the game. That's a thing that developers can do. You do not need to include every permutation of the pointless fetch quest that your designers bring you or because there are a ton of people who seem to actually prefer quantity over quality whatever man you do you you could just rank them and make it clear to the player which is which. You could have had that cross for progression quests yellow for actually good side quests and green for the pointless chaff. Lots of games do this.  That's a big issue that caused me a tremendous amount of agita. I cannot overstate how annoyed these side quests made me.  So when I'm playing a game I just really don't want my time wasted.   Why are games so intense on wasting my time? But the bigger issue with the game story is something much more important. The focus of the plot is on the entirely wrong thing. The most interesting aspect of this game is the political situation. The original plot that overthrows Clive's father Bahumut's father Clive's mother and her plots Titans plans in the Republic and all the wars and politics of this interesting world. The least interesting aspect of FF16's story is the big bad guy God Ultima whose motivations are literally a mystery into the last hour or so. The Ultima storyline is fine. It's not bad or anything but it's pretty generic fantasy action stuff. The game could have had an all-time great video game story like seriously an all-time classic if it had focused on its most interesting aspects. They built a complex and realistic world with interesting politics, social classes and dynamics and this is playing out the entire time but it's almost just background noise to the main story of Ultima's big bad guy plans. Ultima is the least interesting main character in the game. He's the least interesting one. So why is so much of the game about beating an enemy we do not understand? Who literally has inhuman motivations and seemingly no emotions? I love this story in spite of Ultima and certainly not because of him. It's so weird to have him be the whole thing when you've got this amazing song of ice and fire story going on in the back that they did really well. Then there's a host of weird little things that feel sloppy or hand-fisted. Why does Joshua hide from Clive for more than half the game? It makes no sense and it's like frustrating in a three's company sitcom style of way like what's going on? Why would he not just go to Clive and be like, hey, how you doing? Here I am. We're both super powerful monster people. Let's work together. I don't understand. At the end of the game, everyone is saying, oh man, how will we ever get up to Ultima? And I turned to my wife and said, three of these characters literally have wings and one of them is a big Godzilla who seems to fly in half the cutscenes. So why is the story pretending it is some insurmountable issue to get to the top of a mountain? They have wings. They literally have wings. And then other weird little plot things like why would the dominance literally not be the king of every single area? Like how would Clive, the original Clive, not have been the king? If dominance existed in our medieval times, they would not just be people leaving an army. But if dominance existed now, they will be the kings of the world. There's no doubt about it. That's a fact. Like I was pretty consistently arching my eyebrows at some totally implausible things like that. Like how the player character keeps meeting people who somehow survived everything blowing up. See, an all-time great story doesn't have you constantly pointing out plot holes and improbable conveniences. Final Fantasy XVI has a good story with extremely likable characters and incredible best in the world style animation and cutscenes, but it doesn't stick the landing on all the little things. And a bunch of sloppiness takes what could have been one of the best stories ever told in a video game and leaves it as a pretty good story that I started forgetting about three days after I finished the game. There's a big difference between good and great. The first two seasons of Game of Thrones were good. The books are great. Final Fantasy XVI is the first two seasons of Game of Thrones. Odds and Ends. Final Fantasy XVI has been held up as one of the best AAA games of the last few years. I even saw an article saying it's what AAA games should be. Hmm, I think that's kind of weird.  Why does this game get a pass for its awful frame rate issues?  Why do the plot holes not matter? Why does nobody talk about the many boring combat arenas? Why does nobody talk about progression, which is like literally totally pointless, man? You have this like crafting system and it's just so dumb. Why is there even a crafting and armor system? You just get given new armor like through the game. You don't have to do anything for it. You don't end up like having to work for it. You're not questing for it. You just go to a vendor every couple of levels and oh, there's a new sword for me to buy. Oh, and then I go over to this tab and upgrade it twice. It's like really ham-fisted and badly done. It shouldn't even be in there. Or better yet, it should be good. Why is there no talk about how many boring combat arenas against the same few enemies there are? Final Fantasy XVI is a very good game but almost every single system is just shy of great. Performance is bad, pacing is bad, side quests are terrible, progression is bad. Combat is amazing when you're fighting a boss, very good when you're fighting a yellow elite enemy mixed in with fodder grunts, and really tedious when you're just fighting grunts. Pacing is way too slow and the story is good but not great. I find it weird how some games are mercilessly pilloried for every little issue while others are broadly loved while having the exact same issues. Final Fantasy XVI is probably the most I've ever enjoyed one of these games even going back to the late 80s and early 90s but I still found myself wishing it was even better. It's a game I enjoyed very much but will almost certainly never play this thing again. I look forward to the next Final Fantasy game that this team makes because all the ingredients for an all-time nine out of 10 classic were here. The mix is just a little off. So instead of a nine to nine and a half out of 10, you left us like an eight and a half out of 10. It's well above average, one of the better games of the year but this is no all-time classic, man. It's good, not great. All right, 100% my next video is about how I have not played Destiny 2 since the day I finished the Lightfall campaign and why I think I might be done aside from the final shape and that will be it. And it has nothing to do with the story or the ever-versed store.  That's definitely coming up next. I've already started writing. All right, listen, if you stayed to the end, do me a big favor, click the like button and share and subscribe and all that kind of stuff. Thanks for coming. See you next time. Bye.


My name is Chuzoo. I'm from South Korea and living in Berlin currently. I'm founder of the studio Princess Computer. Princess Computer is a music video studio based in actually nowhere because we are working in home office and we are a very self-taught team. I'm the one who was learning all of the processes from Blender YouTube channel.  I didn't know very well about the philosophy behind the Blender Institute and here I could really feel what for you guys are working and the main sprit of the program so I'm very happy to keep these things home. I will be back. I will be 100% back and use Blender.


Okay, so we were just looking at the carbon-free energy requirements and the red curve tells us how much carbon-free energy we would need to produce given the assumptions that we've made here and given the scenarios shown for both carbon emissions and CO2 concentrations. We can also see how our assumptions regarding each of these terms compare to historical numbers. Population growth in the past versus our projected population growth under this assumption that we've made of stabilizing global population at 11 billion later this century or early next century. The assumption of GDP per person relative economic expansion of 1.6 percent per year, these are the historical values, the pluses, and the curve shows what we're projecting for the future. Energy intensity, our curve sort of follows the past couple decades of data as far as energy intensity is concerned. So we might imagine that there have been some developments in technology that have led to a trend in recent decades that may be more representative of the trend we would expect in the future than say the numbers from the early 20th century. So our projected energy intensity over time sort of matches the past few decades of energy intensity information. Finally, carbon efficiency. This is the decline we're projecting as we become more carbon efficient in our energy usage fewer gigatons of carbon produce per terawatt. We are extrapolating the past trend and we project increasing carbon efficiency, increasingly less reliance on carbon based energy as time goes on. So these are the underlying assumptions in our default projection here. And as we've seen in that default projection which corresponds to business as usual, we're going to be upwards of 700 parts per million by the end of the century. If we're looking to stabilize CO2 concentrations at say 550 parts per million down in here, the purple curve, then clearly we are going to need to change our behavior. We're going to need to change these various terms, some combination of these terms in such a way that we lower CO2 increase and accordingly as the purple curve shows, we would need to produce less carbon free energy in that stabilization scenario in the 550 parts per million stabilization scenario. So we can play around with these numbers and try to figure out how we would actually go about achieving a particular stabilization, how, what terms we might be able to change through a technology and through future policy changes and how those changes would translate to a CO2 emission scenario and our ability to stabilize CO2 concentrations at some particular level. So the next thing we'll do is to play around a little bit with these numbers and see if we might be able to lower our projected CO2 increase from the current trajectory, the business as usual trajectory that has us at 700 parts per million well over twice pre-industrial by the end of the century.


Let's go, extracting the signal from the noise. It's theCUBE, covering VMworld 2015. Brought to you by VMware and its ecosystem sponsors. And now your host, Dave Vellante. Welcome back to San Francisco, everybody. This is theCUBE's SiliconANGLE Wikibon's continuous coverage of VMworld 2015. We're in Moscone North this year at the street level, stop by and see us. Good friend Tom Cook is here. He's the CEO of Permabit. Long time, CUBE guest, Tom, it's great to see you again. Always a pleasure. Great to be back, David, thank you. So yeah, this is kind of the Super Bowl of the story's become the storage show, hasn't it? It really has, it really has. You remember the old Comdex days, we'd all get together. Well, it's really VMworld is now that main sort of ecosystem storage show.  It said, okay, the ecosystem's going to rally together. We got all these APIs coming out and the problems of storage are going to go away. Just the opposite has happened. What's going on in storage? Well, it certainly has. When you think about the efficiencies that have been brought to compute by this company, by, and what's happening in the ecosystem, it's really created stress in the storage marketplace and has actually created more storage. And so we're going about the task of solving that, really helping to bring the same efficiency to the storage side that VMware brought to the compute side. And so I think it's a really exciting time. So, Permabit, data reduction specialist, you OEM your technology to a number of high profile players. I'll let you say them because I'm not sure which ones I can publicly talk about but we know you have a relationship with IBM. We know you have a relationship with Atachi and many others. And so you are very efficient, very fast.  So give us an update on your business. Yeah, so you're right. We're working with a lot of the leaders today. So NetApp, HDS, IBM, EMC, NEC, there's a good list of companies that we're working with and having a great time in the marketplace with. And the IBM people are here in force today because it's really a major market opportunity for them to continue expanding their footprint in virtualized world. What we're really seeing flashes become really significant in the marketplace, of course. But when you really look at what's going on in the marketplace, it's products that have data reduction that are winning and leading the marketplace, whether they're hybrid or whether they're Flash designs and almost everything's a hybrid today. And so what's really happened is we've seen this shift where performance today is very available but capacity is expensive in the Flash world. Well, our technology helps to bring that in line. So the same efficiencies that VMware brought to compute we're bringing to storage and we think that's a really important thing to be doing. So it's interesting, when you guys first came out, the Flash was kind of in this niche and it was expensive. You kind of were fighting headwinds to get people running spinning disks to use your technology. Flash would become a huge tailwind for you. Adoption went through the roof. And I want to, it's kind of a nuanced question but the market now seems to be segmenting between what David Florey calls latency storage and capacity storage and fast and slow. Is the attitude toward data reduction on spinning disk changing or do you think that's going to sort of continue and really your sweet spot is going to be Flash? Well, so I certainly think everything is relative. When you can bring savings even to spinning disk it's very important. But the difference between the two medias is something worth looking at. It's kind of nuanced as you said. So spinning disk was very inexpensive for capacity, very expensive for performance. Now we've got Flash that's very inexpensive for performance, very expensive for capacity. So capacity optimization becomes the important technology to bring that to market to really expand the footprint of Flash. And what we're seeing is who wants latency? Flash can inherit many, many more environments when you bring down the effective cost of the capacity and that's what we're doing and others are doing in the marketplace. I think it's become the real trend. And you know, we've studied this and the floor has done some tremendous work here and just putting out the cost curves and everybody can do the cost curves but he's nailed it very well. And at the end of the day it's driven by economics. It's all economics. And so you're seeing is a massive uptake in Flash. I mean, you mentioned a number of the customers, IBM's coming on later, Mike Hewn, whom you guys work with. You guys came on it, was it either interconnect or edge, I can't remember now, but that is really an interesting marketplace. You mentioned EMC and Atachi. So pretty much everybody needs a data reduction strategy. I mean, 100% of the players out there need a data reduction strategy and your approach is pure OEM, correct? It's 100% OEM. So we're working with partners. Our thesis was this. There's enough a raise in the world. What we really need to do is help companies to make their raise better and to make them less expensive on an effective cost basis. And that's what we bring to the marketplace and that's why a lot of companies have come to us. So the OEM business model is challenging from this standpoint. It's a very long sales cycle. So you got to get qualified. You go through, you know, the not invented here.  And that sort of took a long time really to get people to adopt the SDK. Now, once you get them to adopt, you usually get long term relationships with them, but you're always kind of sleeping with one eye open. So how are you sort of as a CEO thinking about diversifying your business? You know, what's on the product roadmap? Maybe you could talk about that a little bit. So first of all, we've made our technology easier to consume all the time. So we started with an SDK, then we moved to a full-fledged application we call VDO and then an appliance with deduplication compression and thin provisioning in it. So we just made it really simple for our partners to consume that and bring real efficiency to their products very quickly. So a lot of companies have started out with us with Sandblocks, our appliance, and now they're moving to embedded models on their arrays so that we'll be embedded, our software will be embedded with them down the road. And what that does is makes the customer experience completely seamless. And we really are just a supplier to the marketplace the same way many other companies have been successful models in the past, LSI, others like that. We're doing it with all software. Yeah, well, what's interesting, right, I was just going to say, what's interesting is what you're doing is software-based. I think that's the advancement of the marketplace too. And Sandblocks kind of breeze life into existing infrastructure. So it's a return on asset play and then you've got the innovation going with your other software products. Let's talk about the industry. You and I, we'd love to sort of pontificate on what's going on. You kind of have, let me sort of set it up. You've got the cartel, which a lot of the cartel buys from you. And they're sort of still controlling the chessboard, but now you're seeing all these startups popping up all over the place, particularly this week. What do you make of the, and I want to add, you've got VMware now, getting more and more into storage, EMC little, giving them a little bit more rope to do things like VSAN. What do you see, sorry, one more thing. Then you've got all this sort of addition of open source, open stack. You've got Amazon now as the $8 billion gorilla growing at 80%. All kinds of really interesting waves going on. What's your take on the state of the industry? So I think there's kind of a classic squeeze going on in the industry. We've seen some upstarts come into the marketplace and these are companies, and they're no longer upstarts. They're becoming pretty well-skilled companies, companies like Pure, like Nimble, and others like that. They've come in with a much better value proposition than incumbent arrays had. And the reason they're doing that is because they really went for more flash and they brought data reduction into their products immediately. So the incumbent products were left in the old capacity world, the old spinning disk world, and 80% of their investment over the years had gone into trying to get more performance out of spinning disk. All of a sudden, that investment's worthless in a flash world. And so we've seen a shift there and the upstarts have come in and done a really good job. In fact, better than I would have thought they'd do because they'd been able to scale and had plentiful capital. On the other side of the marketplace, we've got the Amazon, Google, Microsoft, IBM, and a few other companies. I think that's going to be hugely dominated by very large, very capable companies, ones which pure storage players can't buy. And so all of a sudden, there's this new force in the marketplace that's taking a significant amount of market share. I think the storage players end up fighting for that on-premise, high-performance storage, and that's why we're seeing a lot of play in that market. Did you have an opportunity to sort of squint through the Pure S1? What are your thoughts? Yeah, well, I mean... Were there any surprises in there? I was astounded by the amount of money that's being spent. The growth is spectacular. The execution's been spectacular, but spending $2 for every one can only be sustained for so long. And we'll see where that goes in the marketplace. Overall, I think as a startup company, they are solving yesterday's problem rather than tomorrow's problem. And that's where Google, Microsoft, and other people, Amazon, are living and where they're going today.  Infinidat is starting to have a bigger presence, Moshe's company. So you're seeing others, Papa Brian Biles' company's coming out, or has come out. So it's unclear where those guys fit. What I wanted to ask you is, if you look back when we first started doing these events, like 2010 with theCUBE, and you saw a wave of acquisitions, three-par, compelling, of course, Equalogic went, Isilon then went, Data Domain gets acquired as a public company, as did three-par. Nice exits, two, two and a half billion, couple billion here, 1.8, one, nice, really nice exits. And I've always wondered, well, the Flash guys achieve that same success. We saw Violin spike up and clearly spike down. We saw Fusion hit a billion plus, and sort of get sold. What are your thoughts? Do you think that these new Flash companies will be able to achieve escape velocity? We've had this sort of conversation before, but I like to have it every now and then you've got great perspectives on it. Well, so, first of all, I don't want to rain on that parade because they're doing really well, and big capitalizations, I hope to see investors make a ton of money. But overall, I think they're fighting for a market that's contracting. And, you know, somebody taught me along the way, you know, like my dad said, something about tailwind and things like that, find a place where there's tailwind and it's a lot easier. They're fighting for a market that's shrinking, if you will, the on-premise storage marketplace, and we're going to see this cloud marketplace really grow. I believe Amazon, Google, and others are really coming at that hard. IBM's clearly equipping their business model to address that. VMware is definitely equipping their business model to compete and to be a major supplier in that world. And I think there's some large companies that are going to inherit that space. Well, and I think, you know, a lot of your customers have pretty good product portfolios in the Flash space. It's not like, you know, HP had such a huge hole as it was sort of living off of EVA install base, you know, and Dell had to get into the business if it didn't want to be reliant on EMC, so it just opened up some M&A opportunities for these companies that had achieved critical mass. But you saw EMC and certainly IBM get ahead of the curve this time and pay up for, you know, much smaller companies, but it seems to be paying off.  They've made mistakes and they've had a lot of successes and they really, you know, they have cash. And so it seems to me that the big guys, the cartel, unless I'm missing it, you know, have pretty good strong product portfolios that are going to allow them to lead in this space. Well, so I would really contend that Flash as it comes, because everyone can get Flash, so now we can get performance that makes building an array an easier thing. And so that's not a differentiation point anymore. And I think the innovation that's been introduced is a sustaining innovation rather than a disruptive one in the end. And the big companies tend to do very, very well over a period of time in sustaining innovation marketplaces. And this is a sustaining innovation in my mind. Let's talk about Permabit and your objectives as a company. I mean, you have patient investors. They've really given you the opportunity to find, you know, to hang on until you can thrive. And that's really what's happening. You know, survive until you thrive, as they say. And so where are you now as a company? What are the objectives?  Do you want to do IPO? Do you want to get bought? What's the long-term goal? So great question. So this has definitely taken a couple more years than we thought it was going to take. We thought the market was tipping two or three years ago and it's taken a little longer. But we think the power of the marketplace is even stronger today than we had anticipated. And what we bring to the marketplace, extremely low latency, very high performance, very low footprint, massive scale. Those are things that are not going away in terms of needs to the marketplace. So we're just continuing to really pour into R&D and make sure that we're staying way ahead of the marketplace. And by the way, among our customers, they've had projects in the past that they've either curtailed or canceled and they've come back to us and said, hey, we want to use your technology. So this is a thorny area to develop in. And I think people, it's easy to prototype. It's very hard to make it scalable and make it fast and do all those things. So what we want to do, we want to make our partners successful. They sell more, we sell more. We're OEM partners to them. It's kind of a preclic basis. And so if we can help them to grow their market share, if we can help them to expand their footprint, we win. And so our real goal is to build their businesses, help them to build their businesses. And we get deep channel involvement with IBM right now, very engaged in that, helping that grow. And if they're successful in that, then we're very successful as a company too. Well, the marginal economics of your business should be better than a company that makes a box and puts, let's say, compression or DGIP just inside that box. I mean, your volume, in theory, should be significantly higher and give you that flywheel effect. That's right. Are you starting to see that? Yeah, and we really like being attached to multi-thousand unit array products. And so those are the targets for us as a company because we think the companies that are selling that number can do really well. The other thing that we can bring is a completely cross-platform and complete portfolio solution to them. So if a customer says, hey, I really want this same thing throughout my portfolio, we can deliver it to them. Any platform, any OS, really any architecture. So at this point, TAM expansion is not keeping you up at night. You feel like the TAM is plenty big. You got to keep focusing, executing, innovating, making the products easier to consume, as opposed to saying, oh boy, we're running out of space. We need to go spin up some other skunkworks. So what you can see publicly in our company today is that we're working with the major incumbent vendors and helping them to gain market share and maintain market share and maintain margin. But in the background, I can tell you we're working deeply with the ODMs who will be supplying into the cloud market space and we're working directly with the hyperscale cloud companies as well. So we see the next world occurring and we'll be part of the software-defined storage revolution that's going on. We think we can ride these waves. So big cloud, mega cloud is an opportunity for you guys. Absolutely. To OEM your technology. Absolutely. Those guys, you're finding everyone's different. But on balance, there's opportunity for them to OEM your product because they'd rather buy it than build it. It's not necessarily a core competency and the ones that are interested. Is that right? Well, they've found it to be more challenging than they might have thought. And this is a pretty crowded IP space and we've got some pretty formative IP in it as well. And so that's something we bring to those partners. What's the patent portfolio looking like these days? We're 45 patents today and growing. So we think it's an important area and certainly some other big companies have been active in that space. And so the hyperscale companies are pretty sensitive. So what are investors saying to you?  You have your quarterly meetings. Tom, hey, you're at site to get in some traction. Yes, it took longer, but we stuck with you. Good job. What else can you share with us? They're really psyched about our partnerships and what they're doing with me is giving me a lot of latitude to work aggressively with our partners. Again, with IBM, they have a channel strategy that they go to market with. We've been able to match that by staff additions in our company and working with companies like Aero in distribution and their key vars. It's a real leverage point for us. So my board has been really supportive and what we're hearing from other investors, private equity investors, is they want to participate in this marketplace. They get a little scared with the valuations in arrays and things like that, but they think we're a really interesting place. So we've had a lot of demand in that area. So my last question is impressions of, you know, VMworld and generally, but VMware specifically. With their VMware is kind of theme.  You've got OpenStack or Amazon, all the things we talked about before. What are your impressions of VMware and where it goes? I think VMware is a very well-positioned company and I think it's going to be real interesting to see how this all plays out in the marketplace, whether they're part of, whether they're outside. But certainly, this is a company.  The traction of the product is spectacular in the software-defined world and the capabilities that could be in there with our product phenomenal in terms of value proposition. What would you do if you were on EMC's board? What would you recommend?  What would you always be? Oh, wow. I'm definitely going to create some enemy in this kind of stuff. First of all, there's- Hey, listen, Dave, I'm sure they're having these conversations, right? I mean, there's no easy answer there. Yeah. So let me say, I think that VMware is a great company and deserves a day to shine. And to me, that's part of what this is all about and the way that they partner throughout the entire storage community has been really valuable and I think we'll continue to see more of that. Yeah, well, when your stock price goes sideways for as long as it has and your cash flow is diminishing, investors can get you in a headlock and they can get an extra five or 10 points. They're going to listen. So that's leverage for the investor community. But at the same time, you like to see an opportunity for a large company to continue to thrive. So it's going to be really interesting to see how it plays out. Tom Cook, great to see you as always. Thank you, David. It's really a pleasure. All right, keep it right there, everybody. We'll be back. This is theCUBE, we're live from Moscone right after this word.


Oren, how are you? Great to see you. Good, how are you? Thank you for having me. Very well. Welcome. Thank you. If you're all set, the stage is yours, Oren. Take it away. Excellent. Thanks. Can you see my screen? Cool. Thank you. All right. So nice meeting you all. As an introduction, the role of product manager, whether held by a product manager or by an engineer, any other builder that is acting as a product manager is evolving once AI and machine learning is embedded into the product. And this is a topic of the conversation we're going to, I'm going to talk with you today. I'm going to start with an overview of what is ML, not as this is not an educational session. And what is ML is just to make sure that we're all on the same field as far as terminology. Then I'm going to dive into deeper into the use case methodology of finding the use case for machine learning and AI ML using the PR FAQ of Amazon, the methodology of Amazon for working backwards. We're then going to look at how do you qualify for use cases, a good fit for machine learning or not? We're going to talk a little bit about the POC builder. How do you decide which element do you want to de-risk in the POC and can you do machine learning in the lean methodology approach? People used to think that not today, actually with AutoML and pre-trained model, this is doable. We're going to talk about the team. How do PMs and data scientists collaborate in an empowered team structure along the different journey stages of the ML journey of the company? Whether you start with the first use case all the way through an ML platform that can support an abundance of machine learning models, some in training, some in deployment and so. Data is a very big piece and we've heard a lot about that today. It is a very big piece of any machine learning process and product managers play a key role in us retaining data quality and preparing the data and labeling the data for any machine learning model. The important thing to remember is that a model is not a product. Taking a model into a product requires earning the user's trust. There are multiple mechanisms and tools we can do in order to, you can apply in order to do that. Some of them are called human in the loop. Some of them are called explainable AI and we'll talk about them. And we'll talk about the ML file in the feedback loop. How do we take the data into a model, how do we change the behavior? And then the model will drift because we are successfully changed the behavior. So we maybe need to identify new data, relabel it, retrain it and so on. And finally I'll talk a little bit about how AWS can help you guys with a channel, with a journey of ML challenge of ML journey, whether you're starting it or an advanced stage so that. So let's dive in. What is ML in a high level terminology, just so we can all kind of know that we're speaking about the same thing today. There are three terminologies that are often used interchangeably. So artificial intelligence is mimicking human behavior given in rule, rule engine based method. Machine learning is where we let the machine learn from the data by itself. Deep learning is a subset of that when we have hidden layers. So we have an input and output layer and everything that's happening in the middle is hidden from the human. Though today with explainable AI we can probably interpret and understand a lot of the reasoning behind the machine learning. Another way to look at the classification of different machine learning types is supervised learning where we have a label where we tell the machine what we want it to learn. So we say here's an image of a cat, here's an image of a dog or a million of each. And then for the two million one, we want the machine to identify and classify binary classification by itself. Or unsupervised learning where we do not tell the machine what it is that we want it to detect. We just put the data inside the machine and we expect the machine to identify patterns in the data. Whether it's anomaly or clustering that are attributes of the data that can help say this group is more similar than that group. But what is the meaning of that group is something that we will require an SME, a subject matter expert to later look at the cluster and maybe change them a little bit, name them and give it a meaning in a context. So supervised learning is a place you want to start. It's easier to start but it requires labeling. Unsupervised learning can support in a way. We have a lot of data, dimensionality is in the data and we want to reduce the dimensionality is using unsupervised learning and then feed those into a supervised learning model. Reinforcement learning is where we have a lot of iterative games or opportunity for the machine to learn and design a policy based on a reward. So we tell the machine, hey, this is good. This is bad. Now you can learn from that. The recommendation engine, for example, might learn from how users interact and we can decide as product managers, not as data scientists, product managers can decide whether we want to exploit what we learned about the user so far and give them recommendations based on what we know about them or whether we want to explore new stuff that they may be interested in. So this balance of exploration versus exploitation, I think it's more of a product manager decision than a data science decision. And we'll talk about those decisions and what is your role in those decisions going forward. Machine learning is probably the most disruptive technology for product managers is that traditionally we would look at traditional programming in the following manner. You have data. We have a program that the computer scientists and the engineers are building in order to get the outputs that we want. This has usually high price, a lot of effort and we as product managers often have a desire to do a lot of features, new products, many ideas, but then we hit the wall of, hey, this costs a lot to implement, the software is very complicated, the software is very complicated and so on. In the machine learning world, we actually work the reverse. We say here's the data, here's the output that we want the machine to do for us and the machine takes care of designing its own program. This makes it a lot faster and easier for product managers to actually test, iterate and develop new products provided that you have an ML platform to support this fast iteration of data scientists and engineers ideating together and trying out new stuff. You still need the data, you're still into label the data, but the process of not having to now heuristically code everything actually in the long run makes it easier to design, develop and roll out new products and new features. So machine learning has been around for a while. Why is it now so successful? Something out working here. Can you guys still hear me? We can hear you, Aaron. Okay. From whatever reason my deck... Your presentation is stuck. I guess you can see that as well. Okay. Let me just start that over. I apologize for this. I'm glitched with the computer. So machine learning has been around for a while and the reason we're now seeing machine learning taking up so fast is for three main reasons. Compute is now available for everybody basically. Data has been accumulated in a fast growing manner and research showing that machines they can see better than people in computer vision can almost analyze language better as people in NLP natural language processing and can detect patterns in data probably better than people. All of this is available in the cloud today. So what does that mean for you, the product managers? What is your role and how does that change in machine learning powered product management process? First you can now productize and feature complex situations. Situations are first originally too complicated to condition. When data scales fast you don't have to redesign your heuristics. Machine learning can support that. When the value is a personal hyper personalized path or journey of a user rather than the segments. Let's say instead of looking at an age group or an age cohort, you want to look at each user individually along their digital journey for example. When data changes in an unpredictable manner in real time, machine learning can cope with that much better than traditional heuristics. If you have a lot of data and you're just looking to generate some insights from this data. So you start with visualizing that if you want to go deeper and identify stuff that visualization cannot detect easily. When you want to automate stuff like computer vision like BPM processes or NLP chatbots and so on. And traditionally people used to say that hey we need millions of images or data points in order to train a model and that doesn't really support the lean approach. Well actually today that's not true. In the last couple of years we've seen a significant uptake of auto ML capabilities we now call this auto pilot and AI services. These are pre-trained pre-built models that can be tweaked to a specific use case or specific data set of any specific domain or vertical. But can actually get you started with a machine learning model in a matter of hours or few days. With a poc that has machine learning embedded inside it might be not the best or optimized model for scaling your own use cases. You can like I said tweak it later it's going forward but you can really get fast into a poc with pre-trained and pre-built model. And today you can do that in auto pilot with an open box machine learning. So it's not a close box. You're actually if you have data scientists that just want to save their ramp up time they can start with auto ML and then get on in the background all the ETLs that were designed by the machine and all the algorithm that it checked selected and so on and then you can take it and continue to improve on it going forward. So what is the role of the PM in a machine learning process look like? First we have the problem use case KPI definition. We then want to derisk something with through a poc. The modeling which usually includes data selection and labeling. Feature engineering retakes the raw data into features meaning the longitude latitude can be a raw data but saying that this person in that location is near a ballgame or near the high road or a shopping mall that is feature engineering that is taking two features of location and kind of organizing them in a way that you can see whether it's 50 meters apart from each other. Hyper parameter optimization stands for all the elements that you can control inside the black box meaning that you can control how many hidden layers are in there. You can control whether the data moves in one direction or a circular direction. If you're looking at an example of recommendation of products which Amazon has been developing for the past 20 years, you can look at what is the data that we want to look at historically and when do we want to sample or give weights to the data. So if you're now wanting to look at data pre-COVID, maybe it's not relevant for this specific use case so you want to kind of tone down the weights of the older data or maybe as we're coming out into vaccination or a vaccinated world maybe the data from COVID is actually the one that you want to weight down into the model itself. So these are decisions that are not science decisions these are product decisions that are derived from the data that we put into the model and impact how the data scientists will tweak the hyper parameters of each of the model. Metric selection and optimization again this is where you decide what do you want to optimize the model for let's stay with an example of recommendation engine. Is your workflow is the app the way you embed the model into a product built so that the user sees one button he needs to see the next action and then click next and approve it in which case you have to have one recommendation that needs to be either good or bad or does your flow of user journey is such that you have three recommended items and anyone that he can choose from is good in which case you're just looking to have one out of three further three out of five so all those inputs into deciding what are the metrics we want to optimize for like you see is a product decision not a science decision the model is not a product in between there is earning user trust we'll talk about that we'll talk about human in a loop we'll talk about explainable AI measuring feedback in changing behavior of course that beginning and end of this process are led by the product manager you're the decision maker and what is the problem use case and so on in your decision maker and analytics and feedback and what is the product behavior in the market but you're also a shared decision in productization POC and even in the model process not unlike traditional engineering sometimes that we write the PRD MRD and then we hand it over to engineering to build and we wait for the outcome in the Q&A phase we then we just help to productize and launch it actually here you as a product manager need to be involved in the data selection in the labeling this is just as much a product decision that it is a science decision I gave some examples on the HBO and future engineering and the metrics and optimization selection so now for the next 20 25 minutes I'm going to talk about each of these shavrons and dive a little deeper and give you some insights and hopefully you can start to ideate and lead these processes internally in the next 20 minutes I'm going to talk a little bit more about how to design a product for our companies at Amazon we have what we call the working backwards mechanism when we design a new product or feature we first write the press release with the customer quote inside and the FAQ the external FAQ what our customers are going to ask and the internal FAQ is what our stakeholders are going to ask about this product only then we go to design the visuals and then we go to the business owner could be you the product manager but could be the head of marketing head of sales ahead of operations and so on someone who knows where the data resides and how to access the data and someone who's going to build it if anyone asked them or kind of force them into a process in a workshop manner that answers the following three questions if you knew X you could do Y in order to get said so it's not enough to say I really want to do this. You can drive from that. Well maybe if you're a publicly traded company you could use that in order to report to regulators okay what does that ROI look like for this I mean how much do your analysts today spend time or money to design these forecasting models and does that worth a replacement so keep really asking all these questions sometimes we tend to really fall in love with machine learning because it's a healthy and actionable thing that we can do with it and even if we do we don't don't always have the KPI in place remember that the process of machine learning is can be costly because it's an iterative process and because data scientists are relatively scarce and expensive resource should make sure you are choosing the use case that has a strong ROI here are some examples of how this looks in customers that we've helped within the past which prospects we can get the right message to the right person at the right time or which customers in the last example here will churn so we can proactively mitigate them and reduce our churn there are more examples of course this is an endless number but this is just kind of generic bi type of examples and here are some examples from real-life customer examples from us business tools like in the top left corner here forecasting churn prediction support routing and so on consumer application sentiment analysis in the bottom left top right cybersecurity anomaly detection on logs and other examples or bi examples you know making your analyst use auto ML under the hood in order to generate insights about the business that leverage machine learning without necessarily having to have a data scientist in the team of the bi analyst team once you have a use case try and be critical in the thinking process whether it's an ML or you're just falling in love with idea of doing ML here because sometimes you can say well if you can really solve this in 10 lines of if then you don't have to ML this if the data is gradually scaling you don't have to ML it we talked about the hyper personalization example if the data is unpredictably changing in real time like user journeys in an online manner that's good for machine learning usually if it's confidence tolerant if it's life or death decision well you need some time to make a decision you want to know why the model recommended the surgery versus not but it's not it's easier usually when there's some tolerance for whether false positive or false negatives as an explain the next few slides and whether we have access to the data and this is a critical element typical use cases for product managers to dive into our either computer vision language or the entire realm of patterns in data whether that's for ranking forecasting, personalization, anomaly detection and so on so designing a POC I like the Marty Kagan approach and inspired that you want to de-risk one of these four elements will customers want to use it or willing to pay for it will they understand how to use it can we build is it feasible from a data science perspective and is it viable should we build is it's profitable for us ethically right for us to build this product now you want to time box the POC for four to six weeks you want to make the decision which assumption together with the data scientists and share your insights with the rest of the stakeholders the company it's not always obvious that the POC needs to be machine learning you could be a wizard of Oz sort of pilot when you have a manual person actually imitating the machine that you'll build later just so you can check whether it's valuable enough to your for your users today WS we also have this site where you can actually use pre-trained models you don't have to go through able as console you don't have to use the engineering team this product manager can go into this link here and start to test some of our language and vision services this is an example of one of our customer that use recognitions is a computer vision service that we have a pre-trained one to match people with the art that they like and they were able to get up a prototyping in four hours in production within a week based on machine learning that they didn't have before this process as you grow you will see that the product manager you may first get some help from a data scientist that is a broad company resource or maybe even one of our data scientists can come in and help you in a prototyping on ML solutions lab engagement or a pro-serve and so on but as you grow you'll see that you want to create an empowered team of product manager working alongside a data scientist and these people would want to focus on iterating the ideation and the use cases and the math of things not the IT and the DevOps that supports around it you don't want them to spend a lot of time on getting the right machines up and running or sending multiple GPUs to be tested just for distributed training and so on all of that stuff you want to put in an ML platform so you start to seeing platform engineers and as you grow more and more you have these core teams, multiple empowered teams with a lot of support around it now the modeling side of the work like I said it's not just for the data scientist or engineer this is stuff that product managers can, should and need to actually own and be a shared decision maker especially around data is the data available internally or is it public? Do we have the permission, the IP to use it? GDPR HIP I fear in the health care space privacy is the data fresh we talked about the example of using data pre-COVID or pre-vaccination sparsity of data if we're in a time series use case and we're trying to forecast but we have a lot of missing data in historic data that might not be a good support for our forecasting model very often in anomaly detection we have very few labels because it's anomaly right doesn't happen a lot so we may sometimes want to double index on those few anomalies kind of over sample them so we can really train a model properly but we need to remember data will confess to everything if we force it enough and the way to overcome this was called overfitting is not just to train on everything that we have usually train on 70-80% of the data we keep a hold outset 10-15 20-30% of the data kept free from the training and then we can test validate that the model we train can generalize and that 15-20% we kept on the side is a good place to validate as a general example and then also in deployment we usually don't deploy machine learning model from 0-100 immediately we do some canary testing A-B testing and we deployed gradually in order to make sure that the model we trained on certain data behaves the same in real-world data but very often the problem is that we have missing data and within that missing labels is the most common scenario because we need to manually label manually tell the machine this is a cat this is a dog sometimes we need a professional to do this manually we at Amazon have a tool called SageMaker Ground Truth which helps you manage the workforce of the labelers themselves for example if you want to send a certain labeling job to three different people and make sure that it's only consensus vote that this is a label or majority vote but then these people might be expensive people that are really experts in their field and you have a lot of labels to generate so we have what's called active learning while they start to manually label the data we actually build the machine learning model under the hood that starts to use the data that is already labeled to develop a model and see in the remaining data that has not been labeled yet can the model already detect clear confidence labeling and then reduce the amount of effort that's needed for manual labeling as they advance more the model gets smarter and is better and reducing the amount of work for those labelers you can also use leverage other third parties or you can start by using an example of recommendation and you can recommend the most popular item just to get started and as the data accumulates under the hood you can replace the engine when it's seamless think like a hybrid car it's seamless for the user when it's electric or gasoline powered same here from the popular engine to a machine learning engine you don't have to have a lot of data Amazon personalized which is a our AI pre-trained or pre-built service for recommendation engine requires a thousand interactions between items and users to actually generate the first model can be 25 unique users each having at least two interactions so if I talk about hundreds of thousands of examples and so on very relatively much less data is needed and you can start like I said and transition into machine learning from a popularity based engine you can also use synthetic and data you can use reinforcement letter we talked about before or use again generative adversarial network which is two networks training each other and this is how usually fake human beings for example are created one network is trained on all the images another is creating and trying to trick the first network when the generator is able to surpass the second network then that person is considered to be good enough that it was able to trick a machine for that Airbnb is an example of a customer that's using SageMaker Ground Truth to manage their labeling workforce and it's a good example to remind us that some of the labeling can be sensitive and the workforce that you want to manage needs to be internal or within your customers and users and some can be external because it doesn't have any sensitive data you can use Amazon Mechanical Turk just for the low cost of labeling generic or generally available data now the outcome of the model comes in the form of a confusion matrix in this example for example could be healthier could be cyber if you have a true state which could be malicious or benign in a prediction if you're predicting something to be malicious and it is malicious it's true positive and same for benign but if you're predicting something is malicious when in fact it's a benign that is a false positive false alert but there's also a situation of false negatives when you're actually letting on bypass you know you're saying hey this is this is benign when in fact it was not a benign action now this is the science what is in it for the product manager is actually to decide whether you want to optimize for precision which basically says I want to have fewer false alarms but I can live with missed events or you want to optimize for recall which says I don't want to miss any of the true state situations but I'm okay with living with some false alarms you can see how this is a product decision not a science decision and it depends really on how the product is embedded in the workflow of the user if you're talking about a real-time CISO hot room that now needs to make a decision you need to reduce the amount of false alerts and alert fatigue and so on so you want to really focus on precision but then you got to remember that all the data that was not alerted on is not necessarily clean you are having some missed events in there so when you're doing for example manual forensic investigation in offline manner and the person needs to review 100% of the uses if you can identify 50% that are very likely not to have any malicious activities and that person needs to review only 50% and not 100% you've just created a lot of value for your company and that you've done that by optimizing for recall because you're not looking to have fewer false alerts 30% yeah I have a lot of false alerts but you're looking to see that you don't miss any real true-state scenarios now it also goes into how user product managers set the customer or the user expectations what is the micro copy that you use are you saying hey here is some suggested action decision support system here is probably all of the cases but you may have some errors so there are different ways to position this within the product remember that that's why it's important for a product manager to understand what is precision, what is recall and whether you want to change that in the life cycle of the product maybe you want to start with one and as trust is gained through using the system you want to change to another so I said earlier that model is not a product something needs to happen in between and that is user trust there are multiple ways of earning user trust one of them is adding human in the loop human in the loop is just like in the labeling of an Amazon SageMaker ground truth we call it Amazon augmented AI this is where you take the outcome of the model, the inference it's called of the model, the recommendation or the prediction of the model and you decide whether you automatically plug it into the application into the user interface or you want to first pass it through a human that will take a look at this data you can do that when the model is not confident enough you say below 95% I want a person to look at this you can also say hey I'm screening this now and even if the model is very confident but the person that is the candidate has a PhD I want the recruiter to actually look at this personally and you can also say you know what any 5, 10, 20% of the data randomly just pass them to the human in the loop and with time maybe reduce this number as we earn trust of the product but there is also a tool called Amazon SageMaker Clarify that helps you detect the bias in the data I mentioned earlier when you have oversampling the data in some areas or you may have mis-weight unbalanced data in the different classes so you're an insurance company and you're trying to cohort your age groups but in some age groups you have a very small amount of people because you're focused on older drivers and not young drivers for example and then this tool will kind of alert you say you know what this is not balanced data so you may want to take attention and maybe either synthesize more data or use the cohort groupings so different mechanisms that you can do but this tool can detect that for you can also detect whether you have bias post the training phase and during the model monitoring when the model is running in production and it's dependent on data coming in we're actually creating a baseline of this data when you first train the model and then once there is a drift from the baseline and you can configure what does it mean a drift for you is considered if you can also design your own metrics into that then you can get an alert from that but sometimes what you need is to actually explain why the model why did that not get alone from this bank based on sometimes it's not good enough to say the model said so this is what we're called individual explainable or local explainable AI we use a SHAP SHAP model under the hood we kind of basically took away the under differentiated heavy lifting for you so SHAP is published on research papers we just coded it so you can now use it embedded into the SageMaker platform and you can provide explainable local predictions to your users but you can also provide that to regulators let's say the regulatory wants to know whether you use gender or what weight did the gender feature receive in your overall model for underwriting scoring of this group because it's okay to use it but not too much not to rely on an over matter so global explainable AI and this is also supported by SageMaker clarifying and now lastly once you have the model out there it's very important to remember and especially for us as product managers if the customer is saying something we need to listen how does that behave in real life situation so we call this MLFly we start with saying do we have the data and labels that we can use for modeling great can we generate the model meaningful enough in the prediction great do those prediction and the way they are embedded in the product with all the trust envelopes that I mentioned before the bias detection the human in the loop the explainable AI and all of that are those driving the behavioral change that we wanted to achieve with the product if so great but that by definition is going to change the data because we just drove or generated the behavioral change which needs to be manifested in the data so ask yourself if my model is successful in generating behavioral change how will I detect that in the data will I need to relabel will I need to maybe some of the people that have been successfully altered their behavior based on this then it to be cascaded to another model to provide more granular prediction to different data set with a new label they're different decisions some of them can be made before you get started some of them need to be made on the fly and you just need to get an alert that hey there's a drift in the model now let's take a look at the behavioral data and rethink and retrain the entire process okay so how can AWS help you along the ML journey that you're at we have what we consider three tiered ML stack and in the top tier is what I refer to before as pre-trained models where if your use case fits one of them then you can easily and very fast go in from idea into a POC like I said we saw in the example before from hours or a few days using one of these pre-trained models some of them can be great fit for your use case and to any you never need to do anything beyond that and these do not require any data scientists and so on and some of them could be just a good starting point for your POC in some of them you can also tweak or customize the model for your own language let's say in the example of transcription in Amazon transcribe which is above the speech section in the bottom left in the top tier I'm still in the top tier here and you want to transcribe calls to your call center to detect sentiment to detect what's going on and train your people and manage the entire process you may have a unique domain you're working in the legal domain and you want the training of the model to be tweak customized to the legal area or the manufacturing of the airline industry then you can actually create a ceiling a custom language model for transcription so you don't have to start from scratch you do need to give some data from your own and then you can get the model to be more customized for your own needs but we also have in computer vision the recognition chat bot and lex text comprehend which is natural language understanding how to understand what the language that is within the call within a paper which is can be extracted using text track which columns and graphs and can extract entities even from radio buttons if it's a form that you feel and so on we also have business tools the example I mentioned before Amazon personalized for recommendation engine or forecast for time series fraud detector as an example for any fraudulent activity transaction that undertakes and you want to say hey this IP email credit card how does that impact your score of how likely that is to be fraudulent that can actually enjoy from Amazon so if Amazon has had some exposure to this data points that can influence the score that you get from the model so you get kind of more than just a model here but they're also verticalized solutions for healthcare for industrial and so on and then if you do have data scientists you can start in the middle tier which is the Sage maker you can choose pick and choose so we want to take away the undifferentiated heavy lifting so if you have some things that you think is really differentiating from my business to do my own feature store find and use us for model distributed training use other elements that are non-differentiating for you this is a complete IDE with CI CD that you can manage and it also has the autopilot which I mentioned before if you have tabular data and you want your data scientists to list to get started with some recommendation on ETL's and algorithms and selections but then have it as an open ML auto ML in the sense that then the data scientists can get the notebooks and see what the recommendation auto ML did and do it on their own continue and expand and improve this on their own it's called autopilot as well but if you are an expert and you don't like to use these managed services you want to build everything from scratch and manage it we have that bottom layer of the stack as well optimized for TensorFlow PyTorch MXNet and so on so kind of recapping and we have about five more minutes I want to leave a couple minutes at least for questions so first we talked a little bit about the product management powered by ML I recommend running these discovery workshops with internal stakeholders from business data and technology so you can get a long list of potential use cases qualify whether they're a good fit for ML or not then we can help you go from the prioritizing those use cases and help you choose how to get to the first POC or MVP we have multiple mechanisms where it's a prototyping engagement where we work together with your team hand in hand and your team can learn the process whether we have a pro server we can build for you or ML solutions lab where you can give us the data we will build it and hand it over back to you different modes of engagement and I'll soon show you some slides of the people you can contact if you want to test out different things here in the programs but if you already have several use cases now you want to get to this next level where you want to do an ML platform now you want to create a situation where you have empowered teams and they don't have to worry about the ML ops or DevOps they just want to build the models iterate on them improve in all the DevOps and IT work let us worry about that this is our expertise your expertise is your domain your products your services then we can help in that phase of the journey as well if you want to build everything on your own and just have us help you train your team and get your certification we've got the right side of this slide as well and we will be happy to support and do some training and certification for your team as well so with that I want to thank you we've got a few more minutes for questions and here are the people different languages and we have a very varied and diversified audience today I encourage you to take a screenshot of this screen so you can then know who to contact and if you have any questions or want any follow up or work with AWS on any of your use cases Oren thank you so much what a fantastic talk thank you we enjoyed that very much you emphasised early on in your talk the importance of developing trust so maybe you could just expand a little bit on that in particular what happens when human intuition that gut instinct what happens when that comes into conflict with what the machine learning is telling us that's a great example I think the question is how do we as product managers communicate that to the user so the user may have an intuition let's say you're a professional you're a doctor and now you have a decision support system in healthcare that recommends certain action to that patient but the doctor has an intuition if you position the machine learning recommendation and say I know I'm the machine you should follow my recommendation that can create some clash with the doctor's opinion or said intuition but if you're saying hey this is a decision support here is my recommendation and here is why meaning that I am a model that was trained on data of type X this is the type of patients that I've seen it was labeled by the community of doctors that have all been certified with this grant and this group and this and this and that so now the doctor either identifies that this is not relevant for his patient because the training data was not of the patient that he had or the type of illness that the patient had or that the people who labeled are good strong candidates, good strong physicians that he trusts and he says you know what maybe I will start to trust this because I see that other physicians that I would have called to get some consultancy with to enable this as a tool so I think by being able to provide insights into how the model, what data was used how the process of labeling was conducted and also in that local explainability so I recommended to undergo surgery here because I've seen this size of the tumor was about this big and not that big and because the texture was this and not that this is why I recommended and then the doctor can say okay I understand that's a valid point but you haven't seen I haven't given enough weight in my opinion to this thing that I've seen in the side in this way together a machine and a human can actually make a better decision because either of them can accidentally not notice something or not in not give enough weight to a certain feature and together if it's exposed and explainable and open that decision can be improved okay that makes sense but not just conflict let's consider about possible pushback because humans were not all that rational maybe we can accept that ML is providing us a better solution that we might come up with on our own but then perhaps we're fearful that we're not needed so much this is encroaching on our own role our own job is that something you're seeing yeah I think it's always a question with innovation you know it depends on how you use the innovation right innovation by itself is not good or bad that's why I'm talking to product managers I am encouraging an ethical approach to AI the sense that you think should we build it how do we communicate to the user what to do with it do we force do we say hey this is a machine and this is how it created it's a recommendation and these are its limitation you can decide to overrule it and you want to hey I want to go wild I want to do something different Netflix can recommend the next year is for me to bench doesn't mean I have to take on this recommendation right it's nicer if it says hey you know what I'm recommending this series because of that and I know can now understand and I'm maybe more friendly with that recommendation because I understand where it's coming from so I think that's more of the where this world of machine learning is headed to a collaborative approach of human and machines openly making decisions better and yes the humans can still have the control but the machine can support that can detect things that the human may not have the time or capabilities to do so it can actually augment what the human is doing and not replace that's the way I'm seeing things heading more today. Yeah you're very optimistic Oren I wish I wish we had more time to expand on this this is absolutely fascinating but we are sadly out of time anyone out there who wishes to find more questions Oren please do so via the via the website via the forum Oren once again thank you so much for your time best of luck for the future