Hi, and welcome to another CUBE Conversation. Today we're at BMC Helix's Immersion Days and the Santa Clara Marriott and Santa Clara California and we're having a great series of conversations about the convergence of digital services and operations management. And one of the most important features of that is how do you utilize analytics? Analytics is on the tip of everybody's tongues these days, but it's being applied in marketing and sales. Kind of the cobbler's children that aren't getting the same treatment are in fact the IT organization. So what we're going to do in this next CUBE Conversation is learn more about how IT analytics is beginning to transform IT and facilitating this convergence of digital services and operations management. And to do that, we've got Gaurav Ravari who is the president or co-founder and CEO of Numerify. Gaurav, welcome to the CUBE. Thank you Peter, pleasure to be here. So Gaurav, tell us a little bit about Numerify. Let's start there. Sure, I know I liked in your opening statement you talked about IT in terms of its own use of analytics being a little like a situation where the cobbler's children don't have any shoes because IT stood up pretty powerful analytical applications for the CMO, the CFO, VP of sales, et cetera, but not for IT itself. And so we think, yes, it's ironic, but it's also untenable. And it's untenable because in the age of digital transformation where you're opening up digital channels for revenue generation and the like and customer engagement, IT is really moving up from the basement to the boardroom. So you have CEOs who worry now about things like availability, about sort of a speed of innovation with quality, and things like that. And so to be able to have a decision support system, a system of intelligence, if you will, that across rank and file of IT, across the plan-build-run lifecycle, across the entire IT estate from infrastructure to apps to business services, gives you recommendations and intelligent insights on how to improve the quality of your work, the health of your systems, the reduced the risk of your systems. That we felt was an idea whose time had come. And so that's why we got started with Numerify.  And so it's sort of a string of pearls that you can deploy across your IT organization and it's interconnected. So you can ask cross-cutting questions as well. So that's in a nutshell, the story on Numerify and its vision. So Gaurav, I've been within approximate to IT or in IT for a long time now. And it's not that we didn't have reporting because IT was always doing reporting. We report on stoplights, projects, whatever they were. But I think what you're saying is something a little bit more fundamental. It's really, can we do a better job of really capturing the resources that are creating value for the business, understand how to deploy them more successfully. And we're not just talking about the infrastructure. We're talking a lot about the people. Have I got that right? You hit that nail on the head there. Ultimately, IT is a business. And you have to, if you want to face sort of the epic challenges and opportunities of tomorrow that IT alone can really take on, you have to understand the people, process, project, and product dimensions of the IT business. And so what that means is, if you want to drive down your INO costs from roughly 72% of IT budget, which is what it was the average today, to 50%, which is the gold standard, that's half a trillion dollars for the G2K, right? And you wanna take that savings and reinvest it in agility, in faster app dev with higher quality, right? How do you do that without tapping into things like automation and the use of analytics to drive down your INO costs rationally and increase your dev, your development velocity intelligently, right? So that's why analytics has a huge role to play. But also, it's gotta be, if I can interrupt you, it's gotta be that you have to start with visibility into what resources are generating the greatest return. Why are they generating that return? Why are other resources not generating return? And seeing how all that gets connected across the range of activities that an IT organization is performing on behalf of the business, right? Yeah, no, exactly. I think the how is really about getting that visibility across sources. And it's a non-trivial problem to do that when you have a plethora of sources that were never built to talk to one another, right? You may want to, for example, within IT understand the total open work on each person's plate, right? So they may have a bunch of incident resolution work that they're doing and the signal from that comes from a BMC or a service now. And yet they may be pulled into app dev work, which the signal is coming from a JIRA or a CA. How do you pull that together into a single dashboard that gives you that view of what everyone's working on? And then you can make decisions like goodness with so much unplanned work that's gone Fred's way. There is no way that the epic that he's involved with is going to be completed on time. So I have project risk, I have release risk as a result, I may even have attrition risk. And so the ability to pull together data into a single model, answer the visibility question to the point you make, and then go the next step of predicting likely outcomes, that's the magic and that's the use of analytics to sort of transform IT into operating in a far more intelligent paradigm than it has thus far.  So they forced users to install agents everywhere that there was a single process model that was expected to be employed to administer all kinds of different resources. There were very significant limits on how you consider application development or application management, for example. Why is numerify different? Yeah, what we've tried to do is really take a leaf from the page and books of those who've sort of succeeded in this endeavor before. So if you look at, you know, the solutions that a CMO might have at her fingertips or a CFO might have, right? Fundamentally, it's about pulling data into existing systems, not requiring a change of behavior, but pulling data from existing systems into a canonical model, into a standard sort of analytical data model, that runs on sort of in a dedicated stack. And then you basically have this layer of descriptive, prescriptive, and predictive analytics sort of folded in. And that's the approach we've taken, where we say, hey look, there is such a thing as a change management system that doesn't go away. We would like to mind the accumulated history of all the changes you've ever put into production by tapping into your IT service management system, and then your upstream dev and test system, because change is often a piece of code. It began its life somewhere in a dev cycle.  Who worked on it? What's been their success rate thus far? And then with respect to the change itself, in the past, how often has a change like this failed? You know, if changes were done on a weekend through a combination of an onshore and offshore team, is that implicated in failed changes in the past? And then downstream of the change, in the past, was there a decline in performance or usage or availability as gleaned from your monitoring tools? So we pull data from all these sources without requiring you to re-instrument them into a standard model, and then for every upcoming change, we tell you, hey, this one is a risky change. Go look at it, send it back for further testing. Hey, this one is a low risk change, push it to production directly. And so, inherently, the ability to pull data from multiple existing sources into a standard data model and have best practice reports and insights sort of a layer on top, that's the approach we've taken. Well, okay, I really like this. Let me see if I can summarize something that you just said. So, numerify is not immediately antagonistic to anything that anybody has within the shop. But it starts from a proposition that, look, what you're doing is working or not, but let's start from a premise that you're doing something now. Let's learn more about it. Let's then ask, can we do it better? Yes or no? And you have the analytics to do that. And if it should be replaced, can you actually get to the point of that you can actually indicate or suggest how and when to replace something? Yeah, that's a fabulous question. I think, you know, increasingly what we're seeing is that our customers are pulling us in the direction of making active recommendations on decisions that they could potentially make. Such as, you know, you may wanna consider consolidating a certain class of applications because, you know, given its revenue and usage, the amount of support burden associated with it is too high.   And that process may by itself be extremely expensive and very limited returns. Precisely, precisely. So the ability to transcend now from just visibility and dashboards to providing active recommendations for every action along the way, you know, project risk, release risk, attrition risk, change risk, service quality risk, et cetera, we see that as the vision for us, you know? It's, so the use of AI, not just for automation, you know, sort of, which clearly the AI ops field is investing in, but also the use of AI for decision support, for providing you with intelligent recommendations across the full sphere of activities that IT undertakes. Gaurav Vavari from the, from Numerify. Thanks very much for being on theCUBE. My pleasure, thank you. Once again, this has been a CUBE conversation from BMC Helix's Immersion Days. And we look forward to seeing you again. Thanks for listening.


In other news tonight, a 24-year-old Aiken man has died after driving his vehicle into a lake in Grand Rapids. The Itasca County Sheriff's Department says Jacob Duffney was found dead in the vehicle after it was submerged in Crystal Lake in Grand Rapids. A passenger 22-year-old Savannah Hegdahl was able to escape from the vehicle and was hospitalized with non-life-threatening injuries. The crash happened around 2.45 this morning. If you enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland PBS.


Cloud Computing friends, and welcome back to theCUBE where we are live from Las Vegas, Nevada. Here at AWS ReInvent all week, my name is Savannah Peterson, very excited to be joined by Paul Gillan today. How are you doing? I'm doing great, Savannah, it's my first re-invent. I was just going to ask you if you've ever been re-invent. It's dazzling, much like the sequence. On your top, it's dazzling. Jam-packed affair. I came to the Comdex conference for many years in Las Vegas, which was a huge event, and this really rivals it in terms of the crowd sizes, but I think there's more intensity here, there's more excitement. People are just jazzed about being here, to the extent that I never saw at other computer conferences. I thought I would agree with you. It's my first re-invent as well. I'm about to share this experience together, and the vibe, the calls, I think being back in person is really contagious as well. Maybe the wrong word to use, but in a great way. Energy is definitely radiating between people here. I'll watch my words a little bit better. And in person, we have with us, Samuel Nichols, the director of Public Cloud at Global Product Marketing at VM Software. Sam, is it Sam or Samuel? Depends if I'm in trouble. Depends on who's saying it out loud. Yeah, yeah, it's typically Samuel is usually reserved for my mother, so. Well, Sam, thanks for joining us. We'll stick with Sam on the show. So VM, been a red-hot company for several years, really made its reputation in the VMware world. Now you've got this wholesale shift to the cloud. Not that VMware is not important still, but how is that affecting, you're shifting with it, how is that affecting your role as a product manager and the business overall? Yeah, it's a fantastic question. Obviously, Veeam was pioneered in terms of being the purpose-built backup and recovery company for VMware. And as these workloads are being transitioned from the data center into the cloud or just net new workloads being created in the cloud, there is that equal need for backup and recovery there. So it's incredibly important that we were able to provide a purpose-built backup and recovery solution for workloads that live in AWS as well. And how different is it backing up an AWS workload compared to a VMware workload? I think it depends on what kind of service a user is utilizing, right? There's infrastructure as a service, platform as a service, software as a service. And given that the differences in what is exposed to that customer, that can make backup and recovery quite challenging. So I would say that the primary thing that we want to look at is utilizing native snapshots as our first line of defense when it comes to backup and recovery, irrespective of what workload that might be, whether it's a virtual machine, Amazon EC2, some sort of database on Amazon RDS, FileShare, so on. I bet you're seeing a lot across verticals and across the industry, given the support that you're giving customers. What are you seeing in the market and in customer environments? What are some of those trends? So I think the major trends that we highlight in our data protection trend support, which is a new update is coming very shortly in the new year, is- Do we have to check that out? Yeah, absolutely. The physical server is on a decline within the data center. Via virtualized workloads, namely VMware, is relatively static, kind of flat. The real hockey stick is with the cloud-based workloads. And as I mentioned before, that is partially because workloads are being transitioned from physical virtual machines to being cloud-hosted. But also, we're creating more applications and the cloud has become the de facto standard for new workloads. So you hear about cloud-first initiatives, digital transformation, the cloud is central to that. You mentioned snapshotting, which is a relatively new phenomenon, although it's taken hold rapidly. How does snapshotting work in the cloud versus in your own prime environment? It's not wildly different at all. I think the snapshots is, again, a great first line of defense for helping users achieve very low recovery point objectives. So the frequency that they can protect their data, as well as very low recovery time objectives, how quickly they can recover the data. Because that's why we're backing up, right? We need the ability to recover. However, snapshots certainly have their limitations as well. They are not independent of the workload that is being protected. So if there were to be some sort of cybersecurity event, like ransomware that is prolific throughout pretty much every business, every vertical, when that snapshot is not independent, if the production system becomes compromised, that snapshot's likely to be compromised as well. And then going back to the recovery piece, we're not going to have something to recover from. And it's not a one and done with ransomware. Yeah, so how, so what is the role that backup plays? I mean, a lot of people, I feel like security is such a hot topic here at the show and just in general, attacks are coming in. Unique form factors for everyone. I mean, I feel like backup is, no pun intended, the backbone of a system here. How does that affect what you're creating? Yeah, absolutely. I think, like you say, the backup is a poor to any comprehensive security strategy, right? I think when we talk about security, everyone tends to focus on the preventative, the proactively stopping the bad guys from getting it. However, there is that remediative aspect as well. Because like you say, ransomware is relentless, right? You as a good guy have to pretty much fend off each and every single attack that comes your way. And that can be an infinite number of attacks. We're all human beings, we're fallible, right? And sometimes we can't defend against everything. So having a secure backup strategy as part of that remediative recovery component for a cybersecurity strategy is critical. And that includes things like encryption, immutability, logical separation of data, and so forth. We know that ransomware is a scourge on premises, typically begins with the end user workstation. How does ransomware work in the cloud? And do the cloud providers have adequate protections against ransomware or can they? Yeah, it's a fantastic question as well. I think when we look at the cloud, one of the common misconceptions is as we transition workloads to the cloud, we are transitioning responsibility to that cloud provider. And again, it's a misconception, right? It is a shared responsibility between the cloud provider and in this case AWS and the user. So as we transition these workloads across varying different services, infrastructure, platform, software as a service, we're always transitioning varying degrees of responsibility, but we always own our data and it is our responsibility to protect and secure that data. For the actual infrastructure components, the hardware that is on the onus of the cloud provider. So that's the major. Is ransomware as big a threat in the cloud as it is on-prem? Absolutely, there's no difference between a ransomware attack on premises or in the cloud, irrespective of where you're choosing to run your workloads. You need to have that comprehensive cybersecurity strategy in order to defend against that and ultimately recover as well if there's a successful attempt. Yeah, okay, let's get us out of the dark shadows real quick. And bring us back to a little bit of the business use case here, a lot of people using AWS, what do you think are some of the considerations they should have when they're thinking about this? Thinking about growing their use of AWS? Well, if we're going to stick down the cloud shadows, the cybersecurity piece. We can be the darkness. You're kind of a dark shadows business. We can go rainbows and unicorns, nice and happy if you like. I think there's a number of considerations they need to keep up. Security is number one. The next piece is around the recovery as well. I think folks, when we talk about backup and recovery, the focus is always on the backup piece of it, but again, we need to focus on why we're doing the backup is the recovery component. So making sure that we have a clean, verifiable backup that we're able to restore data from. Can we do that in an efficient and timely manner? And I think the other major consideration is looking at the entirety of our environments as well. Very few companies are 100% sole sourced on a single cloud provider. It is typically hybrid cloud. It's around 80% of organizations are hybrid. So they have the on-premises data and they also have workloads running in one or multiple clouds. And when it comes to backup and recovery of all of these different infrastructures and environments, the way that we approach it is very different and that often leads to multiple different point products from multiple different vendors. The average company utilizes three different backup products, sometimes as many as seven and that can introduce a management nightmare. It's very complex, very resource intensive, expensive. So looking at the entirety of the environment and looking to utilize a backup provider that can cover the entirety of that environment while centralizing everything under a single management console helps folks be a lot more efficient, a lot more cost effective and ultimately better when it comes to data protection. Amazon and all cloud providers really are increasingly making regions transparent. Just at this conference, Amazon introduced failover controls from multiple, multi-region access points. So you can failover from one access from one region to another. What kind of challenges does that present to you as a backup provider? I don't think it represents any challenges. When we look at the native durability of the cloud, we look at availability zones, we look at multi-region failover, that durability is ultimately founded on replication and I wouldn't say that replication and backup, you would use one or the other. I would say that they are complementary. So for replication, that is going to help with a failover scenario, that durability component, but then backup again is that independent company. If we look at replication, let's say the source data were to be compromised by ransomware or there was accidental deletion or corruption, that's simply going to be copied over to the target destination as well. Having that backup as an independent puppy again complements that strategy as well. Do you need it in either, in any scenario? In any scenario? I think the average person would probably say that backup is not the most exciting technology aspect of this industry, but you guys certainly made, built a great business on it. What excites you about what's coming in backup? What are the new technologies, new advancements that perhaps we haven't seen and productized yet that you think are going to change the game? I think actually what we offer right now is the most exciting piece, which is just choice, flexibility. So Veeam again is synonymous with VMware backup of we can cover a multitude of environments, including AWS, containerized workloads, Kubernetes, physical systems and the mobility pieces is critical because as organizations look to act on their digital transformation, cloud first initiatives, they need to be able to mobilize their workloads across different infrastructures, maybe from on-premises into the cloud, one cloud to another. Maybe it's cloud back to on-premises because we do also see that that flexibility of choice is what excites me about Veeam because it's ultimately giving that the users, best in class data protection tool sets without any prescriptive approach from us in terms of where you should be running your workloads. That is the choice of the user. Yeah, Veeam is definitely more than VMware. We actually had a chance to chat with y'all at KubeCon and CaliativeCon in Detroit. So we've seen the multitude of things that you touch. I want to bring it back to something and something kind of fun because you talked a lot about the community and being able to serve them. It's very clear. Actually, I shouldn't say it's very clear but to me it appears clear that community is a big priority for Veeam. I just want to call this out because this was one of the cooler pieces of swag. You all gave out a hundred massage guns. Okay, very hot topic, hot Christmas gift for 2022. I feel like I'm in white right now. And I was actually really compelled by this because we do a swag segment on theCUBE but it's not just about the objects or getting stuff. It's really about who's looking out for their community and how are they staying. Thanks, swag is a brand activation but it's also a thank you and I loved that you were giving out massage guns to the AWS heroes and community builders. What role does community play in the culture and the product development at Veeam? So community has always been at the heart of Veeam. If you haven't looked at pretty much every single development across all of our versions across all of our products is always stood by the community. We have a wonderful Veeam forum where we've got 400,000 less users actively providing feedback on the product, what they would like to see and that is ultimately what steers the direction of the product. Of course, market trends and technology. A couple other factors I'm sure. A couple other factors but community is huge for us and the same goes for AWS. So talking with the AWS heroes, the community builders helps Veeam reach further into that community and the AWS user base and empower those folks with data protection tools and massage guns when your feet are tired from being standing on them all day in Vegas. Yeah. Well, I mean everybody is working hard and it's nice to say thank you. So I love to hear that and it's clear from the breadth of products that you're creating and the ways that you're supporting your customers that you already take care a lot about community. We have a new challenge on theCUBE this year at AWS re-invent. Think of it as an Instagram reel of your thought leadership, your hot take on the show. Key themes as we look into 2023. What do you think is the most important story or trend or thing going on here in the show? I think it's just the continuation of cybersecurity and the importance of backup as a comprehensive cybersecurity strategy. Some folks might say that secure backup is your last line of defense. Again, ransomware is relentless. These folks are going to keep coming and even if that's successful, it's not a one and done thing. It's going to happen again and again and again. So we have a look around the show floor, the presentations, there's a huge cybersecurity focus and really just what folks should be doing as a best practice to secure their AWS environment. Well, Paul, any final thoughts or questions? I just quickly, you've mentioned data security. You mentioned data protection and backup sort of interchangeably, but they're not really the same thing, are they? I mean, what businesses you see VM as being in? I would say that we are a data protection company because of, yes, there is backup, but there's also the replication component. There's the continuous data protection component where we've got near zero RTOs. And then we, again, look at the cybersecurity components of that. What can we do to really protect that data? So I would say that the two are different. Backup is a subset of data protection. Sam, thank you so much for being here with us on theCUBE. It's been a super insightful conversation.  And more of the teams, they seem to be celebrities here with us on theCUBE. Paul Gillan, thank you so much for being here with me. Pleasure, Savannah. And I'm glad we get to celebrate our first reinvent and most importantly, thank you to the audience for tuning in without you. We don't get to hang out here in fabulous Las Vegas, Nevada where we're live from the show floor at AWS Reinvent. My name is Savannah Peterson with Paul Gillan. We're theCUBE, and we are the leading source for high tech coverage.


Our next speaker is as Anand. So we'll be talking about exploring network classes using Python. So, Anand, what do you know? Anand, what do you know? Oh, sorry. I missed the queue. Thanks, Kalyan. Okay, let's dive in. What I wanted to do was talk, actually, a bit about movies, to be honest. We've come to the Python part of it in a short while. But my story starts, actually, with Govinda. See, it was in 2000 that I first heard of Govinda. Maybe it might have been 98 somewhere around there when I saw Bademiya and Chotemiya with Amphabachan and Govinda. And like, who is this guy? And his style of comedy was completely crazy. But after I saw that movie, I was kind of hooked. I wouldn't say like, oh, that's Govinda, right? But while I could name every single movie that Rajnikanth starred, I couldn't have named a single movie with Govinda. That's when I learned about the Govinda, David Davan combination and so on. And one piece of news really clipped me in the recent past. I'll talk to you about that. See, it turns out that in Apki Adalet, Govinda made a statement that he was once offered a role in Avatar, the movie, presumably as a hero. So we said, okay, what if? And this is a pure hypothetical situation. What if Govinda in Apki Adalet had also announced that he had been offered a chance to star with Angelina Jolie in a movie Jolie number one? But in the interview, he declines to say who actually connected him to Angelina Jolie for this to happen. Now, let's make the speculation who possibly could have connected Govinda to Angelina Jolie. Now, it turns out that he's friends with a number of his heroines, and his top heroines include Neelam, Kimi Katkar, Karishma and so on. Now, one possibility is that maybe he refused to share who it was because maybe they were actually connected to, they might have connected him to Angelina Jolie. But the trouble is not a single one of them has acted with the Hollywood actor. So that rules them out. What about his male co-stars? If you look at the male co-stars, the top actors that he's acted with are Shakti Kapoor, Kader Khan, Gulshan Grover. In fact, you could argue that Shakti Kapoor is practically his boyfriend. Now, among these have any of them acted in a Hollywood movie? That's an open question. Let's explore. Turns out that the two people that have acted in Hollywood movies are Gulshan Grover, who's acted in a bunch of them, and Anupam Kher. And there is a chance that these might be connected to Angelina Jolie. But for that, we'll have to start the other way and look at Angelina's co-stars.  But out of all of those co-stars, Dustin Hoffman and Jack Black are the only actors she's acted with a little more. But among the Bollywood actors, it's only Irfan Khan, only one person. So what this means is that the only way Govinda could reach to Irfan Khan is to Angelina Jolie is through Irfan Khan, which can either be through Gulshan Grover or Sanjay Ghat. But there's a third possibility, which is Tabu. And it turns out that Govinda has acted in four movies with Tabu and with four movies with and she has acted in four movies with Irfan Khan. So what that means is that Govinda needs to connect to Tabu, who needs to connect to Irfan Khan, who needs to connect to Angelina Jolie for this to happen. And this is what can possibly cause something like Jolie number one. Now, here's the thing, right? How does one figure this out? How do we figure out what does it take for Govinda to be able to reach Angelina through a co-star network? That's what we're going to explore and through a network exploration. This is useful in a number of scenarios. And every single one of those scenarios is considerably more boring than movies. So I'm not even going to talk about where you could apply network analysis, at least not yet, you can ask me the questions. But we are going to explore this journey. How do we do that? Well, let's get into the code. So the good part is all of the data is available on IMD. And what I have here is, by the way, you can figure this out at IMDb.com slash interfaces. This is a page where you can download the IMDb data sets, the Internet Movie Database data sets from data sets.imdbws.com.  These are all gzip files, which are Tabd limited, and they contain information around all the actors names, the title alternate names, basic information about the title, the details about the actors. And those are the four things that we really need for this piece of analysis. But here's what we're going to do. First offline, we're going to download all of these four files and unzip them. And I've already done that. These files are pretty large, by the way. So if you look at the, let's see, from a nice way, the principal title's file is about 1.5 gigs, and this is about a gig, and this is about half a gig, and so on. So what we're going to do is first use pandas and read the title basics file. What that does, let's run it live. What this does in 3D is have, it has a set of fields identified by T-const. T-const is like the idea of the file, idea of the title, the film. And then a title type, which can be an indication that it's a movie, or it's a shot, or something else. A primary title, that's the name of the movie. And the start here, which is basically the year in which it was shot. Now, that's all the information about the movies. Now, if you want to connect two people, we also need to know who's acted in which movie. So that comes from the next one, which is the principal's information. Now, this has three pieces of information that are relevant to us. The first is the idea of the movie, that is T-const. The second is the name of the movie. And the third is the category. Now here, the category can be, is this name, the person whose name it is, acting in this movie as an actor, or is it a director, or is it an actress, or is it a composer, is it a producer, etc. We realistically just want to restrict ourselves to actors and actresses, though in reality, you could argue that Govinda could just as well talk to A. R. Rahman, who probably has better Hollywood contracts than, let's say, Taboo. So that is certainly possible. But we are going to restrict ourselves just to actors and actresses. And this is really the network. Now, incidentally, these networks are what are called bipartite networks. You don't need to have special data structures to have network graphs. You just need two columns, saying A connects to B. That's it. And you have a network. We'll also further reduce the data set by saying that we are only interested in the movies, not in the shorts, not in the TV series, etc. And we will only pick those actors who have acted in movies together. So if we take that, this list is slightly smaller. And this is what it looks like. So in this particular movie, this particular person was an actress or an actor. The last piece of information that we need is the names of all of the people. So if you say NM101955, who is that? So you need the ID, which is the end const, and their name. And we'll also get their year of birth. It's something that might prove handy when we want to filter across generations. So with this information, how do we figure out whether two people have acted together? Well, or how we connect between two people. So this piece is the central information that we need. This movie has this actor. The library we're going to use is NetworkX. Now what NetworkX does is incorporates a whole bunch of graph algorithms. So the first step to creating a network is saying create it from an edge list. An edge list is basically a table where each row has one edge. An edge is basically a line that connects two things. In this case, one thing is the title of the movie. The other thing is the name of the actor. And we'll then say that I specifically want to connect the titles and the actors. So this gets created. Now what I'm going to do is just confirm that it actually works. So that I'm not missing any stuff. All right, now here's we'll come to this bit in a minute. Now, once you have the edge list, how do you figure out the shortest path between any two actors? Turns out that there is a function called NetworkX.shortest path, which takes a graph. And what is a graph? A graph is the rule. I ran out of memory, did I? Okay, fine. Then question is, there goes my talk. Let's see if I can recover. Here's where, here's where having a saved version of the file helps. So I'm not going to run, but actually, you know what? This is the fun part where you say, can we actually beat the odds? So I am actually going to run everything up to this part and see if we can get it still working. But I'm going to continue talking about some of the stuff that we, that I've at least saved.  And I won't introduce Angelina Jolie because if I did, then that kind of defeats the whole purpose of my talk. Now, how could Manorama connect to Angelina? It turns out that she's acted in Riksha Vodoo with Pareshra, who has acted in What If with Irfan Khan. And obviously, we know that we need Irfan Khan to connect to Angelina Jolie, and they have acted together in the mighty, in a mighty heart. That's the network. How do we get to this? Well, we used this function that I've just written called path. How does path work? What it does is first looks up the name. Given the name, in this case, it was Archie Manorama, it says, give me the source ID, that is basically the end const. Similarly, given a name like Angelina Jolie, give me the target ID and the end const again. And then run network X dot shortest path, which takes G and the source ID and the target ID. The result is an array. And that array has all of the nodes, the movies, as well as the actors who are connecting them. And it gives an entire list of these. So there may be multiple paths to connect from A to B. Let's see if I can show you an example of multiple paths. Yeah. So Sylvester Stalin, for example, can connect to Salman Khan in a couple of ways. The first is through incredible love, which is acted together with Akshay Kumar. And Akshay Kumar has acted in Musseh Shadi Karogi with Salman Khan. The second still connects Sylvester Stalin to Akshay Kumar. And Akshay Kumar has also acted in Jaan-e-Man with Salman Khan. But net net, actor-wise, the only path that Sylvester Stalin has to connect to Salman Khan is Akshay Kumar. But you can see how he has two potential paths. So to connect from one to another, there may be multiple paths. And what networkX.shortest, all shortest paths does is returns, what all shortest paths does is returns all of those paths. There is the shortest path function, which just returns one of those paths. Which one? I don't know. One of those. And then what we do is convert those IDs. By default, it stores it as IDs, just to save space. And we convert those IDs back into the title, if it's ID starts with TT, or we convert it to the actor name, if it starts with the actor name. And that's what gives us this list. So let's see if we have actually managed to get to this and are able to create the edge list. If this works, then Kalyan, I'm going to invite you to participate with me on this. What I'd love to do is have you pick two actors, just choose. And we'll figure out what is the shortest path between those two actors. So I pick Dwayne Johnson and Bianca Chokhra. Okay. So let's create a new cell and construct Dwayne Johnson. Didn't they actually add together? Yes, they were in Baywatch week. Okay. And hopefully you just say Baywatch and there won't be too much of a path. Let's see. Bianca Chokhra. That's doing its computation. And if it crashes, then I'm going to give up. But like, okay. Yeah, they acted in Baywatch. So that's easy. They've already acted together. Two actors who might not have acted together. And let's try and figure out how they might have connected any others. Amitabh Bachar Nagarjuna. You know what? I think they've acted together as well. No, no, you're thinking of another one. Who knows, probably would have actually more connected than you think. Nagarjuna. What's Nagarjuna's IMDB name? IMDB Nagarjuna Kimeni. Yeah, Nagarjuna Kimeni. Let's copy that title and use him and see if they're not together. That seems to be, yeah, they've acted together in Kudagawa, slightly old movie. But yeah, what about his wife? Amla, what's her name? Amla. Nagarjuna IMDB, Amla Kimeni. Yeah, Amla Kimeni. They might have acted together too. But no, interestingly, it's easier for Amitabh to connect via Arvind. Not easier. But he can certainly connect through Vinod Khanna. But that may not be his only path. Let's look at all of Amitabh's paths to Amla, which sounds weird to say. But Amla has acted with Aruna Irani and also Satyendra Kapoor, whom I've never heard of, and Chiranjeevi and a whole bunch of actors. So I guess this really is a pretty long list. He can also reach out to Amjit Khan. He can reach out to Nagarjuna, of course, understandably. Even Rajnikanth has acted with Amla and via Mithun Chakravarti. And these are all the shortest paths that he has. But let's do an exploration of this, meaning, see, it's not just the shortest paths that can be interesting. What can also be interesting is who are the co-stars. Earlier, I had mentioned that, yeah, if we take Angelina, she's really acted only with one Indian co-star. And the way we can figure that out is to look at the network of neighbors for an actress like Angelina and see what all the movies Angelina has acted in and who are the co-stars in each of those and give it a count.  Or if you take an actress like Sridevi, then her co-stars are, well, apart from herself, Krishna, Rajnikanth, Kamal Hassan, Gumadi, and then it falls off. So any actor whose co-stars you want to explore? Mithun Chakravarti and Rajnikanth. Mithun, okay, let's see. Mithun on IMDB has an ID of something like this. Let's see what Mithun's network looks like. So Mithun's acted mostly with Shaktikapur. I did not know that. Prem Chopra next, but all of the others just fade in comparison with Mithun. And let's take Rajnikanth as an IMDB ID of something or the other. Let's copy that and run this. Okay, so this is one of those things that keeps happening occasionally. There are some actors who don't have names. So it turns out, for example, that this particular IMDB slash name slash, yeah, is not necessarily there in the database, but clearly is Padma Kriya and is the same as this ID. So there are these corrections that occasionally we have to make and say that, look, the, where were those corrections that we made? Names of, thought we made a few corrections here and there. Yeah, names. Yeah, corrections like these we need to make. And in this particular case, I'll have to say name.lock of, this is Padma Kriya. And this is the real name, but there is one more ID which actually represents the same person, Padma Kriya, which has been redirected. So let's create an entry for her and add this. When we do that, hopefully we'll be able to see the pairs of Rajnikanth. But while that's coming up, what I'm going to do is now switch over to a slightly different thing. See, we can actually not just get the immediate costars, but we can also count the number of times they've acted together. So if we see, for example, all of the actors between 1980 to 2020, who have acted together in films, and what is the most who are the most common pairs? Turns out that Kavya Madhavan and Dilip Malayalam pair, they've acted together in 17 films. Koel Malik and Geeth have acted together in 12 movies. Prithviraj and Jagathi have acted together in 12 movies. Trisha and Takash Raj have acted together in 10 movies and so on. And it might be possible to create a network out of this to see what the structure of actors look like. So that's where a site like Kumu.io comes in. Kumu is a pretty cool site where what you can do is upload your data set and start exploring the structure of that network. So I've uploaded all of the Bollywood data onto one site and specifically restricted it to only two actors that have acted together, sorry, actors who have acted together only in four or more films. So let's unfocus this and spend a few seconds to look at the network as a whole. This is what the whole network looks like and you'll see that it's kind of a heavy structure. Some of you may not be able to see too clearly but I do hope you get a broad sense of the structure of the network. And firstly, you can see that there are two halves. There's one part out here and one slightly more diffuse part out here and you probably have your guess on what exactly this is. But let's zoom in and verify. This is a network of actors, remember? So if I just focus on the actors, you'll be able to see what this is about. So it says Chittur Nagaya, that says SV Rangarao, then there's Savitri. So these are all South Indian actors and more the Telugu movie industry. And you can see that there's another cluster fairly close by out here. Let's zoom in into that if I net out that in crash. Okay. This has actors like Ratish, Sukumari, Muradi, I don't know how many of you are able to see those but hopefully you'll be able to see some of the names, Shobanaam, Vikar, Nalini. And you can guess that these and well because Mamuti and Ohalal, Nidumuni Venu are here, this is the Malayalam industry. And clearly the Telugu industry and Malayalam industry doesn't mix as much as and now we can start looking at the rest of the structure. The Tamil movie industry would be pretty close. The Kannada industry is either this one or this one. But all of South India is reasonably closely distributed together. But on the other hand, this is the slightly more diffuse network of North Indian actors. But even here you can kind of see two substructures. The major one is and as you might have guessed, this has actors like Sanjil Kumar, Veena Roy, Sujit Kumar, Dharmendra. So that's mainstream Bollywood. But there's also another slightly different cluster out here. Let's pan towards that. And this has actors like Anil Chatterjee, Savitri Chatterjee, Kali Banerjee, Madhvi Mukherjee. So that's obviously the Bengali film industry. And then there are peripheries out here. But it looks like there is a North-South divide among the actors as well. But are there crossovers? Well, let's look a little closely at who are the actors that are at the junction. So let's explore here. So some that are slightly closer to the Hindi movies side, but have still clearly acted with the South are actors like Vajainti Mala, Disko Shanti, who is actually a very interesting actress. Suresh Zarina Vaha, Girish Karnad, who's actually again like Vajainti Mala, South Indian, but became more popular in North India, actresses like Chitra and so on.  Let's see if we can This is NTR, Chief Minister of Anuradhavish, who has acted in some Hindi movies as well. And we have actors like Arthi, who's acted in a few Hindi movies as well. But by and large, you can see that barring a few people that are in the middle, it's pretty much a segregated network. And you can see the clusters, Bollywood, Bengali movies, not sure what this is, South Indian, which has sub clusters Telugu and Tamil movies reasonably close together. Madhyaala movies, slightly separate Kannada movies, slightly separate and then diffuse network of other movies. All of these become possible with a pretty much one library in this particular case that handled all the network structure, which is NetworkX. And what I'd invite you to do is try out the NetworkX library. Apart from the shortest path, which we certainly used, you can also create network graphs. I didn't use NetworkX to create the graphs because graph here because it was too complex and I didn't have time. But you can create these graphs and you certainly can also identify measures and even cluster the networks to see whether the kind of structure that you saw here can be automatically derived from data. More than anything else, the important thing that I realized was networks are not a special kind of structure. Anywhere you have two columns, like name of a movie, name of an actor or product and channel or pick country and language, you have a network. Anywhere where you have two columns that are categories, not numbers, you have the network. And the network exploration, like text analysis and a number of fields that are emerging today, is so underexplored that you just have to scratch the surface and you'll find something cool. So with that, I hope you learned something that was interesting and hopefully useful. I'd love to throw it open for questions. Yeah, and that was a really interesting and fascinating talk. So here we have a question for you. Are network clusters the same as knowledge graphs and any real scenario where people printing data in this format helps? Okay, knowledge graphs are networks and knowledge graphs can be clustered. So let me explain each of these terms one by one. A network is where you have points and lines connected together. That's pretty much it. Network clustering is a specific approach where you, like you cluster things, you say this bunch of points belong to one group, this bunch of points belong to another group. You can do that with a knowledge graph and the question then becomes what is a knowledge graph? A knowledge graph is a network where specifically you have information representing some domain. So for example, you may have obtained a knowledge graph from Wikipedia, which says that this particular person is related to this particular place by being his place of birth. This particular person is related to this particular year by being the year in which he or she was born and so on. So you have various things of various types like places, years, people, and they are related through various ways of linkages. So from such a knowledge graph, which is obviously a network, you can create clusters. What might be an example of creating clusters? Well, one simple cluster would be to say I will only filter a knowledge graph by year of birth. Like we filtered for actors here, we filtered only for year of birth. And then once we start clustering people, we will find that the clusters simply have people who are born in the same year because each person can only be born in one year. That becomes a very simple cluster. Or you could say I will only cluster people with places, but it could either be place of birth or place of death or place of work or any such thing. Now we will find slightly more diffuse clusters. You may find that there's a whole bunch of people that are concentrated, let's say just in New York. But some of them have some of them came to New York having originally been born in London. So they are also connected to some Londoners and there's a London cluster and you'll find those crossovers. So network clustering can be applied to knowledge graph and I hope that what I just gave you was a real-world scenario that helps. Yep, that really makes sense. And so one of the general questions I mean, this is from my side. So what was the motivation or reason behind the choosing actors in order to explain this network clustering scenario? Okay, so see here is how I overrun my PyCon talks. I take something that I've already done and see which is the most interesting and put it out here. We had a hackathon where one of my colleagues, Niyaz, he put together this exploration. So I had it and I said, this is cool. But the reason why it shows this particular representation for a hackathon is that networks are actually things that we don't really understand too well. It's not something that we cover in courses. And very often, at least from my early education, I found that graph theory can be pretty boring. Network analysis can be pretty boring, even social network analysis, which sounds really cool, right? It's covered in a pretty boring way. So Polywood was really my way of getting out of that border. And I just hope that I've passed on some of my way of getting out of the border to you. So yeah, and I think that's all we have for today. And thank you again for speaking in PyCon India. Always a pleasure. Bye everyone.


Two days ago my life was perfect if this boy dies I Die with him. I would trade places with him in a heartbeat if I only could Dear Lord don't take him now Not now sir your name is less right Yeah I'm sorry visiting hours were over an hour ago. I'm gonna have to ask you to leave Do you have a chapel? Yes, we sure do Daddy, I thought I might find you here You've already been here for two days. You've got to go home and get some rest They said we're gonna be in this for the long haul Please go home If anything changes, I'll call you. I promise I just can't understand how this could have happened We ran some coal today. Yeah, we tore the bone out of today boss boys are wearing it out. I Tell you Danny. I've I've been mining coal forever and You all cut it and load it faster than anybody I ever seen. Oh stop.  I know but Little curtain to help a lot. That's what scrubbers are for Look, you know, and I know I'm right It's not just for show, you know, it's for you. I Hear you the curtain don't fill no unit trains start getting ate up with all that safety stuff Crews gonna slow down. I'm telling you remind a good coal. We're looking good Sometimes you just got to rock along to get along Let me think on it. Here we are If Leanne and then grandbabies a hug for Hey, how was your day? She's had a long day Out again today, huh run out of curtain and hardly nothing. We're all right What is it you say rock along to get along right? I heard that.  He's hurt real bad. We must have hit a well there were flames everywhere Somebody call outside tell them to call life flat. We got a man hurt bad down here Get an empty kit in the stretcher and somebody call for the road. They just took him back into ICU They won't let me back there to see him No, I Just can't understand how this could have happened What happened down there? Please Tell me it was all my fault your fault Daddy no, I knew what they were doing was wrong and I just Went along with it the kids called it rock along to get along. I just hope Danny's gonna be excuse me Are you Leanne Smith? Yes, sir. I'm very sorry, Mrs. Smith There's nothing more we could do for him. We did all we could for your husband daddy Yeah, I did Now I have to live with it all the rest of my life My decision will affect the lives of my family Forever all that's left of my life is to help you Please learn from my mistakes When it comes to safety don't ever let yourself just