Time here from Orange Systems, and this is what? 22015 Outer Drive. Normally when we're at a data center, I can't tell you where we're at. I can't deal with legal to let me there, but this is my friend Ali. He does lots of work for us doing lots of cabling. I've got some videos I have to edit with him out there, but we're gonna go look at a data center. Now, we're actually standing out back of it, so we're just in an alley, but this is how you know that this is a data center, not just because of this little address, which clearly has nothing on there, or the front of the building, which looks ominously empty. You can't tell what this building is when you pull up to it, can you? It's so low key. It's very low key. The only hint you get, I don't know how to start, before we go inside is when you have a gigantic Cummins generator here. So yeah, this is behind this little building. For those of you who may already pause the video and googled the aerial overview, you can see this from the sky. I love how big that switch is over there. Don't worry, this is powered down. I can touch these things because this place is not in use. It's actually vacant right now. I will mention here at the beginning of the video, there's no sponsorship or anything here, but this is Ali and friends, and I know them. If you're interested, there's a link down below if you're interested in acquiring this empty building or leasing it or whatever. But we're gonna start with the demarcation. Now, normally, this would be so loud I couldn't record in here. And as I mentioned, usually there's a bunch of legal reasons because I've always asked. We go in a lot of data centers and I mean, what's the first rule, Ali, turn off the phone? Turn off the phones. Where's your phones? Where's your phones? Leave them at the front desk. Leave them in your pocket. You can't take pictures. There's a series of rules you have to go into. But because this is gone, they cut all the fiber too. Yeah. Oh, that's just... Yeah, those are the leaves in. And then the main one, the dedicated circuit, it's still hung up there, worked up there. Yeah, so the main circuit's still good. It's right there. So there's still the main circuit left, but they cut the fiber leading to... They cut the fiber leads for all the other parts. Look at this. This is, that's a travesty. All right, so this is where the demarcation starts. This is where we have actually our first sets of UPS batteries. So, and this is all off. Look at that. There we go. There's all the UPS batteries and things like that. Once again, I love how big that is. Now, walking through, there's offices down there. So this building does have offices. Here's a little maintenance server room. There's a little, what do we have here? Old foundry switch mounted. It's so weird to be in a data center like this, when it's just quiet. Now, data centers are not built with a normal floor. So this may look like a floor. Can we move these Ali? Yes. Give you a hint what's going on down here. There we go. Here's how the raised floors work in these. So this right here, not much light, but you can get an idea. This is so all the cable goes underneath. And then we're going to go over to each of these and show you how they come through. So we're going to open this up. And then you have, this is how the pull-throughs come each at the bottom, kind of going through the racks. And here's an idea of how easy these panels just kind of pop up for each floor for rearrangement. This keeps it kind of minimal, for we do have a ladder rack go to cross here that feeds some of these, that's some of the fiber feeds. But this is how you would get to it. So if we wanted to move any of the racks around, this is that way we're going to be able to get to it. Like this one doesn't have a pull-through because these ones were fed by the trays that come up through the top. But these ones over here, as you can see, they got a little piece of foam in there. Whoops, that is now down there. This is where the wires would come through. That's where the plugs are for each of the racks. So when you're building out the data center, it's kind of different ways of looking at it.  You generally like a lot of it with these floor-mounted and a raised floors for pulling this through. Now, the reason these boards are right here and right here, but that is normally what you would see there. These are the HVAC cooling systems that are gonna pull all the air. Because one thing about having this many servers in one place is all that wattage that this place brings in turns into a whole lot of heat. So you have a system on this side that are paired and the other systems, well, when this system got decommissioned, this place got decommissioned at the time, these sides were taken out, but you have redundancies for each one of these. So you see these different power supplies on each side, which, gee, trying to open it. And there's all the panel boards, everything's in the off position. We already know this is powered down. Normally I don't like to open these things, but it gives you an idea for how each of the racks are labeled and how each one of these would be able to say, hey, this goes to each one of these circuits. Oh, nice little label that they slide it in like that. Looks good. Then you have each one of the panel boards and any individuals here for the power. Now this is the main power delivery room. So this is UPSB. There's UPSA. This looks a lot older. I like this green style, man. It's like from forever ago. That's kind of cool, too. I'm not gonna turn it because I don't know what it is or it's not powered down, but I don't wanna cause any problems. 480 volt, we actually have on this one, I think it's 600 volts in. So these are each 480 volt. And then this system, we have 175 amp at 600 volts, 175 amp at 600 volts on this side. Then we have, this is what runs that generator behind there. So we have the generator, which is in normal mode, available, emergency. I like that they have the instructions here, just in case you wanna do it, I'm not gonna, I wanna turn it, but I'll stop, I'll resist. And once again, nice little warnings that we have on here. One thing that's really cool is just all the different things they've abandoned here. We have, and I don't know if these are good or not, I'm not gonna put my tongue on that to find out because these are 450 volt capacitors, man. That is, these are beefy, these are for the UPS. I don't know, they're all in the box, or they left, what else is some of the equipment they left, is there any cool to look at? I see the water system, which I think is still hooked up out front. That's not, it's always a little bit scary to see in a data center. Let's walk over to the front of the building. Yeah, this is where all the water pressure comes in for that. So, it's again, in case of fire, open door and pull, all right. I wanna touch everything. It's just kinda neat. It's kinda neat to see all of this just kinda sitting here, and it's so weird to be in here with everything just being quiet and silent. I mean, normally when I see this many racks here, I almost see like headphones because there's this drumming on of all the fans and stuff like that. But this is pretty cool. This is a neat little place. I just wanted to do a little walkthrough, show you what one of these places look like. This is not the most modern data center. We had another request we had. We tried really hard to show you guys a data center that was built, brand new, and a couple of years ago, has a flywheel instead of a UPS. Those are super neat data centers to see, and but unfortunately the legal just would not let us take pictures of it or do a video and talk. And we offered all kinds of promotion for the client, like just to let us do it, and they turned us down. But it's kinda neat to see just how a lot of this gets built out and what happens when it leaves. This is a giant Cisco box too. 18T Cisco box. Cisco Optical Network System. I still think it's a shame they cut all these. Yeah. That's a job security.  We're just grabbing the snips and be like, we're just cutting all the fiber in here. That's a shame. Except for the main one. That's a shame. I never found if they had that redundant. See, I don't know. Oh, for the extra feeds on here? Yeah. It's still pretty cool to walk through here. UPS, the A-bay, at least. But that's all. I'm not gonna touch any switches. I'll leave you all with that. This is just kind of a neat little data center tour and a little walkthrough. In case of emergency.


So, the project that I've been working with Dr. Young and Dr. Ganey has been to understand the role of macrophages and monocytes in nanoparticle interactions for tumor immunotherapy. And that's really important because I work in the field of immunogenering where we're trying to develop biomaterials that can actually modulate immune cell function. And this immune cell function is very important for how tumors grow, or grow shrink in our case. We want to actually develop nanoparticles that can shrink tumors. And so when I first thought about this idea, I thought about making a qualitative review, which is what I typically see, where we're then going through the literature and reading papers and then kind of summarizing what we've seen in the field. And I think those absolutely still have a place and what I'm trying to do is not to displace that in any regard because I think the qualitative perspective kind of reinforced literature is very important. However, I also began to notice as a student even that it's hard to read all of the papers in the field. And it's easy to kind of miss a new paper that's emerged or that may not come from a big lab or that may not have the right words on them. And so it's really hard to do that in a way that you just start kind of not paying attention after the first 200 papers, to be honest. And on the other hand, what was also interesting is that the field of nanoparticle delivery is very proliferative. There's so many different types of particles and it makes it so amazing where you can make. And so I wanted to do this type of systematic process to kind of approach both needs to actually be able to survey the entire literature to really get a sense of what the entire field looks like from like a big point of view for the purposes of making sure that I captured everything, which is very hard to do with how much literature comes out. And also to be able to try to develop trends amongst the different ways you can characterize particles and actually load them with different drugs and how they actually benefit different tumor types. And just this real genuine idea of how do we kind of do science in the world where there is just so many papers coming out daily and how do we keep up with that, how do we synthesize information at a pace that's on par with our publication and certainly how do we synthesize that information for any future translational capability. And I think that need is only going to come stronger. So it kind of feels great honestly to be doing some of the work that might be pioneering this kind of idea of thinking, at least in like a more traditional science where again this is the idea of systematically reviewing or mapping is more of something you've seen in the health sciences. And so how do we think about this for our engineering fields? I think doctors, Ganey and Yango remember where we kind of went from saying this is a systematic review to like, no, we're really mapping what's happening here. And so I think that things have evolved as we kind of realized like the complexity of the question we were actually answering that was based on getting to that first some of that first bits of data and thinking about what the real utility was and how to actually maximize utility of the platform. And so I think they helped in expanding how I thought about literature searches, the tools that we can use, and that was certainly really helpful for my students. They searched her papers and they can find papers way better than I ever can. Like I think I'm just going to ask them, can you just tell me if it's even out there? And I know for a fact now that I believe them a little bit more than I would have before that that paper doesn't or does not exist. Did they tell me there are six papers out here that has this? I'm like, that's pretty precise, but I know that they know how to do it. And then ideas of deduplification, deduplification. So how do you then say we've got 13,000 papers? How do we then search through those? How do we organize those files? How do we put them into databases? How do we search by title abstract or author or year publication date? So these were all things that librarians helped do that. I didn't actually realize we're going to be needed in the project and did not predict because, again, not my field, but having someone or people on the team who knew how to do that made it easier for me to focus on the scientific question. I think that's probably the final thing I'll say here is that I got to focus on the what are macrophages? How do I explain to you what macrophages are or what kind of nanoparticles are? Because I knew that I didn't have to worry as much about or I could. I had another part of the team that could help with the the management of data, the accumulation and kind of making sure that we had a thorough listing. Of every single paper that would ever fit here. Yeah, is that great? Yes, I really saw this as an opportunity to opportunity to engage students to kind of get them interested in research and have a community. And so after, you know, a whole summer of looking at thousands of papers in some students cases, we found that students were reporting that they felt more comfortable or confident reading papers. And in particular, scientific papers that they actually really understood how to search or use algorithms or is algorithmic appropriate word, but to be able to search through databases and find papers with an accuracy they hadn't felt before. So they felt more comfortable with those types of things. And to me, that also felt like a real value of this, which is that they were able to learn some skills that are now applicable to them, because one of the biggest challenges I would argue, especially for an undergraduate audience is like, how do you get them to feel comfortable reading scientific literature? How do you prepare them for graduate school? And that's certainly a tool they can use as they start, how they start identifying problems. And for the grad students, this is definitely true where they like are searching and they're using this now when I ask them to look for certain papers. And so what I do is I tell them, this is a really great activity for their first year grad students who are trying to learn the field that they can actually map out their whole niche area. And this is a great way to kind of add a little bit of quantitative data to their qualitative research or their qualitative review, especially now as more people are writing reviews. I tell them that this will help strengthen their search skills. A huge challenge that people face in terms of knowledge integration is trying to figure out what's important because there's so much information that it that and you don't know because you're kind of a novice, like the idea of finding information might be a little bit, yeah, everyone has Google, we all have PubMed. But how do you figure the trend? How do you figure what's more meaningful than others? And the thought process that it takes to define a question actually makes them do that. And it makes you figure out like, oh, we're the opportunities and we're like, oh, that's been done way too much. And like, let's try to like find a space where you belong. So there's a lot of thought processes that help train students. It's a bit independent in a way, so they can also do that without too much insight. And I think that as I continue this work and as I continue this work of Dr. Young and Dr. Ganey, that's one of the things we like to look into, which is to actually explicitly ask the questions of how did learning and confidence change and how do students who do this kind of program, let's say, of like learning how to actually search through databases, make criteria and try to apply them to problems. How do they then apply them later on?


My name is Dr Gareth Bentley, and I teach Critical Media and Communications in If-Cells, SOAS. I teach on FTPS media. I am a long-standing member of the If-Cells, SOAS department. I've been working here for 20 years. I did my master's degree at Glasgow University in media, did international relations, and I studied my PhD here in SOAS to finish that in 2013. I've always been interested in media and alienation. I've been studying it for at least half of my working life, and I now teach it. I did my PhD here at SOAS on journalistic agency and international foreign correspondence. The main topics on the module that I teach are theoretical but critical. From a theoretical perspective, I look at political, economic and structural Marxist approaches to the study of mass media, as well as contrasting that and that with pluralist approaches, psychoanalytic approaches to agency. What's really interesting about studying media today is that media is an extremely powerful structure and agency in the world. We are all structured by it, but we also have the ability to be active media participants as well, and that's what makes it so exciting. Some of the world's most powerful countries and corporations rely very heavily on media, and that's why I would say that studying media today is extremely important. Of course we study the most important media topics including news, advertising, film, television and the internet. I think it's a nice balance of theory and practice. What's amazing about studying media is that it's not just like a traditional subject, it's interdisciplinary, and it's a subject that we're all immersed in, we inhabit, we experience on an everyday basis. So it's got a nice combination of theory where you can test the theory against the practice, and if you like, come up with your own theory, such as complex agents. The purpose of the module is to teach critical media engagement and literacy and to encourage students to be active political participants. The best thing about this module is that it has a healthy balance of theory and practice. I argue that what makes media unique as a subject is that we're all interacting with it on a daily basis, especially in the context of digital media. Although the course has a certain amount of theory, we can excitingly apply the theory to our everyday experience and be creative rather than just passive receivers of media theory and information. The best thing for me about teaching this module is that over the 20 years or so that I've been teaching it, students often struggle at the beginning with the amount of theory, but by the end of the course or at least halfway through they often start coming up to me and saying how exciting and interesting and how they're able to connect the theory with their own experience and practice. And they literally say to me, I really see the media in a totally different way now and it's great.


Live from San Francisco, it's theCUBE, covering DevNet Create 2017, brought to you by Cisco. Hello everyone, welcome back to our live coverage from theCUBE exclusive two days with Cisco's inaugural DevNet Create event. I'm John Furrier with my cohost Peter Burris, who's the general manager of Wikibon.com and head of research for SiliconANGLE Media, which I'm with Susie Wee, who is the vice president and CTO of Cisco's DevNet. The creator of DevNet, the developer program that was started as grassroots, now full blown, Cisco developer program, now starting another foray into the cloud native open source community with this new event, DevNet Create. Welcome to theCUBE, thanks for joining us. Thank you John. Thanks for having us. We love going to the inaugural events because they're always the first and you know, being bloggers and media, you got to be first, first news, first block, comment, always first, and we're the only media here. So thank you. Thank you. So tell us about the event, you're the host and the creator with your team. Yes. How did this come together? Why DevNet Create? You have DevNet. This event is going extremely well. Tell us. Awesome. So yeah, so we have DevNet. We've had DevNet for about three years. It was actually exactly three years ago that we had our first DevNet zone, a developer conference at Cisco Live three years ago. And there we felt like we pretty squarely hit. We've had successes there. We've had a pretty strong handle on our infrastructure audience. But what we see is that there's this huge transition, transformation going on in the industry with IoT and cloud that changes the definition of how applications meet infrastructure. And so this whole thing with applications, what is an application? What is the infrastructure? The infrastructure is now programmable. How can apps interact? It opens up a whole new world. And so what we did was we created DevNet Create as a standalone developer conference focused on IoT and cloud to focus on that transformation. And a lot of industry trends kind of going on and moves you're making. It's company or you, Cisco is making app dynamics, big acquisition kind of speaks to that. But also there's always a natural progression for Cisco to have moving up the stack with software. But IoT gives you guys a unique opportunity with the network concept. So making it network programmable, infrastructure as code as some say in the DevOps world is the ethos. Absolutely. So you guys see yourselves engaging with the community and what are some of the plans and what's some of the feedback you're getting here at the event? So what we've done here at the event is that, as you've seen from the channel is that our content is 90% from the community. Maybe 10% from Cisco, 90% from the community because we believe it is all about the ecosystem. It's about how applications meet the infrastructure. It's the systems people are building together. And there's a lot of movement in developing these technologies. We don't know the final form of how an IoT app, like who's going to build the app? Who's going to build the users? Who's going to run the service? Who's going to run the infrastructure? It's all still evolving. And we think that the community needs to come together to solve this, to make the most of the opportunity. And so that's what really this is all about. And then we think it's actually involves learning the languages, making sure that the app folks know the language of the infrastructure folks. They don't have to become experts in it, but just knowing the language. Understand what part's programmable, what part's not. What benefit can you derive from the infrastructure? And then by really having knowledge of what you can get across and creating a forum for people to get together to have this conversation, we can make those breakthroughs. So just a clarification. You said that 90% of the sessions are non-Sysco or from the community and only 10% from Cisco. That's right. Is that by design? Is that? It is absolutely by design. So when we have the DevNet zone at Cisco Live, that's all about all of Cisco's products, platforms, APIs, bringing in the community to come and learn about those. But DevNet Create was really squarely for IoT and app developers. IoT app developers, cloud developers, people working on DevOps to look at that intersection. So we didn't go into all the gory details of networking, like we very much like to do, but we were really trying to focus on what's the value to application developers and what are the opportunities? Well, it's interesting because Susie that there are, there's a, we're in the midst, as you said, and it was pretty significant transformation. And there's a lot of turbulence, not only in business and how business conceives of digital technology and the world it's going to play. The developer world, cloud disk, cloud that, different suppliers. But one of the anchor points is the network. Even though the network itself is under, is changing in the midst of a transformation, but it's a step function. So, you know, you go from on the wireless go outside, one G to three G to five G, et cetera, that kind of thing. But how is the developer going to inform that next step function in the network, that next big transformation in the network? And to what degree is this kind of a session going to really catalyze that kind of a change? Absolutely. So, what happens is you're right. If something that we all know, all app developers know, and actually every person in the world knows, the network is important. The network provides connectivity. The network is what provides internet, data and everything there. That's critical to apps. But the thing that's been heard about it is it's not programmable. Like, you kind of get that thing configured, it's working now, you leave it. Don't touch it, don't touch it. The mind's working, it's still working. It is, it's wires. Or even if it's wireless, once you can get it configured, you leave it. You're not playing with it again. It's too kind of dangerous or fragile to change it. Because of the sensitivity to operations. Because of the sensitivity to operations. But now that the big change that's happening is the network is becoming programmable. The network has APIs, and then we have things like automation and controller-based networking coming into place. So you don't actually configure it by going one network device at a time. You feed these into a controller, and then now you're actually doing network-wide command. That takes out the human error. It actually makes it easy to configure and reconfigure. And when you have that ability to provision resources, to kind of reset configurations, then when you can do that quickly through APIs, you suddenly have a tool that you never had before. So let me give you an example.  You want to put out a new security system with surveillance cameras. You don't want to put that on the same network segment as your vending machines. You have a different level of security required. You could put in a work order to say- Unless you're worried, really worried about who's stealing from the vending. Unless you're worried. So what you can do now that it's programmable is use infrastructure as code, is basically say, boom, give me a new network segment. Let me drop these new devices onto it. Let the programmable network automatically create a separate network segment that has all of these devices together. Then you can start to use group-based policy to now set the rules that you want for how those cameras are accessed, who they're accessible by, what kind of data can come in and out of it. You can actually do that with infrastructure as code. That was not a knob that app developers had before. So they don't need to become networking experts, but now they have these knobs that they can use to give you that next level of security, to give you that next level of programmability, and to do it at the speed that an app developer needs. So I was talking to Steve Hosty earlier this morning, and he's from Red Hat. He's a lead developer. He's not a network guy.  I care about apps, and he's a developer. And he brought up something interesting. I want to get your thoughts on it. I think you're on to something really big with your vision, which is why we're so pumped about it. And he brought up an example of ecosystems, edges, and margins of the edge, of these, when they come together, creates innovation opportunities. And he used the example of data science meets cloud. And what he was using in particular was the example of most data people in the old days were data jocks, they did data, they did things, and they weren't really computer scientists. But as those two communities came together, the computer science saying, hey, I don't know about data, and the data guy's like, hey, you know about algorithms, I know about algorithms, so innovation happened when that came together. What you're doing here, if I get this right, is you're saying, hey, DevNet's doing great from a Cisco perspective, but now this whole new creative innovation world in the cloud is happening in real time. Bring them together, so best of Cisco knowledge to the guys who don't want to be next to that, can share information. Is that kind of where this is going? Yeah, that's exactly where it's going. Like same example, earlier in my career I was working on sending video over networks.  The same thing. So far, the applications and the infrastructure and IT departments have been completely separate. You would just do the best you can. It was the job of IT to provide it, but now suddenly there's an opportunity to bring these together. And it's again, it's because the infrastructure is becoming programmable and now it has knobs and can work quickly. So yes, this is kind of new ground and you're not going to, like things could continue the way they are, right? And it's okay, we're getting by, but you just won't be realizing the potential of the real, you know, kind of. Well, open source has clearly demonstrated that the collective intelligence of communities can really move fast and share. And it's now tier one, so it's, you see companies go public, MuleSoft, CloudAir and all this goes on and on. So now you have the dynamic of open source. So I got to ask you the question.  How do you see this next leg of the journey because you have the foray now with DevNet Create, looks good, you're really well done. What's next? What's next is going on and making the real instances that show the application and infrastructure synergy. So let me just give you a really simple example of something that we're doing, which is that Apple and Cisco have had a partnership. And this partnership is coming together in that we have iOS developers who are writing mobile apps. So you have your mobile apps, people are writing, we have iOS 10, your app developers are writing these apps. But everybody knows you run into a situation where your app gets congested on the network. Let's say that we're here in Westfield Mall and they want to put out an ARVR app and you want that traffic to work, right? Because if the mall wants to offer an ARVR service, it takes a lot of bandwidth to get that data through. But through this partnership, what we have is an ability, we have to use an iOS 10 SDK to basically business optimize your app so that it can run well on a Cisco infrastructure. So basically it's just saying, hey, this is important, put it in the highest QOS level setting and make your ARVR work. So it's just having these like real instances where these work together. I mean I used to be a plumber back in my day when I used to work at HP and I know how hard it is and so I'm going to bring this up because networks used to be stable and fragile brittle and then that would determine what you could do on top of it. But there are things like DNS, we know we hear about DNS, we hear about configuration management, setting ports and doing this to your point. I want dynamic provisioning or policy at any given moment, yet the network's got to be ready to do that. You don't want to submit a work order. You don't want to have to say, hey, can you provision port whatever?  This is what we're talking about when we say programmable infrastructure, just letting the apps interface with network APIs, absolutely. And I think that you heard earlier that with CNCF, the Cloud Native Computing Foundation, just announced CNI. So that what they're doing is now offering an ability to take your kind of container orchestration and take into consideration what's going on in the network. So if this link is more congestive than that, then make sure that you're doing your orchestration in the right way, that the network is informing the cloud layer, that the cloud platform's informing the network. So that's going to be huge. But do you think, I'm curious, Suzy, do you think that we're going to see a time when we start bringing conventions at layer seven in the network? So we start to parse layer seven down a little bit. So we actually think, so developers can think in terms of some of those higher level services that previously have been presentation. Is that something that, are we likely to see that kind of a thing? So as the pain of the network starts to go away and the explicit knowledge of layer one through six become a lot less important, are we going to see a natural expansion at layer seven and think about distributed data, distributed applications, distributed services becoming more coherence to how that happens on an industry-wide basis? What do you think? Yeah, so let's see. I don't know if I have a view on which layers go away or which layers- Look at the knowledge, the focal point of this. But the knowledge, absolutely. So it comes into play. And what happens is like, what is the infrastructure in the internet of things? Things are part of your infrastructure. That's just different. As you're going to microservices, applications aren't applications, they're being written as microservices. And then once you put those microservices in containers, they can move around. So you actually have a pretty different paradigm for thinking about the architecture of applications, of how they're orchestrated, what resources they sit on, how you provision. So you get a very new paradigm for that. And then the key is to- Is there inherently networked? That's right, that's right. It's all about connectivity. It's all about, they don't do anything without the network. And we're pushing the boundaries of the network. These aren't function calls over memory, like we used to think about things. These things are inherently networked. And we've always, we know we have networked SLAs and service levels and whatnot. But it sounds like we have, I was wondering, here at this conference, our developers started to talk about, jeez, I would like to look at Kubernetes as a feature, a lower level feature in layer seven. Where there's a consistent approach to thinking about how that orchestration layer is going to work and how containers work above that, because I don't have to worry about session anymore. I don't have to worry about transmission. Absolutely, absolutely. That goes away. So give me a little bit more visibility into some of that higher level stuff where really the connectivity issues are becoming more obvious. Absolutely, and an interesting example is that, we actually talked about app dynamics in the keynote. And so with app dynamics, what kind of information can you get from these bits of code that are running in different places? And it comes into where we have the Royal Bank of Scotland who's saying, what's my busiest bank branch? Or people are doing mobile banking in the country. And they're like, well, how do I answer that question? And then you see that, someone has their mobile phone, then you actually break it down to how is that request, that API, how is that being kind of operated throughout your network? And when you take a look, you say, okay, well this called this piece of code that's running here. This piece of code used this API to talk to this other service, to talk to this other. You can map that out, get back the calls of, hey, this is how many times this API has been called. This is how many times this service has been called. This is the ones that are talking to who. Then they came up with the answer, saying that our busiest bank branch is the 9 a.m. Paddington train station. And that's a great example, because now you have, you gain visibility into where the dependencies are, which even if you don't explicitly render it that way, starts to build a picture of what the layers of function might look like based on the dependencies and the sharing of the underlying services. That's right, and that's where you're saying like, what the infrastructure just gave me business value in a very direct way, how did that happen? That's a huge opportunity for Cisco. So it's a big collapse. Well let's get in the studio and let's break down the Kubernetes and the containers. There's Docker's here, a lot of all the folks are here. We had also Abby Kearns, the executive director of Cloud Foundry. We've had the executive director from the Cloud Native Compute Foundation. Dan is here. A lot of folks here in the industry kind of validating your support. Sun used to have an expression, the network is the computer, but now maybe Chuck Robbins should go for the network is the app or the app is the network. I mean, that's what's happening here. The interplay between the two is happening big time. It is happening here, yeah. So just every element, every piece of code, what we saw is that this year, developers will write 111 billion lines of code. You think about that. That we know about. That we know about, there's probably more. And all of that, you're right. These are broken up into pieces that are inherently networked. They have data. It's all about data and information that they're sharing to give interesting experiences. So this is absolutely a new perspective. Congratulations on your success. What a great journey. I know it's been a short time, but I noticed after our in-studio interview when you came in to share with us the show as a preview, Chuck Robbins retweeted one of the tweets. He did. So I got to ask you internally, Cisco, I know you put this together kind of as a entrepreneurial inside the company and had support for that. What is the conversation you have with Chuck and the executive team about this effort because they got to see a clear line that said that the value of the network is creating business value. What are some of the internal conversations? Can you give us a little bit of color without giving away all the trade secrets? Yeah, well internally we're getting huge support. So Chuck Robbins checks in on this. He actually has been checking in, saying, how's it going?  I heard it's going great. Chuck did a couple days ago. Okay. And then Rowan today. So yeah, so we have a lot of conversation. Rowan's a CUBE alumni. Chuck's got to get on the CUBE. Rowan's been on before. Yes, so they're all kind of checking in on it. We have the IoT World Forum going on in parallel in London, so otherwise they would be here as well. But they understand- There's a general excitement. There's a huge excitement. This is not a rogue event. This is not like a rogue event. It's not, it's not. And what happens is they also understand that we're talking about bringing in the ecosystem. It's not just a Cisco conversation, it is a community. You're doing it right. You're not trying to take over the sandbox. You're coming in with respect and actually putting out content and learning. Putting out content. And really, it's all about letting people interact and create this new area. It's breaking new ground. It's facilitating a conversation. I mean, where apps meet infrastructure, it's controversial as well. Some people should say they should never meet. Why would they ever meet? Well, you know, we do a lot of shows. We do a soundpeater that, you know, we were at the first Hadoop Summit, second Hadoop World with Cloudera when they were a small startup. Docker's first event. KubeCon's first event. We do a lot of firsts. And I got to tell you, the energy here feels a lot like those events where it's so obvious that, you know, okay, finally, programmable infrastructure. Well, I'll be honest.   It was a risk. So the question is, will it appeal to the app developers? Will it appeal to the cloud developers? You know, will it appeal overall? And I'm very relieved and happy to see that the vibe is very positive, right? So people are very receptive to these ideas. Well, you know community. The more you give, give more than you take has always been a great philosophy. So I'm always a little paranoid and nervous, but I'm very pleased because people seem to be really happy. There's a lot of action. There's a lot of ideas with Docker stickers on them. There are. There are, yes, yes. We have the true cloud IoT. We have the hardcore developers here and they seem to be very engaged and really embracing and thankful. We've always been covering DevOps again from the beginning and cloud native is to me, it's just a semantic word for DevOps. It's happening. It's going mainstream and great to see Cisco. Congratulations on all your work and thanks for including theCUBE in your inaugural event. Thank you. The vice president CTO of Cisco's DevNet for here for the inaugural event, DevNet Create with the community, two great communities coming together. I'm John Furrier with Peter Burris. Stay tuned for more coverage from our exclusive DevNet Create coverage. Stay with us. Hi, I'm April Mitchell and I'm the senior director of strategy.


All this month, get up to $50 off any Android smartphone from Appalachian Wireless. Any Android you desire. All $50 off with two-year agreement. Better service, bigger savings, that's today's Appalachian Wireless at East Kentucky Network Company. 29-year-old Eric Price of Hatfield Bottom and Mate One was taken in the custody Saturday. He was charged with leaving a loaded handgun behind in the restroom at the South Williamson Arby's Restaurant. Her document said Price removed the gun from his holster and placed it onto the toilet at Arby's Saturday afternoon and left without taking the gun. The gun was later found by another patron and turned in, and Kentucky State Police was called to respond. An arrest citation said management at the restaurant told police that Price called about the gun before troopers arrived and was, quote, verbally aggressive to an employee. The citation said a review of surveillance footage showed that about 50 people, including some children, came into and out of the store between the time Price left the weapon behind and when he came back to get it. Price was charged with misdemeanor wanton endangerment in the second degree and was lodged in the Pike County Detention Center. He's since made bond in his due back in District Court in October. The weapon remains in evidence at KSP Post 9 in Pikeville. The investigation is ongoing. In South Williamson, Chris Anderson, EKB News.


I'm with Amos Miller from a company called Glidance and he's going to show us a really terrific accessibility tool that he's the co-founder of the company Glidance I believe, correct? I am the CEO and co-founder of Glidance, correct? So talk to us about what this device can do. All right, so Glid is, first of all, I'll just explain that Glid is the first self-guided mobility aid for people with site loss. So really developing a solution that addresses the needs of people who need to get around, who are unable to see, and providing people with a, really with a third alternative. Today people use guide dogs and canes and we know that for some people they need some more help, especially people who lose their site later in life. So that's where, that's the origin of the work that led to the development of Glide. So why is that important to people later in life more than if they'd lost their vision earlier or if they were born blind? That's a great question. I mean people who are born blind build a mental map of their world and build the skills and capabilities to navigate in the world from a very early age. And by the way, I'm not saying that people who are born blind would not benefit from Glide. I mean I'm sure that they will enjoy the benefits that Glide brings in terms of this really clear, very clear and guided navigation, which we'll talk about in a minute. But people who lose their site later in life sometimes are reluctant to take those steps to develop the skillset that's required to be an effective cane user and get out and about independently. So maybe it's like learning a language later, you're always translating. If you're born speaking two languages, you spoke two languages, you're not translating. Maybe it's something like that. It is, yes. And it's also being alert to the information that you get from your environment and being able to orient based on that.  And I think one of the benefits that Glide can bring to that is that really very quickly because it actually guides you, a person who knows there, let's say, how to get around there, the neighborhood would be able to take a walk very quickly with it. So I'm going to describe what this looks like to the audience because there's audio only listeners as well. And heck, there's blind people listening as well. So he's got a... By the way, just for the audience, I am actually blind myself. It would be just worth explaining that I lost my sight in my 20s from retinitis pigmentosa. I've been at this kind of quest to figure out technological solutions for navigation for quite some time, maybe three decades now. I developed Soundscape at Microsoft for a number of years. Some of your listeners may be familiar with that. And really, I was very intrigued to explore a solution that can help people who may not be those confident cane users. So let me explain... Why don't you describe what this looks like? Let me explain what it is, yeah. So I'm holding in my hand Glide. I'm holding the handle. The handle stretches 45 degree to the ground and then it rests on two wheels. And the wheels are effectively what guides you. So I will move the device forward and the wheels will start steering left and right to guide me, to keep me on the path, to take me around obstacles, and just guide me to where I'm going, whether it's to a door or a dropped curb or all the way to a restaurant that I set up on my navigation app. So it can respond to navigation from the phone? We're working on those integrations so that you can set your navigation on the phone and then it will guide you to that destination. So you're going to be the first one to full self-driving? Yeah, well it is self-driving but it's also, and that's a very important point, it's also actually the wheels are not motorized. So it's not going to pull me around. Oh, okay. I'm moving it forward, I'm pulling it back and the wheels just steer left and right. The wheels can also apply the brakes to stop me, but the agency remains with me. I'm the one that's in control, I'm deciding do I want to go back, do I want to go forward, do I want to go fast. I can even twist the handle to turn left and right. Okay, I can see the wheels turning when you do that. And when I do that, if there's an obstacle, the wheels, the device will not let me turn until it's safe to take that turn. Okay, so a lot of what a C9 dog does in a way, right? Yeah, inspired by that, but we also have definitely inspired by guide dogs. I want to agree to that point, I mean I'm a guide dog user myself, but we also have speakers in the device, so the device will be able to use voice to give you some more information about the environment that you're encountering. The speakers up in the handle? They are in the handle and you'll be able to also plug it into a headset if you prefer that. Yeah, the speakers are here in the handle. There's also haptics in the handle so that the device can warn you to slow down or tell you that you can speed up or that there's a sharp turn coming to the left or the right. So really working together with the device. So there must be a camera on this somewhere? Yep, right up here. So they up in the handle? So we have, there's quite a lot of sensors on the device. Along the bottom there are short range sensors that really help with the obstacle avoidance and local path planning, and then we have a camera right at the top of the handle so that the device has a nice view of all the environment, the space in front of you and help to determine a good safe path to guide you on. Okay, yeah, I can see how you need to know the obstacles on the ground, but also the situational awareness of voice type. Yeah, and the camera can, I mean we're working on line of sight targets for example so that the camera can detect let's say a door and then work out a path to that door and Glide will guide you all the way there. Very very cool. Yeah. So this is an active product development right now, is that correct? Correct, yes. Okay, and do you have a vision for when this might be a product? I do, of course. One that you're willing to tell me about? Yeah, sure. No, I mean we are really expecting to kick off a beta program this year, as in by this late spring or summer this year, I mean we'll see how successful we are in hitting those dates, but the beta program will really give us an opportunity to get devices into people's hands, start to get a lot of feedback from daily use and prepare to launch the product after that. I heard from one of your other compatriots here that you were having a guy demo at who was blind and he started jogging with it, is that true? Yeah, he went a little fast for running in CES, I don't think he saw how crowded the environment was in front of him. Did he run into anything? He did quite well, right at the end somebody just cut across him at almost equal speed and they had a brush, let's say. But no, I mean the device is definitely working to support good walking speed, I mean I say to people we call it glide because when you walk with it you glide, you know, and that's really the experience that we want to empower of this. It's a very smooth experience, you walk, you're confident, you're upright, you're heading where you want to go and the device keeps you on a good steady pace and you feel empowered and confident. Very good. Do you think we can do a demo of this? We absolutely can do a demo.  We can do the demo at the back, yeah. What do you think? Where are my demo guys? We might need to do a cut because we are not quite organized. We'll do a cut and then we'll move. Okay. Okay, so I'm going to close my eyes. So you've got the device in your hand? I do. Okay. So let's get it to stand first, so now it's sitting down. Okay. It's a nice angle here. You hold it in this hand and when you pull it down to angle, it will be ready to go. It'll get on its legs, on its feet, on its wheels, you'll feel it and then when you walk, just feel where it's going and follow it. Okay. I'm not going to follow directions. Don't let it run away. It's not going to tell you anything. If it starts going to the left, just go with it. Don't kind of let it hang on the left by itself. Okay. All right. You'll get it. Here it goes. I'm closing my eyes now, tilting the back. Oh, there, I felt it. All right, I'm just going to start walking. I have my eyes closed. Oh, it wants me to go to the left a little bit, a little bit to the right. Oh, I can feel it gliding me, go a little bit to the left here. I'm hoping somebody will tell me if I'm going to run into anybody. Oh, oh, I've got to go this way. It just stopped me. I wonder if Steve's standing in front of me. I bet you anything, he just tried to stop me. All right, I'm still going. This is pretty gentle. I can definitely feel just us. Yeah? Okay. Steve wants me to try turning around.  All right. Oh, okay. I'm turning. It's letting me turn. It's letting me do a sharp turn. I'm still going around and I want to start going straight now. Oh, I'm going to go right, left, a little bit jerky back and forth there. All right.  There it goes. I don't know if I'm going to run into anybody here, but so far, so good. Yeah, I can definitely see. Oh, there we go. There we go. I don't know if I went around somebody. It's interesting to do this and not know whether I'm just missing people or whether I've just, I'm completely alone here. I can't actually tell. But I think I'm going to stop here and turn around again. All right. It wants me to turn around. Should I go back anymore, Steve? Go back. Walk a little bit more. Yeah, this is very natural. You can definitely see how this just feels really comfortable. I don't think I've hit anybody yet. All right. I'm going to stop. So I'm going to bring it up at a high angle and it just put itself to sleep. There we go. Yeah, yeah, that was great. All right. Let's close it off here. All right. I'm going to say goodbye to you here because we haven't actually conducted that work. That worked really well. That was very, very interesting. I understand exactly what you were describing. I could feel what it wanted me to do. Now it makes sense. Yeah, it really does. It really does. Thank you very much. So one more time. If people wanted to learn more about Glidance and the guide, where would they go? So the best thing to do is to go to glidance.io, glidance.io.gli, d-a-n-c-e, .io. They can register. We are taking registrations in advance of our pre-order program and they can just register for more information. And we really look for feedback. We look for thoughts. We look for people who want to talk to us and engage. And Allison, I just want to say thank you very much for having this discussion with me. My pleasure. My pleasure. Thank you very much. Bye-bye.


So, today I will be presenting the evolution and scaling of feature store at Uber. Basically, I'm Dhupya Nagar, I work in feature engineering team at Uber and we manage the features, how to scale them, how to serve them, the feature store architecture in details. Before going into all the details, I would be going into the different terminologies I'm going to use in this talk. So, this is my team. We basically manage everything around feature infrastructure at Uber. So, this is the agenda for today's talk. We will be learning what, machine learning, what a feature is, how the features get generated, how do we store them, how do we share them across different models, and some features store operations like offline serving, online serving, model training, feature transformations. We'll go into the details of all of these. So, this is a machine learning model. This is basically a very high-level overview. We have a data, we have a model. We use that data to train the model and then we serve predictions for this model. And then, once we serve whatever data we get in return, which goes back into the loop, and then models get retrained with the new data. And the online and offline here is the data which is used at the training time. We call it like offline serving. So, if I say offline serving or serving at training time, they are the same thing. Online serving is basically serving at the prediction time. So, this is event data and features. So, this is basically what event data looks like. You have certain orders coming into the system. We basically do some aggregations and transformations. And then, we generate features based on that. So, for example, like, in the lift table, we have two orders received on nine. And then the right table denotes how many orders you received on that particular date. So, that's the difference between the raw data and what features eventually look like. At Uber, we have two primary type of features. One is, like, batch features and real-time features. Batch features, which has very low granularity. For example, if you want to say, hey, how many orders this store received in last one day or last 30 days or last 15 days, that's a batch feature, which is, like, large data processing. We process all this data on a daily basis. And then we store this data. And real-time some more, as the name suggests, it's a real-time information. For example, what is the activity happening in last one minute or two minutes or last five minutes, one hour, anything which is less than a day, up to a granularity of a minute. So, it's basically continuous stream of data which we ingest into our stores and so forth predictions. Feature structure is, this is how we basically store features. The reason to do this in a way is we have, like, tens of thousands of features and managing each feature is difficult. So, we basically group them probably at a different entity level. So, an entity can be anything, basically an Uber entity. For example, stores, eaters, orders, different things. So, pallet feature store. This is what we call Uber's feature store. So, feature store is not just limited to serving and collecting the data. There are a lot of other things we do as part of feature store which is, like, making sure features are shared across different models, automatic feature selection, managing the streaming, ETL pipelines, data monitoring, data quality, feature transformations. All these are also part of this huge pallet store or feature store infrastructure. These are just some numbers, what we do and what we manage. So, we have, like, tens of thousands of features. We have hundreds of million of QPS we serve. We serve at, like, P99 is less than 10 milliseconds. So, SLAs are pretty tight there. Thousands of feature pipelines, hundreds of terabytes of data per training for a single model. Now, to manage the scale, moving forward, I'll just go through different scalability problems we have seen and how we overcome those challenges. So, this is just a very high-level architecture. I have abstracted out more details from here, but this is basically what it looks like. We have streaming and aggregation jobs. Those jobs will get real-time data, perform the aggregations. Then the data will be ingested to Kafka. From there, we have an ingestion service, which will write this Kafka data into Cassandra. And we also have a high ingestion, which will write the same data to a high table. And for batch features, we have, like, large aggregation queries. So, those compute jobs will run. They will write the data to high. And from high, we will basically disperse that data back to Cassandra. So, this way, we make sure that both batch and streaming data remains available in both offline and online stores, from where we serve it for training application and inference. So, problems at scale, we divide these problems into four parts. Feature onboarding, offline serving, online serving, and online offline consistency. Basically, how do we onboard hundreds of features with such a small team? How do we perform these large joins? How do we serve hundreds of millions of QPS and manage the infrastructure around that? What is offline, offline consistency, and how do we resolve these problems? So, let's talk about feature onboarding. In order to onboard hundreds of features, the first thing we realize is that we need to reduce the intervention from our team and make the platform as self-service possible. So, what we have done to do that is we have basically automated everything in this architecture. So, what user can do whenever they want to get a new feature or a feature group, they simply write a JSON config. So, they will update the JSON config, and then we run some validations on these JSON configs that everything is correct, and it is not breaking the existing stuff. And once all those checks are done, this JSON config will go to production, and this is what it will do. So, this JSON config will basically trigger a set of jobs. So, it will trigger the ingestion jobs, it will trigger the dispersal jobs, it will trigger the data quality test, it will also trigger monitoring, alerting. So, all these things will basically be automated. The green ones are which are basically automated, and the yellow ones which are mostly self-serve. So, for example, once your feature is ready, you have trained your model, then you can create your inference server on your own, which you can just start sending prediction requests. So, you don't really need much from us at the time of batch feature onboarding. So, this automation basically allowed us to reduce our overhead for our team and basically give a better customer experience to our users. The second problem I'll go into details is offline serving. So, what is offline serving? Basically, this is a very high-level overview of what offline serving looks like. So, we have certain features, and we have base data. Base data is also like your labels and compute configs because we need to run SparkJob to perform all these large joins. So, we take all these information as an input. We pass it to the orchestrator, which eventually starts a SparkJob. This SparkJob will run some set of joins basically. It will join the base data with your features, and then it will generate a training data set. And that training data set is used by the ML models. So, this is what it structure looks like. Now, the fundamental problems with offline serving is these, like how to speed up the joins? How do you do a cascading join over hundreds of terabytes of data? How to reduce out-of-memory Spark errors? How to make it easy for users to write optimal Spark configs? And how to reuse computations? Even when we do all those optimizations, some jobs do fail. How do we make sure that... How do we basically reuse all the computations which are already done in the past and not do them again? So, there are four fundamental optimizations we have done for offline serving infrastructure. The first is batching. So, in order to understand batching, we need to go into the details of Spark sort mode join. So, when we join two tables, basically Spark will do a lot of data shuffling, and then it will make sure all the similar data are available on same executors, then it will perform a join between them, and then it will return you the output. So, when you perform a join for a large data set, these kind of jobs become more failure-prone, and it's easy to get like... Your job will be either out of memory or out of disk. So, what we have done is we have done a batching optimization. So, let's say you want to perform a join between 30 days of data, and it's too large to handle. So, we can divide this data into smaller batches, and then we just process these batches one by one. We can test on these batches, fine-tune the configs for these batches. The jobs become more manageable. The next thing we have done is auto-config. What we have seen in the past is that it's not very easy to configure a Spark job correctly. The reason for that is, basically, you need three things to run a join operation. You need feature, you need the labels, you need the compute configs. In order to write the correct compute configs, what you need to know first is how does Spark operate, and what are the Spark configs which can actually impact your job, for example, like how many shuffle partitions to have, how many cores to assign per task, how does Spark handle shuffle data, how does it clean them up, all these things, and then you can write optimal compute configs to run the job. So, we have seen in the past that a lot of our customers are not able to write these configs correctly, hence a lot of their jobs failed due to this. Now, what we have done is we have written an auto-config module. What auto-config module does is it takes all these input parameters, and it will look at your features, it will look at your labels, it will figure out how large data sets you are joining at the end, and it also knows what kind of code we are performing, or what kind of operations we are performing. Based on that, it will modify the compute config on-flight. If it thinks that these compute configs will not work, and you need more resources, it will automatically update your compute config to have more resources to make sure that your job runs successfully. And if it thinks that the compute configs you have provided is a lot, and you don't need as many resources, then it will basically reduce the resources to make sure that the resources are optimized. The next thing we would go into the details of incremental join. So we have seen batching before. So when we basically perform a cascading join, so this is what your join looks like. A model can be trained on multiple features. Each feature can belong to a different table. So we basically get all these tables, and then we join them with your basis or label data set. So what happens is every time Spark joins these tables, it will shuffle both the tables, then return the output. It will reshuffle output and feature table 2, then again do the join, and then returns the final output. And this operation happens for every batch. And then this is what it looks like. Once it has done all these operations for every batch, we do a union at the end, and we return the join data set. Now what is the problem with this? The way Spark works is it will not delete the data until it has the linear job data in the job, which means that until the union is finished, it will keep all the shuffled data around in either memory or disk, and this requires a lot of data to remain in your memory or disk. So you need a lot of resources to perform the join between these jobs. What we have learned to overcome this issue is we basically removed this union. This is what we call incremental join or incremental materialization. So instead of keeping all these shuffled data, we simply persist this data per batch, and we specifically go and delete all the shuffles. So what basically you can do is the only resource you need to run a large join is as much as a batch. The downside of this is that your job can be slow because now we are adding this extra step of writing to HDFS, but it does make large data set jobs more reliable. We anyway write this data to HDFS, and I'll go into the details why we do that at the moment, but this basically allowed us to run our jobs more reliably. You can perform literally a join worth one year of data and that join would still succeed. The next optimization we have done around join operation is resume job. So what is resume job? So let's say you are performing a join with a large data set worth like a month or three months, and your job went halfway through, and then it failed. It can fail for multiple reasons. External dependencies, power configuration, someone else took your disk which you were using, hundreds of reasons. So what we provide a functionality to resume the job.  You can retry the same job, and we keep this data, I think as I showed, we persist this data in HDFS, and then you can just resume your job from where it actually failed. So we keep these checkpoints. So this allows us to basically not recompute a lot of data if not required. So this is further optimizations we have done to make sure or to run our feature preparation more reliably. Next we will look into the online serving. So this is what the online serving look like. Basically you have whenever you open Uber Eats app, you will see some restaurants, and for each restaurant or your recommendations, there are like tens of models which are being used, and each of that model is using tens of features. So this fan out increase exponentially for us. That is what causes hundreds of millions of QPS in feature store. So this is a high level overview of our online serving. Probably I think this is literally like every standard serving diagram. So you ingest the data and the data is being served. So we will look into the optimization on two fronts. How do we optimize the ingestion and how do we optimize the serving? Let's first go into the feature ingestion. So there are like, here we have listed four things, but all the dispersals, dispersal is basically reading data from highway and writing to Cassandra or any online store. And caching is caching the data during online serving. So we'll go into the dispersals one by one. First thing is efficient dispersal. So we basically utilize again Spark to read this data from highway and write to SS tables. Now, if you just run a Spark job and write this data to SS tables, what ends up happening is every Spark executor will write all these data to every Cassandra. Sorry, every Cassandra node. Hence, this causes a lot of small files, data fragmentation issues, and many more. So we basically rely, what we do is we have basically mirrored the partitioning strategy of Spark and Cassandra, which allows us to write data to specific SS tables and then only not have a lot of fragmentation within our system. The second part is delta dispersal. So the delta dispersal is basically whenever the new events are coming in, what our users do is they take the snapshot of the data and then for every day, they would basically, for every day, they will take the snapshot of data and they will write it to Cassandra and disperse it from highway. In delta, what we can do is instead of writing everything or the full snapshot to Cassandra, you can only write the data which actually changed. So in these two tables, you can see that we don't really need to disperse the data which happened on day two and day three, part of data on day three, and we can only write which actually changed. So this reduces the size of dispersal significantly. So right now we do this on a row basis. So if there is no change in your row, we would basically disperse it. Another optimization which is on the same lines for data dispersal, which is in the pipeline and we haven't really started on it, but we are also planning to do it at a column level changes. So what ends up happening is when you aggregate the data over a long period and there are a lot of different types of features in the same group, every row will more or less change every day hence when you do the column level changes and dispersals that is more optimized as compared to just row level differences. The next thing is selective dispersals. So we have an offline store which is high and we have an online store which is Cassandra. So what ends up happening is not all the models want all the features available in every store. So sometimes we disperse the same feature group into multiple clusters due to isolation or different SLA requirements. And at that point we don't want to disperse everything everywhere. So we allow doing selective dispersals. So what you can do is you take certain features and instead of dispersing everything you can only disperse the features you are actually using in your model. So this optimizes the dispersals, reduces the dispersal size significantly and it also optimizes storage cost. The last part in online serving optimizations is caching. Caching is more or less straightforward. I think we have local cache, redis cache and Cassandra. So if your data set cardinality is low, for example if you are using city level features where we only have like few thousand cities you can just locally cache it. Local cache mostly comes for free because we use the memory of infinite servers. If you have hundreds of GB of data probably like a store level feature then we use redis and if it is a eater level feature or maybe a driver level feature where we have millions or billions of rows in that case we use Cassandra. So it's basically the local cache and redis is much, much cheaper than Cassandra so this helps us in optimizing the cost for Cassandra. Online offline consistency. So let's first understand what is online offline consistency is. In this diagram there are like batch features and real-time features. So when you train a model and you give some data which was, let's say you train a model based on yesterday's data. Then when we serve the model we also need to provide yesterday's data. Then only the expectations of the model will match with the data. And the same thing happens for real-time as well. If you train the model based on last minute's data then at serving time you also need to serve last minute's data. So let's understand this with an example. So let's say you train a model and you wanted to predict how many orders a particular store will get today and you created the model which uses data from yesterday. So September 2 to September 3. Now when your model is already trained it is ready for serving then this is what happens. So you are serving on September 9 and the data which the model is expecting is from September 7 to September 8. But we cannot serve September 7 to September 8 data because we don't have it. It takes some time for data to get ingested in the pipeline. There are some ingestion delays or the large data sets can take few hours to get fully ingested in Cassandra. So instead we serve basically what we have available which would be previous day September 6 to September 7. So this creates a consistency issue between online and offline models. Now how do you resolve this? One way to resolve this is just train your model with probably further old data. Instead of training with yesterday's data you train with day before yesterday's data. But with some models it's not always possible and they want more fresh data for serving. So it's more like compromise between if you want higher consistency or if you want more freshness. So for example if your model is doing a fraud detection then you want more fresh data and you can potentially compromise on consistency. So to do this we allow users to get time shift features. So when we serve the features we take the feature name as an input and you can also give us how shifted data you want in time. So for example time shift zero which is basically you will just say give me previous days data and then we can send the time shift zero. Day before that it would be time shift one and time shift two and so on. You can get either of these you can actually request all these time shift features. It works the same for both batch and real time feature except in real time feature this time shift is in minutes instead of in days for batch it's in days so you can just say hey give me the activity of x, y, z for last minute or for last five minutes or for last ten minutes and you can keep shifting and then based on all those parameters you can make predictions. So that's how we basically resolve online offline consistency and the model owners can make a decision what they want to do or what they want to have more. So feature metadata and data quality as I said apart from online offline serving we also utilize Uber's data quality framework to make sure that all the feature pipelines are running on time the data which is being generated is 100% correct how to detect failures early in the process how to enforce accountability who owns this data who to send alert when something fails. So there are four components of feature metadata and data quality the first thing is unified metadata so we have a metadata store where you can find every information about a feature like who owns this feature how this feature is being generated what is the type of feature where is this feature stored where is it available online, offline what are the pipelines this feature is being generated lineage what all pipelines this feature these feature pipelines depends on if it's offline feature you will see this kind of lineage graph where you will see all the tables it depends on if it's a real time feature then you will see a set of jobs that this is the job which is generating this set of data the tiering of a feature is basically based on the use cases for example if a particular feature is being used in a very critical use cases then it would be tier one we have from tier one to tier five where tier five is the lowest criticality and based on what tier your feature is we register different data quality tests again this framework is automated the tiers get updated automatically so for example you have a feature group which is being used in a model which is a tier one model so all the tiers will get updated automatically based on what type of model is using those features so they will get downgraded automatically they will get upgraded automatically and whenever this upgrade or downgrade happens the tests will also change so for tier one features we have more higher SLA requirements the SLA will be more strict for example it has to be fresh, it has to be complete the data loss should be zero the duplication should be zero and all those kinds of SLAs need to be in place but we don't enforce the same SLAs or same tests for low tier probably like tier four, tier five feature groups so apart from that like metadata and data quality what we also have is feature store search due to having too many features it becomes difficult to search these features or manually navigate to these features so what we have is a search store you can basically search based on your entity feature group, features, specific name you can search based on their databases sources, a lot of different key features and these are some of the areas we are exploring basically embedding vector data type support till now we did not have any vector data type support for feature store so we are adding that better storage I think in general in feature store the way the data doesn't really change very frequently once the data is ingested in online store there are no updates possible so it's mostly immutable so we are looking into potential solutions of immutable key value stores which can be used to lower down the cost of Cassandra or if it can be replaced with something else feature versioning feature versioning is again part of the same effort what ends up happening is when you store some data in any key value store you need to assign some sort of details and if your feature pipelines fails due to any reason the data will basically time out and then you won't have anything to serve instead what we would probably prefer to have is a versioning system where one version of the feature remains until the new version is available so we always have something to serve then feature intelligence we do have automatic feature search we want to enhance this further and near real-time features which is aggregation infra and backfills by aggregation infra we mean is so right now we rely on external frameworks to do the aggregations for us for real-time feature like we have Flink, Athena you can write any custom job in Flink you can generate the data to Kafka we want to provide aggregation framework as part of feature store which can allow you to do basic aggregation with JSON configurations and backfills again it's not we do have backfills for batch features but it's not very straightforward to have backfills for real-time features so that is another work in the pipeline we have that's all thank you any question? could you talk more about how the recent advances in vector space and embeddings tie into your legacy feature store systems sorry, I did not heard the last part of the question I just want to know how the recent vector databases and vector embeddings and their popularity given their popularity how they tie in with legacy system like you demoed today okay, so I think the way they tie with the legacy systems is when we added like embeddings we also had to add support for embedding types so till now like all the stores we are using we don't really have support for added type data types and most of the embeddings are like set of arrays so what we were doing till now is taking them as a string input and at the time of surveying we will basically transform the string into a list of parameters so that is not very optimal and that is also what causes a lot of latency issues at runtime so now as we see that more and more use cases want to use embeddings so what we have seen a shift is more efforts moving into the direction of some sort of embedding store where people can reuse embeddings they can store the embeddings in their native format so we are exploring multiple different databases, vector databases we already have an in-house solution called CIA to serve embeddings so we are exploring all those options to create a new storage system for that I think that's it, no more questions okay, thanks everyone