Live from Seattle, Washington, it's theCUBE at Tableau Conference 2014. Brought to you by headline sponsor, Tableau. Here are your hosts, John Furrier and Jeff Kelly. Okay, welcome back everyone. We are here live in Seattle, Washington for Tableau's User Technology Conference. Data 14, it's theCUBE, our flagship program. We go out to the events, extract the signal from the noise. I'm John Furrier, my co-host. Jeff Kelly, analyst with our big data group at Wikibon. And our next guest is Scott Barnison, business development manager at Amazon. We'll be at the Amazon re-invent conference coming up. It's going to be awesome. Great to have you back on theCUBE. Good to see you again. Thanks, John. So, Amazon, born in the cloud is now the buzzword of this year. Been around for a while, but truly you're seeing the acceleration of Amazon's model of cloud-based computing with all the integrated stacks really powering the developer market. Certainly now, even in the enterprise, people talk about Hybrid Club.  You're doing some BizDev here at Tableau. Perfect fit for you guys, right?  So you're here doing some BizDev, shaking hands. Smiling a lot, shaking hands, yeah. So the relationship with Tableau is an important one for us. They're growing really well. And most importantly, if you heard yesterday in the keynote with Christian, he talked a lot about experimentation. And experimentation for us is one of our core messages. If you can reduce the friction required to get your hands on the infrastructure so you can try new things and if you can lower the cost of failure, you'll increase the rate of experimentation. We believe there's a direct correlation between experimentation and invention. Invention's super important for the history and the lifeblood of companies. So Tableau gives this wonderful view into the customer, right? We have the back end infrastructure, they have the view into that infrastructure to extract the value and the insight. So it's very symbiotic, a great partner for us. We're very happy to do it. I mean, the transformation is obvious, right? So the freemium models on the business side for application developers, clearly working, give someone some taste and they get addicted or they learn, they use it. Tableau's a great example, that perfect example. You guys have the same model. You have one server, you have three, now you got all those other goodness there in Amazon. So the question I have for you is, how tightly coupled are you guys with Tableau? Because as a user of Amazon, of course, we have our crowd chat and our back end analytics all running on Amazon, awesome, awesome service. But I want to push a button and get Tableau. I don't want to have to do a lot of work. So what are you guys doing to kind of get more tightly coupled with Tableau? Yeah, it's a great question and we've been getting that feedback here at the show. Today we talk about a couple different stories with Tableau. One is the back end integrations to our data services. So you take Tableau desktop on the front end, pick your data source on the back end, whether it's the relational database service, elastic map produce, redshift, et cetera, and plug in and get access to the data. The other side of that is running the Tableau server on EC2 on the compute service. Something that I've heard repeatedly at the show, this is a way for folks to get their reports out to the end users in a scalable way at the time when they need it the most. So I think there's a strong story today. We have some technical assets to help people get started and have a good experience. Over time I think what you'll see is we're going to continue to focus on enabling and empowering the users to do this stuff themselves. Do it on your own. Yeah, and you guys had microstrategy in the past. We've used that. I'm kind of not a big fan of that because it's too hard to use, but it should be easy. That's the goal, right? Yeah, I mean I think, again, going back to experimentation, if you make it too hard, people give up. Or if someone needs to stand over your shoulder and do the clicks for you, they might not, so. So is there a lot of overlap in terms of customer base between Tableau? How much would you say is Amazon versus non-Amazon? I could say we have great success without sharing the numbers.  Classic, classic. Yeah, so across the number of industries with some great customer stories, we had a session here, I don't know if you guys, I know you're busy here in theCUBE, but from noon to one, we had a great customer telling their story about Tableau on Redshift. So we've been thrilled with the success so far. I think it's the fastest growing product in AWS history. Is that accurate? It is. It is the fastest growing service in our history. We are growing pretty fast in general, so it's been overwhelming. And I think it's really because this is a technology that historically has been reserved for the few. So if you reduce the cost, the investment required to get started and democratize access to a data warehouse, the count of use cases that are applicable goes up dramatically. So we're really proud of the fact that we're providing more access to more customers and more parts of the organization to solve problems on behalf of customers. So I do want to dig into a little bit more about how Tableau is being applied to things like Redshift, but maybe just take a little bit of a step back. The whole concept of big data in the cloud. We at Wikibon done a lot of research around this. And we think that's definitely where the market is going for a number of reasons, just abstracting away a lot of that complexity, the underlying complexity of the systems. But in the shorter term, there's challenges, right? There's regulated industries, there's compliance rules in some industries where you can't have your data in the cloud. How do you see this playing out? I mean, are those kind of concerns? Do you find that a lot with prospects? They're thinking about it, kicking the tires, they'd like to move, they've got some concerns around regulation, security, privacy, how is that impacting your big data business? Yeah, I think when we think about big data, it's really across industries, it's across vertical segments, geographies. And the areas where you see some of the slower adoption is really, and what we've seen is this is really an education thing. It's a communication thing. So for example, earlier this year at our summit in New York, we had Siemens on stage, Siemens being a large conservative company, launching a diagnostics cloud offering. So this is basically taking medical information from a whole bunch of people to try to triage and get to the right treatment faster to improve healthcare and the betterment of all mankind. This is the thing that you tell your folks over dinner so they're happy with you. So Siemens is a conservative company with a product in a space that's highly regulated, launching a big data initiative on AWS. Let's take them for example, how were they able to overcome some of those challenges? Is it just a mindset or do they have to do specific things to make this possible? I think it goes back to that comment around the conversation in the education. So as we engage with Siemens to show them, hey, here's the security operations and controls that we have and how we take it as our first priority all the time, as well as really digging in and building subject matter expertise so that we can speak the language that they're used to hearing. So we have folks in the team that are healthcare centric, they understand the high tech and HIPAA laws and can then translate those requirements through infrastructure architecture diagrams. Hey, here's how you can do it. This is the way you can maintain compliance and maintain security. So it's a, you know, I think over time, those direct engagements and communications, you know, the friction will be reduced as the word gets out. Hey, this is a, the cloud is open for business or big data of all types.  What are the industries that are kind of forward thinking around big data in the cloud? It's a good question. And of course we're seeing these cases all across the board. I think some patterns that we're seeing are really around customer behavior analytics and that comes in many different forms, whether it's in the gaming world, you know, we talk about Supercell, Nintendo. These guys are looking at how users are interacting with games or using Redshift on the back end to do the analytics. Airbnb is using Redshift again when they launch new features to the website and their business is, right, this web app of how do you connect the inventory of property with people who need a place to stay? That's a big data problem, right? So that's really a behavior, customer behavior analytics job and one market's moving really fast. So within gaming, within hospitality, within other sectors, we see that pattern emerge. Although, you know, the use cases are as diverse as you can get. Yeah, there really isn't an industry that's not going to be impacted by this, whether it's, you know, the example I like to use is agriculture. Not necessarily a high tech business, but absolutely, you see companies like John Deere putting sensors on their tractors and crunching data to help farmers to understand when and where to plant certain crops and there's really no vertical that's not going to be impacted. So talk a little bit about the portfolio a little bit. So you've got Redshift's fastest growing service. You've also got Elastic MapReduce, your Hadoop offering. Talk a little bit about kind of those two services, how they relate, interact. You're seeing a lot of uses where, you know, customers maybe using EMR for some of the big data crunching and then moving things into Redshift. What are some of the patterns emerging there? Sure, so, you know, one of our fundamental philosophies at AWS and specifically in the data services business is that there is no one tool to rule them all, right? So we want to give developers and decision makers a variety of tools they can apply appropriately to their requirements and the problem they're trying to solve. So if you start with, we have our own services, things like DynamoDB for a non-relational database. You mentioned Elastic MapReduce and Redshift. There's also a relational database service. There's a lot of relational databases out there. Yeah, they're not going away. They're not going away. So we have a managed offering there and then Kinesis recently launched our streaming service. And then we work for the ecosystem too. You want to work with Elastic MapReduce and run our distribution, that's great. If you want to use folks in the ecosystem like Cloudera, we're happy to have that as well. So we're really trying to give flexibility in order to align the tools with the customer so it's really, really cool. And I've had some great conversations here at the show is to start seeing those pipelines build together. You might start on the front of the non-relational database when you need high throughput and low latency. But over time, the economics of that data store, you wrote that the timeliness of the data is not commensurate with the cost of storing it there.  So I think the pipeline that's enabled there is a powerful one. I know you can't talk about the numbers because you won't answer the question, so I'm not going to even ask it to, but... Thank you, John. Andrew Sloan Horowitz just put out a great post called Why Amazon Has No Profit in Why It Works, which is a great story, something that we've been tracking. They have three lines of business. This is Amazon, the company, media electronics, general merchandise, and other. The other category is growing significantly, the other being AWS. The other comment in here is about the... Amazon's the master bundler. Bundling stuff has been a key part. Can you comment from a product standpoint and a business standpoint as you go out and talk to all the Clare Cloudera's, the Hortonworks, the MapR's, the Tableau's? You're doing some biz dev. You want to be in good standings with whatever your customers want to use. So clearly you want to bundle all this stuff in. So comment on that other category, Amazon. You guys are obviously doing well, but the bundling specifically. Yeah, so I do believe, so let's take big data. We're at a big data conference. There is a value chain that exists, right? From ingest, to process, to storage, to analytics. And we're trying to look at that value chain in a way in which we start from the customer. What are they trying to solve? And what bits do they need? What parts of the solution are required in order to derive a solution? So packaging matters. And we look to both the technology partner ecosystem, the software providers you mentioned, but also on the consulting partner side. There's a lot of folks who need help to get this stuff implemented. And they may be the ones that are putting the right pieces together in that final package and solution and delivering it to the customer. So bundling is part of, you guys are bundled, basically. I think people want solutions. Yeah, which are bundled, packaged. Which are caught in the cloud. So you can rent a server by the hour. We try to get to the bundled in the cloud, but that's what people want. They want to pay by the drink and have whatever they need to work, right? That's right. Talk about Kinesis and any updates there in terms of success. Obviously Redshift, we love. Big fans of what you guys announced at the last re-invent. Really close the loop on the stack side. The integrated stack has proven to be a great winning formula to you guys. Talk about Kinesis and give us an update on successes, any stats or anything. Yeah, no stats or numbers to share. I'll tell you that Kinesis is very much a developer focused offering today. And we're still, every day, getting new stories of ways people are taking this effectively giant pipe that could stream any number of data sources or any number of end points and doing interesting things with it. So I think you'll hear more about that at re-invent. I can guarantee you. Some really good stories and good customers. You're good.  Mary's trained you well on the media training. I like my job. Exactly. Yeah, we know the culture over there. We love the culture, but let's talk about the Kinesis. I think that's a great example. We're here in the same kind of things that tab below this fast experimentation kind of concept. My take on what you just said was, look at, it's a cool tech, it's getting great reviews. People, we don't really know yet the full magnitude of what this could be. Is that kind of? I think that's true in general of how AWS has gone to market. If you go back to the very early days and the decisions, should we be building real offerings that are packaged in a box versus primitives charged for on the most granular basis we can support? We chose option B because we believe customers will decide how best to put these things together. Whereas if, and by definition, we didn't have a big budget either. But that was the building blocks, right? You start with computing and storage. That's basic. And then you guys are adding more and more building blocks. That slide is getting crowded. Slide is crowded. Okay, so we're looking forward to some updates. What other pre-show data can you share with us for reinvent? What's the drumbeat going into the show? So I think if you, let's take this from a historical perspective. I think you've seen a trend where we've launched a number of services that are more focused on the enterprise. We've done a lot of talking about the enterprise this year. Many folks and ourselves included as well as the community are showing this is the year where enterprise goes mainstream in the cloud. So things like workspaces as an example and some of the integrations around there. Zocalo for collaboration and file sharing. So a lot of enterprise offerings baked into AWS hitting the market this year. You can expect that trend to continue simply because that's where the customers are pulling us. So let's talk a little bit about Tableau. So one of the two areas I was really interested to hear about leading up to this week were announcements around cloud and mobile from Tableau's perspective. My analysis was that Tableau was a little behind the curve when it came to cloud and mobile. So I was really excited to hear about the project elastic announcement yesterday. But how do you grade Tableau on their cloud and mobile approach? Obviously the BI visualization space is getting very crowded with startups that are cloud first or mobile first or mobile only. So they've got competitors coming from come that end of the market as well as they've got the old guard that are trying to adapt. How would you grade Tableau in terms of their cloud approach and their mobile approach? So when I look at the strategy that Tableau has taken it's really about getting the tool in the hands of users listening to what they have to say. And I think there's a group of users that are looking for more cloud first technology, more cloud first functionality. There's a segment of the market that really wants that. They don't want anything to do with the underlying infrastructure, right? And so I think over time, if that group gets louder you'll see more and more from a product perspective from Tableau. But today, taking Tableau online and having a direct live connection to Redshift I think is a big step. And that unlocks a lot of power and value for a number of use cases for customers. On the mobile side, it's funny, we talk, this is a big data show but at the same time, what isn't big data anymore, right? And mobile and connected devices and this is about data generation and then data generation just becomes a burden unless you can do something with it. So analytics becomes a really important story. So for them I think rapid prototyping as well as seeing the value of the visualizations in mobile is important, but I think there's a lot more. There's a lot more there. We'll keep pumping ourselves up and by saying that we coined the term data first because cloud first, mobile first was a big buzzword, then cloud first and even Microsoft adopted that. I think you guys coined cloud first first. But data first is something that David Anvalonte and Jeff and I have been talking about because data is native. So I don't think there is a big data conference anymore. I think this isn't more an application conference. Just think about software in the cloud, if you will. But data is in everything. So it's not, you can't really peg the data market, right? It's like fast data, little data, internet of things. So I think it's pretty clear. Has the internet of things and these new big data, new data areas opening up, propelled Redshift more? I mean, Redshift seems like a great fit for the internet of things kind of things. Obviously Dropcam, who we interviewed at Reinvent, so awesome and then seeing the acquisition was not a camera company, it was a cloud company. I mean, they built, they innovated with Glacier and Storage, some really badass process to create a great pricing structure. They, no one figured that out. Founders said I couldn't get funding. He almost went out of business. But then he ended up crossing over. That innovation was in Amazon. Right. Yeah, I think when you look at internet of things or connected devices, what you need is ubiquitous connectivity, place to store the data and then compute power to do something with it. So it's a natural fit for the cloud. And I think also, because of where we are in the phase of maturity with IoT and connected devices, there's a lot of experimentation happening right now. People are trying to figure out what is the right way to do this? And to do this with purely an operating cost and a low one and a low risk of failure, it's just a natural fit. So I think we'll continue to see. And I think just, you know, just kind of to talk about some of the things that we experienced with you guys just to share for the audience here more data about what we've done is that we've had a similar experience with, I like Drop Can with our CrowdChat app. And I was talking to a venture capitalist to like, it's just a chat application. I can put a database together. I can do that in a second. I go, no, no, you don't understand. It's not a chat engine. It's a DevOps play. Right. And they're like, it's a DevOps play. I thought it was a big data play. No, what is in the cloud because it's horizontally scale. So when they see that level of large scale engineering, they go, oh, I get it. I can see the big picture now. So what things can you share along those lines, similar use cases where, wow, the innovation is not so much what you're looking at. It's in the cloud. Yeah, that's a great question and a long list. I'll take the example from our session here today. So UBM's big event organizer, one of the largest event organizers in the world. And they're responsible for figuring out who goes to the Black Hat Security Conference every year. Okay, it's highly coveted, invite only. So they turned to Redshift. They put all this data in Redshift and they start slicing it to figure out what is our profile of person that we should bring to Black Hat this year? And then how do we actually communicate to those folks and get them to come? So that is a kind of cloud-first strategy, data-centric, using analytics to better understand what is the profile? What is the customer? How do we get the right people in a room? So kind of an interesting story and one that's using both Tableau and Redshift. Okay, to summarize then, we learned that you're here doing Bizdeb with Tableau, great partner, great fit. Obviously, we agree with that. Two, the drumbeat leading up to re-invent is going to be enterprise. And just business as usual for Amazon. Right? We're iterating as fast as we can. Redshift has been on there. You guys always add new stuff. I mean, you guys are always adding new products. I'm expecting to see more new stuff at the show. You can expect to see more stuff at the show. Good, confirm. We're going to see new stuff at the show. Any, anything else you'd like to share? Yeah, I think one thing for folks who are interested in Tableau and trying to understand what this cloud thing is all about, we do have a program we call Test Drive, which is basically a disposable environment with no risk, no cost for people to play around with Tableau connected to Redshift and some sample data. So if you had aws.amazon.com slash Test Drive, there's a Tableau offer in there. And- aws.com slash what? aws.amazon.com slash Test Drive. Be a great opportunity for those who are in the community to try and understand what this means for them to get their hands on the keyboard and try it out. So risk-free for what? Tableau on Amazon? Tableau on Redshift, okay. One question I meant to ask you. Help me understand, help our audience understand the, you've got the, I think the Redshift, we think we understand, but what are you seeing in terms of the usage of EMR and Hadoop in your, among your customers? Are there some use cases emerging? Is it still mostly Test and Dev and POCs? What are you seeing there? Same, it's the same story. Very broad range of use cases. Some long running clusters with very predictable analytic jobs, you know, you're running the same types of jobs every day. Others that are pure experimentation. It's all over the map. We've got some great enterprise customers that are using EMR as part of their core analytic workflow. FINRA is one that we talk about publicly. So, you know, there's some really good stories out there and I think that's the way in which we try to, you know, communicate to the market. There's others doing this. There are ways to find success at the same time. You know, we want to help them find the right tool for the job, so. Okay, Scott, thanks for coming on theCUBE. Really appreciate you stopping by. We'll see you at the show. This is theCUBE live in Seattle, the Tableau User Technology Conference, data 14, hashtag data 14. I'm John Furrier, Jeff Kelly. We'll be right back after this short break.


The mindset changed was the biggest change. How has it changed my management thinking? It's changed everything about it. It changes your dynamic. Changed our cost of production. Really changed our feed requirements. Changed the style of our cattle. Cabbing, that's a big change. And in doing that, we changed our whole outlook on the ranching.


Okay, we've got Joe Gallagher in the studio who's just beaten Gunai Mamadzada. Tell us something about the game, Joe. Yeah, it was a struggle. I played, um, it was a night of Bishop E3 and I played Knight G4 on MiV6, which I haven't played much in the last 20 years or so. She was very surprised by it. But I was actually one of the original players to play it back in probably the late 80s. And I wrote a big article for Neon Chess and the line became popular. So I have some history in the variation. And um, so I prepared it this morning, I don't know, I just want to play something surprising, my opponents really. And we had a position on MiV 16-17 with a peace sacrifice, which can either in a draw, still complicated, but or she can go into an ending with two pieces for a rook and two pawns, which is a bit risky. And that's what she did. She played, well, I mean, she quickly got in some trouble. I think I was just completely winning. And then I made probably a mistake in time trouble and maybe she could draw, she'd draw in chances, but I managed to get into her three pawns against Bishop, just no other pieces. And even though I put all my pawns on white squares, they were basically going through. She couldn't defend. Okay, let's have a look at the game, then Joe. Okay. Okay, Joe, let's have a look at your game. So yeah, I played Qd4. I don't know what's the best here. This was the end of my prep, basically. It looked, you know, unclear. And she played Bishop d3. Now I was wondering, I had a choice of rook f8, which I played or knight e5. Knight e5 was interesting because she can go Bishop b5 check. And then I can take, and after Qd4, I've got knight takes f3 check, winning back the queen. So if we go back instead of Bishop b5 check, she can also go, she can also go Bishop g6 check, and now I mustn't take, obviously, because then I can't, I don't have the fork. But after so instead, I didn't play knight e5. Yeah, I would go King e7 in this position. And it's probably, but it's a mistake Bishop g6. That was just a fun variation. Yeah. So I didn't do this because if we go back to move to Bishop d3, I didn't play knight e5 because maybe she goes Bishop e4. And I wasn't sure in the end in how it was. Why should I go for an end in here now? I've got to play rook f8. So I played rook f8. And I wasn't sure.  So it was quite interesting. She castled and I took on, took the pawns. She played knight e3. Now I didn't know what to do with my king. I didn't really want to have it on a white square because there were many tricks with that. So I, or castles as possible. But I calculated a long line and I went for it. And then I realized I made a mistake in the calculation. So she played rook b1, I believe, was it here? King g2. King g2 first. Oh yeah, because if she moves her rook, for example, if she goes rook e1 or something, I can play a rook takes g3 and knight e5. And this is rather tricky with threatening knight f3 check and yeah, even. Yeah. I mean, this is good. I mean, even if after rook b, rook c1, I can do this, I think. So she didn't do that. She played king g2, rook f8. And then she played rook b1, b5, and she played rook c1. Now I had a long thing I miscalculated in the variation. Originally I'd planned to play rook takes g3 check here. Rook g3, takes, knight e5. Oh, sorry, one way. Yeah, it has a life of its own. Knight e5, rook c7 check. Sorry. Rook c7, king d8, queen a5. And I thought I could play queen takes e3. And I thought there was no good checks. But then I realized that after rook g7, king e8, she's got bishop takes b5 check. That's the end of the game. So I managed to spot that in time. Otherwise it's probably winning for me, that position. So if we go back to move, yeah, so I played my reserve variation, which was knight b4. So now she's got, I mean, it's a good line. Knight b4. I should probably have wasted time on that other line. I should have just played knight b4 because it's a decent move. It's a decent move. It stops things. I didn't see, I don't know. Now we were short of time. So the game went, she played rook c3. I played knight d5. There's other moves as well, I don't know. But this looked good and safe takes. I mean, she's got to take queen takes. Now king g1. Now I played queen d4. I don't know. I could have, I repeated once. Yeah, I wasn't sure. There was many tempting moves, but I ran out of time. So I went for a run. Well that is, but yeah. One of the tempting lines I took on g3. I took on g3 and played rook f3 and now, well, she's got a few problems here. She has to play king h2 and then queen e5 is very strong. Maybe there's other moves. I haven't really had time to look at it, but now I'm threatening to take on g3. She played queen g2. Only move and now, wow, I've got many moves, but I should, I can just take on g3 and then take on c3. Yes. And with rook c7, I've got king d8. So I get four pawns with bishop. Should be winning quite easily. But I overlooked, I mean, I ran out of time. I went queen e3, which looked even stronger to me. And then I realized she's got bishop e4 or bishop e2. Yeah. So it was just a blunder. Typical of my games really. So now I can't do anything except exchange. Exchange rooks and she took with a queen. Maybe she could take with a bishop. I don't know. I don't know if that's probably better for black, but it's not easy anymore. I mean, played d5, I suppose that bishop is very passive. Yeah, but that's another game where she took with a queen and I wasn't sure. And then we're in time trouble now. So it's difficult to exchange queens. I thought this is probably at the best chance. And I thought I'd activate the king straight away. Her king is far. So she came with a king to g2. I played, I don't know if this is called king e5 or not, maybe. And she played bishop g4 to try and stop me playing king d4. I played b4. And then king f2, a5. I mean, so I've always got this a4 and b3 tricks coming. King e3. I just played d5 because I can always get d5 and e5 in the system. She can't stop that. So she waited king d3, king d6. Maybe here she might have made a mistake. I don't know. She just gave me a tempo, which might be I'm losing. Maybe I don't know if she's got time for bishop d1 here. Right. I mean, bishop d1 might be drawing. I don't know. To stop a4. Okay. Anyway, I think bishop d1, I think after what she did, she's just lost. She played king d4. She just gave me a tempo and I play, now I play e5 and then I play e4 so that this is a pass pawn a4. Bishop d1 probably only now b3. And now if she takes a3 obviously, and she goes king c3, I've got d4 check. So that just comes through. So she played a3. Maybe she thought this was okay, but now she's in a zugs way. It's like she can't move the bishop because of b2. So I just wait with king c6. And she's forced to move it. She can do things with a pawn, but if she plays h4, take, take, and it's actually very easy because my king has a square. My king hasn't got a square in some lines. That's why she almost had chances. So she's got no move. So she went king c3 and now king c5 this morning. So I'm coming forward to bishop h5. Now d4 check. She had a choice here, but I think king d2 loses at once to something I can't remember what. I think b2 or e3. Probably just e3 and if king d3 then just b2 and d3 check and e2. Just they go straight through. Don't need anything. So she played king b2, but now my king comes in. So it's all over. King just comes into here and she can't sit around. So it was interesting. I made a stake, but it was quite a good game. Still came through. Yeah, it's good. Good stuff. Thanks very much. It's nice to beat the young ones. You sound like Victor Korshner or your former teammate. Well, I'm not as grumpy as him. There's still time, Joe. Still time, but yeah, I mean, he was he's grumpier than me. Thanks, Joe. Thanks.


Welcome to Authorization in Software, the podcast that explores everything you need to know about authorization. I'm your host, Damian Schenkelman, and in each episode we dive deep into authorization with industry experts, as I share their experiences and insights with you. If you're a software developer or just someone that's interested in the world of authorization in software in general, you are in the right place. Let's get started. My name is Damian Schenkelman, and today I'm chatting about how BOX approaches authorization with John Halfaker, distinguished engineer at BOX. Hey John, it's great to have you here. Thanks for having me, Damian. It's my pleasure. Could you maybe give your listeners a brief overview of your background and what you're doing at BOX nowadays? Yeah, for sure. My last job was at a faceted search company called Indeka in Boston. We powered sites like New Egg and things like that. Whenever you search for a part or a video card and then there's the price range and things down the side, we enabled a lot of that. We did a lot of analytics apps and eventually were acquired by Oracle. I then moved over to BOX a decade ago. The first team I joined there was the team that was launching our public APIs for consumption by platform partners and third-party applications. One of my first big efforts there was on our metadata platform. I launched our metadata facility which lets customers define templates, whatever you want to call them. Say they want to store a contract with us. They could say the amount, the counterparty name, the expiration date, things like that. We use that as they fill out values. You can use that for faceted search, but also workflow. It kind of goes in a lot of different places. We do this a little bit right now, but hopefully over time it also helps drive authorization decisions. During building, BOX is a pre-AWS company. We had our own data centers and I saw how challenging it was to get. We were evolving from a monolithic architecture towards a more service-based one. I saw how hard it was to get a new service launch at BOX. I moved over to work on one of our internal platforms built on Kubernetes. We're super early adopters of Kubernetes. Interesting story, BOX built the initial version of KubeCuttle, which people use a lot today. I didn't know that. That's an interesting fact. One of our co-founders actually ended up building that. We were just super bleeding edge. Deployments didn't exist. We were manually managing replica sets super early on the Kube bandwagon and went through all the different growing pains. It was a great opportunity to interact with Joe Beta, Craig Mclucky, Brian Grant, Tim Hawken, all those people that started that project and worked with them on how to use it and round out the ecosystem. We built a lot of early components there. It was definitely challenging. From a business standpoint, probably hopping on the bandwagon a little bit later, might have been a better move, but super fun as an engineer to work on a project that early in its lifespan. Then after that got stable, fairly well adopted internally, I moved back to working with our core team, which is if you think about the backend platform for everything you think about when you think about BOX, like files, folders, users groups, enterprises, all the just sort of core, high-scaled objects that you think about. That's where I'm working now just across a very, very, very wide variety of efforts there, including still interfacing with our internal platform teams as we do big migrations, as representation from the core team. I think that covers all of it. Hopefully your listeners aren't going to sleep yet, but... It's interesting because you've been through a few different areas of a very large product and platform, right? In my case, I've done some similar things here at OZero and AOTA, and that gives a unique perspective whenever you're building authorization capabilities or any kind of platform capability in general. I think that perspective will be very interesting for anyone listening to the episode because we'll be able to mix general product and technology perspective with a more security and authorization perspective. Totally. Yeah, it's BOX. I like to say about BOX. I've been here a decade, but I've had three very different jobs over that time period. And BOX has big company problems, but sort of small company infrastructure. And so anytime we go to build something new or develop something new, we're always looking at it from a very basic level all the way down to the platform level to think about how we can best serve both our internal and external customers. So yeah, it's a fun place to work. I'm sure you probably have a lot of similar experiences at OCTA, just working on very high-impact, basic stuff all the time. Yeah, one of the great things I like to tell folks on my show is that we get to experiment and figure out, in our case, what the identity industry is going to look like maybe in the next five years, but we are the ones shaping it considering the impact that we can have. Yeah, it's very fun as an engineer. Yeah, definitely. So as always, I like to start with the why, right? And this is why I was saying that I think your context is not just working on authorization, but in general, across the BOX program, the work platform will be useful. Why is authorization important at BOX and what happens if authorization is not working properly? Yeah, I mean, there's lots of layers to this answer, and we can talk through both. But the high level is BOX, you have to think about it for a minute, but BOX is secretly a security company.  But share sands or shared drives is the security risk, right? Like you're putting your files on the hand. You don't want it to get out there. You don't want the various governments or state actors to get access to them.  So we spend an enormous amount of time as a company thinking through all the different security aspects. So our security organization tends to break down on two lines, like there's the application security and then there's like the infrastructure security and our application security team is just very focused on like, how do we ensure that the application that we have is not, you know, like somebody doesn't add a new line of code and make files accessible to anyone that has access to the system or, you know, various application concerns. How do we build the product securely? Like running PEN test. We work with Hacker One a lot, which has been a really fruitful relationship in terms of getting early PEN test results. But also there's like the whole other side to that, which is we're basically a file storage, the file productivity platform. How do we give tools to enterprises to help their users not expose things, right? Like there's always some level of sharing that you want to do with a document either internally or externally and allowing those your internal users to do that safely and securely is a big deal. So we all, you know, all the way down from like shared individual shared link exploration and things like that and passwords all the way up to more advanced stuff like classification policies. We just launched information barriers functionality that helps keep different parts of the enterprise from seeing each other's content. Email uploads, secure file request features, things like that. So yeah, it's securities a very big deal to us and our customers. And all of those are different types of authorization decisions. And then like the more low level infrastructure oriented authorization decisions are, I'm sure your customers are fairly aware of all these issues of making sure that, you know, your IAMs roles are locked down, don't have have least privileged access to various resources, ensuring that an attacker that somehow manages to get access to one node can't pivot and start moving laterally to the infrastructure is another place where we spend a lot of time and we use things like micro segmentation using Calico. And now we've got some local policies down at the infrared layer validating that changes that the person making a change should be able to make that change. Yeah, so just layers and layers of authorization decisions floating around here. Yeah, I was I was going to say right like you folks store a bunch of sensitive data on behalf of customers and like everything else like security is all layers and layers and different kind of like ways of protecting things. One thing that you mentioned was that again, there are a lot of things that you can do right there. You need to authorize users, you need authorization at the different level. There are multiple ways in which you can file share, you can file share to a user, you can file share with like temporary time limits, you can upload like in an email, things like that. And then there's also kind of like you mentioned the notion of at the beginning of having an API and working with partners, which means there's also kind of like up permissions. So yeah, a lot of things like at a high level and we dig deeper across kind of like the show, but like how does box handle authorization? Can you maybe give us an overview of that? Yeah, so we have lots and this is something that's a little bit in flux, but we like obviously depending on the thing that you're asking about there's lots of opportunities for authorization checks. But the bulk I'm going to talk specifically here about the applications authorization decision. So like if I fire up my web browser and I browse to a folder and it shows all the items there and then I upload a file, those are all, there's all individual authorization checks being done in each step in that UI. Can I view the folder? Can I view the folder's children? Can I view that file? And can I upload a file into that folder or all different authorization checks that we need to do? And given the nature, given our nature is kind of a web application, we tend to, and given the complexity of the object model and authorization scheme, we repeat the same checks over and over again on each request. Yeah, so and even if I'm like viewing seeing if I can see a user, we have to see can you see that user through your collaboration graph? Are you in the same enterprise? Can I see a comment, et cetera. And so the way this is orchestrated, and so this is something that's changing as we migrate as we break down our monolith. But I'm just going to talk specifically about our monolith for a moment. So we have effectively the old boxer as a classic lamp stack way back in the day. And within there, you know, we have like a kind of active record model that was home built. And on each of those at any kind of point in the code, you can say, does this user, does this context have the ability to see, does it have this permission, whatever the verb is on this object? And there's some like some that are built deep down into the data layer. So like any time you try to fetch the files metadata, it's just a metadata like name or the size that kind of like core object information. We check if you have view access. And there's a few other places like that. And within that, that kicks off that like do I view access? It's off a call to our permissions policies, which are just more PHP code that basically break down into two flows, like a grant phase and a reject revoke phase. And at the end, we check, hey, did you did you get granted view? And is it still there? And if yes, so we, you know, return true and we return the object. And those policies are just like, and nor it's probably the single most complex thing within the box system. It acts in access to the number of tables, very performance sensitive. And it checks, it checks, like, oh God. No, I was going to ask a bit about that, how kind of like these two ways of evaluating access for the positive and the negative, but since you can do it from anywhere on the code, and all of these things require state from both code review perspective, performance perspective, security perspective, this might be a bit tough to do, right? Like knowing when you should call it and having to optimize things so that like you minimize calls and so on. Yeah, and this is one of those things that's specifically interesting as we move from the monolith to the micro services universe. Honestly, a little bit, it's a tiny bit easier. There's different set of challenges in the monolith universe. You know, we, the request is largely happening within the monolith. And so multiple, we, we cash multiple calls or memoize multiple calls to the function. And so adding superfluous checks to the same object isn't too much of a, isn't at all a perfect, it basically short circuits out. In the services version of the world, but also on the monolith side, it can be somewhat, yeah, you, you want to do a balancing act between calling it too much, that also like the worst thing that can happen in box at some level is that you don't call it, right? From going back to the previous thing there. So that is one of the challenges with our monolith is we, you've got a bunch of library, it's a normal mod, you got a bunch of library functions, some of which may call the permission check, some of which may not, and you need to ensure that the permission check is happening. But you have some of those library functions are used in permissions calculations themselves and so you don't want to use, and some of them you don't want to be calling using permission. So it gets to be a real struggle reasoning about where the permission checks are done, where it isn't. On the services side of the world, we've landed on like a really simple model. We have domain services, they have endpoints and effectively every one of those endpoints has a permissions policy associated with it. So there's no question there about like where it's applied, when it's applied. It's applied very rigorously versus with the monolith. There's a lot of weight put on the testing to ensure that we're doing, I mean obviously we need to do testing in both cases, but it's more error prone and so you have to like really back yourself up with negative tests to ensure that all the permissions policies you expect are being enforced on an endpoint. It's very interesting that I'm checking things now, like 18 years old, starting in 2005 and only now you're starting to kind of like go on the one hand towards microservices and there's a whole industry debate always going on about like what's the right time, it's not the right time, but also starting to extract policies away from code and into kind of like more policy logic. Yeah, I'll dive back into what we kind of do on some of the policies in a second, but just to answer that. We've been pulling services out, like we're kind of taking like the strangler pattern and slash like peeling an onion with our service based approach. Even when I started, we had some initial very infra centric services that were already being pulled out. And so at the places where it was easy to extract in the services we did, so like a lot of the unstructured data handling is already pulled out. A lot of our user extensible metadata is pulled out. A lot of the clients have had a bunch of their logic pulled out. So it's just kind of like we're pulling off the top and bottom of this monolith slowly but surely what's left in the middle, which is what we've been we're starting to tack or we've been tackling for a year ish, but it's also it's very complicated and sensitive. So it takes longer are the kind of core business logic. And so yeah, it's just kind of a long, long running process here box has a lot of stuff to do. So we go in and out of dedicating more or less resources to an effort like this. But yeah, to dive back to just for a second to the permissions policies themselves. So within any like taking a file, for example, in the grant flow, we check, hey, is there a shared link in the request context? Are you we call them collaborations, but you can think about it like an ankle. Are you collaborated on this file as a viewer as an editor as a whatever are you collaborated up the folder tree because we do this thing called waterfall permissions. And that's kind of the bulk of the grant phase is a few other steps in there that we could talk about but the less interesting. And then the revoke phase is like much more interesting like is the thing deleted. Are there legal holds are, are you at quota you can upload more of this folder if you're a quota. And that gets a lot more complicated. So yeah, so. So that's kind of the high level of authorization. I'm curious if there are places they'd love to talk about more. Or things that were unclear. I'm used to doing this of the whiteboard.  Yeah, I get it. Maybe in the future season, we can do another episode with video podcasting, which is also becoming a lot. I'm curious to learn more about kind of like where you are, what you're evolving towards right like what technology is the service oriented approach going to use for for policies how you picked it. Yeah, so, and we've, we've gotten in and out on this decision and we like reconsidered it a few times. But for our services on our floor, we're using a zackamal engine open source zackamal engine. And I don't know how you actually pronounce that x a c l. We're using Balana, which we have a few developers that are ramped up on and are able to like make local patches to in terms of like multi attribute prefetch and things like that. So we have quite a few domain services on the floor that are using that or a lot of basic policy. And as of as of right now, we're in the middle of pulling the giant PHP permissions policy over to it. And that that's got that's been an interesting discussion. I mean, zackamal is obviously pull it separating the permissions policy from the rest of the codes obviously has some really nice benefits in terms of the one thing I previously mentioned is like it's very clear where it's enforced and when it's enforced. One of the challenges in that monolith land, like I mentioned is you've got some things that need to run like the permissions itself needs to run like in sort of a God mode supervisor mode in order to fetch the data in order to see if you can even see the data. And so it creates a really clear boundary there where like the permissions policies have a lot of access internally. And, you know, we we switch over to using we call it a domain service token, but a version of our internal token that has basically indicates that you should because we want to be able to reuse the data access logic from the service to fetch the files, but without inflicting the permissions calculation on, you know, without creating an infinite loop basically. So yes, that's one super benefit is it becomes very clear, which like where the permissions are going to be enforced are going to be enforced on the endpoint, and the permissions policies themselves have broader access to the data than the normal natural colors, the endpoint. We we looked at we looked at a few things I mean obviously just like homebill policy in Java was one thing. Zachimal was another thing and we revisited as like open and off that and all those things have come out, like whether whether those would be better choices for me for me like for me having started this problem for a while. I would love it feels like some machine learned approach here that like minimizes latency of the policy by by using any features that you have available at request time either the enterprise ID user ID certain request features, being able to find like a fast path through that to the grants would be a wonderful thing. And I was super excited about Opa for a while. It ended up and it's all like I don't I haven't looked recently at Opa so I apologize if I'm not stating any of the changes in direction they made but they seem like they were really heavily focused on infrastructure security at that moment really focused on like low data scale, like almost like IP titles level like sub millisecond response times. And we, we have a lot more complicated policy issues here where, you know, we touch 60 plus tables. It's a large amount you couldn't fit it into like kind of the static small database they have like very large data sets including the files folders their collaboration relationships and things like that. And so it didn't it wasn't quite a fit and I felt like we'd kind of be a use case that they would have liked but then kind of way outside the where they were getting a lot of success and traction at that time. But yeah, we, there, there have been concerns about, you know, we were handwriting Zacamole for a while, which is obviously no developer enjoys that so we actually have a bunch of libraries written in Java that effectively represent they generate Zacamole at the other side so it's closer to like a Java like we've experienced with the IDE and things like that. But yeah, the big kind of question as we move the items permissions policies over are going to be and as as we move to microservices right like the repeated checks thing starts to become more of a concern. And so the parallel project, trying to enable graphql for some of our internal clients and so like, obviously naively that things going to try to, you know, hit the same endpoint over and over sometimes. And so the permissions, the permission recheck thing figuring out how to do either request level cash which we had for a while, or some other type of caching in order to avoid the penalties that is going to become important. And then yeah just like ensuring that the underlying underlying policy, ideally is faster than the thing we currently have hand tuned in PHP. But at least at parity with the thing that we have is important. I in my my secret hope is an architect here is, you know, the current policy is, you know, like I said hand tuned which is a polite way of saying very, very, very hard to change right like some of these policies have ordered appendices on the other parts and can move them around without potentially creating like creating a security issue. And so I would love to just get the policies into a place where they're naturally expressed the way you'd want to express them without thinking through the purse implications and then allowing the underlying platform to intelligently make decisions about evaluation order and things like that. So, yeah. Yeah, that makes sense. So I think there were a bunch of interesting nuggets there. I'm going to try to see if I got them. So, yeah, so because you already have a set of policies, when you started looking for alternatives and only one hand you took out all barn and that being kind of like more infrastructure oriented which I would tend to agree I think we go as a language allows you to express everything but like all bond and the tool said it's more thoughtful kind of like infrastructure when you think about like how they handle things like cash and unstable in general. But at the same time, you already have policies, which means that you need to keep compatibility with that and maybe you were used to expressing things in a certain way which you probably want to maintain for security reasons for like simplicity reasons and so on. So, yeah, some sacrament at least made the cut and then you were okay, we're going to go with this implementation, which is banana. And then from that point on it's about okay how do we improve things so again you got the kind of like higher level maybe the SL library in Java so that you don't have to write the technical code. You mentioned also that you're making changes internally to how the library works which might be again you're making improvements to the catching logic and so on that you kind of like have a small fork of banana that you're kind of like tuning for a box and that's kind of like what you ended up doing as part of that migration. Yep, that sounds right and we're always open. I think we're always open if like something revolutionary shows that we're always open to considering other other options here. Yeah, the authorization space is kind of like fairly effervescent right now and new things are coming up every day like I know for example AWS recently released open source at least data policy language which is kind of like yet another thing and has a few interesting benefits from like static analysis perspective so that would be an episode on that that is a bunch of new SaaS companies kind of like looking at the space open source project so yeah I imagine over the next few years you will be able to either replace or complement parts of what you're building with some of these solutions in the space. Um, yeah that would be great and if as soon as we got out like the key goal right here is getting out of sort of this thicket of permissions logic that we have like we have things that will like short circuit for perf reasons that creates like the lack of commutivity of the policies um or a lot of things that you know reject all permissions and things like that so getting into zackamal and like having the stuff naturally expressed puts us in a good place to experiment with other technologies more easily than kind of the free for all PHP code basically have. Yeah, no I get it that makes sense and I'd like to go like double click on that because you mentioned the notion of paths earlier we were talking about like all the ways in which you can share documents with folks so as I think this is a problem that that I've been thinking out for for a few years now which is when you get into granular document sharing like anyone can view or not view like specifically just one file it's not the folder it's not files but it's on like any attribute or something it's like they can either be you or edit one file when you get into performance when you get into box scale. You are a kind of like dealing with complex authorization requirements so how does authorization work for file sharing what happens when I share a file with another user and give them maybe read permissions or write permissions how is that handled internally. Yeah, so I guess is a click down in terms of detail from the previous answers but basically like if you imagine going to the the our preview page to look at a file. And it you've been collaborated to use our term what happens is you know we go and dive into that endpoint and eventually at the permissions logic and what'll happen is we we pull back. I mean the first thing we see is we try to access the file which then triggers the permissions policy the file metadata. Which triggers the permissions policy which will look through for that user. What are all the the collaboration that it's a part of. So the file metadata includes the collaborations for the file which would be the ways in which the user can interact with the file. Yeah, not not fully quite so we have two tables under the covers we have like a files table folders table and a collapse table club being short for collaboration. In the collapse table it's got the inviting user the receiving user as well as their role on that thing and basically because of the way things are done like you'd think like oh just look at the the the clubs table for that user ID is trying to access the file and the file ID. Unfortunately we are or fortunately we maintain things very normalized under the covers. So if you were collaborated at like a parent folder somewhere. We have to give you access based on the fact that you've got access at that parent folder is referred to as waterfall permissions. So basically what ends up happening is on each request where you're accessing a file or folders we pull back all the things all the roots or all the folders or all the files that you have access to. And then you know we quickly look through that list and see which and we we keep that cap. We recommend customer standard like 10,000 of those but it does require some thought on the customer side about how they want to store their permissions in a normalized way. But we scan that look to make sure you have access to that file or any of its parents and then if we do is actually rarely do you just want like we drive a lot of UI on the other side. Rarely do we just want to know can you view view the file right we want to know if you can delete it so we can know if we need to show the delete icon or not. We want to know if you can comment on it we want to know a bunch of things about what you can do with this thing. So actually in the policies we like compute and mask say largely use the same data. We compute and math what what permissions do you have on this file can you comment on it can you delete it etc. And then so so it's not that you don't have to ask the same question multiple times it's not like saying hey can the user view then can the user then can the user delete it's it's more of a hey what like from all of these tell me which ones a user can do something like that. That's basically it and then that function that I was talking about earlier can you view it all that thing does is looks to see if you have that permission amongst that array of permissions. This is something that you know we call it like bulk or batch permissions on the Balana side but that we have to do something slightly interesting here to fit into like the you know you called get we can just evaluate the get. On the Balana side but we do want to be able to bulk calculate the permission sometimes because we do return to our clients. Can it be deleted. Can it be changed etc. And so yeah yeah yeah that makes sense and how so like let's try to get off like mix things up a bit because you're saying okay yeah. You you can have kind of like all of these sides but you might have kind of like this cascading or waterfall permissions where you might have the ability to view a file not because someone shared the file with you but maybe they shared the fights patent or maybe even like grand pilot folder so this is kind of like I'm like everything is really a graph traversal problem if you look at it but again you might have multiple ways in which you read the file reach the file. And then you also have groups right like there's the notion of hey I can be part of a group and we the group has the column. So it seems that there's a bunch of state that you have to kind of like put or how go through the body see so at the body see gonna arrive at the right decision. I'm curious on like I want a couple of things one of those is like again you mentioned some recommendations for customers or like keeping things under like 10k objects for roads but I'm not sure if I followed so I'd be curious like how you limit that. I'd also like to understand a bit more how you handle the groups with these kind of like traversal denomination like broader right. Yep yep. Yes so you're correct it is a lot of data and groups is an interesting question in here. We make very extensive use of memcache and redists along the way through a lot of these calculations. So like that that saying I mentioned around like what collabs of my on and like what what groups of my and those those pull extensively from cash and that's probably one of our larger expenses. But so in terms of groups specifically so groups is a fun one for us. In my like when I joined box I you know there was no groups and I'm like this is quite great like as an enterprise software company that groups would be. At a key point of scale like giving companies your customers some of which are very large a scalable way to kind of manage access to content. We did add it a little way and I wouldn't say it's necessarily 100% where I'd like to see it. But as a yeah as of today just in terms of that specific example we're talking about we do pull all the groups that you're a member of and we pull all the collabs for the groups that you're a member of and we kind of match that all in. And there's this kind of a deep reason why you like if you if you stare at this use case and you think about it for a second and you're like this seems absolutely ludicrous to be like pulling all these things back each time like pulling all the things you have access to when you know like the destination is this one file. Obviously like a database index would solve a lot of that for you pretty rapidly. The hidden thing and a lot of this is so we have three or so and potentially more experiences one one obvious one being search where there's an implicit filter when I type like show me the documents containing my stocks. There's there's an implicit filter there right show me my stocks and all the documents that I have access to people don't think about that I have access to you know part of that sort of choose is challenging. Yeah and as part of that we basically and we can talk about the low level details of how we index that. And we also have to do that for like metadata query which is another facility we support but also like the main landing page at box is this big experience we call our all files page. And it's basically kind of a virtual view of everything you own everything you have access to and a few other things and so across these three different experiences. We basically end up having to act like look at everything you have all the roots you have access to each time anyway. And so you know it ends up being we end up just kind of relying on that facility for a lot of. Well basically all of our access control and even though if you look at any particular use case there'd probably be some way of optimizing the access there. And so yeah groups specifically and like more interestingly like you're all enterprise group which we've started supporting in certain limited cases. Yeah we we we go through might we go through we expand the group or we see which groups your member of we start from you walk out groups your member of and then figure out the collab objects on the other side of that and feed that into that giant machine. My my ideal hoe over time here like right now we don't really index the groups that have access access to a file in our search indexes and in our metadata query indexes. My hope over time is we would start doing that and then just start passing in the group IDs there and order to reduce some of those like those those cardinality challenges. Like in terms of that 10 K limit that I talked about before they basically any like or or system problem at box is like there's just this hard trade off between storing it normalize and having to deal with like an expansion and like bumping up against that 10 K limit. A folder IDs that we pass to a search index or denormalizing it and making it fast but then potentially bloating out the amount of data we have to store or like just having like an operation that was fast before like adding a operator which was like Oh of one before is now potentially an operation where you have to like reindex everything. Yeah. And then you get into depending on how long that takes. If it's a few milliseconds maybe it's fine. If it's going to be three minutes until the change is propagated you might have issues from a security perspective like different systems have different trade offs there. But where you were just constantly like which one is it in this case and thinking through all the implications of the normalization versus denormalization. Yeah, I get it. Let's go look at some kind of like all of the numbers behind this so first like for each authorization decision. What data are you using like we talked about a lot of data. Can you give us an idea and what the performance of authorization decisions typically is. Yeah, absolutely. So I mean box in general and these numbers are going to be non specific so I didn't run them by any anyone. But we have hundreds of billions of documents that we're storing like nearly an exabyte and scale of like the actual underlying content hundreds of thousands of enterprise customers. And then you know like the customer size tends to follow like standard power law where we have some that are like over 100,000 users and then a lot of like really a lot smaller ones. I think our average customer size like 30,000 users or something. And then just in terms of like, in terms of performance, it's highly variable like you can imagine based on how I described the authorization decisions. It can be very highly variable. Like for me, like when I go to my all files pages is like my initial landing page. And I'm kind of at the harder end of scale it's to just like get the basic items which is mostly just a permissions check on the items just like fetching them fetching our rebacks not that expensive. It's it's kind of close to a second, which compared to like, you know, some of those numbers we see from hope up and right goal is is probably two or three orders of magnitude higher. And then this but you know you have if you're a smaller customer or you don't have this complicated permission scheme, you're going to see lower numbers, but still probably in the hundreds of milliseconds range is probably our minimum there. But we do have like, I mean our CEO ironically, you know, he's, you know, been there for I don't know what you said 18 years has been there for the 18 years. 18 years has all that data accumulated. And so he, you know, his, his all files page experience, which he loves dearly, it can take him upwards of like three to four seconds to load, which is far from what we'd like it to be. And we obviously were working on a lot of projects to drive that number down. But at some level, some of this data that we have to process just takes time to access and run through. In terms of like, overall throughput, we're dealing with like millions of requests per second. That's it with that. That's a lot of, that's a lot of requests. And so this is a kind of like when you compare to some of the public metrics from our framework. So this has been a system that's in print for a lot of years. X, Y, J, so it's, it's, it's not an uppers to uppers compiler. So let's just put it like that. Yeah, of course. Yeah, I mean, when you're serving an end user app, and we have a really, I think we're going to maybe talk about integrations in a minute. But, you know, we have a lot of things that integrate with us, least of which are like our desktop clients, which do basically try to maintain some replica of your view on the file system on the local local system. And so, you know, we're pushing events out, they're making calls back. And so those drive a tremendous amount of load. We also have third party, like products like CASB virus scanners, things like that that are just constantly hitting the API's, ensuring that the data is protected. So, so yeah, we have a lot of things going on. And then I ran the numbers, we're doing like hundreds of thousands of permissions checks per second. As part of that, like anything that's accessing a file, there's some amount of a permission check going on in there. And so yeah, so it's quite large scale, makes it fun. It makes some things like we use, you know, as we're extracting the core stuff doing a lot of parity testing, it makes things like that feasible, like taking the old policy, the new policy, and like running them through comparing the results and sharing the same. It makes those techniques more viable and more fruitful. Yeah, then I can imagine, again, any change you make, you have to like maintain the same kind of a backwards compatibility, let's say from a result perspective, while at the same time making improvements. So it must be an exhaustive set of checks from multiple perspectives. Yes. You mentioned apps, which we discussed, we would touch on as part of the episode. And we mentioned them at the beginning, where you started working on APIs for some of your partners. I know Vox integrates with a lot of other apps, I would say, typically, like between. How does authorization work with apps? How does a typical Vox partner and a Vox application that uses Vox and depends on the Vox API work from an authorization perspective? It's largely an added layer. So the API came later after all the permissions policies firmed up. And so the way this tends to work is if you're going to build a third party integration or even a first party integration, even our own teams use this. And provision, we call a service but applications a better name. Provision application, you get your client ID, client secret. And from there, as part of that creation process, you specify what height of these apps. We do have a different mode where the apps actually become a user within a particular enterprise, and those are more limited to the enterprises. But for your standard client use case, where it's doing something for a user, you specify what of that user's underlying permissions you need. Are you just going to be, are you an admin app and you're just managing users for the enterprise? Or are you like content processing app and you need to be able to access their full tree, their full folder tree?   And then when we publish it, it's able to be used by people outside of your own kind of local enterprise. But basically what happens is, as a standard, we allow a different, different sets of the offloads, but we use O up to, and as part of that, we don't, I know there are other ways like people use like scope. And this might be the standard way, but when people use scopes, they'll like effectively pack that whole precomputed permissions process. Into the scopes and then it's just a matter of checking whether like the scope has access to the underlying file, given how dynamic our permissions are, what we actually put in the scope are more of that clients permissions on the underlying users permission. So it's kind of an allow list on the underlying permission. So it says this service when it's accessing a file can, you know, read or write all their file, or their whole folder tree. And so we basically check, hey, you accessed file 12345, we compute the permissions on it does this user even have access to file 12345. And then when it comes out through the API response handling, we check, does this application have the ability to read or write that user's files. And if yes, then we kind of will return the result or allow that that API client to make that change. I mean, this is all this is all done much more generically than how I'm talking about it. But that's the basic flow. So yeah, I get what you mean like that we actually the first episode we did of the podcast back in season one was with Victoria or touchy who worked with us and we talked about how for scenarios like these were authorization is dynamic because like resources and permissions and there's lots of them. Scopes and embedding things in access tokens is not the way to go. So you kind of like in boxes case you're saying hey, we have scopes as a higher level notion which might be hey, we are going to be managing users, but you're not going to say which users or I'm going to be reading or writing some folders. And then when you actually have to kind of like make the specific call you just make sure that the access that the user consented to kind of like encompasses the specific thing so that if you're changing a folder's name, it would be hey, can they write this folder's name. And did we actually get on access so that this token would be able to edit folder names or whatever there is. Exactly. Yeah, that makes sense. And I'm here. Where is the authorization decision made for the kind of like API decisions. Yeah, it's all done. It's all done as part of that underlying permissions policy. I believe yeah, it's all done in that underlying permissions policy. And so it basically as like the last phase it computes all the permissions like I said that you have access to like can you view can you comment can you delete, can you edit, and then as like a last step, it will check like the if the it was a get API, it's going to check do you have view, and it'll take the application scope and use that as kind of an allow list on the underlying permissions. And if the scope was like let's say manage users, but they're trying to access a file via that app, it won't have you on that underlying underlying file even if the user has access, because you didn't grant that access to that application. And it's all just kind of all happening in the same place. Okay, it makes sense. So you essentially you reduce the kind of like the pre computed set of collabs, as we're calling that that the user has on the file and then saying, Okay, this API code actually requires this collab. So going back to hey, does this scope the correct one and does the user have the kind of makes sense. Yeah, it's just sort of two phases we can you compute the user's natural permission on objects, and then we we basically filter out anything the user didn't any access the user didn't grant to the application as part of the OAuth to flow, because it as part of the OAuth to flow right we send the user to a screen that says, Are you okay, allowing this application to read or write your files or manage your users and so there's, there's part of that all slow that we have to maintain. Yeah, yeah, the off consent. Yes, exactly. Yeah. Okay, but John, it's been, it's amazing, man, we did a very deep dive on box, which is a both a complex and like large system on and I'm sure the visitors will really appreciate learning how to affect a large battle tested lonely production sass deals with authorization. It's, I really appreciate your time. Yeah, thanks. Thanks for having me. And if there are any listeners out there that are working on authorization systems that are interested in more detail, we're always open to conversation with open source developers or companies about what they're building. Excellent. Thanks for the, for the ask. And again, as John was saying, you should reach out to him if you're building something that that might be useful for box. That's it for today's episode of authorization and software. Thanks for tuning in and listening to us. If you enjoyed the show, be sure to subscribe to the podcast on your preferred platform so you'll never miss an episode. And if you have any feedback or suggestions for future episodes, feel free to reach out to us on social media. We love hearing from our listeners. Keep building secure software and we'll catch you on the next episode of authorization and software.