And what happened with these two passports was, is that better? Okay. Yo nací en Mexico y tengo los dos pasaportes. And from a very young age, I learned about this divide, which is something that's still part of my work. So Sheila mentioned this in her talk earlier about imposter syndrome, one foot in, one foot out. So with WordPress themes, you may have one foot in code, one foot in design. You may, a lot of us in this industry have introversion. So we like working at home on our computer, but we have a, in a venue like this, we're on the extroverted side of it, right? So we're sharing externally. So this, the reason that this was an awakening for me was because I realized this background, where I was the one who was the foreigner, right? A white kid in Mexico, playing soccer and trying to learn everything about that, was that I got really good at being a bridge between those two worlds. That's really what my work is today, bringing people from, that are not part of the equation into the equation. And so I wanted to talk about inclusive design in that way, because it means good design for everyone. And I've discovered that it coincides with the same principles and philosophies of WordPress. I'll talk about that a little bit today. But before we get into what inclusive design means, let's hear from some other people, some products, some real companies that you may or may not have heard of. Barclays, which is a bank in the UK, really understands this. They started doing product development, starting from one person's perspective, which was a blind user. This is Robin Maylan. And she wanted to be able to use the ATM without having to get help from someone else. So they started developing a mobile app, which had voice control. Well, what they found out was, it wasn't just a little bit of work for one person or an edge case, it was actually something that a lot of people wanted. So it turned into a commercial opportunity because that her specific solution led to a broader development for a lot of people. And if you think about mobile banking today, it's pretty standard, right? You can upload a check, you can do a lot of things that you would normally have to go in before. So Barclays figured that out by starting from someone who was typically excluded from that experience. And this is another blind person, Robin Christofferson. She talked a little bit about the same where they started from her access problem and came out with a solution that was really good for everyone. Oxo figured it out. This is one of my favorite product stories. It's not software, but it is a product that I have in my kitchen. Maybe you probably have seen it too. It's a peeler. And this story, Sam Farber, he's famous for making enamel cookware in the 60s. He was with his wife and she was trying to make an apple tart. She couldn't hold on to the metal, you know, those really sharp, small, old fashioned peeler. She just couldn't make it work. She had arthritis. So he thought, well, what if I could make something better? And it turned into what we now see, this kind of black handle. And if you get a chance, check out the video. I have a link to it in the slides. You don't have to write any of this down because all the slides have the links to all these videos. But basically, they go into a lot more of the detail. They took kind of like a rubber handle from a bicycle, you know, like a BMX bike and stuck that metal peeler into it and that kind of led them to discover this design. But it was also a commercial opportunity for them. It turned into a $60 million company in just a few years. So really going from one to many. Roku is another really great example. Who has experience with this? It's basically a TV alternative, right? So you can get rid of your cable and you can do streaming. But really what they got was simplicity. They started with the customers in mind who didn't know tech very well. So someone who didn't want to deal with like a hundred button remote. Or if you've looked at the interface, it's like really blocky and you can really only go one direction and then click. One of my favorite features is the one click Netflix button. So you just click Netflix and you're right in. But there's CEO. And by the way, I wanted to give a shout out to Leo who was not here. He injured his wrist. He was supposed to speak. He pointed me to this example last word camp Phoenix last year. They started with what customers want is easy. Again, starting with, I think they had in mind elderly folks who weren't really familiar with technology who just wanted to plug and play. And they also went to a very, very commercially successful. I think their market capitalization is like $7 billion now. This is an example from my work on WordPress.com. We scratched our own itch. We had a very popular blog that had 43 million people subscribed to it. And when I first joined, I was doing theme announcements and I would just break out in a cold sweat every time I had to post to this blog. Like what if I make a typo? The whole world sees this. So we had this little plugin that was just a JavaScript alert that said, are you sure if you click the publish button in WordPress? And it just popped up in your face and said, are you sure? If you hit cancel, it would go away. And if you hit yes, it went out. Well, we decided, why doesn't everyone else have this? Why are we just solving it just for ourselves? And so recently, just last fall, we launched this new screen which gives you one more confidence check before you go out to the world with all your content. And it also includes a visual preview, like in a tablet and a desktop and a mobile view. Basically just trying to reduce the anxiety around publishing. And we learned that we could solve that for, not just ourselves, but for everyone. So what do these product examples have in common? The first thing that comes to mind is they start with people, right? They start with a specific person. AJ in the last talk mentioned this. They were starting with someone really specific in mind. The other thing is, they point to the purpose of technology, which to me is to help people start. That's what it's for. And this coincides with WordPress. So going back to why I wanted to give this talk, this has kind of started to line up for me as I learned about inclusive design. This shares a lot of the same philosophies with WordPress. This philosophy of openness. At the same time, offering a practical way to expand your market by recognizing who's excluded and saying, I'm gonna open that up. And I'm gonna find a bigger, addressable market because of that. And WordPress has always been about websites, but it's really more than that. And this is something that's been learning for me. It's not just like technology. It's about freedom. It's about possibility. And really it's about carving out a livelihood, whether you're within that WordPress ecosystem, like a lot of us here, or if you're someone with a website, like the customers of our customers. We're trying to create a livelihood for ourselves and for them. And inclusive design really resonates with this because it talks about just like WordPress, democratizing publishing, but it can go beyond that. It can go to democratizing marketing or democratizing small business. You don't have to figure out all those same things because it's already been solved for you. I'd like to give a shout out to two strong voices in the inclusive design and general design community that have really made a big impact on me. They are Cat Homes and John Mita. Cat was formerly of Microsoft. She did a lot of work in this space. And I'm gonna mention and show a few of her slides from her research. And John works at Automatic and he's the head of design and inclusion. So it's part of his job title. And he's been a mentor to me and I really appreciate what he's taught me. One thing he does every year is he does a design and tech report which comes out with South by Southwest Festival. If you're familiar with that, it's in March every year in Austin. And part of this is like a broad overview of everything happening in technology, but from a design perspective. And one thing that captured me last year in this report was that adopting a more open approach means you have a larger market. And it finally kind of clicked for me. From Cat, I learned about turning accessibility all the way upside down. So instead of saying, that's an edge case that I have to figure out kind of at the end, you start with that. And realize that that person solving that person's problem may solve a lot of people's problem. And also really just, I'm gonna talk about this from her research too about building empathy and really getting to know what some of those problems are. So if there's one thing you take away from this talk, I want it to be this. Inclusive design is for those that wanna make great products for the greatest number of people. Instead of minimizing access and empathy and diversity into kind of a corner case, something that you maybe a stat in a chart that you might look at and say, oh, it's too small to worry about. Really keeping it central. Because you broaden your reach, you have a broader market to work with. It really is a design for everyone. And so I've been told that every presentation needs to have bullet points, so I put on some here. Just kidding. But this is what we've come to. So starting with Cat's work at Microsoft and our work in the last couple of months at Automatic, we started to come up with what we consider to be inclusive design. This is kind of a loose definition and I'll go into each of these. So recognizing exclusion, who's not part of the picture. Broadening perspectives to build empathy. Bringing diversity into teams and processes and that comes in many different forms. And then as these product stories showed us, solving for one case and then extending that to many. That's really the root of the inclusion part of it. This slide and one of the next ones is from Cat's work, Cat Homes work at Microsoft. And this was a really big turning point for me because I previously thought of accessibility as what basically was a medical condition. Like a permanent injury or someone who's deaf or blind. And in her work, she describes it as a mismatched condition. So instead of being the system that the person needs to adapt to, like a prosthetic leg or a hearing aid, the software, the product should adapt to the person. The human is at the center. So I realized that instead of thinking of them as edge cases, they should be more as mismatch between the person and the product. Here's an example. So imagine a permanent disability, someone who's like maybe an amputee, one arm. Then someone like Leah, who just got in an accident and has an injury, right? Or someone carrying a baby. They all have the same interaction with that product or software, but it's a very situational. One of them is temporary, one of them is permanent. Another one that Cat talks about is picture a video screen and with captions, right?  So they need to see the words. But pretend you're, or imagine that you are at an airport in that crowded kind of gate area and you're trying to watch the CNN news that they have, that will help you because it's so loud that you need to read it. Then imagine another situation like for example, someone learning a new language and the words are being spoken, but they don't know how to spell it. So then it's popped up on the screen as well. Those all three cases are mismatched where we need the help of the visual of the letters, but we don't necessarily have that same permanent disability, but it still applies to us. Microsoft has a fancy name for this, which is persona spectrum and there's a lot of other examples. In the slide notes, there's a PDF that Microsoft made called the Inclusive Design Toolkit and they go through quite a few of these. Building empathy is something that we're trying to do more at our company and we give each other these challenges so it may be something like go in a ride-along with a one-on-one which is like a video with a customer or use a different, if you're using iOS, switch to Android and see what it's like so you can see something different. And the reason for this is to see things from others' eyes. If you're not familiar with empathy, it's basically being able to understand the feelings of others, put yourself in their shoes. And for me it's been about, like I said earlier, not thinking of them as just stats but as real people. A lot of times it takes a real connection to do that. So last fall I did an empathy challenge where I worked from an iPad. I left my laptop at home and even at home I used it and I didn't have my big screen and everything. No mouse, which was hard. And it was difficult but it was actually very eye-opening. In fact, one way it was better was on an airplane. This was on the way home from Ireland. I got onto a Zoom call with my colleagues and I was able to do it from my plane seat which was kind of amazing because it was so small. I didn't have to bother my neighbors. But what it taught me was, now when I get back to my regular life, my computer with all my fancy stuff, I no longer think of the person who's one way we talk about this is across the digital divide meaning they do not have all the fast wifi, all the fancy computers that we do as tech workers. Someone like that, I don't think of as a stat anymore. Like, oh, I have 5% tablet users. I should maybe think of a responsive design. Now I think of it like, that's me, right? That's, I couldn't work, use the editor. I couldn't click on something, I couldn't target it. So it really puts it into human perspective. Another part of building empathy was for me was broadening, opening up the voices that I was listening to. Ted Talks are really good for this. If you're not familiar with, just go to ted.com. But this was a specific one that really hit home for me. Sheena Iyengar, she's blind. And the topic of her talk is not what I'm talking about. It's something else, very interesting about psychology of choice. But she, at the very end, she gave a snippet of her life as a blind person about picking, I think it was about picking color of nail polish or something. And the person she was talking to said, well, why would you care? There are two versions of pink or something like that. She's like, I care a lot. Just as much as you do, you know? Just because you couldn't see it doesn't mean she didn't care about it. Videos like these really helped open my eyes to how other people see the world or how they experience the world. So inclusive design, that started to realize building that empathy connected me with that other person. So we've been talking about the higher level stuff, right? How it connects to WordPress, the philosophies of openness, freedoms. What about bringing this into our work? So I do wanna spend a little bit of time on actual practical actions we can take. We've been trying to put this into practice in our work. So we have a checklist that we've started. And I brought a few of them with me today if anybody's interested afterward. But it's always available online at design.blog slash inclusive. I wanna pull out three of these and speak of them as examples. So the first one is helping your audiences inform your design, which as AJ mentioned, it's as simple as doing a survey. We're talking to people directly. Especially important if your team isn't diverse or if you're one person solo or you aren't going out of your house. We're pretty diverse at Autobanac. We speak about 79 languages, including Español. We're around the world, but we're still fairly technically and skill heavy in one direction so we can still learn and adapt more. But the thing that I've been learning is that diversity isn't just gender or race or background. It's a lot of different things, neurodiversity, right? The spectrum of autism, the spectrum of other types of mental health comes into play when you're in design products. And this came a practical example from WordPress just a couple of weeks ago, February 5th. Tammy Lister in the WordPress core project mentioned updating some of the error messages in WordPress. There was one that was said the word cheating. And someone from the community, Peter Wilson, I think he's from Australia, said, hey, it's kind of alarming. It makes me feel anxious when I see that. Like I did something wrong. And so he made a change to make it something like an error has occurred so it was less anxious. So that's an example of letting your audience inform your design. That goes along with choosing copy and imagery with care. These examples, these screenshots are from this inclusive design checklist. Yes, yes, what you're saying is it comes across as an affront, like an attack. Instead of you might want it, but not right now or not in that way. Let me make the action. Yeah, it's pushing it toward you. It's funny because I think this wasn't intended to be that originally, it was meant to be humor. But this was also when WordPress was a few people in a room that were mostly from Texas, I think. And they didn't have a necessarily inclusive view at that time, right? So it shows the history of the project as well. So one of the principles of choosing copy and imagery with care to more represent, in the case of WordPress and WordPress.com, is our customers are a diverse group. We went through a process where we hired an illustrator named Alice Lee to update the imagery. Previously, when you would go to sign up for WordPress.com and you would go through the sign up form, a lot of the people or illustrations look to kind of like me or like some of us are like young-ish, white-ish, guy-ish, with like an Apple watch and a coffee, you know? I mean, that's great because I identify with that, but a lot of people don't. And especially like I mentioned, the different types of diversity. Someone who doesn't have access to a fancy coffee shop with wifi may not wanna sign up for that service. So Alice went through this project where she updated, this is showing it in on real articles on our blog, but she updated the imagery to be more representative so that people could see themselves in the design. And this wasn't as near and dear to my heart, so designing for low bandwidth. This came up earlier in one of the talks about automated marketing, if anyone was in that one this morning, where some people were abandoning their carts because the images weren't loading on mobile. This is what this is about. So there's a trend in, as companies like Twitter, Facebook, Google go into new markets, whether they're like India, where there's less access to fast internet, they're developing these alternate products like Twitter Lite, Facebook Lite. And if you read the copy of the marketing, the value proposition, as AJ would say, it's about faster, it's about less data that's being used on your plan. Think of someone who pays by the megabyte or by the gigabyte for their phone plan. So Facebook was losing market share to WhatsApp and other apps in India because it was using up too much bandwidth. But really it's not just about the bandwidth or the data use, it's about getting people to what their goal faster, making sure it's accessible to everyone. So in our project, one of our software projects is called Calypso. It's the interface you see when you log into WordPress.com. And we have this site that's called iscalypsofastyet.com. And every single one of these pointers is when someone has changed code, pushed a change to GitHub, for example, and you can see how there's kind of a huge spike there. Well, it alerts that developer or designer to say, hey, you made a change that is gonna adversely affect how fast this loads for someone. And we take it pretty seriously. We want it to go down. It's not up and right, it's like down and right. And I also want to bring a reminder from Wapoo, mind the mobile, super important. And I would say before last year, I didn't think of this as inclusivity. I thought of it as more of a technical challenge. But there are people that are definitely gonna not come back if they don't have access to whatever message or whatever product you're selling. So those principles, like it's available online, I have a few hard copies as well. Design.blog inclusive is the URL if you wanna look into more. There are quite a few other checklist items there. So bring this back to the bigger picture, inclusive design and those mismatch conditions. I realize that you're solving for one person, but you're really finding a universal solution. And I also wanted to share, if you're interested in sort of how things can go wrong with tech, check out this book by Sarah Wachter-Betcher called Technically Wrong. She goes into more detail on some things like Snapchat, Facebook, especially things like artificial intelligence and machine learning, which are new to us. And research has shown bake in some of our biases, even though we don't think of it that way, but the people who program it have biases and they come out in the algorithms. So if you're interested in that, I don't have time to cover that, but check out her book. She talks about inclusion and design as well. And she has a book from a few years back called Design for Real Life that really motivated me to learn about this topic as well. Once I discovered Cat Home's work and with John's prompting these voices, right, standing on the shoulders of giants, I finally got it. I finally understood how my work and my thought patterns are influenced by my biases and by my background. And I also noticed that I wasn't always thinking of those who were left out. I was designing for myself. I was designing for what I already knew, like a default, right, or a normal person. And really, now I've opened up to it. I'm thinking more broadly. I'm thinking more openly. And I'm starting to see that when I don't minimize the access, which is a key part of inclusion, right, those mismatch conditions, if I don't minimize that into a corner case, I can start with that. I can start with those people. What's their story? What does success look like for them? What are they not being able to do that I'm able to do? And I've just come to realize that inclusive design is designed for everyone. People like you, people like me. Thank you very much. Do we have time for questions? And all the links, so on my website, sensible.blog, you can find all the Roku and Oxo, all those case studies. If you don't only watch that Oxo, the one about the vegetable peeler, that's a really cool story. My question is... Oh, hold it really close. My question is, how does this principle relate to the principle of progressive enhancement versus graceful degradation? Because I'm working on some sites that need to work for people who can't read at all in lots of different languages that aren't normally represented on the web, but we also want it to look nice and slick for people who spend money. Yes, that's a great question. So the question is, how does this relate to graceful degradation and progressive enhancement? That's exactly what this is. So starting with a mobile-first layout, the same idea is that make sure it works in the most simple case, right? Black text on a white background, on a say edge phone or something like that. Yep, or worse than that, right? Dial up with what would be, what would be a visually worse than that? Yes, exactly. And this is exactly what those light apps were addressing, was the, if you take everything else away, what's left? It's like the title, your avatar and your messages. And what's interesting, if you go to Twitter now, when I'm logged in, I started to see their light version pop up for me, even on my desktop, because what they realize is that everyone likes it. It's easier for, there's fewer navigation items, there's simpler labels, and it just loads faster. So that's, so the progressive enhancement in saying, what's their starting point ends up being the end product? And so, yeah, it's the same exact philosophy, actually. So it has everything to do with that, thank you. Yeah, I'd be curious to know what pain points that you guys encountered as you maybe mentally have gone this direction. Do you mean on WordPress.com specifically? Yeah, and maybe personally, like is there, there is a trade-off and so how, I think it's a better way, but what are the pain points and how you've addressed maybe that? Sure, one of the problems is finding folks to talk to to learn from. So I mentioned Cat Holmes. She worked at Microsoft and they had Xbox and they had Office Suite and they have, like all these different products. And they had people on staff who were like paraplegics. Super cool. They're the ones, so for example, she gave an example of, they're trying to make a mobile coding, so like I don't know if it was for C Sharp or something, but it was like a mobile app that you could code with. And the person who developed it was someone in a wheelchair who talks at the ceiling, right? Like who better to work on that than someone who has no keyboard? But that's been one of the challenges for us. So when we do those empathy challenges, we're limited by our local capability. So actually Cat came on as an advisor to Automatic because of that to connect us with some of those people. The other challenge has been around, it's hard work and so we have a lot of pressures from our stakeholders, which would be, so in my case, I may be the project lead and want to get it out faster. So anyway, we may be attempted to skip some steps, like test, so there's a couple of things you can do, like in Chrome there's a rate limiter and I think Xcode has one for iOS where you can make your network connection really slow on purpose. But that's annoying, so I may not do that, but that would be the way to start. And so probably one of the things is just getting into the rituals and habits of thinking of that first, thinking of that edge case, like I said, as the central thing, but it's really been an educational experience, just learning from more people. And as I mentioned, hiring and diversity onto the team, that should make a difference too because someone will say, hey, I'm from Kenya, right?  I spoke at a work camp in Nicaragua and I couldn't even load WordPress.com on the mobile broadband, which I had T-Mobile and it was like a 2G, nothing happened, just got like text. So like when I got back home, I said, okay, I'm gonna file a bug for that because it didn't work for that entire country, you know? Kind of eye-opening. Hi, my name's Mike and I have a question. Do you have any experience working with China and some of the firewalls that they have and the ways of working around that and some of the things that you found to be successful? I do not, actually. We're, as far as I know, we're blocked in China on WordPress.com. If not that, some parts of it are blocked, so I don't have experience actually, but that's something that I would love to learn more about. Do you have experience with it? Oh, gosh, I'm just a small amount working with China. Being able to use like WhatsApp, we're able to talk 24 hours with that, but with video has been difficult. I did make contact with them and they're using office programs and they sent me some attachments, word attachments that I downloaded to my computer, but a week later we were not able to download from that same email, so it got cut off. These are just some of the things that I've experienced, thanks. Thanks for the question. First I want to say I love your talk. It's amazing. But I do want to ask for those of us who either freelance or just are in the design business and really like this stuff and have an interest, but don't know how to phrase that to someone who might be the key decision makers. So for example, like a business owner, a business leader, your boss, coworkers, or teams who may not be too convinced on this or unaware, how do you convince those people or stakeholders that inclusive design is important, profitable too, which would be one of their concerns and good and useful for all. Great question, what's your name? Robert. Thank you, Robert. That's exactly me of last year. So I was not convinced. And some of it, part of it is the commercial aspect and part of it is personal. So feeling the pain myself, right? So that's where just those empathy challenges. If it's someone who's the business owner, I would definitely recommend looking at some of these case studies. It's hard to argue with $60 million from a fish to popular, but there's other better examples. So that design and tech report would be one. So it's put out by a venture capital firm in Silicon Valley. So it's not from a known name. So from Arizona, I work from home. People may not respect what I put out to the world, but this is a very respected, I think it's called Kleiner Perkins, Coughfield and Byers, but it's KPCB. Anyway, designandtechreport.wordpress.com I think, or designandtech. I can send you the link. But the point is, show them that this has been studied across the industry, and it's a trend that we're all going in that direction. Design is becoming more human. It's becoming more human-centered. The commercial opportunity is because you're, what business people call total addressable market, like the amount of people that may be interested in your thing is now gonna be bigger because you're not excluding them because they don't have a certain ability or they don't have a certain language, right? So I would talk to both of those things. I would get them involved in the research, and then I would get them personally involved, seeing, so one example would be making them watch a video where someone's interacting with what they're working on and that has trouble with it. Okay. Does that make sense? Yeah. But yeah, that's a great question. Thank you. The resource that I've gone to is the people, the guys that make opera, the opera web browser, they're like, they're very forward-thinking. They know that the smartphones that people are buying in Asia, about half of them have less than a gig of RAM on them and they're very low resource and they have a lot of resources for developers how to design for the next million, your next million customers because there are more people in a small circle in Asia than there are in the rest of the world and they're the people you need to design your web for. Yes, and opera has that really cool feature where they cache all the content so it's already kind of locally available and it's not, you don't have to fetch it across the world. I can't remember the name of it, but for mobile. Proxy servers, thank you. So you mentioned the next one billion Facebook in their marketing for the Facebook Lite. It says this is for the next one billion people, right? Just like you mentioned, that I would say across the digital divide or don't have any kind of technology that we would call like laptops, the feature phones. What does the web look like for them? But it's also huge, speaking to Robert's point, it's a huge, how can you argue with the thing that you're making available to one billion people? Even if your product is 99 cents, that's a lot of money. So it makes business sense to make it available but it's also the right thing to do. That's one thing I didn't really mention but what an insight for me was it's not cause we're good people, but it is, right? Like we use WordPress because it's free to use but it also has the freedoms. I think Matt Mollin like says, we make software, not for free, we don't make software for free but we make it for freedom. So that's kind of keep those in mind. Like there is a commercial aspect to it, like we wanna carve out our livelihood like I mentioned but we also wanna bring more freedom to the world. So I think that's the message is that it opens everything up.


7, 0, 1, and then that would be 2, 5, 0. Whoa, dude. I just saw a meteor. I just saw a meteor. Let's just go through it and fix it up.


So that was a basic example of how you cheat at a fun spot but it's thinking outside the square or should I say outside the t-shirt apparently the official rule is that some part of the actual body to its substantial limit of inches has to be inside the t-shirt. Don't count apparently because you don't wear a t-shirt on your feet. Well not in this part of the country anyway.


This study found that older recipients and those who received thymoglobulin were more likely to develop CMV disease after kidney transplantation. Additionally, no association was observed between the immunosuppressive regimen and CMV disease except for thymoglobulin. This article was authored by Viviana Navarro Rodriguez, Alvaro Herrera Muñoz, Adrián Castro, and others.


I'm really excited to meet the the whole new cast of Shane saw man since everybody died in the last episode Please gentlemen, welcome back to the channel. My name is Jack the Buster. I should know be a little bit of night of our chance I mean watch her called from Kyoto the last of the four episodes remaining in this season and like I mean last episode everything Just popped off more than it has the entire show thus far with everybody getting shot and him and no Sacrificing herself and I keep still being in a position of death not looking good If you knew the channel top of something hit that big what bell and let's hop into it surely more cannot die Oh, we went back a little bit and then I was alive Oh power ran away dude, then she is so important his heart. Oh wait.  His laugh is so like on point So did she trade a nail for her snake? Come on did she I mean they might be even match especially if did she is like much more powerful than he realizes Which I'm going to assume since they're after his heart It's blue on the blades as well from the ghost I'm guessing so red and blue. It looks crazy. Oh It's worth a heart, what's he doing?  I'm guessing Oh, we got caught. Oh Maybe not tranquilizers I do not think he cares about him then she it doesn't work like this to them.  Oh wait, no, no, no, no, this is pouncing position. Oh My god. Oh, he's so fast It was kind of like a personal thing too. He's he's what's his name's grandson the the mafia guy Not mafia Yakuza, what is happening? Metal isn't even here Honestly, that she was gonna be like the villain Whatever or Tendo Well, they're all getting word other devil hunters That's a good question is mark about that who knows anymore she has a scar going that or I guess on the right side He has one on the left side. What if it was like from a similar cut like they were near each other. Oh This is a train that they were on. Oh, yeah, they were waiting for her. Oh my gosh. How is she alive?  Hey, did she like take their hearts? What were those holes? Ideals meal. What is makama man? Somehow she's not who I'm most scared of right now though She knows that the enemy is probably after denji so she knows of his importance Maybe the higher-ups of the devil hunters organized like sort of like a coup since they have the most information about him I'm gonna put my tent foil hat on and get to the bottom of this and trust me I will get to the bottom of this that just seems like an event that would occur at like the very beginning of a season like just madness Oh, no, oh Makama's back that she's got back. Oh my god So I'm guessing these two were to because she was coming here for a business meeting with like big wakes I'm guessing they were to like escort her. Oh wait. They were bringing the prisoners to Makama. Oh, what? Bird is watching. It's the government Brother I can't predict the goddamn thing anymore Is it Makama wait, is she doing that from the bird? He over Is happening man What is going on? What is happening? This is some ability.   It's like a it's like a like those dolls They're like like witches have where I can they stab the doll it hurts you But she has to use a human to do it. Oh my god all of them But she doesn't know the two most important names or can't kill them or what I don't know Oh my gosh, and they're just not allowed to see I mean it makes sense If there's any loose end about how that works and how her power works, and if that ever got out I mean he would keep their identities a secret They're all just dead So they know who it is from what's happening Damn this show is crazy man The grandma miss her shots Ain't no way comedy bro a knife wait, we don't know her double contract. She was she didn't share There's no way Coleman. He is as strong as them if not stronger Oh She fast She can't kill didn't she so she can't shoot his heart So brave yes, she used another fingernail to summon the snake again So she has like wait for them to regrow if she runs out I guess they that's a really like light price for how strong that snake seems Oh, it's forgiven for that performance. It's chill man. It really is in the grand scheme of things, you know Damn, he really took the shot How did we end up here man? You can tell they were pretty close because they were both sort of like that that's shy type as well shy kind of nervous unsure of themselves Oh, she doesn't want to be a devil on there, man. I wish goodbye. Just just leave. I'm sure she'll understand Oh, he lived Madoka Madoka She's so chill Did Jesus for right everyone is decks for the non-humans The the gun that will just got shooters like that I guess All the visions absorbed into four All under Makama too. Oh my gosh, no, but they're both kind of scared of her now They don't know they have a continual idea of what she does I don't blame you brother.    It's not just like raw fighting It's devious man It does feel like it's it's like death note mixed with that the head crusher from the boys If you've ever watched the boys except it's yeah, like you have to use I think my My analogy of like the little straw doll that like like are used like stories of stuff I give you stab it or anything you do do it happens to the person it's connected to it's like that But she just needs like a human sacrifice to perform it on someone else I think that's completely accurate based on what we saw unless it wants us to believe that's how it works But it's completely different, but that's genuinely like around what it seems, but hell yeah, Copini having her moment I'm proud of her. Well, that has been it for episode 9 of our chainsaw man watch your phone called from Kyoto I can't believe there's only three episodes left I'm guessing the season's gonna end off on a decent cliffhanger if you need a channel drop us up and hit that big What bell I'll catch you in the next one. Peace. Peace


Hi, everyone. I'm Kadeshia. And today we'll be looking at how to use the new deployment protection rules feature with GitHub Actions. I have software engineer Bo with me today from Datadog who will be showing us how to use their new GitHub app to add an extra layer of protection to your deployment environments. Bo, take it away. Thanks, Kadeshia. I'm a software engineer at Datadog. And one of the most stressful parts of my job is production deployments. I have messed up many deployments and brought down many a prod. Any additional bit of certainty I can get from my CI CD process is so valuable to me. And to that end, we've made it so that Datadog monitors can now act as a pass or fail gate for GitHub deployment pipelines, which means that unless a Datadog monitor is in an OK state, the deployment pipeline will not execute. So let's get started and go through first how to enable this feature. It's very simple. From the Datadog integrations page, you can search for the GitHub tile. For customers that already have their GitHub app installed with Datadog, you can just select your GitHub app here and make sure there's a green check mark next to the deployment protection rules. If there isn't, not to worry, you can go ahead over to the GitHub UI and under the GitHub app settings for permissions and events, you can go ahead and check the deployment protection rule here. Alternatively, if you want to either create a new GitHub app or you don't have an app already registered with Datadog, you have to click this button here, the Create GitHub app button, and just make sure that the deployment protection rule is checked from the feature list. So for this demo, we will be using an existing GitHub app, the Datadog DD Sandbox app. And this app has access to a single repository, the GitHub Gate Service. And the goal of this demo is to deploy or attempt to deploy the GitHub Gate Service to production and hope that our Datadog monitor protects us from doing so because this is going to be a bad deployment. So first things first, let's go ahead and create a new monitor that will protect us. Any Datadog monitor can be used as a deployment gate. I'm going to be using the Composite Monitor. And I find that the best monitors for production deployments are pre-production monitors because pre-production is the staging ground for production. As you can see, one of my monitors that I made before is an alerting stage that is on purpose and will play a role later. So I'm putting some demo text here, but the very, very important thing to note is the tags of this monitor. The way that Datadog knows that a monitor should be evaluated given a GitHub deployment pipeline is through the monitor tags, and specifically two very important tag key names. The first being git underscore m. For our purposes, it is going to be prod. And the second very important tag key name is git underscore repo, which is a combination of the owner, which for our case is DD Sandbox, the org, forward slash, and then the repo name, which is going to be GitHub Gate Service. So once we hit Create for this monitor, we are hypothetically now protected from executing any bad deployment for production. But let us test that theory. I'm going to go ahead and navigate to my actual repo here. As you can see, the git repo here is DD Sandbox slash GitHub Gate Service. And just to double check the MF, we can go here to Settings. And under Environments, we have a single MF configured called prod with one protection rule. OK, let's go ahead and actually attempt to deploy to prod. I'm going to execute my deploy to prod workflow on the main branch. And we're going to watch it go. OK, so as you can see, our deployment failed because it didn't satisfy other protection rules. And fortunately, we can see exactly what protection rules that was from this comment. If we click this link here, it will actually take us to the monitor that we created just two minutes ago. And as we knew before, we do have elevated 500 type errors from our service. We should not deploy to production until that is resolved. Thank you for watching. We're really excited to see what people use this feature for. Keep in mind that monitors and deployment rules have a many to many relationship. You can have as many monitors as you want configured for a single deployment protection rule. And you can have as many rules as you want all reading from the same monitor. The sky's the limit. Thank you so much. And thank you to the GitHub team for making this so fun to integrate with. Thank you for watching. Thank you so much, Bo, for taking the time to show us how to use the new DataDog GitHub app to add deployment protection rules to our environments. If you have questions about this feature, be sure to check out the GitHub community and leave us your feedback and comments. Be sure to like this video and subscribe to our channel so you don't miss any of our future videos. And until next time, bye.


Welcome to another edition of RCE again. This is Brock Palin. You can find RCE online at RCE-cast.com There's a RSS feed there as well as a link to iTunes for your favorite hot catcher now I have again Jeff Squires from Cisco Systems and open MPI Jeff. Thanks again for your time. Hey Brock. How's it going? All right So we also have off a website a link to your blog and you've actually answered some of the questions I've had recently being an MPI noob myself and thank you very much for that and I see it actually sparked a little bit of discussion with some of those questions. I think that was good. That's all good for people Yeah, yeah answer these kinds of questions MPI could be quite the mysterious beast. Yep Yeah, and so you can also find all of our Twitter handles and usual contact information on there as well as a list of all of our Backshows and things that we're looking at getting a hold of if you have contact information for any of the shows We haven't done yet. That's on that list. Please let us know on the contact form Today is kind of we've been going down a little bit of a path, right? We talked about Hadoop a while ago and Hadoop has come up more recently and Today's show is kind of related to that, right? So, yeah when I was keep finding these different pieces of software Lucene Hadoop, Notch, and Solar, they're all kind of related and they all live out of patchy.org and They're all Java based, but they're also all designed to scale very large scales when we had Cloudera on earlier talking about Hadoop You're talking about massive, you know multi-pedabyte clusters and I'm sure those have only grown since then so I think something coming on You know the definition of HPC is just kind of changing and this is kind of a contentious topic We know it's it's not just the same traditional HPC that we've we've seen No, there's a lot of big data number crunching going on that you know loosely could be considered HPC It takes a lot of compute a lot of scale a lot of processors disk memory all that stuff You know as a matter of fact in open MPI We're just going to be adding Java MPI bindings to our development trunk in the in the near future and the reason for it Is because the Hadoop guys Have become interested in using MPI as an IPC channel And you know they're in the in the map reduce world their reduces are becoming so computationally complex that they want to go parallel and they need a good IPC mechanism for it So, you know some of these things are starting to have very interesting Overlap so what you didn't traditionally consider as HPC, you know, it's starting to get in there And I think today's project kind of fits in that same category Yeah, anything bigger than what you can do in a normal traditional programming and server environment Gotta start doing these things at these massive scales. Yeah, so let's go ahead and introduce our guests today Yeah, so our guest today is Simon will now or I believe he's over in the UK And he's one of the people involved with Lucine. So Simon, why don't you take a moment to introduce yourself? Hey guys? Good to talk to you Thanks for giving this short introduction My name is Simon and I'm a patch of Lucine committer and I also happen to be the Lucine PMC chair Which is kind of an official position. I've been working with Lucine since 2005 and I'm a committer It's since 2006 and I spend a whole bunch of time and working on this project. I'm pushing it forward you know spreading the word and in my day-to-day job, I worked like 50% on patching Lucine specific stuff and The other 50% I usually spend on customer projects I Share a company with a couple of other people. It's called search workings This is also related to search workings.org. It's a community portal about Lucine solar and all these open source Search stuff and I run a conference in Berlin. It's called Berlin bus words. It happens annually and this year. It's in June So what can you give us rundown what Lucine is exactly? well Lucine's basically in a contrast to all the other things I've seen on your podcast It's it's just the library, right? So it doesn't do anything. It's not an application. You can't start it up It's just basically Java library Which solves the hard problems in information retrieval the basic information retrieval like building in index retrieving documents and not analyzing text And making everything in there are very very efficient. So it's basically a high performance search engine library entirely written in Java So does that mean you entail indexing as well? So I mean you mentioned accessing documents and searching So assuming Lee there's there's a bit more to it than that, right? right so In Lucine we have we have a notion of a document which is which basically corresponds to some kind of an entity But everything is a somewhat schema less So you don't have to really define the different entities you can just throw it in it works and each document has a set of fields The you can you can just imagine it as a spreadsheet a database with one table All right, and the Lucine takes care of building up this table and making this table accessible So the point of this is to be able to search it really fast, right? So do you build a separate index for this? I mean how I actually know very little about search embarrassingly little about search So how does you know a typical search work and how do you make it fast? Well in a contrast to a database in Lucine or in an Information people library you have something called a reverse index or an inverted index. So you basically index the terms the unique terms in a document and then map those terms to the documents where the term occurs in and Everything you do is when you type a query is you get a list of documents those terms occur in And how we make this fast. Well, there's a couple of things we do on a technical level, but Basically, I mean, where should I start? What what we do is we every data structure in Lucine is read-only right and we have some some standard Algorithms in place which writes stuff to disk makes it makes it persistent and then people can can load this indexes and fire the searches against it and You know once you have updates you write a new little index to disk and then you work it in the background together And everything is basically read-only. You don't change anything all the data structures are highly efficient You we make efficient use of of the infrastructure file system caches and all these kind of things In contrast to to a sequel database or something like that You don't need to do update everything right you don't need to update anything in place not a disk that I think makes a big difference and Lucine is a library which is very very much tailored to the purpose of full-text search Once you do something else with it you could easily get into problems, but we can elaborate in this later So Lucine would not be good for kind of replacing a traditional database if your information is changing rapidly Well, that's that's basically two questions One is the data is changing rapidly. The other one is can you replace a traditional database with it? The their answer to the second question is you can but you have to denormalize your data model, right? You only have one table. You have to put everything in one table. You can't do more than one There is some developments which implement joins based on Lucine But they're there they have a lot of limitations. This is not they don't have the power of a sequel store When you when documents update rapidly Lucine still works. We have something called a near real-time search But it works slightly different and it involves More disk I O and it's it's basically less efficient space wise because in Lucine You don't have an update procedure. You only have a delete and an at so basically if you want to update a document You delete all or mark all previous documents with the same key as delete it and then add the new one on top So it sounds like you know, it doesn't have all the features of a full database Things like MySQL and postgres SQL and these other databases where we've traditionally stored a lot of data have full-text search and Table partitioning and you know the MySQL MBD cluster Why would I want to use Lucine over one of these traditional systems? Well, the the first problem is how is this full-text search implemented? I'm I'm not sure about every every implementation But full-text search is more a bit more is more than than just putting something into a database and start indexing it Right, you could have a whole bunch of analysis in front of it Like you want to you want to remove thycritics you want to tokenize your text you want to put synonyms in remove stop words do stemming all these information retrieval related things Put some machine learning in front of it. Lucine supports all these kind of things and it is actually made for for a full text Full-text-ish queries Just go one step further and say you have some positional information like term a occurs next to term B How would you do this in a no sequel in a in a MySQL environment, right? You don't have those notions and the basic reason is Those databases were not made for information retrieval. So let me take a step back here. Why? Why is text search important? What kind of fields is this used in what kind of applications? I mean obviously the the most obvious thing I can think of is you know You go to Google and you get like the quick search and things like that, but what other areas is this useful in? Yeah, you're absolutely right. I mean if you go to Google on type of web search and you definitely at some point hit an inverted index There's much more about that in Google. There's a lot of ranking stuff going on Lucina's ranking too. We can get to this later There's a whole lot of applications Starting from your little mobile phone running Android There you could implement some kind of on-device search Searching your emails Go on your operating system. You want to index your file system and find things quickly You have a web shop and you want to find You know things things you sell But you you don't want to rely on the database which only gives you exact matches You want to have stuff like spell checking and query suggestions and all these kind of kind of neat features Which I'll do sink and give you like faceting But it can even further and say hey you have a you have an application related to Geographic information and you want to find something within a certain geolocation bounding box searches This is where we sink and help a lot Yeah, all kinds of all kinds of information for basically every website has a has a Search engine behind it. Like there's a lot of content management systems integrating Lucina solar Yeah, there's almost no application these days without a search box, right? Okay, that makes perfect sense. So How did this get started at Apache and and what's the relationship with? Lucine and Hadoop and solar and others Well, let's talk a little bit about the history I think in 1999. Well, well, that's a long time ago for computer science, right? 1999 duck cutting came up with a little pet project. I think it was his first Java project he wrote a search engine and I call it Lucine and A couple of years later. He he donated the code for Lucine to the Apache Foundation, which is version 1.4 If I call recall correctly, that was in 2001 so the scene turned 10 last year and Well, what happened is that this library is further developed and gained a lot of traction a lot of people invested in it a lot of Also famous people or well-known people in the Java community it contributed stuff to the scene but at some point The the problems you want we wanted to solve in Lucine became bigger Like, you know, you want to index the web a Little library doesn't help you really right? Do you need to write so much code on top of this? Like you need a crawler you need a database where you store the link information in et cetera, et cetera and somebody came up with the idea of nudge and nudge was built as a web scale Crawling search engine It should be capable of indexing tons and tons and tons of data are basically in the entire web Well using Lucine for this work pretty well But you had to deal with all the the distribution between machines And you also had to deal with storing the the websites and the link information and run run algorithms on it like map reduce Or page rank algorithms map reduce came later and out of this when Google published the papers about HDFS GFS and Map reduce duck cutting founded yet another project as a sub project of Lucine called Hadoop and Basically out of this it became another top-level project Nudge became a top-level project and Lucine was left alone with this task for information retrieval So those are the relations between those projects So then is does Lucine require Hadoop to actually use like is it a library that assumes map reduce functionalities available? No Lucine is actually one of the project in the Apache foundation which which has zero dependencies So the core of Lucine doesn't need anything to run except of a Java runtime environment Yeah, there's a couple of modules we have that need third-party libraries But the relationship is the other way around so if you want to do some kind of indexing things You're pro and you're using Hadoop. You have enough data where that makes sense You probably want to use Lucine to actually build your indexes Another question kind of going back a little bit here. Does Lucine Only index text or does it index other things as well? I mean you mentioned documents, but can those documents contain things other than text? Yes, and no Let me elaborate here So the latest version of Lucine Is somewhat balanced to text so it it basically accepts a string for for a value For a document consists of out of one or more fields and each field can have a value And those values are strings, but in the current trunk developments. We moved from strings to bytes And we index those bytes. So basically if you if you can you know Materialize your data to bytes and you can make some kind of sense out of this Then you can index it with Lucine So once i've done that how hard is it to make a an index environment that's aware of say My collection of genomes or my collection of images That's an interesting question. Um, it's To begin with Lucine is probably not the right tool to search your genome database I'm not sure. There's a lot of people came up with crazy ideas like You know Some somebody came up lately and we were talking about to build a chess computer Based on lucine, right? You what you basically would do is index all the the chess games Published in a in a in a in a big chess database and then make good predictions out of this But you know everything you need to do is you need to you need to transform Your genome representation into bytes and if you if your application needs to do something like you know Give me all give me all genome strings starting with this string or or a similar to this string with in a certain distance Then that would be probably straightforward, but Well with lucine 4 Um, I would say if you if you break out of the of the common model of full text search It probably involves way more work than Than a classical application, but it could do it so The the major purpose for moving to bytes is actually A different one. So I can elaborate this if you want, but To answer your question. It's it's it's way harder than just doing a normal full text search engine So then on a on a related question here does lucine do other languages? Well languages other than english Things that are multi byte representations of characters and do the same, you know phonetic and search types of contexts span languages Yes So basically internally to be seen everything is udeavate bytes Right, so we represent everything in udeavate. We moved out of the java model of What java basically uses 2 byte for character or for a code point and We moved down to udeavate representation because it can save us a lot of space on disk and in memory But yeah, you can you can basically search everything Which you can represent as as udeavate you can also use some unicode compression If you you know if you search some asian scripts You can certainly go and say use some book who encoding to to save space The question if you can search another language It doesn't really relate to how lucine is implemented internally lucine offers a ton of language analysis tools for I don't know if never counted, but when I look at the source probably like 50 to 60 different languages From from hindi to polish japanese English Yeah, brazilian portuguese It's all there Some some of them are better. Some of them are worse, but um, you can basically Write your own if you're not if your language is not supported. It's all possible to see So you keep mentioning this is all written in java. What was the rationale behind using java for such a system? I think I mentioned this before it was duck cuttings first project in java and it just it just You know, it was just meant to be a little pet project That it took off in a way it did. I think was never planned in a way, but um You know somebody could argue you could do the language like c plus plus where more freedom with memory Here and there and you probably get more More performance out of it I say probably i'm not sure if that's true.  They mostly do the right thing Sometimes they don't and you wonder why but yeah, there's there wasn't like a real decision Hey, we write a search engine and let's write it in java. It just happened by accident Yeah, I feel your pain there, you know, whenever you're writing optimized code, you know, a lot of rules can go out the window unfortunately to to extract really high performance but Java has also come quite a long way since 1999 Compilers have gotten better and the jits have gotten better And the performance has gotten quite a bit better. It's still an interpreted language But you know, there are a lot of enterprise class applications that depend on speed and Optimization that that run in java today, but by the same token with all that is kind of a prelude to my question here Do you have some parts of it that are not written in java? Like I I know very little about java to be honest But I know that some applications will actually kind of branch out into c for their optimized parts Actually, we don't Which is not entirely true. Um, nothing nothing is written in c which is actually I know of the people use it in production we have a couple of implementations of our lowest level file system representation For a couple of reasons like we've seen there's a lot of i o especially when it merges in nexus in the background and When you when you write in write to an i o stream or to a channel on java You basically always hit the file system cache right you invalidate your cache And there's a couple of implementations which try to work around this and do like direct writes and Or do write-throughs and Talk to actually the the the operating system the teleoperating system. Hey, I'm gonna read this sequentially, etc. Etc Stuff like stuff like I operations you don't have access to when you're in java but To be honest our benchmarks show that The implementations are not faster than the java stuff The problem with calling into native interface in java is that the native interface call is pretty costly So if you have a piece of code, which is pretty hard or a method is called very very often You should rather rely on the chip compiler to go compile this in the native code and keep it around Then trying to work around this and use the java native interface. That's my opinion So what is lucine's core model like what's the What's it look like when i'm passing something into lucine does it Get kind of mung into a sequel kind of thing or is it completely its own deal? No, it's it's it's completely different. It doesn't do any sequel kind of thing Um Well, the most of most applications I would say they just pass in a user typed query was like I don't know you type you type some words you want to find let's say david bowie, right? And um, lucine internally runs a query parser on it. It understands a couple of um Boolean parameters and or not um And it has a couple of things like range arrange syntax you can you can search Numeric ranges or text ranges date ranges You can do wild card queries and prefix queries phrases and fuzzy queries but You know that that that syntax is Is only bound to the lucine standard query parser and there is like five or six of them around Which have totally different syntax it all boils down to the api and the kind of queries you can instantiate And I would say the a lot of people really do this built their own query parsers come up with their own syntax um Depending on what they really need and this is in my opinion one of the biggest advantages in lucine Is gives you the freedom to do it. You're not bound to anything um Syntax wise Well, I think you just hit a first there in uh rce history. I think that's the first time we've ever mentioned david bowie on one of our podcasts Uh Awesome. I I just read something about lucine and somebody had a problem on on stack overflow with Ranking david bowie against some other david and that's why I came up with it Hey, you know great real-world examples. This is all good All right. Uh, my next question is uh, what kind of back end storage do you use? I mean you mentioned, uh, you know native file system stuff. You mentioned one big table like interface What is it? Is it an actual database a customized kind of database or what do you do? Well, this scene writes its data structures itself So there's no database behind it all the all the low level code is is contained in the core jar Um The the actual persistent storage or Volatile storage, whatever you want to use is hidden behind an abstraction what we call a directory and you can build directory which operates exclusively on um on heap space Or you want to do some memory mapping or you want to use javas n i o classes, but it basically boils down to either a file system Or some kind of other Voluntiles storage There's a couple of people try to build directories for kasandra another no sequel storage um well solutions That in my From what I can tell it doesn't work that well or it doesn't scale that well But usually it boils down to uh, you go to the file system Your basic thing is your hard disk and and then all the data structures are written on top of this Does this answer your question? Yeah, it does let me ask you actually a further one. What happens if I use uh, What's becoming popular these days cloud based storage right if I've got A whole chunk of data and I'm a small startup And I host it out on you know amazon's cloud storage or or google google's cloud storage or something like that A How does lucine react to that? Is this in i o intensive and so the latency would would kill me or Is it more tolerant of that? How does that go? Well, um, that really depends. Um for the indexing side of things it can be extremely i o bound For the for the read part for searching. It depends on what kind of um file system implementation You use what kind of directory? Um, I would probably recommend to use memory mapping. So it basically pushes everything to memory. So you don't suffer from really slow disks um But in general lucine is not doesn't contain any any code for distribution It's it's it's meant to be on a single box If you want to do this in a distributed mode and you have like massive amount of data Uh or or a super large amount of documents You probably use something on top like solar or elastic surge of kata some frameworks on top of this So can you go into that a little bit more? What exactly do these Other frameworks that i've seen were built on lucine like you mentioned solar, which is the one that keeps coming out What do they do to really make this scale to a massive level? So Solar is basically a full flash application. You download it. You put it in a surflet container. You start it up and Put some documents in we are rest interface and then just search right And it uses lucine under the hood. So it's basically built on top. It's the lucine's official search server this other project like elastic search would focus on real-time search and You know document replication large scalability and sharding solar adds these features too in this in version four, which is the upcoming version But yeah, basically it gives you everything lucine doesn't offer out of the box It puts the it integrates the api Into an executable application. That's the major difference between it So what what are the resources for actually running one of these things? Does a scale with a number of documents or how many documents you're indexing today? What's what's the hardware to make this go well? So I I personally have experience with running lucine on very very low power mobile phones, which works right You're certainly not gonna index millions of documents on this Well, unless your battery runs out but The hardware requirements are not really the issue here It always depends on you know, how much data you have how fast you want to deliver your search results And what kind of searches you're executing there is some kind of searches are very very computational extensive Some of them some of the operations like indexing is very very io bound But um, you know, um, when you have a let's take an example Let's say you have something around 20 to 25 million documents All of the size of let's say a wikipedia page, which is roughly like four kilobytes of text And you want to you want to serve those? this data in in a very nice speed Let's say, you know normal queries like turnip or is a boolean queries coming back in something between five and 20 milliseconds Scaling with the number of requests you probably use some some hardware like You know two 12 cpus two times six cores Um, you probably need something around eight to 12 gigs of ram Your heart disk is not super super important, but SSDs usually give you a better performance here, too So do you cash much in memory for subsequent queries? So because i mean you mentioned a pretty low amount of ram there actually only one gig per core or so or even less right well We absolutely rely on the file system Cash, right? We don't try to cash a lot of stuff in the java heap space For the actual reverse index There is some data structures held in memory But those data structures are a tiny fraction of the actual size of the index So we basically rely on the fact that the file system keeps hot pages in memory as long as memory is available So you don't necessarily need tons and tons of memory to make this run Really really smoothly, but if your index starts to grow, right your file system cash needs to get bigger and bigger and bigger so um Basically, yeah, you can run this on a very very restricted environment and still have reasonable reasonable response times um If you if you go much bigger I use the term bigger here because I can't just say more documents or bigger documents. It really depends What kind of stuff you do and if you store the entire document or you only use it for indexing and then retrieve the ids Things like that Yeah, you can you can do this kind of stuff with almost every requirement every environment You could run it on your notebook and probably you know for a webshop nobody would really see much of difference to a big machine So where does a hide dupen come into this whole thing? I mean hide dupes is big cluster massive scale out thing How's how's lucine play with that? Huh That's a that's a tough question. Um to answer in a couple of seconds So let me give you an example. Um a lot of people have lots and lots of data and um to get all this data and index this data Um, they usually fetch it from a database But let's say you have so much that you have to put it in a on a on a file system like hadoop Where you need to process? a tons and tons of data um You probably want to build your indexes on the fly in the process of a map reduce job and um Basically when you do this what we've seen there's nothing special about it You probably have a couple of reducers in hadoop Which receive text and out of this text you're gonna You're gonna build a lucine index on on a reducer Um, and once this is done you copy the whole index to hdfs And your search engine fetches the new index from hdfs So hadoop can be used to lose to to build lucine indexes, but it doesn't have like a real relation There's nothing where you say hey, here's the component you use then it works with lucine There's a framework called kata Or it's actually an application That uses hadoop and lucine or hadoop underneath and lucine on top dual searching index building and um Use hadoop as a persistent storage, but there's not this close relation anymore Would a setup like that let you scale Lucine from you mentioned, you know a couple million documents on a machine with 12 gigs of ram and 12 cores to say You know 100x that you know get into the billion It also depends. Um, you can i i've seen indexes with 200 million documents with small documents And i've seen indexes with a couple of million documents where each document is massive Right and something around like five megabytes of text per document It always depends on what kind of stuff you index But usually is a rule of thumb 100 million documents is probably the limit for a single box So one interesting thing is the google book scanning project U of m's involved with that and last time i talked to the guys involved with that they were using solar interesting I think i'm guessing that's lucine underneath. Yeah. Yeah, absolutely. Absolutely. Um You can you can scale the scene to to a super large amount It always depends on the distributed system we build on top lucine itself Doesn't do any Distribution is not related to you know, it doesn't have an rpc implementation. It doesn't talk to Sockets or anything like that. It's completely agnostic Um, you put something on top like solar where you have replication and um sharding and distributed search And all this kind of things or elastic search the same kind of thing Then you can scale out massively. Absolutely So speaking of large indexes, what's the largest index that you've you've heard of that somebody's doing with lucine? the number of documents or How do you want to qualify it? huh I've seen a couple of people they're hitting actually the limits of a single index. So just for instance, we have a limit of um 2.14 billion unique terms per Per segment in the index and There's a couple of people actually hitting these limits When I say unique terms, right? So imagine how many unique words you need to have to hit 2.14 billion border This is a massive index in in terms of in terms of Size it really depends On average or not on average, but a common compression ratio is 30 percent of your original text When you build the index If you don't store the source data into this in lucine you can do that too You know this I've seen lots of indexes being more than 500 gigabytes per machine It really depends on what kind of documents how many documents So you can index 200 million documents and you end up with 25 gigabytes But you can index 2.5 million documents and you're gonna end up with 250 gigabytes of the index size So it really depends, but it scales out that way. So what's coming for the future in lucine? um The current version of lucine is lucine 3.5 and we've been working on lucine 4 since basically 2009 and the scene 4 is Almost I would say almost a rewrite a lucine. We changed tons of internal apis We made great improvements on the indexing side. There's like 300 percent interest rate improvements on indexing Fuzzy surge is 20 000 faster than 3.x searches can do There's tons and tons of new features regarding different scoring models we we support language models and BM 25 scoring all those kind of things researchers would like to use There's a massive amount of performance improvements And solar comes with new cloud features like, you know, it manages its instances automatically and shards Can join and leave and if machines go down it starts up new machines. That's replication Um, so yeah, there's it's gonna be great. Um, there's a lot of improvements coming up and we hope That we can release it within the next six months Okay, simon. Well, thank you very much for your time. We'll have this show out soon Again, you can find us online at rce-cast.com. Thanks again simon. Thanks simon. Thank you