I'm sure you should just share that. She's righty well, hello everybody. Welcome to Smart Business Moves. I know we probably didn't, I don't know if we announced this or not, but new face on the host side over here. It's not time or years, what's going on? What is happening? Who are these people? We've hijacked the live stream. No, so- It's the Twilight Zone. And as we were talking earlier, age reference, now I'm aging out on you. We'll have a couple of those, I'm sure. So my name is Aja. I am a member of the Made Central team. I work closely with Tom. I've also worked closely with Liz over the years. And I'm here with Jonathan to talk about a lot of things, but particularly about using AI technology for hiring some data around cleaning professionals and just a lot of stuff around culture, data, psychology, and kind of how all of this works together. So welcome to the live stream, Jonathan. It's very great to meet you. Thank you. It's good to meet you as well. And I enjoyed our little pre-conversation. So I'm looking forward to the next period of time together. Yeah, I think we're going to have a great time. I think we have very similar interests. This is going to be awesome. So I guess tell me a little bit of kind of how you started in some of this and where your interest in this kind of data comes from. Yeah, so I've always been fascinated by human performance and what causes people to be uniquely qualified for a job or a profession, what creates passion in them and skill in the work they do. And I actually got started in business right out of high school. I sort of have a soft spot for the cleaning industry, which is why I wanted to be on this podcast particularly because my first business was a commercial cleaning company. And I later sold that and went on into tech businesses and then really developed a love for data and how can data inform some of the decisions that we're making? And so the current company that I am the CEO of is Perception Predict. And it's a company I came across three years ago. At a conference, I was actually speaking about sales because I wrote a book called The Sales Boss. It's about building sales teams and it went on to be an international bestseller. And so I've been in, you know, the grind of what actually, how do you actually build a sales team that's sustainable and always a big believer in assessment technologies. And so I came across Perception Predict. It was started by a gal that grew up in Taiwan, literally in a rice paddy, no running water, no electricity, at first of her family to go to college. And she is an amazing data scientist, I-O psychologist and she was building models to help companies not only know someone's fit for a role, but also be able to predict in whatever KPI matters. So what's the dollar revenue they're going to produce? What's the customer satisfaction score? And doing that before they hire. And so we, I came on as an equity partner in the company, moved it here to the U.S. And we've been bringing this to large companies, CrowdStrike, Yext, those sort of companies. And always had this thought of, how could we bring this to smaller companies who have smaller teams? So it's even more important that they get great hires on their team. And so that's what we've been doing is building that for the cleaning industry. We've just actually released a model for predicting cleaner performance. Yeah, well, I do want to get onto that. A couple of things that I'm not sure if people might know, but what is I-O psychology? Organizational psychology. So it's that study of how do people operate inside of companies. Okay, awesome. And then the other thing, Oh, I had one other question there. Oh, yeah, and the idea of for smaller companies, how much of a bigger part this kind of plays like the expense of training is insane, especially for smaller companies, like the amount that you lose from turnover and not knowing who's going to do well, not even thinking about performance and productivity and their KPIs, but just the initial expense of turnover of training somebody and losing them.  Exactly. Yeah, no, that's great. So I guess, well, let's jump into some of the juicy bits. Tell me a little bit about the data that you've kind of been using for around or that you guys got around cleaning professionals. Yeah, so I'll tell you a little bit about our process. So there's for many years been online assessments. You've probably taken assessments that tell you if you're introverted or extroverted, so on. And so that's not really a new science, but what we've done is we've looked into all of the research around what can you measure reliably in humans? And we've developed about 500 unique traits that we can measure in humans. And so you might think of those things like efficacy, continuous learning, hope, hurry, impulse control, job agility, job burnout, those sort of things. And so the position we take when we go into studying a particular role is that we don't really know which of those things lead to performance or lead to somebody burning out. And so we get a group of participants to come together and do a research study. So we just did this in the cleaning industry and we took a number of companies with hundreds of cleaning technicians and we asked them to take an online assessment. And so we looked at about 80 psychographic measures, things like I just mentioned. And then we actually brought in their actual performance data. So we used data from QDS, which some of your listeners will be familiar with Martha Woodward. Yeah, I've used quality driven. And then actually also physically rating those cleaners. And we rated them on five different areas. The first one was cleaning efficiency. So how would you rate the cleaners efficiency? Would they perform the cleaning past in the least amount of time with the least amount of effort? And is it done correctly time after time? And then we had them rank them on cleaning service quality. The third thing was customer happiness and loyalty. Then we looked at work ethic and then the fifth thing happiness at work. And so we fed that into our AI analytics engine and it processed hundreds of people and 80 measures. And it actually came up with a fingerprint to say here's what actually matters for cleaning performance in the home. And so now our customers can send a link to a candidate. They're going to complete about a 10 minute survey and we're going to output two things. One is from zero to 100, 100 being the best. Where will they fall in quality as a cleaner? And then the second thing is their flight risk which is how long are they likely to stick and stay? That is so cool. I would have jumped on that. You know, it's very, it's very impressive. And I have to tell you when I first came across Perception Predict it was actually almost too good to be true. I was like, people are messy. How do you predict things like that? But what we've learned are people are unique but they're predictably unique. And when you have a large enough data set you can start really understanding how somebody is going to be talented or satisfied in a role and how does that satisfaction lead to performance? So as an example, we did a performance fingerprint for A1 Garage which is a large garage door company. They have hundreds and hundreds of technicians. And we developed a hiring model for them that predicts what's the average ticket size in the garage when they're selling and then what is their customer satisfaction score going to be? It's something they call TTL. It's a mixture of like Google rankings, callback work and that sort of thing. And they've been having great success in being able to hire technicians that train faster and stick and stay longer. And that's what we've launched in the cleaning industry as well. That's awesome. So I guess for people that are watching primarily being in cleaning industry, what are some things that we should be kind of on the look out for or are there big red flags that we should be concerned of that we might think are green flags? You know, what's interesting is we tend to think in terms of what are the things we could ascertain inside of a common interview? And we're sort of taking a different approach. Many of the things that we are measuring are things that might be difficult to measure. And that's where an online assessment helps. So I'll give you an example. When you're thinking about hiring somebody and you're sitting in a job interview with them, would you tend to rate somebody's sincerity as a good or bad thing? Typically, I feel like most people see that as a good thing. I think that it is typically seen as a good thing, particularly in cleaning industry. You want to see somebody who's sincere. That is going to be caring and want to be helpful and whatnot. You hear it all the time that like if they care enough we can teach them anything or we can get them to believe in anything kind of thing. Typically, that's really... Our human nature is especially when somebody is sitting in front of us they're more likable when we see them as sincere because they're being transparent and honest with us. But after we analyzed the data for home cleaners in particular we found a negative correlation between the amount of sincerity and how long they would stay in the job. Meaning the more sincere they are, the faster they burn out of your organization. And that's an interesting insight because it's sort of counterintuitive to how you might rate somebody just in a physical interview. And when you think about how does sincerity impact home cleaner in a negative way, you've been in home cleaning, right? And you're going to show up and someday you're going to have a client that's not fair and they're rude or any number of things. And if you're super sincere, you're going to take that personally, right? And it's going to weigh heavy on you and it's going to burn you out. Or you might not manage your response appropriately to the householder because you're going to tell them how it is. And you see that as a quality that you should be proud of. So it's just one sort of insight that came out of the study. And we include that now in our assessment. So as we measure somebody as more sincere, we're showing that as a potential flight risk or going out on the job.  So I hired for four years for cleaning and just kind of thinking about some of the things that I would kind of look into or think about. And as a cleaner, I remember vividly the like three bad complaints I got so vividly. And I still have those imaginary conversations in my head of what I would say to like defend myself. They were such small things and it was six years ago, seven years ago, but I still remember absolutely vividly how much that bothered me. And at least you got out of the home and into the roles that you're in now. Yeah, exactly. And then again in hiring, I think that there's like, there's times where that kind of thought of, you know, typically somebody, if a technician is slow early into their, you know, job position, slower in homes, our normal response was, oh, they just care so much. Like they just care so much, they're not, they're trying to make sure that they don't miss anything and that's why they're taking longer is because they care more or whatever. And that ends up biting a little bit because yes, that is, that is, it tends to be part of their motivation for going slower, but then they take it so negatively that they need to be faster. Like that helps that almost pushes some of that burnout. And I just remember so many times technicians coming back and saying, oh, they made me, they made me have to go faster and they just didn't care about the quality of it. And it was just because they cared so much about it that the idea of going faster, even with the same level of quality and care wasn't something that they could really connect for themselves. So that's, I feel like that all, that makes a lot of sense to me actually. That's an interesting insight. And then on the other side of that, you have sincerity. One of the other things that we found correlated strongly to length of time in the job and quality in the job was that they are tough-minded and tough-minded from a psychographic meaning is that they tend to take a critical approach when they evaluate others and they don't tolerate mistakes and are more inclined to voice or express their disapproval. Interesting, okay. And we suspect that that's because the willingness to be tough-minded and critical of others allows them to be self-critical as well and hold themselves to a higher standard. Okay. So that's an interesting, how would you? It's interesting as you start measuring each of these things. It's how do I balance that difference between sincerity and tough-mindedness? Yeah, definitely. In kind of like the development for these things, how do other psychological quizzes, tests and things like Myers-Briggs, disk, things like that, how would those, how do those kind of relate to this? Yeah, that's a great question. In my book, I write about a lot of the assessments that are popular to use and I'm a big believer in them because they give you insight into the person. But almost without exception, every other assessment tool that's on the market is really based on 30-year-old established science. It's really a measure of introversion, extroversion. And when you pair that in different ways, it shows up in different ways and also some form of cognitive intelligence, which you might think of, you know, how does somebody think they're IQ? And there's almost no data to suggest that because somebody is introverted or extroverted, they're gonna do well in a particular job. So tools like Myers-Briggs, disk, they're helpful in the sense that they're going to tell you how that person fits on your team. Like how do they show up, right? Are they into the detail or are they more expressive and they're a people person? So that's certainly helpful. But what happens when you make the mistake of using that as a hiring screening tool is you're hiring people with a certain personality and not really looking at how does that translate to their ability to do the job? So I'll use an example in the selling world. People tend to think, and they would be true, that if you're an extrovert, you tend to do better in sales. I would say as a broad-based assumption that's true. However, if you exclude the people that are introverted, the unintended consequence that you have is that when somebody's introverted and does well at sales, they outperform by almost 15 times the extroverted person. So by sort of taking this blunt approach, you're really excluding a whole portion of the population that could be your very top performer. And so that's why we take this approach of if science says we can measure it reliably, why don't we measure it and then let the data tell us what's important? So we did a study for Samsung retail store executives. Think of people selling like cell phones and that sort of thing. And they were having a turnover issue, which I won't share publicly here, but they had a well-defined hiring process that was based around two things. One was customer orientation. So if somebody was customer-orientated, they thought we're going to hire this person and if they had a love of technology, that this would be a good idea. After we analyzed the people they had doing the job, crunched all the data, we found there was actually zero correlation between customer orientation and how somebody would do in the job. Yeah. It's sort of humorous, right? That is so great. That is so funny. They're just like, I hear it all the time. I've said it so much, but it's just like you can't trust your gut feeling sometimes. You got it back and out with data. People think that they have a good gut feel for it. And the truth is there's plenty of data to suggest that our gut feel is really loaded with bias, right? Yes. So it doesn't mean you don't want to hire somebody that's going to treat your customer, right? It means let's don't use that as a filter. The more disturbing thing they found was that the more somebody professed a love of technology, the worse they did in the role, right? So that's where letting data speak really helps us hire people that are going to ramp faster, stay longer. Here's another interesting maybe insight coming out of our study is when it comes to cleaners in particular, there's this trend towards performance pay. And there's a lot of data that suggests that performance pay ends up getting better performance from your cleaning crews. However, the data also suggests that someone that's going to do well in that role is not money-driven. Their primary motivator is to be disinterested in money or high social status. They don't mind being in that serving role inside of somebody's home. Yeah. Yeah. Actually, so we had one question really quick. I have a couple of things, but I just want to make sure we don't forget this. Nina had asked, what was your book? It's The Sales Boss. The Sales Boss, Nina. The Sales Boss. Yes. So that is the book. And The Sales Boss is really not focused on selling. You might think that from The Sales Boss. It's really around how do I build a high-performing sales organization? And I use the term Sales Boss for a couple of reasons.  Like, if you ask somebody how they're doing, you don't want somebody to say, well, I'm managing. I want to hear like, I'm doing it like a boss, right? I also use it as an acronym, B-O-S-S. And those are the four levers for diagnosing performance in any role, but particularly sales. So if somebody's performance suddenly drops off and I'm in a leadership role and I have to coach or develop them, make decisions about their employment, I'm going to go back to my boss acronym. It sounds for behavior. So the first thing is I'm going to look at their behavior. So, you know, what is normal for them?  But there's also normal for me, John, or for you, right? Yeah. So as a leader, you have to know what's normal behavior. How many calls do they do? How many, like each of the tasks. So if I look at their behavior and their behaviors way off of normal, I can probably say, well, their performance, right? The return they're doing as sales is behavior. The second part of boss is outlook. And outlook is how do they see themselves? How do they see their company?  What's their mind telling them as they go throughout their day? And sometimes you can be experiencing something in life that might be personal and, you know, that outlook has is ruining your performance has nothing to do with, you know, your ability to do the job, but it's impacting how you show up at the job. So behavior, behavior, outlook, if both of those things are, are steady, then I'm going to look at the next one, which is skill, like, how do I do it? Right. And, and most people when they're thinking about performance, they start with skill. Do they know the skill? And I actually think you start back at, let's analyze behavior, let's analyze the outlook. Then let's look at skill in the last s is around stature. So how do they, how do they show up and, and, and what's their reputation with, with the customer client or in the industry? Okay. So with sales boss. Awesome. Um, yeah, I had one little follow up there. You'd mentioned the introvert performing, um, at a higher performance level than the extrovert. Um, something that I guess I have in my head is that, um, as an introvert. Um, that, uh, you know, I can sell really well and I have a very amazing sales track record, but wow, does it drain me? Um, and that I think is where that kind of extrovert introvert kind of line up like comes from, at least in my head. Um, is that like, not that I can't do it. It's just that I get so drained by it. Yeah. I think about that like a, like a rubber band, right? Here's what's natural and comfortable and this is uncomfortable. And just like a rubber band, if you stretch that really wide, you can do it, but eventually get tired, right? And if you get too tired or stretch too far, eventually you snap, right? It snaps. And that, that's never good on the job, but, or especially with a customer and that's sort of where behavioral styles do come into play. Like, is it more natural for me?   That's where, that's where personality sort of profiles come in. But it doesn't change the being able to predict the skill level at which you can do it. And so what we're, what we've done is we're taking a platform called who hire.com and that's are going to be our product for the SMB market as opposed to the enterprise clients that we've been doing at Perception. And inside of that, we're doing everything from helping post your jobs dynamically to the job boards, but have two-way text messaging, auto scheduling and also being able to add in a prediction model for your particular role that's going to give you a high degree of, of indication of how they're going to do in the role. So we have customers who are literally hiring people into their organization. The first time a human's interacting with them is that final face-to-face interview. So all of the screening and everything else has happened and it feels human and it feels engaging to move people quickly through the process, but you're not, now you're not having that, especially in the cleaning business, you always have to be hiring, right? Always. Always. And so this, this gives companies the ability to keep their lines in the water, so to speak. And especially what happens is sometimes you get mid performers in an organization and the hiring process is so exhausting that they slow it down and they sort of take a breather because, hey, we got every, we got every opening filled, right? And so let's get off that treadmill. What our system allows you to do is let's keep fishing and let's keep bringing candidates in and qualifying them. And the only time your executive is going to get, your hiring manager is going to get notified is when there's somebody in the pipeline that we're predicting is going to do better than somebody that you have on the mid-sized area of your team. Okay. Cause we're often hesitant to replace our mid performers cause you think, you know, it's the old saying better the devil I know than the one I don't know.  Anyway, why don't I stick with what I know? Yeah. And I mean, there's, there's, you know, the, like catch 22 of that situation of having, you know, the either slightly low level of staff that you need or just the right amount of staff that you need that you haven't seen in a long time or whatever. And you've got that person that is on the lower mid end of performance and you're just like, ah, but it's better than nothing. And, um, but then you also have, how does that affect the rest of the team as well? How does that affect the perception of your culture? Like there's so many other pieces of that that can kind of interact a little bit, but like the idea that you can have that consistent funnel of people that are more likely to be in the top mid top level of performance is that would be so reassuring. So you, you moved out of the day-to-day cleaning managing a cleaning company and now you're working for a tech company. So how, how are cleaning companies using technology in that, in that aspect? Oh, yeah, made central. Um, like just to, you know, uh, technology for scheduling, reporting data. Um, trying to like, you know, bringing together the different, the different systems of, um, processing or card processing, um, yeah, operations. Yeah. So what's the value proposition for them? They're like, if I'm a client and I sign up and made central, I'm not trying to pitch your software. I'm going somewhere with this, but when they, when, when a user signs up and they get started with you, what's the outcome that they experience? Like what pain are they trying to fix and what's the benefit for them? Yeah, the, I guess the real, the real like pitch for us is that it's going to reduce the amount of time that they have to spend on the minutiae areas of their business. The areas that should be on auto essentially, you know, scheduling, billing, invoicing, those things aren't things that you should have to be worrying about. Yeah, exactly. And so you're decreasing that, you know, friction in the system, but you're also, um, helping them understand their pricing and should they increase their pricing and how does that balance out over the year? And some of those things aren't at an instant intuitive or they take so much effort. And what we found is there's a lot of tools for businesses when it comes to the operation side. Tools like made central, right? They really help them granularly understand what's going on in their business and we're doing that for the people part of the business. We want to give such deep insight into what's driving people that come in and thrive in their organization versus those that come in and sort of, you know, just wilt and never thrive in that and we believe that when we can drive that turnover down and the time to productivity up, that ends up at, you know, real money in someone's pocket. So absolutely. And I just taking it back to the cost of turnover and as a small business turnover is it cannot be stated enough how important turnover is as a small business, like how important reducing that is. And especially at Made Central, we we do really focus on the idea of being a profit focused kind of business versus a sales focus where it's not that how many homes are you cleaning? How many customers do you have? But how valuable are those customers? How valuable are your cleaning technicians and what they're capable of doing? How does your efficiency and productivity pricing pay rates? How does that all work together in the bottom line of your company? Because you can keep bringing in customers all day long. But if your pricing is at where it needs to be or your productivity of your technicians is at where it needs to be. You're just working really hard for to work really hard. Working really hard to actually move yourself forward. Yeah, that's exactly right. And I've noticed an interesting thing as I've worked with business owners over the years is that we sometimes settle into a routine of taking our A players because everyone has somebody that just, you know, they're a rock star. They outperform everyone else. They're easy to manage. They're sort of you can sort of put them on autopilot. But business owners, they tend to almost treat that as a unicorn or as an anomaly. Like I couldn't ever hire somebody like them, right?  And what we want to propose is this radical idea that your unicorns should make up the widest portion of your business. It should actually be rare that you're getting somebody into your business that can't thrive. What one of the things that I write about in my book and I talk about five truths about humans, right? Yeah, what cause them? And one truth about humans and performance is that the only reasons that matter are their reasons, right? Why why they're doing and I believe somebody will stick and stay in an organization as long as they believe they can still do their best work for you. As soon as they start thinking that that they're not growing, they can't do their best work, then they start looking elsewhere and then it becomes about, you know, what's my drive time? What's my, you know, what's my pay rate on this? And when it just then it just becomes transactional. But when they believe you're the home where they can still grow and do their best work, they're going to they're going to stick with you.  That idea that their their major driving force is not financial. I mean, it helps for sure and having a, you know, that keyword of livable wage or whatever is obviously part of that, you know, bottom layer of needs, but it's not necessarily the one thing that happens in the trades quite often and especially in home cleaning is that your best cleaners oftentimes go out and start their own companies. Right. Because they see, okay, I think I can do this. I've got clients already. It'd be easy to go out on my own and it's actually a fear of a lot of business owners, right? What if I train somebody up? I give them access to a lot of the tools in our organization all of a sudden I'm competing with them. That's one of the things that in our prediction model, we're looking at when we think about tenure and flight risk is, is this somebody who's so entrepreneurial driven that they will go out and start their own thing or are they somebody that thrives in being in a team environment and being a follower and on the team. And so for our hiring managers and our companies using our platform there, they're always able to, you might have somebody with a high flight risk because they're very entrepreneurial, but they're also predicted to have really high performance. And so you can sort of, you know, decide is this is this somebody I want to invest my time and training and energy in? Or should I be looking at other candidates? Yeah, that's that's huge because, you know, the idea of training somebody to be your competition is probably one of the biggest fears. You know, I hear about from other operators and owners and stuff like that is just that idea of, oh my gosh, I've given them access to our training, our materials, our customers, things like that. And let's let's to stop them from doing the same thing that that we're doing here. And it's funny because yeah, I don't see the people that are good that I've seen in the past that are really good cleaning professionals as the people that would want to start their own business or have done that successfully even. If we go back to the thought we were talking about earlier of having A players and not treating them as an anomaly. There's this also also a truth. There's a I've learned there's only one value of writing a book as you can quote yourself. So I'm going to in my book I say things are only good or bad by comparison. And what I mean is like every human has developed their sense of what great is by things they've experienced.  So if you if you if you put Pete somebody in a group of mid performers they'll quickly just become a mid performer.  I think as a business owner or what I would be encouraging people to do is always looking at your team and saying how do I up skill and make sure that I'm hiring somebody that's going to compete with the best that I already have because that's going to pull the entire team forward. Yeah, somebody who I think about this you know most of the cleaning services have like a quality score you know, minimum quality score to retain employment or whatever and they'll set it at like 96% 95 94% whatever that might be and it's very interesting to see the technicians that think that that is unattainable and like how they how how quickly that thought can spread like wildfire through a group and suddenly that goal that KPI goal that hadn't been an issue for so long is suddenly this wildly unattainable expectation and the company's being unreasonable and you're being so unreasonable to expect that and it just took you know one person to really pull that idea down yeah that's that it totally makes sense group thing is a major major thing also I just the tennis reference was funny to me I definitely do not play tennis but I play a lot of video games I'm a very big video game player and I've been recently playing with two extremely good people in one of my favorite video games and they carry me so so well in the games that we play together but I have gotten so much better by playing with them by having them on my team it's been a lot of years since I've played video games but the my sons were playing video game of farming yeah the farming simulator thing was I'm just since you're in the video game world is there a video game for cleaning like you know you go around and you dust really quick and you vacuum or is it there's a couple of things similar to that so I do get a lot of satisfaction of like before and after stuff hence why I became a cleaner in the first place there's nothing necessarily fully about cleaning they have pieces that aren't cleaning related but there is one that I see way too much of that is about this office and it's like a dermatologist office where they're like fixing skin and cleaning skin and stuff like that so I know that there's that I haven't seen an actual cleaning one yet but I'm I'm sure there's one out there I'm surprised that I wouldn't see it because the cleaning algorithm and all of my social media is very strong what's the game of choice for me right now it's overwatch that's my that's my go to game right now it's it's a pretty fun one so but that was interesting that just by being with people that are higher skill levels than me I I've learned so much about it I'm able to be a much better player than I was previously and I feel like you can see that in any area that you have you know people working on the same same goals I guess yeah makes sense so I guess tell me a little bit more about so you've done all this data for cleaning services cleaning platform for hiring how yeah so the platform for hiring already exists it's out there you can go to who higher dot com right now and sign up and we're what we're doing is we're integrating that prediction model in there so if you're a cleaning company you would just reach out and say I'd love to use the cleaning module and we'll we'll implement that in your system for you what other modules do you guys is this just the first one for the small business management or do you have other ones for small we're we're doing sort of industry by industry so HV the plumbing air conditioning painting exterior cleaning Christmas lights so we'll eventually get through through all of them land landscaping so one by one we're doing doing all of those industries yeah do you kind of go into this with a little bit of a hypothesis of what you might see is like to how it will compare to kind of the industries that you've done in the past you know I think there's always natural drive me is what's the finding that's going to come out that's going to be completely shocking there's like there's some things that sort of make sense so as an example in the cleaning finger fingerprint being organized and neat is a high predictor of success in the role that's sort of make sense but sincerity and being too sincere sort of doesn't you know at least in my mind didn't make sense until I actually thought about why why might that actually show up and even more fascinating to me is when the data suggests that something is true like this trait is going to lead to poor performance or higher performance when it doesn't make sense is the ability to trust that the the why doesn't really matter it just matters that it is true and having having the confidence to just trust it so as an example you know with associates having customer orientation not being a predictor it's sort of like why is that true but why yeah yeah it's so my curiosity like I want to unwrap but the the reality is probably any of the things I would come up with which it it's just doesn't matter yeah I mean it doesn't it doesn't but man the curiosity and me the the need to know why and me would just I would be I would be trying so many things to figure out if I could figure out the why behind because I mean I love data and I love understanding how things relate to each other and how you can get predictable outcomes based off of you know what you have now and what actions you're taking now but the idea to do that on the personal side the psychology side of predictable outcomes of how a person interacts with the world around them or believes in the world around them or themselves that why is still like it's almost like that ethereal concept that we're never going to be able to fully grasp it's just the idea of psychology and its own right is never fully defined it can never be fully defined so you're right it doesn't matter but me and I want to know and what and what happens is people get what I call their hero trait so if they think somebody needs grit then they're gonna over index on boy this person's gritty right it's almost like saying if they played college athletics or which is not necessarily true so people over emphasize on the thing that they personally you know put value in and which is why we rather than producing a score saying this person is fair and genuine organized and need hardy and committed not money driven humble focus whatever we actually just return a score because we don't want the the hiring manager to over index on any of them because it isn't any of those by themselves that makes the difference it's how they compliment each other yeah person it's almost like you know if you sit down at a restaurant you eat an excellent meal the individual ingredients don't matter with you know the salt by itself wouldn't make a great like I'm just looking for a dish with salt in it that's it yeah but you might over index on it because you like salt what you really want is how do all those ingredients and we find it pleasing it's a similar thing with an employee we want all those ingredients to come together in exactly the right mix so that they can enter our organization and thrive that's really cool so they get a score back that says you know they're up is it out of a hundred or out of 10 it's it's one to 100 so anything anything really above say 80% is going to be a really high fit and we we return three recommendations one is proceed quickly to the candidate now the other the and you might move faster for those certainly if you have a lot of people coming into your pipeline you're going to prioritize your day by the highest fit and then the mid tier candidates is going to be to to to look closer right so you proceed slowly doesn't mean you're going to rule them out but you may spend more time in the interview process you may make sure right that you really Val validate all of the other things that go into the decision and then we have what you know what we would think of as no fit live will internally we call liability hire now I've said that extra externally but the person that's recorded now yeah it's recorded it'll be everywhere but what do we mean by a liability hire that might sound negative but it's actually a liability for the the person you're hiring it's really unfair to hire somebody into a role where they're going to struggle and they're going to lose a part of their their job history right and the weight of taking on a job and losing a job is really a heavy burden for people so we have an obligation as members of our community to do the best we can not just you know selfishly for us to grow our revenue which we can do great things with increased revenue but also for like what's not just view it as a human mill where we're just churning through people because that churning through people has an impact on them so it definitely does the idea of going through another hiring process is just as costly to I feel like the people that we hire and lose quickly as it is to the business so 0 to 100 on that and then we give you a flight risk score which is also a 0 to 100 it's a percent obviously flight risk score you want lower performance score you want higher got it got it with with the initial pathfinders group and the pathfinders group of those companies who volunteer to give us their data they let us work with their technicians which QTS and Martha so she was a huge helping getting this done so I definitely encourage you to take her approach to thinking about performance pay and how do you measure that because it's fascinating data but to be able to feed that back into the hiring process it it is is really key and I went on such a rant I forgot where I was going with that we were going through how it was scored and so we did our flight score yeah so the flight risk score is be a percentage based score okay that's really cool and so they get both of those scores I actually I I forgot this question you'd mentioned the unicorn earlier and how you can use this to find where unicorns essentially or that your unicorn should be your standard and not your your one in a million white you silver unicorn that you found or whatever how to I guess I guess what do you define that you know how how would a company define their unicorn and would that be something that they can alter and kind of how they're searching or would the expectation be how the search is coming back that it would be finding or those unicorns yeah great question so our you know our 95 do a hundred percent percent on there are really the people that not only that their psychographic profile predicted them to do well and they're actually doing well in the field this is data from real cleaners in the field showing up in people's homes every day yeah we just set your standard really high and I'm I'm I'm going to quit hiring people that are maybe's and sort of can because what there's also the reality is that you have time pressure so if you have a whole in your company's like sometimes you're going to hire the best available candidate yeah for sure and I think you there's a mine shift that needs to be I'm going to not hire the best available candidate I get the practicalities of business I gotta have somebody absolutely what what that means is I as long as I know I have somebody on my team that's not the unicorn I'm going to be actively working to recruit people that can take that role when they leave the organization and they might lead by choice right they they their life circumstances change and they leave in a but I'm going to always it takes a while to find the one out of a hundred which means I shouldn't be looking under pressure I should be constantly looking what's in who hire today is we're enabling people to constantly have their posts out we automatically refresh and rewrite their job posts for them we're screening their candidates texting them declining them all that's happening automatically and you can just have a score to say here's when I want to actually have a conversation so oh yeah they score 95 and all these other things are true you know let them schedule on my calendar I'm happy to do lunch man I need you to get this on the software side I'm doing interviews right now and me well so so we do have prediction models in a lot of different industry so we could we we could talk afterwards for that you know going back to the flight risk model though in the core group that helped us build the cleaning module of course we assessed all of their existing staff and the first time the owner logged into the platform there were four of her existing employees that we had indicated were her initial response was these people aren't leaving me and she went out on indeed and she found all four of them actively looking for jobs and she was able to approach them and have conversations get close right reach or re-engage and able to say able to save two of the people that she that she wanted to save that were her high performers and there were you know reasons they were looking and she was able to overcome that and two of them were gone within the next 60 days so oh my god that flight risk model is can suggest how somebody's going to be in the role yeah have you I guess so this is really good at helping for people you know moving forward what about you know that idea of being like the the idea of flight risk being time-bound to me I guess that you know for those two people at that moment they were very flighty but the idea of that of a conversation helping to reduce that flight risk is there I don't know I'm seeing this is like an on-going thing of like ongoing surveys of my I mean professionals we wouldn't we wouldn't in this case do an on going because we're really interested at the point of hire the reason I shared that is it was it's just an interesting anecdote around some people are sort of hard wire to deal with stress and life circumstances differently and so what we're really saying is this person is more likely to transition out of the job early might be might be things in in terms of how do they deal with stress and all of that it could also be on the other side they're really entrepreneurial and they just don't stick around because they like yeah they like change the right so the wide it's gonna pick up on both sides of those and I think it's important with any like assessment tool to realize that it only it picks up on we would say about that contribute to performance that but there's about 20% that that we can't impact like we don't know what's going on in their personal life they might have what I would call hygiene issues with drug use or an abusive relationship or you know they've got a child that's it going through illness or some other thing happening in their life and even if they're a really high fit for it and you would score them as a 95 they might still lack getting to that performance level just because of those outside factors but what that allows you to do as at least when I was you know running my organization I like I always believe the best about people almost to a false I was like if I just give them another week another month or if I retrain them and if I'd had this tool back then it would allowed me to say okay my desire to be patient with this person will pay off right they're going to make it but if if I compared their predictive really low and their they were felt failing to thrive in in practice then I would probably feel much more at ease moving quickly to move them out of my organization yeah I've always been really good at the encouraging them to leave I I got really good at not having to fire people and having people be able to come to the agreement that we are making the right decision for for them to um what was the hardest thing for you as a as a manager of people that you when when you were hiring them I always just wanted to help people you know I I loved the idea of seeing somebody's potential of doing well for themselves and being able to take control of an aspect of their life that they may not have had control of previously because like as a cleaning professional one of the things that I really love about the industry and about being a cleaning professional have a lot of self sufficiency throughout the day that you're out you're in people's homes you might be with one other person but you might be alone too and it's just like that idea that the work that you're doing is solely based on your ability to do it and finding people that would would get so much out of that out of creating those results for themselves so for me the hardest part was just believing the best of people all the time I used to joke that I'm just I'm optimistic to the point of being naive sometimes and I'm not a naive individual typically I'm just very optimistic and I believe the best in people so fully that when I was hiring the hardest part was just not being able to face the music when it was there right being able to make excuses for people after it was fairly clear that that was not going to work out really well I think we're similar in that regard so I developed a habit over time of being in business is when I hire somebody I have a conversation with them up front and I say look I am completely in love with your potential for this role I believe you're going to do great at it and I'm going to I'm going to really like you like there's no question right but I also have performance pay and so we have a performance standard so at 90 days or 120 days if you're not meeting the standard I'm still going to love you I'm still going to probably want to keep you on the team and I'm going to still believe the best value so there's just sort of a hard performance metric that we would both hold our ourselves to and I would I would explain to them the reason is is I know I have an obligation to the company and then if we're investing this amount in marketing and training that are very best in the company if you can't perform in that love that level one you're not going to make income that's satisfying to you and secondly I'm actually stealing from all of the other members of the company we can't invest in all the things that we want to do in the community and our standard of giving in the community is less and that that tended to to serve me well I'd still get sorted to the 120 days and be like well they're only a few percentages off they're so close no but I I was definitely seen as kind of a hard manager for that like once there were set expectations for their different KPIs and we had it very obvious like okay this is meeting expectations this is below expectations you can only be below expectations for such and such amount of time before we're we got to make decisions and so I got really good that I didn't I didn't love that I got good at that but I did get really good at that so it's since I you know since I wrote the book the sales boss I know we've been talking about hiring but I just shift gears right at the end here I'm going to go all the way back to the the the two things from the time I spent and I was doing commercial cleaning so this I know there's a difference yeah but two things that really one out for me one is when it came to hiring people I'd always say I can pay X or Y one is that you're managing yourself and I'm going to pay somebody look over your shoulder and manage you or I can give you the entire wage but I'm not going to pay somebody to manage you but this the first day that we decided we're getting customer complaints and you're not managing yourself you're going to go 100% of people said I'm going to I'm going to manage myself right yeah likes to feel micromanage so that tended to work really well and and and I really focused on peoples who whose life situation made working for me as a cleaner important so a lot of times it might be a mother who needed to be home with her kids during the day and so being able to come in during that late evening hour while they were in bed was really important to her might be a young person putting them college or some other thing and matching that life style need with the hours and pay that we could offer what was important so that's sort of tip one I don't know that helps anybody that's listening sure but the second thing is like when the economy changes the problem with being in the cleaning business is anybody who has a vacuum cleaner or dust mop or you know anything they can show up under price you take your business they're not paying for insurance they're not paying for the infrastructure and clients so the way I'd dealt with that and I should patent this is it just a great idea I'm going to pat myself on the shoulder but there we go I would go into a organization let's say it's you know doctor's office and I'm bidding a quote I always made sure my bid was outrageous like I would have to be at least I would say 40 50 percent above the next highest bid they got okay and you know most people think I'm going to bid as got my follow-up meeting and ask them why I didn't win the bid it was always like well we're even you know before they were they well it's the price I said well you you know you you gave me a list of things that you needed to have done and we had the discussion that the reason we were having the conversation is because the person wasn't doing all those things I only have two people or two costs it's the chemicals that I'm using and it's the people so which do you want me to use less of because I'm going to I'm committed to doing a hundred percent of course they would say well I don't want you do any of them so they would agree to the higher price which wouldn't buy itself you know protect my business because times are going to get bad they're going to cut so the way I would build a mode around that customer is I would get three months into cleaning for them and I would send them a letter and then I would followed up by an in-person visit and my letter would say look we've cleaned your building for three months and we've now projected our costs and we realize that we priced ourselves twenty percent over what it actually cost us to do a hundred percent of the items that you've requested and we would give them a refund check for those three months and there you go right so we really embedded in the mind of the buyer that we were only charging exactly what we needed to be able to charge in order do a hundred percent of the things right from that that person that was going to come in and under bid and and for the first services that we did from them we even if we weren't contracted to we would drop the light lenses clean them out steam clean all the chairs the tile would be shiny and we would put a huge bouquet of flowers on the receptionist desk it would probably be half my height and so for the entire next week when people came in they'd be like oh who got you the beautiful and on every seat in the building we put a little card that says your chair has been professionally clean please make sure it's dry before seating we knew it would be dry by Monday we just wanted to draw attention to somebody's chair and they go oh it's really clean so I could just a picture my competition walking in that front door and the first person they're going to have to talk to to get to the owner is just blown away by your service the receptionist who's like just got had flowers right an impact yeah and and and that person got a lot of praise for making a good good decision in the company they hired and anytime they come in with a lower price I've planted that seed in the buyer's mind that well what what of the hundred things I need done are they leaving off the list so that's an enclosing that there's a little couple to highlight great I love related totally unrelated to hiring it's been great chatting with running up our time yeah we're running up on the time so yeah definitely your book sales boss correct yes the boss that's the sales boss I will be reading that so that's not so much fun that you've got who hire to hire dot com and if somebody wants to reach out to me personally and ask questions about this happy to get on it's John at who hire dot com but it's J O in got it there's no age in my yeah so if you're interested in getting some AI assistance in your hiring process I'm blown away I want to I want to learn more like if you guys want to you know publish your findings I would spend so much time looking at all of that I love it so much we will actually be publishing some papers around the work that we've done and fantastic nice thing is our our models aren't static so the more cleaning companies we have in and who are giving us their data on turnover the more we're adjusting that model in the better it's getting business of hiring people to clean homes I love it so on that note thank you so much it was really great meeting you I love this conversation this is such up so up my alley so thank you for taking the time to talk to me I'm glad that I got to sub yeah so I thought I would I thought I would miss Tom and Liz but you can tell them they were un missed and sorry Tom don't come back and and and hold down Kansas city for me yeah I will do so thank you so much bye everybody


Hello everyone. I'm Sanjay Gupta. I welcome you on Sanjay Gupta Tech School. In this video, I'm going to demonstrate a scenario on record trigger flow. Let's understand the requirement that I'm going to demonstrate in this video. So I'm going to differentiate dollar record versus dollar record underscore underscore prior global variables. Those are available in the record trigger flow. So these variables are available whenever you trigger your flow on record update. So if you're updating a record, so there are two state of that record. So if I take you to the account record, so here you can see we are having this account record. If I update any value, for example, here in phone, we have this value. If I update this value to some other value. So let's say here instead of seven, I am having eight. Right. So now this is the new value and earlier one was the old value. Right. So if you save this record and the record trigger flow runs so that record trigger flow will be having both the values old value as well as new value. Right. So old value will be available in this dollar record underscore underscore prior and the new value will be available in the dollar record global variable. So now I'm going to demo this with the help of a use case. So we need to create a record trigger flow. So whenever we update a phone on account record. So we need to populate phone fields prior value that is old value, as well as new value in the field of that account record. Okay. So now I'm going to create this flow. So I clicked on record trigger flow. Now here I'm selecting account from here you can choose a record is updated or record is created or updated. So right now we are just focusing on the updates. We are selecting record is updated. Here I'm selecting account phone equals. So you can select is changed through if it is changed, then your record trigger flow will run and we need to update the same record so we can select fast field update and I'm clicking on that. Now here. I'm going to select update record. So update. Okay. So before that, let me just create a resource. So I'm just creating a text template. So it will be having phone new and old value. So here I'm going to create it. So I'm selecting you as plain text. Now, first I'm going to show old value, and then I'm going to show new value. Now here I want the resources. So if you go here and So first we need to show old value. So you can search for record. So here you can see we have these two global variables right record and record underscore underscore prior. So I'm selecting record underscore underscore prior and then phone. Then in next line. I need new values. I'm selecting record dot phone. Both the values will be available. Now I'm clicking on done. Now I need to update the account record. So I'm selecting update record. So update account. Then we need to. In case of fast field update, these options will be disabled. So because by default it is selected use the account record that triggered the flow. So here you can just put description equals and then this text template. Right and done. So this way our record trigger flow is created. And now I can save it. I'm activating it. Now, if I modify this, so here you can see, so let's say instead of 7,000, I'm putting 8888 and I'm clicking on save. So if I go to description, so you hear you can see old value is this and new value is this. So this way, if you update any record. So dollar record will contains the new value of that record updated report and dollar report underscore underscore prior will be having the old value of that updated report. And one more thing, if you if you use any other thing like if I go here and let's say I search for description. For example, okay, it is available. So let's say I am selecting alert node. And here if I search so these record and record prior, both will be available here as well. So in create record get record update record so everywhere you can have these record dollar record and dollar record underscore underscore prior. So this way I hope you understood the difference between dollar record versus dollar record underscore underscore prior. Always remember they will be available in case of update only because in case of create and delete, you have only one state, you don't change the values of that record. So these will be available in case of update.


start. Good afternoon everyone. My name is Seky Vazquez and I'm presenting you this session Drupal Extreme Scaling. For the next hour I will be talking about a project that I had the pleasure to participate about one year to make this project work. I'm going to focus on infrastructure on this project. This project was successful and I will give you the details under the project and I will focus on the scaling features of the project. When I start to talk about the project itself you will see why the session is titled Drupal Extreme Scaling. First things first, who's me? Who's this guy? Well as I said my name is Seky Vazquez. I work as a DevOps freelance mainly and also I do backend Drupal Development. I focus on Drupal science for the last three years and a half. I also am a PhD student on the University of Seville. My PhD thesis will talk about cloud computing to be precise. I will try to develop a formal method to optimize cloud deployment options. I'm very involved with hacking and security, IT security stuff. I really like it. I like everything in life is code. There are other things. I always like to let you know about some of my hobbies. For example, I like rock and roll. I play the electric guitar. I have my own rock band. I like video games and I love to read books, special adventure books. After this introduction, I will review the main points, the main sections of the session. First I will give you a brief introduction of why scaling, why extreme scaling. After that I will describe the project, the requirements and the technologies and all the stuff. The main points is the second. After that I will focus on some problems we found when developing the project and how we solved those problems. After that I will show you a little demo. It's recorded. No more live demos. Finally I will share some conclusions very fast. Let's start with this introduction. When we think about Drupal scaling, scalability, performance, availability, we all have some modules and some techniques, some tools on mind. For example, the ones that we have here. Drupal has a problem with cache. Sorry. Drupal has a problem with cache. It uses by default database as cache system. So we can use main cache.  We use varnish to improve the speed of our application for anonymous users. We use redundancy to be able to use the code to make sure we are confident about availability. You can choose whatever technique, whatever tool, modules. There's a lot of modules to improve this kind of stuff, which are non-functional requirements. But we have to consider new technologies that come from the last years to nowadays. They represent not the future, but the present. I want to focus on this introduction on two of them, the two main technologies that we use on this project. I'm talking about cloud computing and containers. The main advantage of cloud computing is elastic computing. What does this mean? Very simple. We have resources. Sorry. We don't have resources fixed. I mean, we don't have, for example, four servers, six servers, and then we have those resources in the same amount of time. Sorry. I will repeat what I was talking about is, basically, we have a fixed number of servers, but with cloud computing, we have the ability to have an elastic amount of resources. For example, when our application has a workload, a peak on the workload, those resources are able to scale up. I mean, we will have more resources, more RAM, more CPU, et cetera, to be able to manage that peak on the workload. And when we have no more need for those extra resources, those resources just scale down and disappear. So we don't spend too much money on this stuff. In theory, all the cloud providers sell this stuff as, well, we have almost 100% scalability and availability, and this is not true. Why? Well, it's not the first time. It will not be the first time that one of the CPT's of, for example, Amazon Web Services goes down and one availability zone just disappears for a couple of hours or, you know, so you have to be very careful when designing how your infrastructure will work on the top of the cloud service. And of course, this is something that is not talked about by the providers, the cloud providers, is the budget. You can easily deploy more machines and more machines and more machines and more resources and more resources, but after the end of the month, the bill will come and you will just go crazy when you see the number. So you have to be very careful with the budget and you need to design properly so you can avoid excessive costs. About Docker containers, well, today Docker is mature enough to be used on production environments, but I have seen a lot of people who use Docker containers just to have, quote, virtual machine and, of course, for local development, but we use it as a technology based on Docker and it works very well. We will see in some moments. Okay. Having done this brief introduction, I will focus on describing the project itself. Okay. First, I will describe some basic requirements. At the moment when we started to build the project, we use it to use Drupal 7 and we will use a multi-site install. We are talking about a platform to which each user on this platform will have a Drupal site for him or for her. So at first, when we started development, we were talking about 30,000 users. That means 30,000 sites and business wanted to scale, to be able to scale up in the next two years up to 100,000 sites, a lot of sites. They had problems with the previous infrastructure, with the previous version of this project regarding the availability. Sometimes the service became unstable and they wanted to, you know, cloud providers give us almost 100 percent, so we want 100 percent availability. Of course, high performance as always, lowest possible cost as always, and, well, due to the need to manage so many sites, we need a tool to control all of these sites. So we needed to research an external application. We wanted also automated and not disruptive deployments, I mean, zero down time when possible, related to the availability percent. And we needed also to do a migration from the previous system and, well, one of the problems were that our team was made of only three depths. We have a front-end guide, a backend guide, and me as a DevOps. We also have another non-technical people, but, well, three people one year, and a lot of sites to support and, you know, all these requirements. So our face was like, what? It was like a bit crazy, you know, three people and this monster project. But, well, that seems scary, but don't panic. No problem. We talked about the project, the requirements, and, well, we agreed that we would need a bunch of technologies that could support us in our work, in our mission of giving support to that number of sites. Well, it looked like a great challenge, very challenging task, but, hey, we are computer scientists, aren't we? So we started to review the possibilities, and, well, we focused on open source. So definitely we decided to use, on one hand, cloud computing, and, on the other hand, containers, as I described before. We found Apache Mesos is one of the tools. Well, I will describe in a moment, and, well, we will use Docker and all this stuff. I will describe. We will use Amazon Web Services as a cloud provider, okay, our low-level infrastructure. We will use Apache Mesos, which is a software that let us abstract. It's an abstract layer, which let us to just assume all the resources available on a group of machines. For example, if we had four servers with four CPUs each one, Apache Mesos will make us look at these four servers as one server with 16 CPUs, for example, okay? We will use Marathon, which is an Apache Mesos application, which let us run Docker containers on the top of our cluster. And here is Docker, and for the Drupal, Drupal will be the CMS, of course, we will use Nginx better than Apache. We will use Node.js for the external application to control the multi-site to manage it. We will use MongoDB for the field storage, so all the field data and field revision tables will be out of my SQL. And to automate all, we will use Ansible, because Ansible works very well, pretty well with Amazon Web Services with the API. So, well, let's describe. I will go from, if this would be a pyramid, I will go from up to bottom, okay? I will start describing the Drupal installation, and then we will go down. So, as I said, we will have a Drupal over Nginx and PHP FPM. Nginx is very flexible when configuring it. That is a two-sites blade. It gives us a very flexible configuration. PHP FPM increases performance significantly comparing to mod PHP on Apache, and we use it mod security just to improve the possibility to use, improve the security of the security system and avoid some basic attacks. And, well, we developed custom roles for Drupal, but well, I need to work on them a bit before I can release. So, like this picture, we have a Drupal Docker container. We, as I said before, use Node.js to develop an application to manage the sites, the multi-site. So, why? Why so asynchronous? Node.js is asynchronous, as you know.  You tell the application, well, when you finish to execute this, just execute this callback, this function, and you forget about how it works. And, well, we managed this with Node.js app, which connected to DynamoDB, which is a Node SQL database on Amazon Web Services. We have a JavaScript SDK, so we can interact between the Node.js application and the DynamoDB table to manage the state of the different sites and so. Well, we developed it also on the same app, an API, so we can do batch processing, for example. Hey, there are 2,000 new users that want their sites created by Monday. Okay, no problem. We just send a JSON to our API, and the sites will get created, and we use Qwis and so. Well, now that we have the two main applications, the two main level, the top level applications, we will go one step down, and I will describe the container itself. Okay, remember, we have a Drupal running on top of NGnex and PHPFPM, and we want these technologies, these tools inside a container, a local container. So, we need a stateless Drupal container, which is this stateless Drupal container. We don't want to have any mutable data inside of the container. We want to be able to kill the container and raise up a new container without losing any data. What that means in practice is that Mencache, Mencache, MySQL, MongoDB, must be external services. And the files, the directory with the user-uploaded files will be out of the container too, so we use it as storage support. For that purpose, well, Mencache, MySQL, and MongoDB, you can configure it at the end point on the sit-ins PHP, and to use S3, we use the S3FS module. That module is a great advantage, and I will talk about a pitfall that we found and how this module helped us to go on. Okay, the e-mails will be sent with an external service, which is postmark, and we use it in the new relic for monetization on each container. You can see here a small diagram, okay? Elastic is blue instead of green, because it's not persistent. So, having this in mind, remember, we have now the container, and now we have to go another step down, and I will talk about the cluster. I mentioned before, we use it in Apache Meso's cluster with two masters and a bunch of workers. On the workers, we will launch the Docker containers using marathon, and all the clustering is orchestrated by ShowKeeper. We also had a hyper proxy on the bottom, and well, we use it also another tool, another Apache Meso's basic tool, which is Kronos, to let us execute Chrome jobs. We had some problems to execute Kronos in the Docker containers itself, so we use it Kronos, and it worked very well. So, you can see here the, oops, you can see here the diagram. We have ShowKeeper orchestrating the two Meso's master, and we have one main worker and then a group of another workers. I will explain now. Very important is that marathon and Kronos works in a way that they have a REST API exposed, so to launch a new Docker container, a new app, or maybe a Chrome job to set up a new Chrome job, you just only have to launch an HTTP request to that API, and then you have your staff done. We will see an example later. Okay. So, you put on top a Ginex and FPM on a Docker container, on a Meso's cluster, and Meso's cluster is not elastic, but using Amazon Web Services, a cloud provider. On the bottom of the Meso's cluster, we have an elastic Meso's cluster, which is a very, very good idea. So, we use EC2, an auto-scaling group of EC2, and I will explain, okay, how it worked. We have, this is the diagram we have seen before, and take a look at that, at this. We have an auto-scaling group with the Meso's workers, not the main workers. And, well, the Route 53, which is the DNS, we'll point to this auto-scaling group. We have all the external services, and MongoDB, we use it MongoDB, MMS, which is the managed system from, from, from, from, from, from, from the company of, which created the MongoDB. And, well, why is this? We had three different containers which are, first, varnish, second, the Node.js application, and third, the Drupal containers with Ginex and all the stuff. But, one of the problems I will describe later implied that when an auto-scaling group just scales up, it adds, it adds a new EC2 instance, okay? More, more containers are deployed. But, when the auto-scaling group goes down, if you don't separate, for example, we had only one varnish container and one Node.js application container, and a lot of NGinex Drupal containers. So, if when scaling down, the container with varnish and Node.app gets skillet, then we lose uptime. Our application just won't work. So, we decided to separate those two groups. And well, the, when you, when the, when the auto-scaling group goes, sorry, when the auto-scaling group scales up, there's no default way to deploy new containers. So, we have to create an script and execute it from atcrc.local to get those new containers deployed after the new images is up. Okay. Let's continue. And, well, of course, we have now an idea of how the infrastructure looks. But, it's a very large infrastructure to be managed by hand. And remember, a three-people team. So, I'm a bit lazy with my work. I don't want to repeat work. So, let's automatize. Lazy DevOps is burst DevOps. I really like the way Homer thinks. Well, we use it Ansible because it's a very lightweight tool. It uses SSH connection to manage all the hosts. And it integrates very good with Amazon Web Services. You know, a simple playbook is enough to execute one command and have a full environment deployed. We use it Docker half, a private Docker half account to store the Docker images. When we deployed, we made a new deployment. We just created that Docker image. We uploaded it to Docker half.  Not in that order. But it worked. We use it a rudimentary way to configure the containers itself, which is this. Make file and Docker file. We use the Docker file to configure the container, the static content of the container. For example, the Immutable configuration files and the packages installed and so on. And the Make file will help us to deploy mutable content, mutable settings. I mean, for example, the settings PHP file and so some configurations about the virtual host and so on. The best point here of using Ansible and this combination of Make file and Docker file is that we have a very easy way to create and destroy environments on demand. Okay. But that's not all. Devil is on the details. We have to pay attention to a lot of details as this infrastructure is very complex. So, for example, backups. Amazon lets you do automatic backups on RDS, for example. But we created a new recovery plan, a custom recovery plan. So we didn't depend on the Amazon default staff. You know, we have a firewall. We just used it. This is basic. It looks basic. HTTP authentication for all these APIs, but not everyone does always. You will be surprised. And, of course, lock centralization. We later, when the project was a couple of months running and working well, well, we started using Mesos. Mesos has inside of each, easy to instance in this case, inside of each worker, had a directory in which, which is shared over the network with some little configuration. And we use it to communicate and to store the logs of the site creation process. But this is good. But we finish it using elastic search given a log stack. But, well, Mesos is very powerful in that point. So I will focus now. I will talk about some problems that we found and solutions. And databases. I want you to think about the problem of having a lot of databases on a multi-site install. You remember a multi-site install will have one database per site. And if we're talking about more than 30,000 sites at first, and we want to scale up to 100,000 sites, there's a lot of databases. So, yes, the cloud scales, but that's crazy. Remember that MySQL creates one folder per database. And inside that folder, it has, depending on the configuration, of course, one file per table. So if we use 100,000 folders, okay, it's supposed, we suppose that Amazon will manage low-level stuff. But, well, we test it and it does not, it was not working properly. So, well, we just decide to make some changes. Another pitfall was MongoDB. When you create a new MongoDB database and you add one collection and one item to that collection, it will pre-allocate about 600 megabytes, so it's 655 megabytes. That memory will not be used at this moment, but it's pre-allocated. So if you start creating a lot of collections of the same database and you start to add in data, that will be growing exponentially. So we have to split because both MySQL and MongoDB will not scale up so easily. It was basically unmanageable. So we use a strategy, a classic strategy, which is divide and conquer. So what we did, we identified each site using an unique hash. So each site with its hash, that hash will be the prefix for tables on MySQL and MongoDB. MongoDB module, MongoDB field storage did not have the prefix feature, but we modified it. I have pending to contribute that change. What we did, so we can now mix on both MongoDB and MySQL, we can mix the tables on the same database, but we don't want to have one database with a lot of tables, millions of tables virtually. So we use it group by. We grouped both MySQL databases and MongoDB databases on groups of 500 sites. That way we get a maximum size of 4 gigabytes for MongoDB databases and each MySQL database had 62,000 tables, which is more manageable. So it gets more ordered. Well, we had to perform a couple of operations to determine which database belongs to which database belongs to which database belongs to which database belongs to which database belongs to which database belongs to which database belongs to which database belongs to which database belongs to which database belongs which site. So, what happened when we received on the web server a request, an HTTP request. Okay. We used the settings PHP file which is executed on all the HTTP requests to perform some changes. So, before creating the databases arrived, we just detected the subdomain which is, yes, we used subdomains to separate each site. So, as I was saying, yes, we received on the signals PHP the request and we identified the site, the subdomain, the subdomain name for the site which received the HTTP request. From that subdomain, we obtained the hash, the unique hash, and we used that hash as the database prefix on both MySQL and MongoDB. We asked DynamoDB, remember we used DynamoDB to identify which sites are deployed and which just don't exist. And on DynamoDB we have a column which identifies the database to which the site belongs. So, we just created on that point the databases arrived and we can access, Drupal can access to its tables without seeing the other site's tables. And well, to avoid multiple database, sorry, DynamoDB calls, we stored the database reference on Mencache on a specific key so we don't have to call DynamoDB once and once and once. So, we had another problem. This is a very strange problem which happened when we started testing. And when we created a new Drupal site, we had a timeout which we could not detect on the first days. It happens when the Node.js application just launched the site and it just got cut. We were under very heavy pressure and we just developed a quick patch which basically is take advantage of the Node.js asynchronous feature and we developed a Fire and Forget method. You know, we launched the site, we informed a container, a container, hey, create a new site with this subdomain, this user and this data. And when the site gets created, notify me to this new URL. When we forget about that site creation process and when we get notified, we can continue with the processing. After when the pressure went down, we could investigate and the timeout was produced on Haproxy. Well, that kind of things happened. And well, other problems that we found were instability on Auto Scaling Group is the kind of stuff. This is related to the stuff that I talked about before, you know, the containers for varnish and Node.js app being destroyed when the Auto Scaling Group went down. That happened randomly, so we decided to split and have one fixed worker, Mesos worker for those containers and have the Auto Scaling Group with only the NGINX containers. It was crazy when we were in the middle of migration with just three days to finish the migration, we were at 20%, I think, and this happened. MongoDB instances went out of space. Be careful if you use MongoDB MMS system because it creates easy-to-instances. It creates a replica set and a chart of MongoDB instances on EC2, but the default storage size, the default volume size, are 50 gigabytes. We just realized it when it was too late and we lost about three hours. In the middle of migration, it's hard to fix that. We also found on PHP FPM some stability on some request. Just remember to disable slow log when you are using NGINX and PHP FPM. On the first approach, we created one single bucket for each site. 30,000 sites means 30,000 buckets. What's the problem? Amazon only lets you create 100 buckets. That's a big problem. As I said, S3FS module saves it as because it lets you configure a single bucket for all the sites, but each site will have a directory inside of that bucket and the directories just don't see in between them. It was a good solution. When you are applying a new update, for example, we have a new Drupal code release, you have to deploy. The Docker container will get the code. We create the image, upload to Docker Hub, and then on the Apache Mesos cluster, we just deploy. But what's the problem? We have to apply a database update to a lot of sites. To do it, we created a custom script that will let us call drash on each of the sites using the DynamoDB list that we have of all those sites. It took about 24 hours, a full day, to call drash on all the sites when they were 30,000. The migration, I have spoken about it. We use the Migrate module. It was a Drupal to Drupal migration, so no problem with that. One thing to mention here is that the previous version of this project was a single site Drupal. You can imagine how big it was. 30,000 users with a lot of nodes each one. Subdomain module, it was crazy. But it went migrated and it worked good. One comment to add, when you are using RDS Relational Database Service on Amazon, if you allocate more size for your instances, then the instance will have more IO speed. We realized it and we just used it. We had some problems with the database speed. We tried first to approach by reducing as possible the size of the RDS instances, but we realized that it's not a problem of storage. We prefer to have instances storage so it can scale best as long as new sites are deployed, but also we would take advantage of this feature. I will do a small demo. I have prepared a video. I will show just Apache Mesos. If you don't know it, you will see. This is Apache Mesos HTTP console. I have a cluster on my local with two workers. We will see these are the two frameworks which are Cronus and Marathon. These are the tools on the top of the cluster which let us run CronJobs and run Docker containers. At this moment they are not used. We will see we have two workers, two machines with one CPU, each one with 244 MB of RAM. This is the disk and the IP address. It's very complete. Looking at the Mesos main screen, at this moment we have no task deployed. This is a summary of the number of CPUs we have. Remember, two Mesos workers, but we see two CPUs. The memory also gets zoomed. We have them idle. The active tasks are known at this moment because we don't have... This is Marathon HTTP screen. We have no application run, no containers run. This is Marathon. This is just the user interface, but remember that we have a REST API. How the REST API works. We send it to a specific URL. This file, for example, is a JSON file where we describe what kind of container do we have. In this case it's a plain NGINX container. This is the configuration for the network. We have the container port, which is 80, the default HTTP port. We receive the... I will stop the video for a second. If we receive a request on the cluster port, 1088, we will redirect it to the Docker containers, 80 port. We have one Docker container in this case, one NGINX container. It will use only half a CPU, and we will use 64 megabytes of RAM from the available. There's a lot of options, a lot of configuration where you can fix the host, the worker, where the containers will be deployed, and so on. We have here the URL post request. We use JSON, and we include the JSON file itself, the contents. And 8080 port to the apps URL. We receive a very interesting response. We can even use it to manage it. And then you can see we have one Docker container using the features, the resources that we ordered. We have also access to the JSON file. We use it, and remember, the port open, the number of containers, the CPU, all the data. And now another interesting stuff. Well, you can see here, okay? If you can access, well, let's see. You can access to the container on a fixed port. In this case, it's this one. Okay, we will close it after. Okay, here is the NGINX. I don't think it gets, you know, it's 31. Okay, here. Okay, so if we open the fixed port, we have the 31, 484 port, and we have access to the NGINX. This is direct access to the Docker container, okay? But we do not want to access to each, different ports each time we receive a request. We want to access through the 1088 port. So we will scale up to two containers, see how easy it is. Just pushing scale, set two apps, and we have another Docker container deployed on top of our marathon. And we have one of them on the 10 and another one on the 11 IPs. We can access on the direct port, okay? But if we change to the 1088 port, we update, and we have now access to the NGINX. As a best practice, we close the ports on the range. It's a configurable range. And we let only access to the 1088, so we can manage it on a proper way. So this is a marathon. You can see here, the resource is being used on the Mesos cluster summary. And now you can see here, too, the tasks which are the Docker containers, our references. You can also access a sandbox, so you can take a look to a standard output, standard error. Okay, so it's very easy to manage the containers. You can see here the access log for NGINX. Okay, so it's very easy to concentrate the logs there. That sandbox, it's based on that shared directory that I mentioned before. And this is Kronos. Okay, this is an example. We can launch KronJobs. In this case, it's just a data. So it's very simple, and it will be executing. This is the next scheduler execution, the time it takes. Well, it's very interesting. And that's it. Okay, that's the demo. You know, I just wanted to show you the most interesting part on the technology stack that we use on the project. Just to mention, the project has right now, if I don't remember, about 70 CPUs, 70 cores, and about 120 or maybe more gigabytes of memory are available, and almost all of these sources are being used. So those are very great numbers, and well, just to be finishing. As conclusions, well, this project went live on April, last April. It was a success, yes, but I slept two few hours that week when we went live. You know how this works. I learned a lot about those new technologies, and this is something I have in very high value for me. And well, the combination of containers and cloud providers is an absolute success too. Just after we launched it, Amazon released a new service which let you run Docker containers on the top of Amazon itself. You know, no need to Apache Mesos, no need to EC2 instance. Well, it was a bit late for us to change at that moment. And well, another thing I learned, well, I already knew it, but Drupal is very flexible when someone just say, hey, I will use another CMS or another tool. This project has demonstrated, at least for me, that Drupal is flexible enough to basically don't use it on every project. It's very flexible. And well, about these tools, I recommend you to learn about them. But I recommend you not the quick way. I mean, don't try a couple of hours and so. Read the docs, are very good docs, both marathon, Mesos and so, and keep practicing with them for some time. And well, when you just found a way to reach the volcano and throw the ring, it will be more useful and more satisfying for you. So, please practice with them. And well, that's all. Thank you. Thanks. And if you have any questions? Yes, please. Sorry? I can't understand. Can you show the site? No, I don't. You ask if I can show the site. I cannot because I don't have great permission to do so. That's because I have not mentioned the company. If you remember, I have not seen. But I don't have permission, so I can't. Sorry. Yes? Yes, so my question was always everybody's very positive. If you create a new social media site, you will never usually think of the use case that people leave. So, first of all, obviously, great work on scaling at really amazing levels. But there will be complexities probably when you will remove users and then you had 500 sites per database and then you need to somehow scale. So, will that be how have you thought about handling that? So, your question is? When a user removes a site, what will happen? Or is it impossible that you cannot remove a site? Yes, you are asking how we manage that sites. As you remember, we build a Node.js application, which is a custom application. And we use the Amazon API and we connect in that application. We design it to be able to locate a site from the subdomain or from the user identifier, the unique ID. And just calculate the database and which is the data, the tables. And then remove them or move them to another database and perform whatever action we needed. Maybe there were best approaches, but we use it this because a custom Node.js app will let us know a lot of flexibility to do these tasks. Yes, so thank you. Thank you. Any more questions? I have two questions. First, simple one. Do you have any problems with S3 file system? A problem with? S3FS. S3FS. Well, at the beginning, yes, because we were using the first version of the module, one point, et cetera. And then we had a problem with requiring an S3 bucket for each site. When we realized that that wouldn't be possible, we switched to the dev version of the module and it worked like a charm. We had no problems with the modeling in this case. We tried some options, but that worked best. The other question or idea is, did you think of use this kind of scaling for one site only using some load balance or something? Could that be applicable? I think I didn't understand. You have multiple multi-site installation. Yes. Did you think about using it to scale only one domain, one site, with the same technology you made? Well, I did not take the decision on the technologies. I just implemented everything, but the architect was the one who decided. But, well, the previous version of the project was a single site, you know, a single multi, a single, sorry, a single site installation. And, well, maybe due to the design decisions took on the moment or whatever, it does not work it fine. It was just slow. We were talking about five seconds per request, non-catcher request. Well, the architect took the decision to use this kind of monster infrastructure and it worked. It was a good experience. Maybe we will have considered, we should have considered that option, a single site, but at that moment this man decided to use the auto scaling that way. Thank you. No problem. Hi. I have two questions too. Yes. You said about the main worker and the other workers. Is there any difference between the two? Yes. What we did to get the, to avoid the problem with killing an easy two instance and getting the varnish and Node.js containers down. We just separate and we have one worker with fixed containers, one for varnish and one for the Node.js app. Okay. So in the auto scaling group instances, we have only Docker containers with the Drupal itself. That way the auto scaling group can scale up and down. It can add instances and kill instances without problems because the load balancing occurs inside the mesos itself. But we, that way splitting the two kind of workers, we get confident about not being killed on the varnish and Node.js site. Okay. And the other question is simple. Do you run dras with Ansible or not? Are you asking about running tests? Dras, drus. Sorry. Dras. Ah, drash. Yes. No. To run drash on all the sites, we built a custom script. In fact, it's a plain PHP site for it. And we used it on the, over the network. But that was a decision that took on the last minute basically. But it worked. We just prepared a launcher, PHP launcher and another PHP file which was listening on the container side. And well, it just executed the instruction, the command that we ordered from the other endpoint. It's not the best solution. I would have used another kind of solution, for example, to raise up a new Docker container just specifically to run drash there. You know. But we had no time and we had a very heavy pressure at that moment. So we took that approach. And well, it was fixed later to use the single Docker container to run the drash. But what? At that moment, we used that solution. Thanks. Thank you. Thank you for your speech. It's been amazing for your job. But I have a question. It's running in my head.  But I would like to know who... Try to speak with a higher volume, please. I would like to know how you make... Because you speak... You say you make like 3,000 database. And I would like to know who you make that database. You put all the stuff inside that database or you just separate some tables. Or you put all the tables all together. I mean, user, cache, everything all together or... Yes, you're asking about how we manage the tables on the database. No. We needed to split them in different... How did you separate all the tables? I mean, for example, when you make a multi-site database, you can, for example, say to settings, PIP, you can say, well, the user, the table user is going to be in that database and the fields table is going to be in another database. So what did you do? How did you make it all database full stack? I think I get what you... What we did was two main steps, which were we just created the new sites on the database as normal on the main SQL, on the database that it functions. But we don't have two kinds of tables on that database. For example, cache tables were not on the main SQL database. We used main cache. So we just removed them except cache form. That has to be persistent. And on the other hand, the field data and field revision tables were on MongoDB. So we just don't have those tables. That way we have per site about 100 tables on my SQL. And once we did this, we split all the databases. Imagine we had 30,000 sites. So we have a lot of tables. But we mixed tables for each 500 sites. I mean, each bunch of 500 sites, we create a new database and we start to use that database until we reach 500 on there. And the database, we can be sure that they don't have a conflict because each table has a prefix, which is the hash from the subdomain. What exactly did you do with the user database? For example, if I'm in one site, if I go another site, for example, subdomain, I will be keep logging or I have to log in again? No, no, no. You are logged. It's a multi-site in Drupal install. So it just behaves. So you manage with the model? Yes. That's it. It's plain. Thank you. Just one more question. Well, after that, I will be out. But we need to let the room for the next speaker. Please go to the microphone, please. My question is, on this setup that you had on the sites and it is in a multi-site approach, when you log in to one, it log in to every single site, you are using like a separate set of users for each one of the sites. Yes, the logging is separate. I mean, a user logged in on a single site will be an anonymous user on another different site. We just had some mechanisms to show the admins could interact with the sites, but that's a part. That's not related. Okay, thank you all for coming. If you have... Thanks. Thank you. If you have more questions or you have to discuss something or whatever, I will be out or maybe down with a coffee. So feel free to get by me and ask. Thank you.