Alrighty, hey everybody, welcome to our panel. I'm here with Igor Kozlov who gave us a great talk on how they use data science, specifically machine learning at Bell Canada to do detection engineering. We have Machu Sonier, I think I got that right this time, who just finished his talk, which was full circle detection.  And it has some really great contributions on the community end of detection engineering and detection response in general. So guys, I appreciate your time and hopefully we get to talk about some things. I know we have some community questions that we wanna go into, but first I wanted to kind of set the stage based off the talk that we just heard. Machu, how important is it? So I kind of in the intro talked about how one of the big differences between kind of like the red side and the blue side is that the blue side is very much a team sport, right? So like you don't just get to, if you're trying to do this thing all by yourself, you're gonna have a hard time, right? So we need to, and that could be both internally and externally. So internally we need to work with our partners who are on other teams, maybe they're on the infrastructure team, maybe they're the incident responders when we're building detections or community wise, sharing that information. And I'll kind of get everybody's opinion on this, but how important is it to kind of share or work as a team to try to bolster our ability to perform detection and response? Yeah, I think it's one of the key thing in this thing and that's the whole thing that I was talking about at the end of the talk, right? I think that the red teamers share a lot, they do blogs, they share their tools, they share their TDPs and everything. And the defender, there's always this kind of almost false sense that everything is intellectual property, that if we share, we're gonna expose our defenses to the attackers. But as I said in this talk, and I said it again, but lots of the things that we're bailing detection for are actually already known. So there's nothing, it's not an intellectual property to look at LSAS or to make sure that we get notification when LSAS is touched or in this case, when those files are created. So yeah, and that's for the whole community and inside. It's a bit harder when you're a team of one and that's when I think you need to rely more on the community. And that's why I think it's so important to share with the community. If you're lucky enough to be in a team that has many people that specialize in different things, some such as threat hunting for Carlos or machine learning like Igor or threat hunting or whatever, building detection or extremely strong on as you mentioned on the data collection on building pipelines. If you share all of those things that you're building as a whole team and that the smaller player can help, I think the whole community benefits if we're gonna see less breach, we're gonna see better protected company and it all goes together. It's one of the main reason why I like to do talks is to try to share the knowledge that I was lucky enough to gain through all of the other people that I admire. Awesome, yeah, great. So one of the questions that just came in that I think is kind of moves through that kind of line of questioning pretty well is the idea that attackers are quickly coming up or very rapidly coming up with new attack techniques. How do, like what's a good way? I think it's important kind of implied in the question. It's important to have a goal for your detection efforts.  That way you can validate whether or not you achieved it, right? So like how do you judge it? So does anybody have good ideas on how you can actually prioritize what you're looking for? So like for instance Igor looked for credential abuse or Matthew you were looking for outlook abuse. How do you choose what to look at? Is it just, hey I saw something cool and I think I should go after it or is there actually like a process for determining that? I'll go to Igor first, we'll say. Yeah, thank you. So yeah, I just wanted to go first because I also wanted to comment on something that you said before my presentation and also which was exactly related to what Matthew was now talking about. So I think what you mentioned just to remind everyone is that you said that analysts lack context often times that the scientists say, oh, this is anomalous and then analysts look into it and like why? So and I think this exactly relates to what Matthew was mentioning and also to his idea that people working together always stronger and I just want to say that I worked with Matthew personally and I know that when he was trying to continue that belt. So he said when he was building a team who is extremely successful, we have a lot of superstars there. So he said, I want people very different. I don't want the same type of people. So and here I would like to again to circle back to what you said. So why analysts don't know have lack of context? It's because other scientists never work with analysts and from my point of view, this is the hugest problem. So people coming together with very different backgrounds. So this is when we have the power. So now going back to what you said. So priorities are usually set in corporate environment by other people but just my personal comment would be so I was mentioning credential attack in the presentation as an example. I was also saying that it can be extended to lateral movement and other TTPs. So credential attack is just one that everybody knows about kind of it's easy to sell. And so indeed, not everything can be equally easy attacked with, for example, machine learning techniques that are out there box for you. For that reason, whenever somebody tells me let's use machine learning here, I say, well, can I please look at the data first and talk to people who have a lot of experience in this field and then come up with what will be the timeline for the solution? And sometimes managers would agree that, okay, so indeed it seems like a simple problem but solution would be very complex so we don't have data or we don't have experts. So I would say we indeed prioritize according to the enterprise. So they have their decision metrics. They involve a lot of smart people like experts in the fields to say what is priority, right? We have a lot of industry research on that but also when it comes to machine learning, I think it's important to understand that data is the key in the sense garbage in garbage out. If you don't have good data, it will not work. Thanks. I want you to go ahead. Yeah, so this is an interesting topic in particular because I hear it often, how do I get started? What do I do? And I think it's important to start with the notion that there are different business and business sizes and organization sizes. So it would start with that, right? So taking into account that because you wanna start simple regardless of the size, the key is that you find a way to scope things and you really wanna narrow scopes and there are multiple ways to do that. Either you use the intelligence, you take the intelligence approach, what are the APT's actors that are after all is, you can look at vulnerabilities. What operating systems do we have and what are we vulnerable to? You can look at a project where you design what your crown jewels are, what are the systems that I'm worried about? Maybe you're worried about insider thread. So there are multiple ways. You have to pick one and try it. And if it doesn't work, go to the next one. That's probably one of the most important things that wanna make sure people know. There isn't, we're here but it's not like we, there's a solution that applies to everyone. You have to try a few things and you'll find how things work out but you have to iterate. You have to try the methodical and that's the other thing. You have to find a way where you can develop a process and a process that you can repeat so you can improve it over time. And as you go, you will start seeing the benefits. So to recap, there are many ways to do it. Find one that works for your organization, the size and the type of work that you do. Develop some methodology or borrow one of the many that exist and some of those have provided and then try it.  If in doubt, there's plenty of forums out there where you can come back and chat with many of us and we will have to dive in further but that will be kind of like the TLDR on this idea. Yeah, I often tell people when people ask me that question, first of all, it's like something is better than nothing. So don't like get to where you're paralyzed not being able to make a decision and like use some intuition. If you're just getting started, think about like what are the things that I hear about the most and like just go for that, right? Otherwise there's like the threat intel approach. There's like an entity-based approach like for instance, we worked with an airline at one point that decided to start building detections around their loyalty system because they knew that there was some interest in hacking their loyalty system. So they decided to focus on that. Maybe active directory is something you wanna look after. Maybe there's like a tactic-based approach to where you say, hey, like we know that persistence by definition is persistent and going to be there for a long period of time. And so maybe that's a good place for us. We don't have great collection capability right now, but we know persistence is something that's always gonna be there and it's meant to last for a long time. So let's focus there first or maybe we only have collection on domain controllers. And so we wanna look for active directory type attacks first because we know that we have the telemetry to do that. Carlos, one thing that you kind of touched on was this idea of continuous improvement, having a process. One of the things that we like to talk about at my company is the idea that we want to differentiate between when we're successful because we have a single person which is like individual competence. You have a really smart person who does a great job versus an organization that is organizationally competent meaning that you can replace individuals and or you could hire new people or a person could leave and that like the new person's gonna come in and plug right into the process and know what they're doing because there's documentation, there's relationships between organizations and expectations and things like that. How important match you is that in the type of things, the type of concepts that you were talking about with that detection life cycle of being able to provide that IR playbook and that validation, those types of things. So just before I go to that, I just wanna add two little things about the previous question if you don't mind. So I think that having metrics to show how you improve over time is extremely important as well to show your management that you started at point A and that maybe in Q2 you are at point B and then at point C by the time of Q4 and that you're improving. I think this is very important. I made also another talk about that called the SOC counter attack that is based on using the attack framework from MITRE as a starting point to build your detection program. And the other point is when you ask where to start, I think that if you have a strong security solution, you might want to know where your current security solution is actually failing. If you have a very strong EDR but it's weak on the network side, maybe you wanna focus on network-based attack and the other way around if you have very strong IPS system and very strong firewalling and zoning but a bit weaker on the endpoint, you might want to focus there. So it really depends I think on what you have and it will depend on for each company. So testing some new techniques, they're testing some known technique for me it's extremely important like a kind of a purple team-ish type of approach. I think it's very important. Now to your other question about replacing people in a SOC and having one very competent person versus a full team and I think it's extremely important to teach your new people because a SOC is typically somewhere where people will turn very fast. You'll have new hires, people will get promoted the better ones, they won't stay in that analyst role for a very long time. So it's very important to have something in place that makes those people great, fast and make sure that everybody is at the same level. So you don't wanna have a rock star and I don't like that term but like one analyst that it's very, very good whereas your other one are average, you want all your tickets to be in the same way every time of the day, every time of the year. You don't wanna go during Christmas and think if this guy is on shift and this type of attack happens, it's doomsday, right? You want that everyone has the same level of understanding and that's why I think that the training part is something that's extremely important to bring your analysts to the next level that will bring the whole SOC to the whole level and that will bring the whole corporation also to the next level in their security maturity. Yeah, I think one thing you reminded me of is it's probably okay to have a superstar, right? But like as a leader or as somebody that's running the program, like your effort and maybe even some of their metrics for job performance should be related to how do they give back to the organization and make the organization better as opposed to kind of remit, like whenever you ask like as a consultant we'll go work with a company and you'll say how does this work and the person or like where is this documented and the person will point to their head and you're like, okay, well that's bad, right? Because that's like the bus theory where if that person gets hit by a bus you're in big trouble. One thing that there's two questions that I think are related and I think everybody probably has an opinion on this. The first question is what data sets, I'll give them both and let everybody kind of give a quick answer on for both of them. What data, the first question is what data set is always gathered but is ultimately mostly useless? And then the kind of like corollary to that is what data is rarely gathered but would be very useful if it was. Maybe I'll go to Igor first. Sure, well, I can start with a funny story, right? Because when I just started, Mathieu said, so what should we do first? And showed me a metro attack and I was like, dumpster diver is the easiest one because we can just set a camera, we can just detect people very easily with machine learning and he was very surprised because for him it was something strange, like why would it be an interesting source at all? And in terms of useless data, I am the guy who likes data. So for that reason, for me, there is no useless data. I think what is useless is if I look at the data and I don't understand it and then I go to somebody who knows what it is about and they can provide insights, this is that becomes very interesting. So I think not having people who are experts about certain data around is what I don't like. And if this also relates to the superstars, I totally agree with everyone that it's important for everyone to be able to pull. I just want to say that superstars I like on teams because they inspire with their passion, everyone. And really when you see a person who is burning with the idea, it's really something that other people also grow and shoot for the stars. So yeah, thanks. Go ahead, Mathieu. Well, useless, it always depends. If you want to build detection or if you're talking about forensic. This is like a religious question, by the way, basically. Yeah, I know, I know. But I like polarized debates. I think firewall is slightly overrated. It's hard to build meaningful detection and I would say actionable detection based on firewall and the other end, when you have an incident, definitely, definitely, definitely, you do need those firewall logs. So it's a double-edged sword, right? You don't want to not have them, but they are extremely chatty. So it's a difficult question. And the one source that we should have that a lot of people don't have is probably Sysmon because it's such a rich data source and lots of people, and I have seen that in many companies where I worked or where I was consulting, is that they say, well, we have an EDR. It already collect all of this or the equivalent. That may be true, but most of the time you cannot do everything you want and you cannot control if you want a very specific event type or a very specific value because there's a new attack, there's a new vector. You don't have any control. So when you deploy Sysmon, you might have it very toned down, if you will, and not duplicate what's collected by your EDR solution or any other solution, but at least get the things that your EDR is not collecting and then you can build meaningful detection based on that. So this is actually a very interesting question and I'll go back with that. It's about the size of the organization and what you do. In general, I have this idea that data is only useless when it doesn't have or provide any context and it's our job to find the context of the data. But even then, there is a lot of data that just gets accumulated. If you're not gonna use that data, then it brings absolutely no value to you, even for forensic purposes. So you need to count to the fact that that becomes problematic. If you have a lot of data that brings a lot of noise or you have an study, you wanna think about it. Is it time for me to bring that data in for me to use or not? Otherwise, sometimes it makes things even more difficult. I come from an organization where it gets terabytes of data and so sorting and looking through that is very difficult and it becomes dense, right? So I think it's important to be mindful of that. And one way in which you can figure it out whether data has value to you is to actually do something that it connects to the previous questions. You really wanna develop or implement a visibility gap assessment. It's really not difficult and people make it very difficult. Just map all of the applications that provide some telemetry, right? And then identify which of them will bring you value for whatever is that you're going after. And then prioritize that onboard it, look at it, investigate it. What makes data much more valuable? And this is where I think you get this weird notion that some data is useless is you need to enrich the data. If the data is enriched, it becomes a bit more powerful, but when you consume it raw without that enrichment of context, you make the process much more difficult. So I will say that that is very important too. As far as what data is readily available, I will say in the context of what I do, which is threat hunting or incident response, most of the faults that I know even on threat hunting, they just heavily concentrate on the host side and you really need to have network telemetry. If you don't have network telemetry, you might not, you're not gonna find many things. And that network telemetry is key beyond the host. You really wanna know what's happening on the wire. And I will say that that will be one of the best things any organization should concentrate on. If you don't have it, if you're good at host, start spending time on the network because it could be the flip that tells you. In fact, sometimes the network telemetry gives you the fact that someone is doing a lot of activity that otherwise you didn't notice on the host level. Yeah, so to add on to that, there's so many things going through my head right now, but to add on to what you're saying about the network is you could also do correlation, right? So for instance, let's say, so attackers use, I've been really interested in service creation recently. So like one of the reasons why that's interesting is because it's frequently used for lateral movement or privilege escalation. However, it's also something that happens a ton, right? On any network. And so like one of the things that you do is you say, okay, how do I know when a service has been created? Like I wanna know every single time a service is created or I wanna be able to identify every single time a service is created regardless of how it's created, right? So you find that like, I kind of call it the base condition of service creation. But then you also wanna figure out features or contextual factors that will basically, you can analyze, right? And so like one of the things that you might be interested in because services are used for lateral movement is was this service created over the network? Well, yeah, the best way to get that is by collecting network data, right? So like you could get Sysmon even ID three which gives you network information, but it doesn't have that granular context of like, this was an RPC request and this is the, you know, RPC protocol that was used and that type of thing, which is really valuable for us to evaluate. And so I think that gives you a better picture of what's actually happening. Yeah, and if you mind actually to sound the network because it's a topic that I hear often is there are attacks that you actually need the network data to identify them. Beacon is a perfect example, right? If you don't have data about a browser extension that has been deployed that has been used for Beacon because it's using DOH, then the DNS data over the network or some other information will be valuable. You also get JJ3, J3S hashes and other type of hashes. So it provides not only context, it allows you to connect, but many times it will be the bullet that will help you identify that there's more, you know, going on on the network. So definitely my hit is network data, it's a must. Yeah, for sure. Cool. Okay, so one thing that that kind of reminds me of is we are really focused on, we have this world where we're really focused on reducing false positives. And I think that's kind of what gets us going towards the machine learning kind of path is how do we reduce false positives in a smart way? But then one of the worries, not necessarily with machine learning, but in false positive reduction is the introduction in false negatives. And like the problem that I see with that is that false positives are apparent, meaning that you have an alert that tells you that you have to explicitly say as a false positive, but false negatives are transparent, right? And so like one of my, I guess there's a couple of thoughts about that. One thought is how do you manage the reduction of false positives? And Igor, I think you probably have a really well thought out idea of this. How do you manage reduction of false positives? Well, also managing, you know, your rate of false negatives. And then the second is, how much should we be like, we're not building detections in a vacuum. So like, you're not just building one detection saying, this is what I'm putting, I'm putting all my eggs in this basket and it has to work. You're creating a kind of series of detections. And so like this one may have a false negative because of X, but you know, people don't just perform one attack technique. There's a whole chain of attack techniques. And so like how important is it to be, you know, worried about false, like the threat of false negatives? Thank you very much. I think it's a great, great question specifically because it also highlights one thing. So what people usually know about machine learning is what they learn from popular sources. And I see this a lot. And so when you ask this question and you're talking about false positives and machine learning, but false negatives doesn't sound the bell of machine learning in a sense. It's because this is the source, not the textbooks and the professors who've been thinking about it for 70 years. So why I mention it? Because so there are very different metrics. So when we're talking about false positives, precision, and when we're talking about false negatives, that's recall.  And a lot of other stuff. And there is a lot of research about it, which also says that you should also not only look into false positive, false negative, you look into everything over all four parts because otherwise you can be misguided easily. And what I want to tell often to people who have power in decision making is that when they're talking, let's reduce false positives and say, okay, let's turn off all the others and we will have no false positives and we will have very good accuracy because most of the times we receive false positives and they don't understand it. So let me just please clarify this idea in terms of poker because in my organization poker is a big thing. So the idea is that if you play only when you have top four aces, you will be always winning. But will it make you a successful player? No, it will not because you have to play with the hand that you have and win with the hand that you have. And so for that reason, it's absolutely paramount not to focus on just one metric. So one metric by itself. So there is again, a lot of studies in psychology that shows when people's organization starts following one metric, first they start affecting this metric just to look good and it never is a good thing. So here I would say when we're talking about false positives, we should be always talking about false negatives at the same time, at least. And so we should be saying at this rate of false positives, this is the rest of false negatives, do we accept this? And so once again, there is a lot of research so let's not deep dive into it. But what I want to tell you is that there is only one way to improve both at the same time is to build better detections. So once you build better detections, you can reduce both false positives and false negatives. And sorry for the plug. So in my presentation I specifically show that if you just use one type of detection and supervised machine learning and rules, you will not be seen what you will be seen with addition of supervised machine learning on top. And so this was just one of the examples. Regardless of what you're using, you need to include more and more ideas into the thought process. And as you are saying, so you need to bring a lot of a lot of dimensions into the game and you also need to prioritize. This is what the other speakers say and absolutely agree with this. So machine learning just could help you to prioritize a bit better if you have a lot of data. But if you don't have data, go to the experts, security experts is your data. They've seen it all, they know how to help you. So to summarize my answer, so look into the problem having multiple dimensions in the sense that bring all type of experts into the game to discuss and a must is security experts, if you don't have data, they will tell you what to prioritize. They have intuitive understanding of situation. Just make sure to include also people who know a bit about data because sometimes they can't help when, because we don't know what we don't know. So bring as much as possible of what you don't know. Bring as much as people that whose background you don't know and they might help. Thanks. Cool. Anybody else have any opinions on that or should we move to the next question? I have some, definitely. But I wanna see if Matthew has some before. I think we might all have opinions. I'll give you mine in a moment. Matthew, you wanna go first when we go to question? Go, go, man. So this is actually a very interesting question because you're gonna encounter as an analyst, as an incident responder, your presumptions, right? And even when you are seasoned with quotes, those might come back to bite you. And so you might encounter things that definitely look good, that there's nothing wrong, but they turn out to be bad, right? You get those things. So I think the first thing is to address that, be mindful of the presumptions, right? But then challenge them. I have this notion for me of what trait hunting is. And the way that I define trait hunting is just a methodology that allows me to proactively look for the unknown unknowns. And I use this concept of basically that the unknown unknowns, right? So if you challenge the presumptions, you're gonna find a way to uncover some of the false negatives. It is not easy, right? And it takes time and iteration. But one of the ways that I find that you can, it can help you and even actually helps you with the notion of improved detections is unfortunately with one of the side effects that I perceive happen when MITRE came up, right? The attack framework is a lot of, a lot of folks have zoomed in too much into the atomic type of detections, right? One detection that will help you detect this variant of Mimicats or something else. So I have this concept of molecular detections. These are detections that are more broad. They try to cover more grounds and they're prone to be false positive, right? But what you do is you compliment with the atomic detections. So you eventually end up with what I call high confidence detections. It's not easy and it takes some time but I really encourage people to think molecular. Go one step up and then start looking at the data. And that forces you to do one thing that reduces in my opinion, false negatives to understand your data. What happens to your network is unique to your network. And that allows you to determine is this really bad or it isn't. And how you're gonna go about it to challenge that, to determine whether there is one or the other. Hopefully that makes sense. I hope it did. Yeah, I like to, I usually talk about detections as being precise or broad and there's a spectrum to where you're on either end. Like precise would be a file hash would be very precise. Basically no false positives, like no false or potentially tons of false negatives depending on what your goal is. So if your goal is to detect Mimicats and you're looking for a single hash they could change Mimicats by changing a byte.  And then broad is like, I very broad would be like, I wanna detect lateral movement. Well, you're gonna have a ton of false positives, maybe less false negatives. Now like the question is, where does something get labeled a false negative, right? Because like you're gonna detect it and maybe alert on it, but somebody's going to mark that as a, close that ticket because they're just inundated with tons of alerts. All right, Matthew. Yeah, so that's one of the things I wanted to say for that subject and specific. And I think I brushed the subject in my talk, but is that not all detection that you're gonna build are ticket worthy. So if you go broader, you might want to have a dashboard or a weekly report that you're gonna produce on that output and someone can look at it manually, maybe once a week or maybe once a day depending on how verbose it is, but you don't need, your detection don't always need to create a ticket and it don't need always need to be handled by your stock analysts. They can be handled by other people. I worked for an organization in the past where they had people that were not security specialists, but they looked at report daily and they kind of became the anomaly detection by themselves. They knew that every day you would have, and I'm just throwing numbers here, but let's say 10,000 firewall block and 100,000 allow. And if these margin change by a significant amount, they just raise the flag to another, to an analyst and say, look into this. This kind of deviates from my manual trend and they were making these trends in Excel at that time. I was a few years back, of course, but it's still, I think now, that's a little bit maybe what Igor is doing with machine learning and now these things exist, but for teams that don't have that, it's still possible to have people that are maybe not security specialists and do this type of work once. They are even a security specialist to review things and understand what is normal. We often talk about when you need to know what is normal in your network and this is one very good way to learn what is normal. You make those dashboards, you look at your different security solution and you look at the ins and outs or the block and allow and you can quickly make trends for that. So, yeah. And if I may add, there's also a term that my family's team that works on a lot of the kind of measures we deploy uses is the risk-based, RBA, risk-based analytics. The idea is very simple as there are events, many events that trigger things, but there are events that require context and depending on the risk and the score that you have, they become notables. They allow analysts to jump in and say, this is meaningful, I wanna go and look into it. It takes time and this is, again, iteration. You really wanna continue to go into that cycle to is this working or is this not? And I think one of the things we haven't talked about but I wanna kind of mention briefly is I found that for me to be successful in threat hunting, there is obviously the team but there are other people that are in the trenches, you know, the SOC analysts know a lot and the engineer is his admins and they can provide a lot of feedback into what's normal or not normal, what could be a false positive or not. So, find a way to communicate with them, to interview, to have meetings, to get them involved into this detection cycle that you build because they will bring a lot of value and they will be your partner. So, it's worth to account for them too. Yeah, so unfortunately we're running out of time here but I wanted to close up on just one little point. So, we were talking about ways that you could handle these types of situations. And so, Dr. Anton Chewbaccan, who I'm sure a lot of you are familiar with from Twitter, but I think he works at Chronicle these days, he has a blog post called On Threat Detection Uncertainty, is what it's called, On Threat Detection Uncertainty. And he says, like uncertainty is like, how do you know that you're doing a good job at this, right? And he says there's three things that you can do to kind of deal with this false positive problem. One is improve alert triage, which is one of the things that Igor touched on in his talk, which is provide the analysts with the things that they need. Also, Matthew, you talked about this with the playbooks. Provide them with the things that they need to efficiently and quickly and accurately triage alerts, right? Another one is use multi-stage detection, which is also something that Igor touched on with the unsupervised and then supervised machine learning aspect, right? So, like you can have multiple steps to your detection process. It doesn't have to become an alert immediately. You can step through. And then lastly is split bad from interesting, which is what you just kind of described, Matthew, which is you have things that you know are bad and you have things that you know are interesting, but you don't know for sure that they're bad. And maybe you have different pipelines that those go down and you treat them differently and you have different resources allocated to them. So that's kind of, I thought that that was a very informative blog post. And I think you guys kind of both touched on those concepts during your talks and I thought it was interesting. So yeah, unfortunately, that's the end of our time. Hopefully we can continue this talk at some point in the future. I really was enjoying it. It's always a shame how short of a timeframe you have. We could just talk for hours, but thank you again to Norsak for having us on for the detection engineering section of the program. And also thank you to everybody. Igor Kozlov, Matthew Sonier, and Carlos, AKA Plug. So appreciate it guys and hope to talk to you soon and continue these conversations. Just one thing, today is Igor last day as a non-dad. So I want to raise my glass, Igor, to that.  Thank you so much. I appreciate it.  Cheers, congratulations. Thank you so much. Awesome. So back to the organizers. Thanks guys. Thank you. Thank you.


Welcome back to the Aldis Bersen show presented by Buffalo Wild Wings. I'm Sam Murray joined now with Stevenson men's volleyball head coach Aldis Bersen's and coach off week last week didn't have any matches. How's the week of practice? It was great. The guys pushed really hard They do two days of weights and then five days of practice and we got a lot of Some fun stuff in we had a doubles tournament. I like that where you pair up With every other guy and then at the end I can rank the team from 1 to 15 They don't have any like excuses like I didn't play with him or so So anyway, we had you know, that's a lot of fun for him just goofing around but we We're starting to play some sixes this this week and prep for Wilkes on Saturday. Yeah talking about that preparation They're a brand new team. It's a it's a first match for for both teams in the inaugural Mack tournament How do you go into a game like that a team that's brand new has only played a handful of games? well, we search for tape on the internet and You know, we know We talked to other coaches too. So we try to get some info on them But it's mostly on our side of the net right now at the start of the season. So We're gonna focus mainly on our side And what are some of those things that you're focusing on in this week of practice that you want to see your team Execute in that game. Well We want to hit better a hitting efficiency has been down. I that has to be in like the two three hundred So you're competing we've had some matches where we're hitting close to zero and that's not helping a lot Then you just have to ace them and block them, but we've been working on our hitting and And also we've had some injuries. So I'm not to give anything to Wilkes or to anybody, but we've had players step up and play in different positions and So we've been going through that in practice, you know having guys that aren't used to playing middle or or setting Just stepping in and doing a good job of it. Oh Coach we wish you the best of luck in your in your match on Saturday. We'll see you back here next week Okay, see you there for coach Aldis Bursons. I'm Sam Murray and this has been the Aldis Bursons show


So, you know, these are the replacements we have been seeing on TV about how to profile test publications. Well, can we turn off the lights? Excellent. So, anyway, let's get started. Let me show you some bars to scale. I won't really tell you what these bars are about right now. We'll just go one by one to see, you know, the proportions of different things. So, that's it up here.  So, Doug is a smart man. He's writing a new debugger's content. He's written a bunch of Excel stuff which I can really not understand. So, Doug's brain is 20 centimeters long, right? I don't know how he manages to understand all these 20 centimeters. Say you're working in your computer. So, you have your beautiful wooden desk. Like here, as everyone else. You know, your computer. And you don't remember something. So, you reach for a book. Your radius of action in your desk is generally about 80 centimeters, right? Let me take someone else. What's your name? Okay, so can I measure your arm? Okay, 80 centimeters. Number eight. So, if you have to go out and reach something on the other side of your desk, it's 80 centimeters, you know? It hurts in your back as you try to stretch more than that. So, it's the ones that you have in your desk. If you have to go to the other end of the room, the cheap one, so it only goes to three meters around here. You have to stand up, sit down, read the book and say, oh, okay, now I have it in my brain, so it's something that's really far away. So, it's not inside your apartment. It's more than 10 meters away. So, let's get it back. It's not at the end of the hall. It's not even 25 meters either. Scaling down. This thing is far away. It's not the corner store. You can see the little, you know, the people who agree with eyesight, unlike mine, they can barely see the orange stripe here. So, what is this thing that is so far away? Don't these people looking at the speed of their piece of crap? So, this is getting... We're starting to see something here. This is the campus. Right? We are somewhere... I guess we're something else. So, it's much farther away than that. Let's go back to 3,200 kiloseconds. Right over there.  Now, instead of making it this distance, let's make it nanosecond. Something that we programmers are very familiar with. So, one instruction, every half nanosecond. Sorry, it only has to be considered that it takes half a nanosecond to access the register and to run the instruction. And, you know, we have the first part there. This is exactly to the same scale as this brain, the desk, the room. You have to do something that is just in the L1 cache in the processor, which is, you know, not in the registers, but it's something that is already fetched from memory. It's really fast. It's only two nanoseconds. But it's still four times slower than a single. You really have to go to memory source cache. Memory accesses take ten nanoseconds these days. Right? So, if your program is accessing this huge array, and you end up jumping nanoseconds for each access instead of 0.5. So, that's a factor of 20. Scaling it down. That's one disk six. One disk six is eight milliseconds. Your program has to access a file on this, and it makes this disk change the position of the hard drive head. That's eight milliseconds. That's six orders of magnitude larger than accessing something in the processor register. That's five orders of magnitude larger than accessing something in memory. And there's nothing in between. You do things near here. It's going to run. There's nothing in between going to run and actually hitting the hard drive. Right? So, you are either really fast or really slow, and there's nothing else possible. There's this famous man, Colonel Packard, Mr. Packard, who once said this, you know, as processor speeds increase, processors are getting faster all the time. And hard drives have not improved their speed for years.  So, we are making our machines wait all the time for accessing the hard drive. And later, this famous man, the beautiful, like we seem to know, Thomas Postman, and he wrote this on his blog a little while ago. He bought a new power book. ThinkPack, which is much, much faster than this power book. A lot more powerful processor, no limp operating system like MacOS. And he feels his new machine is just about, you know, he feels that it's about the same speed as his old machine. What's the problem there is the hard drive. In both machines are accessing the hard drive just as much, so both machines are equally slow to him. You know, this is a very informal chart of how things have been going in terms of speed. Discs have barely improved their speed. At some point, processors overtook memory, that was like in between the 386 and the 486. The 486 had a memory cache in the processor because the processor wasn't actually faster than memory access. And it's been going on since then. So, there's no way our programs can be fast if we do too much this access or too much memory access. Where are we going to bother optimizing things, you know?  You end up sitting for minutes while your machine reboots. It's really uncomfortable to look.  I mean, we can just wait it out. So, has everybody seen the new version of Microsoft Office, Office 12? They, okay, they took Office. They re-signed the complete user interface to make it more usable. And this is one of their usability experts. So, they actually found that when people found something that was really slow, they had a lot of trouble using it. Because they said, okay, what the hell is going on here? I moved my mouse and nothing happens. And a little time afterwards, oh, something moves. Actually, catching up with your mouse. So, they discovered, and they are slow, as long as the programmers start working to make them fast, they become instantly more usable without any changes to the user interface. So, the other reason we want to make things fast is to dominate the world. I mean, the whole world is going to be using all these machines pretty soon. And they really don't have it.  They have no hard drive. They just use a flash drive. And we're certainly, I don't know, probably, I don't know, 100 times slower than RAM. I have no idea. It's still like one or two or five or six orders of magnitude. So, you know, there's nothing in between. Also, wouldn't you like to be able to buy a cheaper laptop, rather than a cheap one, rather than the latest model, which is always $3,000. The laptop you want is always $3,000. Yeah.  When people want to make their programs faster, they immediately say, okay, I'm going to make this faster. Boom. There's their big idea. But how much faster? This doesn't let you set a goal. This doesn't let you see when you're at one. This doesn't let you see regrettions. If you make your program really fast, over time it will get slower because you will keep adding code, modifying things, changing APIs, and eventually it will be just as slow as it was before. So how do we keep that from happening? How do we optimize a program and make sure that it ends up being fast after a number of years of changes? So the thing you have to do is to set a goal. You have to say, I want this pop-up menu to appear in under one tenth of a second. Once I reach that goal, I'm done. I don't need to optimize it anymore. The user's goal is not going to perceive any difference between the menu appearing in one tenth of a second and one hundred. So there's absolutely no point in going faster. I feel bad for working on a new mail program and his goal is something like, I want my mail program to use less than three megabytes to show a mail folder with fifty thousand messages. He has been working towards that goal. Or watch a program in under one second. Open Office these days takes like eight seconds to launch in my machine and it's like the fastest laptop I could get. That's just ridiculous. So once you set a goal, you can blame people when things go wrong again. You are not going to do what you go. You have to get your timings over and over again. By that time you will have created a little benchmark to test the thing you are doing. I'm not going to let you take the picture. How can you create a benchmark that's easy to reproduce and an easy way to reproduce your measurements? And then you can use it as a regression test in the future. That's what they've been doing with Tyro. The new performance test treat for Tyro, it's a big list of performances. You know how long it takes to draw a polygon, how long it takes to draw a gradient, how long it takes to draw a big complicated laser curve. And the way they do it is they record all of the sessions of their performance test treats and they keep a history of that. So they can compare the timings of their benchmark at any two-point in time and see, okay, something went wrong between this version and this version because now things are, like, five times slower or something. So we didn't know that a few years back. So when, you know, Stefan Kulov in Susse, he made a lot of work for Susse 9.3 to improve the boot time. So boot time in Susse 9.3 was actually pretty nice. And then he went to help for Susse 10.0. He went to help because we never had a way to keep taking those timings consistently. So in Susse 9.3 and Susse 10, people started adding more crap to the boot process, changing the demons, changing the start of things. And now it's just as well as before, you know. Right now Susse takes, I don't know, one minute to boot, two minutes to boot. So all the hard work that this person did was worthless because we didn't test it all over again. So there's this guy who is one of the pioneers of performance stuff in general. So let's see. Let's say you have, how do you decide, how do you decide what to start optimizing in a program? Say your program has two parts or a little part of your process has two parts. For example, to pop up a menu, you have to create the menu, populate the menu items, load the icons for the menu items, create the corresponding widgets, and finally show the menu. And you think, by intuition, okay, I think that creating all those labels takes a lot of time. So let's imagine that the menu only had two parts to it, like it had only the labels, only the icons and the labels. And you say, I'm going to optimize the label because I'm really sure they are really slow. So if you had taken your timings instead of just doing it by intuition, you would have seen that part A takes this much time, and part B takes this much time. It doesn't matter if the parts are not like big, nice, continuous chunks. This could be a little part here, a little part here, a little part here, but if you spread them out, they look like that. So you work for two weeks, you work really, really hard to make this part faster. You make it five times faster. And everyone gets really happy when they make something five times faster. I mean, that's a lot. But in the end, in the big picture, it's not really much faster. If this took, what, 15 seconds? Maybe it only takes 11. That's not a huge change. I can still go make it half that time. But instead, if you had worked on making the red part twice as fast, which is not such a big improvement, this wins.  You cannot do it by intuition. You really have to take your measurements in the machine, and then you'll see what to optimize. There's another consequence to this, for example. Let's say you, let's say that the absolute fastest you can make each of these parts is, like, the fastest you can make the red part is twice as fast as before. And the fastest you could make the blue part is this little square. So if you, if after doing this, the red part, you do the blue part, you will end up like this. All was to get here. There's no way you're going to be able to achieve it if this is the fastest you can go. So by setting your goals and later analyzing how much faster you can make each part, you'll actually know if you can, you know, if you can actually reach your goals. Well, we use a profiler. Profiling tools in free software, in general, they suck. G-Prof is completely useless. It doesn't work for shared libraries. It doesn't work for multi-threaded programs. It doesn't work for things which you use with DL Open. So it's very much for the examples in the G-Prof panel, because otherwise it's useless. I don't have SysProf installed on my laptop. Does anyone have SysProf? Okay.  You can start seeing which functions took the most time. SysProf, unfortunately, only shows you CPU time. If you have an application waiting for this IO, it won't show you that. Or you can use S-Trace to monitor SysCalls. So what's the problem with S-Trace? Once you run S-Trace, and you give it the options to say, you know, give me the timings for each SysCall.  It says, you know, process ID, timestamp with microseconds, name of the SysCall, parameters. And that's the patient person you can go through that log. So this is the kind of log that I was getting a few months ago when I was profiling Nautilus. Not the start-up time of Nautilus. Trying to see how the Nautilus takes like four seconds between the time you run it and the time it actually paints this for the first time. So Nautilus start-up is 196,000 SysCalls. You know, and it's just impossible to see what the hell is going on there. So, back to DotG's brain. Can I? Can I? So 20 centimeters by 15, well, 15. Okay, thank you.     Pictures, and you say, I like... like that, it's automatic. It's really, really fast.  And it says, you know, this is a timeline. It says one second, two seconds, three seconds, four seconds. And those little lines connect the, you know, the checkpoints in the program stack.  So, it's not worth it to optimize this because it takes, you know, 6.32, 6.38. From here to here, it takes six hundredths of a second. While between here and here, you can see that it takes, you know, almost two-and-a-half seconds.  And that's actually GNOME BFS Demon launching. The next thing I did is, why the hell does GNOME BFS Demon take some... take such a long time to load until I start going to GNOME BFS Demon? I fix the problem now. This has gone to, you know, being pretty much instantaneous. The same problem up there, you know, let me see if this works. So, we have the same problem. This is GNOME 1.4. Nautilus used to have this little function that initializes all the icons that Nautilus will ever use. So, it reads a ton of files. It reads a big ton of files. Later in GNOME 2.0, something very recent, we got GNOME to do all the icon registration and initialization and X-ray path. So, it turns out that Nautilus no longer needed that all, really old way to load icons. But nobody knew, because that code was really old. Nobody had ever measured the timings, and nobody really remembered that it was there. So, when we looked at this profile, we saw, oh, why the hell is it registering the icons? It doesn't mean to do that anymore. So, we removed that function and removed the one line called that function, and those what, 1, 2, like 2.2 seconds, they just went away. Right? So, if you spend a little time making a little automatic tool, you will have much easier time. I ended up making like 30 of these charts, and they were just, you could just look at them and see the problem instead of going through this graph. Yes? One question, how do you specify the login points where you're loading them? Good question. The thing they did, the only thing that stressed print is these codes. So, different points in the program, I wrote this little function that is called the access, access system code, and to access, you pass a string, just to see a file. But instead of passing a file name to access, I just pass this string, which is you know, you could do that today, you could go to the top of books to actually get this, but it's much easier to just put it in different points.  But then my little Saxon program looks for those failed access codes so if you take the time to write those little tools to signature in a program you know, you'll have much easier time because I ended up getting like 30 of these charts, and they were really useful. So, what do you do when you're promoting things? You have to take the same measurements over and over and over again. Especially when you're doing a lot of I.O., things get very unpredictable.  So, you have to run things many, many times to get some apparatus, do some statistics to get, you know, to get meaningful results. Otherwise, one time you get a measurement of 4 seconds, the next time you get a measurement of 7 seconds and it doesn't make any sense. So, you have to take some statistics across a set of results. So, you actually have to take the time to make it automatic because if it takes two minutes of typing things by hand to get one single set of measurements you're going to do it twice then you're going to be bored and go, you know, look at the poor part. So, you have to make it automatic. If it takes you five hours to write a little program to get those benchmarks, it's going to be much better in the end. Things you measure. You want to measure tasks. You want to measure fine-grained, well-defined tasks. A lot of trouble goes and a lot of GUI programs have the same problem because they are very, very asynchronous. It's not like, you want to measure the time from here to here in the source code because that source code will set up idle handlers and those idle handlers will set up time-outs and those time-outs will launch threads and the threads will later notify other threads when stuff happens and, you know, you never have any idea of when that stuff finishes. So, naturally, stuff like load the file list in an idle handler then also asynchronous and it repaints the whole screen on its idle so you never know when that shape is finished. So, what do you do?  For example, in your remake function there, you could check your pending thing of your pending list of things to do to say, do I have any other file spending loading their icons? Do I have any other icon spending scaling? Do I have any whatever? So, I'm not going to meet the checkpoint until all those conditions are met. If you have never profiled an application, if you have just been adding code to it all the time, it's very likely that the problem is that your application is doing too much work. For example, you know, G-Edit, the simple nice text editor program it had this problem where G-Edit had this problem where you would use the search and replace function and it was really, really slow. If you loaded a document with 20 pages and you hit search only, but it would take a few seconds to go through it and it's really slow. I mean, it's just scanning text. There's a problem with that. When we run the profiler, we saw the following thing. We saw that G-Edit lays it was doing this guy, search next was doing, okay, search for the text was changing the sensitivity of the menu item that you used to search for the next item so that G-Edit could show you whether there were any more items to search for by disabling the menu item, you know. And of course, change widget, flip the page in one of the widgets and then it will repaint. Every single occurrence of the text that you were searching for, it was causing a full repaint of all the menu items. To make the search function return whenever there are actually any other items you collect that information here and only re-variate when you update the widgets. Do you understand that? Yes, no, I said way too much work for a simple equation. If you have never profiled a program the next thing you'll notice in your application, it's probably that it's doing too much IO. Once you have removed all the excess work, once you have gotten rid of all the crap, then you'll actually start to see real problems, like too much IO then bad algorithms then smaller stuff like that have high sweet memory access only trashless cache. And until the end you get to see really tiny things like, oh, you know you should be using better instructions or other compilations, because that never matters in the beginning because it's way too much stuff anyway. Some examples of bugs we have found if anyone wants to fix those, feel free you have to characterize the problem you have to say, okay, my problem is that this pop-up menu takes half a second to appear. From time I click the mouse and it appears on the screen, it takes half a second and that's way too much. Then you make an hypothesis, you say okay, well you make your hypothesis by asking the provider not by intuition, you know, because we are not so you ask the provider and you say oh, okay, the problem is that to create this menu I have to load these 10 icons. Your hypothesis is if I didn't load these icons it would be you know, 10 times faster.  If it's really faster then you found the problem, you just have to fix it. You have to change one thing at a time. That's the control panel of the Mark 1 computer. The first computer is in University. So you have to to restrain yourself from changing many little things at a time. Oh, I'm gonna remove this code and move this one while I'm optimizing. Now we have to change one thing at a time. Then you confirm it, fix it, fix it. Did you reach your goal? Are you done optimizing it or not? So why do our disasters happen? Why is our performance so horrible back? So programming is like building a sand castle.  That never happens in reality. In reality, people build bridges by putting two big pile of columns on the ground and then they build the bridge. So programming makes it very easy to fix things out of thin air. So we keep adding and adding and adding code to our programs. Then the sea comes along and what happens to the sand castle? So how do we fix that? Well, we keep adding sand. That's not gonna work.  So we have to do more engineering instead of finding the top of things. We actually have to go measure things, establish our performance goals, fix things to reach those goals, and then have an infrastructure to make sure that those goals are made for pieces. We have to have the benchmarks. We have to run them frequently. And whenever they start going more slowly than we intended we know that there's a problem. During the last wave, we discovered that someone changed the caching scheme in Fountainfield to cache the rendered cliffs. And before it used to be that most languages were, you know, about equally fast or equally slow. But after they changed the caching scheme everything is okay except Japanese, Chinese, and Korean. Those were 300 times slower. Why did nobody notice? Well, because nobody has run the profiles. I'm sure someone in China was really happy about that. So, for example, in 1997 our scheme to load icons was really simple. Our load icon function took an icon name. We built a file name just by preventing, you know, user-share icons to known whatever, name.png load.emash. Done. It was really, really simple.  So now our icon loading thing looks like this. File names can have different extensions. And we can have many, many paths for the icons for each possible extension and each possible path we look for it. If we look, if we finally found it, then we say, oh, is it an SVG? Okay. Then load the SVG, scale it, and carry it just so fast for that. So, the SVG afterwards we load the big map. After loading the big map we scale it. Then we actually take the scale. We match and add some cute shading to it to match the theme.  This is my GNOME icon path on the machine. This comes from the distro. The distro wants to be able to load icons from user-share images. Blah, blah, blah KD Blah, blah, blah. KD is not even installed on my machine. I don't know where the hell those things came from. Finally, the GNOME one starts and every time my machine has to look for an icon, it has to go through all those paths.  So, why do disasters happen? People just start writing a program and call out beautiful APIs. And to people, an API is like a flower. Every little flower in a pot. People see that flower and they see load icon.  That flower pot actually says load icon. People don't know what's going on under that load icon function. If we look closer, if we look under the earth there, you know what the hell is going on inside our pretty functions that we give them to use.  They don't know that the monster wants to scan a million paths for a million extensions. And it's going to load SVG files and render them and scan them and shape them. You know, people just don't know that. What if our documentation said things like this? This is the function we used to take the hash we used to compute the hash code for a string. It's really fast. It just goes through each character in the string once. What if we said, okay, this function is really fast. It's running time is proportional to the length of the string. And you know it will run really quickly. However, the performance of this icon is completely unpredictable because it could go to a thousand paths with daily images, get a helper program to get the images for you. You know, that would help because if people write the documentation oh, that's a problem. What if we made these annotations machine-readable?  So this is like the header files installed in your system.  This is your paper icon which name, you compute the idea as a draw. Then you load the icon, then you paint the machine. If we had machine-readable annotations, then your friendly text editor could highlight that code and say, okay, you wanted to write a fast function here, but that's not going to work because we're calling a really slow function here. This is not my idea. This is an idea from one of the performance experts at Microsoft. They are actually adding these kind of annotations to older .NET libraries and they are going to make Visual Studio do that kind of stuff. So we have to catch up again. That would be a really cute hack. Well, Michael Migs did ask to Martin yesterday. That would be really lovely. We need a way to know, the question was what about tools to detect things like page faults? Which you cannot really control. I mean, you can control when you load a file but you cannot control when your program doesn't page fault and the kernel has to go to this tool to fetch your data. I'd love to have those kinds of tools. For example, we have the problem where you're using your machine for a while then Mozilla goes and eats all your memory and everything else gets paced out but what happened there? Why is my program slow again? Well, it's possible, but anyway. So, yeah, we need a way to know this is the code that got paced out. Why is the... Just look how I was running OpenOffice right now and I bet you that accessing my menu here is going to be really slow even though I just used it before I suspended my machine. I just clicked on it. It took, what, like a second? The next time, it's really fast. What happened? It got paced out. But there's no way to know that currently. So we need hooks in the kernel to be able to tell us these are the VMAs that got paced out those VMAs correspond to these libraries. Then once you know the address within those libraries you can see oh, okay, GTK paint menu that got paced out or it's all the icons you had in memory that got paced out. Yeah, we'd love to have those tools. Yes? How to tackle the problem of people not being interested in fixing the wrong stuff? No.  So it's a lot of fun but it's a kind of fun that takes you know it takes some getting used to because the tools we have are really bad and when you don't have the right tools it's just too cumbersome to get started to have access system calls and then get your logs and do it by hand, that's a pain in the butt. So we just someone needs to write better tools and someone really needs to know what kinds of tools we need to know fixing them. Maybe we don't complain enough when speed disasters happen. Maybe we just assume something has happened but nobody does. However, I'm pretty sure people will fix things if they corrupted their data. Maybe they don't care as much as speed.  In the moment it took a while to get used to the idea that we have to make things usable that people at some micro-system find on that same idea for a couple of years for everyone else so maybe we just have to beat them in the head for a couple of years before they start going on to make some things. A reason. We really think such a meta-data principle is the way to go, the last thing we showed. Because always somebody can't completely access the speeds because you need to have a lot of correct information everywhere otherwise the system breaks down. If you have a large group and some of the people are unable I think it would be better spent, for instance in defining the thing with access automatically placing synchronization points logging points to be able to profile well I mean this is a preventive measure not a corrective measure so if this could keep up from doing some disasters that would be good yes oh with that you are totally screwed with that you are totally screwed I think one pattern we need to start using more is you know often you kill things in idle handler so put them in an amazing operator whatever but there is no easy way to know when everything is done for example in GTK a lot of the time we have GTK do something and it does it in an idle handler but there is no object signal limit on that thing that actually finishes so if we look for those places and start adding I don't know, geogic signals like operation finish in some parts of the file that would help because then you could hook your log into that to those this one would be better sure that would be good yes in general you have to find those places and add some notification back then yes in the end you would need a thread and the way the thread notifies you that it's done is by you know some synchronization which would happen in the idle all that actually brings up a good point I'm really sure that they were writing every single time they run into a performance problem they said, oh, let's create another thread and do the impression there that's just like showing the trash you know that work needs to happen regardless of when you do it but they just created another thread put everything there and it will the user will not notice they actually do, I mean the stupid window takes 5 seconds to open this is how long now it takes to open a window I'm going to open my folder here click, click may not come under it's that dangerous it only does a read there every file to print them to the screen it doesn't try to load icons it doesn't try to compute fancy fonts to the fonts in the screen I mean it's much simpler it takes place application but if you run Windows Explorer it's a lot faster than any of the crap we have right now they have actually done their homework performance problem that is putting the versions in a thread anytime it's just going to be synchronous and your GUI is not going to respond while you're doing this it's a lot easier to measure when it's not when it's not asynchronous but often it's hard to change because the rest of the program assumes that so you actually have to take your time yes did you try to oprofile oprofile to kernel compiler for the kernel base compiler did you try to use it? I tried using oprofile and I found it ridiculously hard to use one thing about profiling is that you want to get started really quickly so the thing about usability also applies here you know we actually need to do some usability work in our development tools we don't know when to use oprofile well but perhaps it's too hard to figure out and all the kernel people really have well I don't know where a bunch of kernel people really have to use syspro because you run it you can start, you run the program you can stop and boom you can't use all the results immediately and oprofiling is like well that's a complication of the file now that didn't work let me look for another option it's really powerful actually I've collected a lot of statistics from the system it's really powerful but it's really hard to keep going yes did you try cold heat? the software in my machine looking for performance problems just some of them


Angela says, how long is too long for a guy to make a decision and move ahead? It depends on what kind of decision you're talking about. Are you talking about like a decision about being in a relationship with you? This is something I've talked about before. I'm just going to assume it's that since I don't see any other kind of message about what you're saying here, Angela. If you're talking about a guy making a decision about being in a relationship and moving things forward in a relationship, the longest that you should be in a relationship without knowing where it's going, the longest, the absolute longest is six months. After the six month period, if you've been with a guy and he tells you he doesn't know or he doesn't, he's confused or whatever, you should walk away. That is the longest period of time that you should be in a situation where you don't know where the relationship's going. After six months, he knows. He knows where it's going. I hope that's what your question was. I hope that's what your question was. That's what I read it as.


Well, hi there. I'm Sandy Alnok, and today I've got some more coloring in the Human Rainbow with a stamp by Mo Manning. When this blog hop came up and I was invited, I went to Mo Manning's website to pick up some digis. She does really great people stamps, and when I get in the mood to color people stamps, I often will turn to hers. They're much more realistic than you get from a lot of other companies, and I just love the freedom that she can draw with. She's such a great illustrator. When I saw this stamp, it had two great people of different ethnicities happy together. I thought that is what I want to see. I want to see us being happy together and not caring what someone else's ethnicity is, what their skin tone looks like. It doesn't matter other than it's beautiful. I love coloring all different kinds of skin tones. You might know I do the Human Rainbow. I have color charts available. There's links to all that for free in the doobly-doo down below. I have always been one of those people that's drawn to diversity. Maybe it's my love of color, and I love looking at different skin tones and picking out what colors would make up that person's skin, what makes them unique. That could be part of it. I don't really know, but I know that we need to get past the racism that has been rampant in America, and I really want to be part of that solution. I have been trying for years to be part of that solution and have some of those difficult conversations with my black friends. I have this one friend who had to come over my house at one point for something. I won't get into the long story of what that was, but when he came to the door, he said, no, I can't come in. Can we talk outside? Well, we sat on the front porch and had a conversation, and I remember it being really stunning to me that he said he couldn't come in my house. And I said, why? And he said, because I would need to ask permission of your man before I go in. And I said, what do you mean? He said, I'm a black man.  The thought that my friend can't come in my house because we might have, I might have a neighbor who gets scared and calls the police over something, and it could mean his life, and he knows that. I don't have to live with that experience. And it was really shocking to me at the time, and I've tried to make the effort, and I haven't been very consistent with it, but I've tried to make the effort of having some of those difficult conversations with my black friends of late, especially when it comes to what they think of what's going on in society around us right now. What do they think of the protests? What do they think of the solutions that are being offered, and the reforms that are being offered? Because I can certainly have my opinion as a white person, but I don't know what that experience is for someone who's not white, and I need to learn to listen, and I'm not really good at listening sometimes, so I'm working at that, and I hope that everyone is, because while it's great as an artist and as crafters to make cards that show diversity and that celebrate that, the thing that's going to fix all of this is not the art that we make. The art that we make is going to help to convey that message, but the art is not going to change our hearts or anyone else's. It might inspire some conversations, but it's those conversations where I find the biggest growth is happening in me, and no matter how much art I do around the topic, I need to be in that uncomfortable place to ask those uncomfortable questions, to get those uncomfortable answers, and try to find out how I can change my behavior and my treatment of other people so that I am not part of the problem. And I know there's a lot that I can't do anything about because I can't change the fact that I'm a white woman, but I can change some of the things that I do and the things that I say and the way that I treat people in order to be part of that solution. And that's my prayer for all of us, that we can become part of the answer and not to further the problem. Because we have enough people who are deliberately trying to do that, and we don't need to accidentally do it. That's what I'm always afraid of is that I'm going to do or say something that I don't know what that means. And I think that's a big part of it for me has been learning that the language that I use is not always received the way that I say it. And the language that someone else is used is not received by me the way that someone else means it as well. We need to learn how to ask people what they really think about something, what they really care about, and what how something really makes them feel as opposed to just assuming that they think the same way that we do. And that goes for all of us on all sides of all issues, not just race, but all sorts of things where we have misunderstandings. So there is a rather large group of people who are blog hopping with us today. There will be information in the doobly-doo as well as over on my blog about all of that. If you'd like to see some more people creating on this topic, and maybe you'll hear some experiences from some of them that might open your eyes and your thoughts in different ways and hear about some other people's growth, that inspires me to see other people taking part in things like this as well and gets me excited to have hope for the future that we can be different. We can do things differently. We can resolve our issues. So there is my little coloring for today with of course some white pen out of at the end because nothing is a sandy piece of art if it doesn't have some white pen in it. And that is about it for me for today. I hope you enjoyed this. If you did, please click the like button and I will see you again very soon, as always, because I'm always putting out more videos. And if you haven't yet subscribed, please do that so I can send you some more creative inspiration. Take care. Have a great day. Bye-bye.


Coffee is a billion dollar business. It is super competitive and saturated yet It is super rewarding at the same time, which is the reason why if you want to be successful You must be unique in this video I'm going to share with you five unique coffee shop business ideas from around the world so you can draw inspiration and Creativity to build out your own coffee shop.  It is very very competitive So for you to succeed you need to go one step above everyone else Above the white wall minimalistic coffee shop. That's always around the cities or the hipster wooden frames With some plants. So to really really stand out I'm sharing with you inspiration from Asia where competition. It is over the top. It is furious Now I'm not only gonna be sharing with you the five unique coffee shop business ideas I'm also gonna share with you why I think it works for you So then that way you can take these concept ideas and apply it into your own business now Let's dive right in first unique coffee shop business is Main Street cafe a massive New York City themed cafe, which has several stories in Korea They have several floors which has its own unique themed rooms That is perfect for photo opportunities on their main floor They have a wall mural that shows you like that you are now in Times Square They have Lego room for kids and even old-school New York City trolley buses Grungy graffiti rooms to Coca-Cola themed rooms Even the exterior is painted with mall murals of the Empire State Building and the iconic yellow cabs And to really pull the whole theme together the food they offer are these massive sugary milkshakes that are in chocolate and strawberry flavored American cheeseburger quesadilla barbeque chicken wings and the swords They're so popular that even K-pop stars were filming their music videos and their drama shows in this coffee shop So now the breakdown of why I think this works It is because a lot of Asian they fantasize about the New York lifestyle the American dream And which is the reason why they would want to visit this store because they feel like that wow I'm there without actually having to visit New York And this shop has been super intentional by creating multiple themed rooms these photo opportunities for their guests to come in and take Creative photos and videos of their store because of this they have gone viral because of the UGC User-generated content when people see that there is such a cute little place for people to take photos What did they automatically do all their friends and family start seeing it starts coming in as well And this acts as a form of marketing free advertising for the Main Street cafe This is also the secret of how Main Street cafe is able to rank all the way to the top to Korea's search engine neighbor Just imagine you being at the top of Google the second unique coffee shop business is touch 24 cafe Now this is a concept that I've been noticing quite a bit Recently these unmanned cafes where there's no staff no baristas in the shop at all You select your drink you pay and you get your drink and you can customize everything yourself pretty insane That's what touch 24 cafe is and they are operating 24 hours All you have to do walk in order your drink from this massive screen just like McDonald's and Automatically your drink pops up It is like a giant modern-day vending machine where you order your drink and you get it and you also are able to order food From this shop as well Now the only caveat is that this type of location is not going to work in every single city out there If you understand the Japanese and South Asia culture, you understand how safe it is Sometimes you can leave your laptop out in the open for hours and people still won't take it And that's the reason why it works in Japan it works in Korea now This is not to say that this concept won't work in North America You just need to understand the risks because I've seen certain states that they do have these unmanned convenience doors Now why does this work well? Not everyone is looking to spend hours in a coffee shop where it's super aesthetics being able to have these relaxing music No, some people just want to go in a coffee shop grab They want what they want and be able to be on their way now the beauty about this whole concept Is that you don't need to account for much Operational labor all you really need to account for are the ingredient costs the cost of good sold and your rent in addition to that you can also Uplift your average order value by having different types of food in this location Now once again if you're really considering this type of concept You must understand the neighborhood that you are building this type of concept for because You need to account for the theft and the vandalism as well The third unique coffee shop business idea is the space filled with you This is a cafe that is in South Korea now The special thing about this cafe is that not only are you able to enjoy a really good cup of coffee with aesthetics around you But also you get to write a letter to your future self something like a time capsule So why I love this coffee shop and also this whole gimmick or just this whole Experience as a whole so so much is because not only are you able to escape for a second in This current moment and to think about and ponder on your life escape from the real world for a second and to write something for your Future self, but this is a gift that you get to receive a year from today How brilliant is this idea how this works is that you go to the coffee shop and you buy this letter set for around $3 and if you want to add it a wax seal on top of it you add an additional dollar After you write your letter you put it into a slot This slot indicates a future date that the cafe owners will go and mail the letter back to you And this whole experience you get to do yourself from the moment that you are writing the letter to the moment that you Are burning the wax and putting it on your letter and actually sealing it this entire experience Goes along with this coffee shop and the reason why I think this concept is so great and so Lovable is because Everyone loves themselves. They love talking about themselves And the second thing is that in the food business it is all about their experience and this food concept has them both talking about yourself and Experience that you can really cherish. Yes, I do got to admit that this is more of a gimmick This is something that you do once and most likely you're not going to come back to write another letter for another year However, this whole experience is definitely going to bring more word of mouth and get more people to come to your shop This would be a great concept if you're at a tourist location where people come And you have a lot of traffic and you're able to actually attract part of this big funnel of people going through your town Because everyone wants to be able to reminisce What an experience that they have in this tourist city and you guys want to know what would make this even a better idea It is for you to smash the like button so then that way we can show the YouTube algorithm That this is the type of video that you enjoy shows us that this is the type of content We should make more of for you It's going to help us a lot and it's going to mean a lot to us So go ahead and smash the like button. The fourth unique coffee shop example is coffee doses Now I know that I talked about how this video is all about the unique coffee shop ideas Ideas that are aspirational that you can bring the whole family together on but coffee dose is a little bit different They are much edgier even their tagline tells you this is not your grandma's coffee shop Because their brand really pushes the envelope really pushes the boundary with swear words Offensive words, but something that has a lot of character. Let's look at their branding anti Serum or morning stickers on their coffee cups their drinks and their food items are named Really to give the identity of the bad girl like Phil's side chick or the gateway drink These are all different items to give an identity to a very particular type of clientele They went on to even be more creative with their sandwich board that says we survived a pandemic We will survive even with your shitty yelp review or Saying that no Karen you can't talk to a manager to top things over They even have collagen add-ons or CBD add-ons which are quite unique to their concept The reason why I think that this coffee shop really works is because they take a stand a firm stand of who they serve In today's age, it is so difficult to take a stand because we need to be political correct We need to always be serving everyone So when you see a brand willing to take so much risk willing to stick their neck up and show people who they are And what they really stand for you can't help But to resonate with that and to be able to support them if that's how you truly feel It's easy for people to go and check out their store because they appreciate their rawness They appreciate them being authentic to who they are which also makes for great PR aside from their edgy character interior designs They also make their drinks so then that way it is Instagram worthy once again is free marketing for this coffee shop And because they know exactly who they stand for and what their brand really appeals to they went on to collaborate with other Brands who have similar values such as nasty gal, which is an online Retail shop something that you can take away from this is that don't be afraid to take a stand of something that you believe in stand out by really firmly and Loudly telling the world what you stand for being authentic to that and really embrace that Being polarizing is what's going to set you apart from the crowd Fifth unique coffee shop idea is the peach gray cafe a very popular Watercoloring cafe in South Korea in this cafe not only are you going to enjoy cute drinks But also jiggly fluffy pancakes while you make your own artwork All orders comes with a set of watercolor paints paper and paintbrush for you to fulfill this entire Experience and the reason why this works is that not only are they able to provide a coffee experience But also an experience for couples and friends to get together Who doesn't like a paint night with their friends while they catch up while they talk about some gossip While they can have a full-on coffee experience at the same time The cafe is also very Instagram friendly with some very great drinks very cute food items And also the interior of it makes it Instagram worthy now as something that you should be really aware of If this is a concept you want to deliver an experience for your customer Turnover rate is a really really big deal because if your customers are taking your spot for hours on end Just buying a coffee just buying a sandwich or a pancake Then you must understand how much time that they spend and how much money you can make from this one person And if all of a sudden your whole place is packed with only five people every single hour That's not going to be enough for you to make your money And that's the reason why I highly suggest having time slots on slower times that allows for people to do that While the busy times you do not have this as an experience Or on the other hand if this is something that you truly believe in that is an experience for everyone Then what you can do is set a maximum time slot for your customers Or have them book online in advance so then that way you can pack your full day full of experiences just for your guests So there you go friends the five unique coffee shop experiences that you can model after to create your own dream coffee shop I really really hope you enjoyed this video and if you want more of this You review if you want to learn how to make a proper coffee shop business Then you must attend our free masterclass in the link below That's where I share a ton of information with you to share with you how you can go from an idea To build in an actual business to actually building a coffee shop everything from scratch So definitely check out the free masterclass in the link below Otherwise, I really hope you enjoyed this video if you guys enjoy it Smash the like button and I will see you guys in the next video


This study concluded that hydroxyapatite, HAP, is a more effective agent than fluoride in controlling dintin hypersensitivity and may be superior to other desensitizers. This article was authored by Hardy-Limeback, Wachiman-Axe, and Frederick Meyer.


We got to watch him. He ready to hit somebody. Oh my God. Got them bunnies. Let me see real quick, Coach. We got one more. I can do that. Yeah, all right, bad, bad, bad, bad, bad. Crazy work. We got them small bags.  I don't like the way he said yeah, bro. Oh, gotta go. Four bags? See, I'm feeling the fire in his rookie. I'm feeling the fire in him. Did you get the limit, the limit to? The line, listen, go to Walmart. They got everything. They're going to have to put me down with the best hair on the team though. I'm going to just say that now. Ain't nothing like that old school fro. I lift weights. Let me put this old jury is back in here. Remember that time you farted in the meeting? Oh, you cold. Oh, you cold. I do. You got that liquid kept on you? Man, chill out, bro. I need that, dude. You're mowing. Don't worry about it. Give me no action today. They told me I'm mic'd up. I can't do nothing. Hey, I should put my helmet back on so I can look like I'm doing something. Oh, yeah, if it ain't there, I got you. Just slaying that thing. Oh, yeah. Oh, yeah. Little action in there. Get in the mix with the big boys. See, I just try. I got to get in there and get me some kind of love. He for so bold, like. Hey, cold. Hey, P, you bold like it? I see. Look, don't try to hit the jog, too. Don't try to hit that bow leg. Jog, baby. He's going to stand over there and put our helmets on. Look at the camera. Did they set my intro? My intro picture. That thing look fine now.