Bi-factor models are an important tool for modeling scale dimensionality. Bi-factor models are also used commonly to address method variance issues, even though the term bi-factor models is not commonly used in that literature. One challenge with this kind of models is that their identification status is not clear. They are not always identified and they may be empirical under-identified. Unfortunately, as far as I know, there are no rules that you can apply to establish the identification of these models. So if you need to show that your model is identified, you must either prove it yourself or there are certain special cases that have been proven, but as a general rule there are no rules that you can apply. So that creates a challenge for an applied researcher. This challenge of identification has been acknowledged in the recent literature. This great article about Win and Young about empirical identification of this kind of models states that identification is one of the more challenging issues in empirical applications for fractal ecosystem modeling. It is challenging because identification is something that you need to establish by working through the math. So if there are no rules for identification, then you need to understand where the model implied covariances come from and then be able to solve from those model implied covariances to unique values of the parameter estimates. This is sometimes easier said than done. For the three indicator factor, this is simple to do, I showed that in another video for more complicated models, proving the identification or non-identification can get very tedious. In practice, people do identification checks empirically by trying to estimate the model and seeing what happens. I will talk about the identification through math in this video. One particular problem in identification of these models is that they may be also empirically under-identified. So it's possible that the model is identified generally, but for specific values of sample or population covariances, the solution to the parameter estimates does not exist. So the idea of identification or proving identification is that you work backwards from model implied covariances or sample covariances to the actual coefficients or parameters. And that gives you the parameter as a function of the covariances. Sometimes some of these covariances are zeros and you may end up with a solution where parameter value contains a division by zero. And that indicates that it's not identified because you can't divide by zero. And that is the problem of empirical under-authentication. A model can be identified for certain sets of covariances, but not every possible covariances. And that's the empirical under-authentication problem. So the by factor model was a model where you have minor factors. We have four minor factors here, each measured between five to three to five indicators. And then we have one general factor that loads on all indicators. And this model here is identified. I will not prove that. I will prove identification of another variant, the simpler variant of this model. But one important thing to understand about this model is the special case where we allow these minor factors to be correlated. So would the model be identified if these minor factors are correlated? Typically in a by factor model in a textbook example these minor factors are said to uncorrelated, but there is an important application of by factor models where these correlations must be freed. And to understand the problems of that application you need to understand first the identification of this by factor model, where the minor factors are constrained to be uncorrelated. The important application is method variance models. Briefly, the idea of method variance is that the variations in your model and variations in your measures are due to the measurement method as well as the constructs of interest. This is a concerned particle and cross-sectional surveys where a skeptic of your study could argue that the correlations between x-indicators and y-indicators is not driven by correlation between x-construct and y-construct, but it is driven by how people tend to respond on the left-hand side of the scale versus the right-hand side of the scale, given that the x-indicators and y-indicators are measured using the same scale format from using the same informant at the same case. So this is the method variance argument. And method variance issues are commonly diagnosed and to some extent addressed, or at least we try to diagnose and try to address the problem by using this kind of models. So this is a by factor model where we have one general factor that contains three unique indicators, and then we have two minor factors that are correlated. This happens to be identified, and I'll show the identification later, but then we have this unmeasured method factor. So this is a very common approach. Typically researchers who have not done much surveys, they do a survey study and then they send a survey study to review, and then the reviewer says that by the way you may have a method variance problem in your data, please address it. And the way that people address the problem is using this kind of model. So you have one factor that explains all the indicators. This is a bit of a problematic model. I'll explain this model in another video. But to generally understand one of the problems is that this is basically a by factor model where you allow all the minor factors to be correlated. If you were to constrain these minor factors to be uncorrelated, then you couldn't be testing your theory because your theory is that these constructs that the minor factors represent are correlated and that's what you want to estimate and test. Articles that talk about this modeling approach typically mention that identification of these problems is challenging. So this is in Pozhekov's paper. They say that there may be potential identification problems. Unfortunately, these same articles don't really go and explain the identification problem in detail. They just state that the model may not be identified without explaining why it is not identified and under which conditions it would be identified and would not. This is of course important to know because if your model is not identified, then your estimation will not produce any meaningful results. So in Pozhekov's paper they tell that, well, there is a disadvantage of identification that concerns the number of indicators. It is related to the number of indicators, but the number of indicators itself is not the reason for non-identification. So it's not that you would be running out of decrease of freedom. The identification problem with this kind of models is something a bit different. And then Pozhekov points out that people are solved this problem by fixing some of the factor loadings to be the same. That will identify the model, but it also produces a mis-specified model unless the factor loadings are known to be once in the populace. And of course estimating a mis-specified model just to get the unique estimates is a bad modeling strategy. If you estimate a mis-specified model, particularly in this case, the estimates can be really, really misleading. So let's take a look at why these by factor models are not identified or are identified and under which conditions. So we'll start a simple model, so we have two factors with three indicators, each and one general factor. So we set the scale by fixing the variances instead of fixing the first indicators because that simplifies the math a bit because now all the loadings are symmetric in the sense that the equation for solving the first loading is the same as the equation for solving the second loading. Instead of having to treat the first indicator as a special case because it's constrained to be one. So this gives a bit of symmetry to our analysis, but it would not be required to have this scale setting. You could easily, just as well, fix the first loadings and prove the identification status of this model that the equations just would not be as nice to work with. So in this particle scenario. So we have 21 simple covariances and we have 18 estimated parameters which gives us three degrees of freedom. So this model is, if it's identified, it is over identified. So we know that we have some constraints in the data but positive or decrease of freedom does not guarantee identification case. So we ask, is this identified? How do we know? Well, for identification we know that the the latent variables must have scales which we do and then we know that the three indicator factor is identified but that rule doesn't really apply to the by factor model. It applies to a set of indicators that load only on one factor. So how do we then know if this is identified or not? Well, we have to work through the covariances. So that's all model-implied covariances for this model and we can see here that we have actually three groups of covariances. We have these variances for indicators. Then we have these covariances for one minor factor covariances for between minor factors. We don't actually need to use this first set of variances because that is only used to solve the error variances. So if we can solve the loadings, then we can quite easily solve the error variances. We just put the loadings there and then we arrange them to the left-hand side and that gives us the error variances. These equations are not useful for solving the error variances of the factor loadings either because we can't know the error variances. So they are uninformative of the factor loadings. So we'll just be looking at this subset of the equations to simplify the problem with. And how do we then show if the factor loadings are identified or not? Well, we can just start solving this set of equations and we solve by substitution. So this is high school math. It gets a bit tedious because the number of equations is very large, but it's doable. So let's just do it. So we'll start solving. We'll take the first equation here and this is just a convenient solution to start with because this is a simple equation. It has only two terms compared to these other equations that have more terms. So we start solving using the simple equations. So we'll solve lambda g1 and lambda g1 is sigma 104 divided by lambda g4 and then we substitute this solved value for lambda g1 to the equations and now we have a set of equations where lambda g1 is eliminated and we have one less equation. So we eliminate the equations and eliminate the unknowns until we have just one equation with one unknown and then we solve that one unknown and then we start to plug the values of the solved parameters back to the models until we have a solution for every parameter. So that's lambda g1 solved. We take then lambda g2, we solve it and we substitute back. Then we take lambda g3, we solve it, we substitute and so this is, we have now eliminated the loadings of the first minor factor m1 here lambda, the loadings of the general factor on the first three indicators. We can then continue with lambda g4, so we solve lambda g5, we solve lambda g5, we substitute back and this starts to get a bit tedious but we can solve for lambda g6 now and we substitute that and now we have this set of equations here that only contain lambda g4 and four equations with one unknown. So that is over identified, if it's identified. Let's simplify these equations a bit and after simplification we can see that actually the lambda g4 disappears. The reason why it disappears is to be basically we have something multiplied by lambda g4 multiplied by something divided by lambda g4 and when you multiply and divide with the same value then that value disappears. So now what? We have actually these are constraints so if this model is correct for the population then these constraints should hold in the model so these are over identification constraints. Now we have a problem because we have four constraints but decrease of freedom is three. So if decrease of freedom is three and the model is identified then we know that there are three constraints but now we have four. So how is it possible that we have four constraints but only three decrease of freedom? It's possible because we actually have here six equations with seven unknowns. So these constraints will not give us any information about the unknowns so we have six equations and seven unknowns that cannot be solved which means that this model is not identified and we just prove the non-identification of this model. So this is the way that you can prove non-identification status, prove the identification status or non-identification status of your model and quite often if you have a hunch that the model may not be identified then proving that one part of the model is not identified is sufficient to prove that the model as a whole is not identified. Some parameters can be identified but typically working with these partially identified models is something that I would not recommend that you do because that requires lots of expertise to understand the results after that and you will get warnings from your software and to understand whether those warnings apply to the part that you know to be under identified or the part that you think is identified is challenging to do. So what will happen if we try to estimate this model using SCM software? So I'm using R and I have the hosting and swine for dataset that comes with law on software and I'm estimating this model. I have from the example model visual and text of three indicator factors and then we had a general factor we can see that the software gives us a warning that this is potentially not identified we get missing standard errors which is a symptom of non-identification and we know that these estimates should not be trusted. So what do we do after that? So if you want to estimate this kind of model but you don't get the standard errors and you get the warning what do you do? Well you have to first understand the identification problem that I just explained and then make informed decisions on whether you want to fix some of the loadings or whether you want to for example try adding more variables to the model which is not always doable. Let's try to understand why this model is not identified. So this is the sample covariance or the model implied covariance matrix for the model. So we have the parameters here and they form the implied covariance from X1 to X6 and what information do these between factor, between block correlations give us? So we can see from these X1 correlations that they give us information about the relative magnitudes of lambda G4, lambda G5 and lambda G6. So if we know that this X1, X4 correlation is twice as large as X1, X6 correlation then we know that lambda G4 is twice as large as lambda G6. And the same works for these X4 correlations. So the correlation between X4, X1, X2 and X3 give us the relative magnitudes of lambda G1, lambda G2 and lambda G3. These four covariances are actually redundant. So we could just as well shift this instead of looking at X1 we could be looking at these correlations between X2 and other variables or X3 and other variables. So these are redundant if we know the proportionality between lambda G1, G2 and G3 and the proportionality of lambda G4, G5 and G6 then these are just four testable constraints. So the problem here is that the relative magnitude of the sets lambda G1, G2 and G3 and the set lambda G4, G5 and G6 is not known. So we know the relative magnitudes of lambda G1 and G2 and G3 but we don't know which one is larger lambda G1 or lambda G4. So it's possible that these G1, G2 and G3 loadings are actually very large and these loadings for lambda G4, G5 and G6 are actually very small or vice versa. That's not we cannot empirically determine which set of these loadings is larger. So the correlations if X3 and X4 are only weakly correlated it's possible that X3 is actually highly loads highly on the g-factor and X4 loads only weakly on the g-factor or it's possible that X3 loads weakly on the g-factor but X3 loads weakly on the g-factor but X4 loads strongly. So we don't know which one it is and that's the identification problem. So these will not help for identifying whether lambda G1 or lambda G4 is larger. These correlations within the minor factor one indicators, minor factor two indicators don't help because we have 12 covariances and 12 parameters for these loadings. So this three indicator factor has six parameters so we have three loadings, three error variances we have two factors, 12 parameters and we already consume that 12 units of information by estimating the unknowns for this factor. So we don't have any excess information that we could be using for estimating the g-factor loadings. Let's take a look at a bit more complicated model and see if that is identified and why. So what if we have an eight indicator model? So we have the first minor factor, four indicators the second minor factor and another four indicators and general factor loading on all the indicators. Is this identified? Well we need to first establish a scale we again fix the variances we have 36 sample covariances 22 estimated parameters gives us 14 degrees of freedom. So if this is identified it is over identified but is it identified or not? In this particular case establishing the identification of the status of the model it's useful if we take a look at the subset of the model. So sometimes this is a large model so we have 36 sample covariances trying to type all those sample covariances on a slide would fill the slide with equations but we don't have to do that we can just take a smaller part and establish the identification status of that part. So let's take a look at the first five indicators we are ignoring information on these indicators and these indicators provide us information about the M2 loadings and M2 variants but we are not interested in improving that we just want to prove if the loadings for the minor factor one are identified if those loadings are identified then the loadings of the second minor factor are identified because of symmetry and if those factor loadings are identified then the general factor loadings are identified and so on. So if we start, if we can prove the identification of some part of the model then that typically helps in improving the identification of another part of the model. So we are just looking at these and why we can eliminate these indicators from the model is that these indicators don't have any observed consequences in the model. So we have all the causes of X1, X4, X3 and X2 and X1 included in the model and that is enough for us to prove identification. We cannot prove the identification of these variants so this is a unique source of variance for X1 now and X1 has also E5 unique source of variance so the variance is not explained by the g-factor in X5 we don't know based on these indicators whether that is due to the minor factor two or due to the error variance here but we don't need to. So we just look at these identification of these loadings. So let's take a look at the identification of this model. If we only work with the first four indicators so that's our model-implant covariances for the first four indicators this would not be identified. The reason for not identification is that this is basically an exploratory factor analysis model so we have two factors that both explain all the indicators and from our reading of exploratory factor analysis models we know that those models are not identified but we must establish a rotation criterion for identifying the model. So this is like an exploratory factor analysis with a rotation criterion and that is not identified. If we add the X5 indicator then we add four more covariance to the model. Instead of proving the identification which would be doable from these equations it's easier to consider the logic of why this is identified and the reason why it's identified is that adding this indicator X5 establishes a rotation criterion for this factor analysis model and the rotation criterion is that the loading of M1 on X1 must be zero so it's uncorrelated and that establishes the rotation criterion which is that M1 and X1 are uncorrelated and that establishes the rotation and identifies the model. This argument can be understood better if you understand how exploratory factor analysis can be conducted using confirmatory factor analysis framework and for example this web page on Stata's website explains you how you can run that explains the principles and shows an example using Stata so if you use that software this is a useful exercise to do. A model with three or more just identified minor factors would also be identified. I will not prove that but it basically follows from the identification of three indicator factors. So this is the problem in identification of this model this G factor is basically the same as the problem of identification for a two indicator factor. You can know whether the average loading or the product of three loadings for two indicator factor but you don't know which one is higher and that's the case with here in the G factor loadings if we have just three indicators but if we have a minor factor three or more then that would be identified because of the identification rules of three factor models three indicator factor models. Okay so that's the identification generally without any consideration for empirical identification. Let's take a look at another example. So this is the model that we just worked with so we have eight indicators, two minor factors, one general factor we just proved that this is identified and this is a model discussed by Green and Yang and they discussed the concept of empirical underline so their article has this kind of example they have a factor loading matrix here so all indicators load on a general factor and each indicator loads on a minor factor and one minor factor only. All the loadings are equal and all the factor loadings are equal. What will happen now is that we have just three unique equations with four unknowns so how is that possible? We have like this quite a few covariances here but only three equations. How is it possible? The reason why that happens is that these are actually the same so all the variances are the same because these factor loadings and error covariances are constrained to be the same and well these are the same. All these between minor factor correlations are the same and all these between minor factor correlations are the same. So we have just three equations but four unknowns we can't solve four things from three things which makes this not identified. Particularly it is empirical under identified. So empirical under identification are in a factor model context if you have a two indicator factor empirical under identification occurs if that two indicator factor is uncorrelated with any other coefficients, any other factors. In a three indicator factor if two indicators are uncorrelated then that leads to an empirical under identification problem. But this example shows that the empirical under identification can occur also for models where some of the covariances are non-zero. So in normal factor models zero covariances are problematic and by factor models also non-zero covariances can be problematic. So the fact that all your indicators are highly correlated does not mean that your model is empirically identified if it's generally identified. So what's the identification rule here? Identification requires that loadings differ and this is a bit of a problem because quite often when we design for example survey items we want to design the items to be interchangeable. So each item should perform roughly equally well and if that is the case then we run into this identification problem. And so what do we do about this problem? Well, there are some things but one thing to understand about the problem generally is that this empirical under identification can occur also if you constrain the loadings to be equal. So even if these loadings are different in the population and they will be different in the sample so that you don't have this kind of uniform covariance pattern if you constrain the loadings to be the same then the model would be empirically under identified which is kind of counterintuitive because normally when we constrain things in the model then the model becomes more strongly identified but in this case if we constrain the model then it can actually become empirically under identified and understanding why this under identification occurs is critically important if you work with this kind of models and particularly if you follow there are recommendations of fixing all factor loadings to be equal for the method factor if you use these method factor designs because that can actually lead to problems that you didn't have before. So what are the solutions to this identification issue? There is a recent article by aid and co-authors in psychological methods and they talk about different strategies for identifying this model and particularly they focus on two identification techniques one is that we have indicators of general factor that are not associated with any minor factor so we can see that these three indicators here that are only for the general factor they are sufficient to identify the general factor because of the three indicator factor rule once the general factor is identified then the remaining identification of the reminder of the model is very easy to show. Another way of doing this identification is to have at least one indicator so that's another way one indicator that is constrained to be uncorrelated with all other factors and that establishes a correlation criterion for the factor model and that's another solution. Whether these are applicable to your scenario of course depends on your theory so is it theoretically feasible to say that there is a general factor that influences some indicators and there are no other influences on those indicators for example if you would have these three indicators would be measures of social desirability and then you have two scales that you suspect could be affected by social desirability bias then this kind of model would make sense so you have a social desirability measured by three indicators and then social desirability also affects these indicators so that would make sense also embedding the by factor model in a larger factor analysis model helps with identification but that's not always possible but if you work with minor factors and buy factor models as a diagnostic tool in a larger model context then typically that would identify the model because your general factor only applies to one scale and not every indicator in your analysis but if you have a general factor that you model as a cause for every possible indicator in your model then identification issues are surely a concern for you


How weird is it that when you say pride people immediately think of gay pride means gay? Yeah, how wild is that? Yeah, they did that. Yeah, they own like that's it's came flasks has brown pride tattooed on his chest Yeah, now people like what do you try to say?  Was that what you trying to say? You know it used to be you were proud right now. Now.  It's man. It definitely means why we're here pride month Queer month.  Yeah, you say pride month How insidious they snuck it in they slowly took over pride like they took over the rainbow. It's yeah rainbows are that is It's so funny to think that University of Hawaii used to have rainbows on their helmets can't have it anymore You just listened to a portion of Joe Rogan's podcast where he bemoaned queers who insidiously took over pride and Hijacked rainbows because this is definitely a very serious issue that serious people care about You know, I think this just goes to show you that rich people have too much time on their hands when you are So rich like Joe Rogan when you are a hundred millionaire You literally have to invent things to be mad about otherwise your life will just be too boring I guess but my favorite part about that clip was Tom Segura just sitting there uncomfortably nodding along and then trying to gently move the conversation into less Explicitly homophobic territory because I think that he knows like if he pushes back too much here Then he's not gonna be invited back on but also he doesn't want to appear homophobic So, you know defending gay people getting invited on Joe Rogan show It's hard to walk that fine line, but I do believe that Tom Segura straddled the fence Perfectly, but when it comes to Joe Rogan, I mean he's an asshole, right? And watching that tells me that he's just devolved further into being a Fox News grandpa But it's even worse than that because he even sounds like those whiny evangelicals who are part of the reclaimed the rainbow movement Who accused the LGBTQ mafia of misappropriating their religious symbol? But I mean at the end of the day, we all know that contrived outrage over the rainbow being stolen is just a more Respectable way of saying that they hate gays and they wish that we'd go back into the closet and the same thing is true for Joe Rogan Right, but we'll come back to that because first I want to discuss how angry he is that gays stick over the word pride Now you can argue that the word pride. It's become pretty synonymous with you know the fight for gay rights I think that that's fair sure, but there's also something known as context and That changes depending on how you use particular words, right? I feel like I shouldn't have to explain this to someone who is twice my age, but regardless we're gonna do it But first let's get back to basics So the definition of pride is a feeling of deep pleasure or satisfaction derived from one's own achievements or Consciousness of one's own dignity now missing from the definition is gay Now I feel like it's useless to even go to the definition because Joe Rogan is somebody who lacks common sense and basic reading comprehension for example He reacted to an MSNBC article about how the right and the red pill movement is fixated on health And he just responded with something to the effect of what the fuck being healthy as far right now So we're not dealing with the brightest bull, right? But the point is that pride is used in different contexts still.  I hope to realize this the problem is that He's irritated with gay visibility and he doesn't want to say it, right? And this is the only way he knows how to articulate that without sounding overly homophobic But I mean, it's it's a little bit too late.    It's like they can't win.   Yes, cis people have gender identities, too But if you're cis and straight the difference is that you're not shamed because of it, right? You're not shamed for just being who you are gay people and trans people However, they have been historically discriminated against and shamed because of who they love and who they are So pride isn't just about hey everyone look at me.   Yeah, they well.   I don't want to see like Yeah, tuck pants.  First of all calling a trans person confused is explicitly transphobic I mean gay people were called confused to right when I came out my family said that I was going through a phase So if you can extrapolate that and apply it to trans people It's the same argument same bigoted playbook and it's ironic because Joe Rogan is literally receiving gender-affirming care himself in the form of steroids See this is the double standard because when middle-aged men put let's say Rogan on their scalps They're not confused that gender-affirming care is perfectly fine But when trans people receive gender-affirming care when they start using different pronouns and different clothing well, they're confused It's a double standard and it's transphobia period. That's all it is and I think that what Bothers him is that you can tell that they're not confused trans people Especially Dylan Mulvaney who he's talking about seem pretty confident with who they are right? You just don't want them to be comfortable and happy. You don't like that it makes you unhappy But the main point is that he feels like all of this LGBTQ plus stuff is being shoved down his throat And he doesn't want to see it he doesn't want to see tuck-friendly swimsuits for example at the stores and Theo Vaughn his guest went along with it and every single Joe Rogan guest should feel deeply insulted by bad invite because Understand that he is bringing you on specifically because he knows that you're a beta cuck and you're never gonna disagree with anything That he says or push back too hard So if I got an invite from from Joe Rogan if I were Theo Vaughn, I take that as an insult It's the same as oh, are you calling me stupid because that's effectively what it is But I mean that clip it demonstrates what I mean when I say that Joe Rogan is hiding the ball He is homophobic. He is transphobic, but he doesn't want to say it when you're to the point where LGBTQ plus visibility triggers you and you have to say oh my god. I don't want to see it.    He hates queer people, but he doesn't want to say it, but Joe We know what you're thinking so just say it with your full chest.  We know that he's thinking it, but he just won't say it I mean I for one I'm so sick and tired of seeing Joe Rogan everywhere every time I lock him to Twitter There's another Joe Rogan clip every time I'm on Spotify.  You went mask off years ago But I mean I should be clear.   He's been on both sides of LGBTQ plus issues I mean he supported the Santas's don't say gay law and then months later He subsequently condemned Republicans for being against gay marriage He also talked about how Republicans should lead gay people alone, but yet you're supporting their crusade against gay people He also drank but light following the hysteria over Dylan Mulvaney, but then attacked Dylan Mulvaney because she's confused according to him I mean, it's this weasley way of giving yourself plausible deniability So that way when somebody accuses you of being homophobic or transphobic You can simply point to the times when you weren't as evidence that you're not But I mean there's really no both sides in these types of civil rights issues regardless if it's explicit or implicit or over its or implied bigotry is still bigotry and He can try to hide the ball and cloak his disdain for queer people in disingenuous language and disingenuous rants about target But at the end of the day, we all know where he stands He is complicit in spreading anti LGBTQ plus bigotry and any guess who just comes on that show and Nervously sits there while he's explicitly condemning the queer community throwing an entire community under a bus If you don't call it out, you're complicit and Tom Segura and the Elvan they were complicit there So either way Joe Rogan is a moron and he's also a coward Ideology


I took one of those Leonardo da Vinci naps over lunch. Just 15 minutes, right? You ever hear about those Leonardo da Vinci naps? And I had the worst dream, and the dream was that I was giving a talk at a type conference, and I had to follow Louis Feely. And you were in it, and you wake me up when it's over. I'm really happy to be here. I was very, very fortunate to lead some study abroad tours between 2009 and 2014 at the University of Minnesota. And on one of the trips, we were hosted by Dr. Klaus Maival of the Gutenberg Museum. And he had these four journals in his personal collection. And in the journals were roughly 6,000 ornamental capital letters from the Renaissance. And so you're looking at a page on screen right now. And 350-some pages, about 400 books, different books represented. And the tragedy, of course, is somebody had to cut these out of the original books. These are printed initials from the Renaissance. You see here, they're from 1508 and 1511. And around about 1890, somebody decided with the scissors, you know, these would be prettier if I just cut them out of the books. So kind of an act of vandalism there. But it was very common to trade these like baseball cards back then. You'd have a little envelope of all your favorite woodcut engravers and your printers represented. And so not the same level of respect for historical books the way we have today. So in these journals, and as I said, there's four of these different journals, you have notations like this. You've got these beautiful Lombardic capitals you see there. And then you have these handwritten notes. L-U-G-D is right here, 1511. And L-U-G-D, what is that? Well, after talking to some colleagues in printing history, that's the Latin name for the city of Lyon in France. And like, why didn't they just call it Lyon? Anyway, a long story. So these are from 1511. And so you would have this envelope on the left side of the screen you see there, and then a little notation that got put in the interior of it. And I was absolutely fascinated by these working for Hamilton Wood-type Museum. These were wood-type. These were engraved either in pear wood or in boxwood. And they're just exquisite. And they're about the size of a small postage stamp. These ones happen to be about one inch square. So think for a minute about carving these letter forms in reverse and in relief, which means that if you're trying to do the angry eyebrow of the angry bird, which is where the idea came from, it's like you don't carve the eyebrow, you carve everything but the eyebrow. So as we did our research, we were looking at classifications. We enlisted the help of Dr. Paul Gale from the Newberry Library in Chicago, absolute genius of a man, and very generous with his time and energy. And he helped us come up with these classifications. So you've got fluorated, including plant or flower motifs. You've got historiated letter forms. Those are people that are either from actual history or from fictional history, which is, yeah. Inhabited letter forms, those characters that have people or mythical beings or faces in them. Speaking letter forms, those are the design. They use the first letter of the word included. And I'll show you some examples of these in a minute. Zomorphic letter forms that have animals. Crible, which is this kind of a stippled dot pattern in the background. Calligraphic, that which shows the stroke of the flat nibbed pen. And ornamental, in this last category, decoration that's non-representational. That was actually the most difficult bucket category to establish because it's the largest bucket. And we'll talk about that in a second. So what I did was I put some basic borders on these. And you kind of get an idea of how beautiful these letter forms are. And again, just think for a moment about size. They're five feet tall on this screen. And these things are the size of postage stamps. So what you see here is in the upper left-hand corner, you've got an inhabited letter form. It's historiated. And it's a speaking initial for St. Cecilia. So C is for Cecilia. Right side, upper right-hand corner, you've got Crible in the background and Fluoriated Adam and Eve, of course. Back then, serpents had ears, which I didn't know. So that was a cool thing to discover as part of my research. And then you've got on the lower right-hand corner inhabited, somebody being stabbed in the back really hard. A couple more of these. We asked ourselves questions as we looked at these. Where did they come from? Why weren't the names of the books included in these notations? And so there was a lot that we didn't know about them. And you see a lot of hidden notations. This guy here hiding inside the queue, he's super sinister. This one here, I don't know what to say about this one. This Z is really beautiful. But on these slides, you can see the cities and the years in which they were printed, not the years that they were cut. So very important to understand that these letter forms could be used on multiple books. The other thing, this is me when I've had enough coffee.  I don't know if you remember that from when, if you're old like me, there's a long story. So what we wanted to start doing was when we needed to ask some questions about why we had some information and why we didn't have other information. And Dr. Gale pointed out to us that the notations were taken from the colophons of the original books. And as you know, colophons are the printer's notation at the back of the book stating the name, excuse me, the city that they were published in the year and the name of the printer. And so, but not the name of the book. And so if you're holding the Nuremberg Chronicles, you don't need the name and the colophon, you know that you're holding the Nuremberg Chronicles. So that was an important discovery for us. So the first thing that we, of course, tried to do is to make wood type with these. And the technique that we wanted to begin with is CNC routing. So CNC routing gives you incredible detail. It's a 14 degree bit. Actually, that's more like 14. Yeah, so a 14 degree bit that goes down and cuts out all these letter forms. In order to do that, you need vector files. And so here you're looking at the vector content of it. And so in the previous slide, in order to hold on to this kind of fidelity, all of this imperfection, all of the perfections and imperfections, scanning it, tracing it, and then telling a CNC router to do it, the CNC routers like, come back tomorrow. I'm gonna be busy. So we decided that that was too long of a pursuit and we switched to laser cutting. And laser cutting is a really great option because it lets you use TIFs or bitmap images. And so you filter these things down. You get mostly to a black and white stage. You don't have to deal with tracing because the interpolation with tracing is nightmare if you've ever done detailed tracing and I'm sure you all have. So laser cutting on end maple blocks. We have this maple in our collection and you see a couple of different renderings of this block and notations about how much power and speed. So S represents speed, P represents power and you can vary those things. And if you're trying to letter press print with these blocks, you have a balancing act to match.  I like to call this light, medium and dark roast. So there's a couple of the images that we wanted to test with and what we had to do is go to worst case scenario, find the letter forms that would be, present you with the most challenges. Related to that. And so there are simple letter forms and complex letter forms. And so we found one setting laser settings that were just right. So this is a close up of a finished letter form with a little ink on it. That's the exhibition poster in the background that is a pig playing a bagpipe which apparently they did back then. So on the left you see a printed example of it and on the right you see the original block. But when you encounter these prints off of your printing press and then you look at the original, the presses are a little bit better now than they were then but the skill involved in carving these and the complexities of inking, large areas of type, large pieces of type and then 10 point type. Those are two very different inking requirements on letter press. So we're having to anticipate the printing process, the printing presses that you're using in order to make this happen. Here's another example of a nice letter V and next to the finished printed poster. So from here I'm just gonna run through about 150 of these letter forms. Yes, I'm really gonna show, no maybe not that many, we'll count. All right, so this is from Leon in 1520 and one of the things I'd like you to pay attention to as these go by is look at the design of the letter forms. So we've got sort of a Lombardic letter form. Paul Shaw was saying these were Versa letter forms but beautiful examples of these bifurcated terminals and look at that crossbar, right? We've got a beautiful foliated letter form there. This just blows my mind, this is an inch and a half square and I'm just absolutely in love with that with the letter forms that are there. This one I included because it's an example of a beautiful letter form that's poorly printed and so if you put too much ink on the type, press comes together, it squishes it out like peanut butter or jelly or both. But when you look at this, this one and then compare it to this one, you see the amount of skill that's involved with printing with an incredible amount of detail. St. Cecilia again, we've got that nice angel in the background. This baby is either trying to wake up his dog or scare the fleas off of it. I'm not sure, the dog seems unimpressed either way. This ornamentation in the background is what some of us would refer to as arabesque. You have a lot of influence of Moorish art that you see in Europe during the Renaissance. Our friend the pig again and that, you look at all the nuances of the tonality and all the great imperfections of this one and then you print a new one and you go, I like the old one better. This is an example of an inhabited sea, that little face you see kind of hidden in there is a very important thematic idea. We'll talk about that more. Our friend the streaker, this is an early form of streaking. I didn't know that it was popular back then but apparently it is, but only in the letter C. So that's good. Combination of a caterpillar and a butterfly which is cool, maybe showing metamorphosis. This is a service, that's the Latin word for deer. So this is what we call a speaking initial, so sea for deer. And one of the things that you think about at this point is literacy. A lot of the people who may have been carving weren't necessarily readers. So it's a really interesting extension of that one. This person's either being beheaded or knighted, I'm not sure which one. We really beautiful inhabited initial here. So you've got not only people inside the initial but the initial itself also being made up with facial features. Beautiful griffin there. This is an example of what we call the green man. So a green man is a face that's embedded in foliage usually. Folage comes out of their mouths, their ears or what have you. I like this bird because it's eating a cactus and this slide shows you a book that this letter form appeared in and the context that you get, the whole side to side comparison of, well in 1566, this is what clip art was. And it's important to note that these woodcuts were not made specifically for these books. You had a tray of ornamental type. You'd pull the D that you needed. This is a history of the wars between England and France during the time of Mary Queen of Scots. We've got Saint Ezekiel, another speaking capital. We got back to the stab in the back. And one of the other things we wanna pay attention to is the major cities, the major printing cities that these appear in. The other thing I like about this E is that it shows the like epigraphic typography we're cutting into stone. So you've got that three dimensional feel to it. This is from Aldus minutius III, the Aldean press, Venice 1576. People dancing with the letter F, which I guess was a thing back then. Did someone just clap? Good. All right. Early veterinarian care, I guess. Checking to make sure that the dragon has teeth is okay. This letter G is really beautiful. The Crible pattern in the background. This is what we refer to. Actually, Paul Shaw just told me this, but this is a curly G. And this curly G you see is a much older predecessor. Here it is again. This is on the cathedral door of the city of Mainz. This door was cast in bronze in the 11th century. So this curly G is really beautiful. Side trip through some great ligatures. Look at all that great stuff there. Next lecture. Architectural and G, this is really beautiful. H, you look at the way the shading is handled. Sometimes heavy on one side. Sometimes true drop shadows, et cetera. This is an owl and smoking pot. It's a Portland Joe. Sorry. Thank you. Adam and Eve again. And the thing that I'd like to focus on with this kind of engraving is the engraver, it's not that you engrave the eyebrow. You engrave everything but the eyebrow. And so you look at the shading in these and look at the detail. And the fact that you have to carve away everything. This bird is an inks, I-N-Y-X, which makes it a speaking initial. Beautiful sun and moon from Mayans. This is Moses receiving the Ten Commandments. That scene with Charles and Heston where his hair turns white, I think that's what's happening. And it actually looks like Charles and Heston now that I think of it. A really beautiful phoenix. Again, a nice three-dimensional letter M there. This is a seraphim, those baby heads with wings. This is one of the oldest letter forms in the collection. This one, I don't know. It's just so cool, I can't even believe it. And if anyone knows what this is, come and tell me afterward or shout it out now. But it's just an exquisite figure of someone's imagination. This is a manticore. If you saw the Harry Potter films, you'd know what a manticore is. So here you see the cutout version on the left and another example of the letter form placed inside the book. And again, we look at these beautiful classic Roman letter forms, 1559. Look at the beauty of the italics and the decision on the part of that printer to put these beautiful things together. This is the goddess Minerva. So the artistic director saying to the engraver, yeah, she needs a blindfold. How do you want me to engrave that? So I'll just give her a blindfold. That's what she needs. So Minerva is a justice, I think. Anyway, this one's really cool to me because it's a letter casting a two-dimensional shadow on a three-dimensional space, right? You got that landscape in the background, but somehow the shadow floats in space. So I really like too. This is what we refer to as the Holbein-esque style. Hans Holbein, an amazing artist from the 16th century. He himself wasn't an engraver, but he would commission engraving to be done. Again, me having enough coffee. And as we look at these, these are the dates that they're printed, not the date they're carved. And so in many cases, we find books that are 60 or 70 years apart with the same initials in them.  This is rearranging a pee, I guess. This is a cool one. This is another speaking initial. Pee is for Piper. And either the dog that he rode into town on doesn't like the music or doesn't like the fact that his owner is riding him into town with a stick and a key. This is a good example of an ornamental letter form when you see these beautiful ribbons. It's a very common architectural detail, but it's an ornament. There's not a category for ribbons, but that was one of these things that, well, what should we call this? This is another one where I'm thinking about the conversation between the engraver and the printer. Can you make the bear really angry? Yeah, I can make the bear. And make the birds really angry too. So just work those eyebrows. Really exquisite cue. I like this guy hidden in the cue there as well. Really great owl. And as I look at the detail of this letter form, and I compare it with this letter form, this is really a sublime, really sublime craft. No magnification, very few eyeglasses being used back then, all being done by daylight. Balaam is a Old Testament prophet from the Book of Numbers, in case anyone was curious. This is a Seder getting ready for a Halloween party, I think. I wanted to add that one in.  I don't know if that's for sure, but that's what I've been told. And he's got his laying his hands on this great amphora of wine. So clearly a good time is going to be had tonight. I was hoping that this was a speaking initial, S for snail, but the Latin word for snail doesn't start with S, so they just happened to go together. One of the cool things of these designs is they all have these borders on them. And when we printed the exhibition poster, any letter form that you use that didn't have a border on it, the paper can settle, which means that any wood that's lower than that surface is going to show up in the print. So the borders become not only an aesthetic consideration, but a technical one. Really nice, unshel U. Really beautiful V. The other thing that we know is that U and V are the same rendered, can be rendered by the same letter. V is for virgin. I don't know if that's a U or a V. Sorry. I'm here all day. I'm almost done. It's OK. Hang in there. I like that W a lot. Inline letter form is really great there. This is probably a speaking initial. The West Wind is known as the Zephyr. And so you've got the wind blowing. Amelia Hugo was speculating that. She's a great printing historian at RIT. And then a beautiful Z. This is absolutely exquisite. This is a short illustrated book on artillery and fireworks printed by Johann Lechler from 1564. And again, looking at one Z custom made for a variety of jobs and then the type version of the metal type version of that Z. And as always happens, when Z shows up, that means that's the end of the talk. So 16 seconds. No, OK, go ahead.


talk of today, we look at something practical, which is kind of, and I think if you were there for Frank's talk, network traffic analytics is something that's quite important and getting more important and more like cyber-attacks we get and stuff like that, intrusions. And so I welcome Martin and Mirko from Kroder to talk about that, how you would do that on Hadoop clusters. Thank you. Thanks everyone and thanks for staying for the last talk of this dev room. Let me... Yeah, I think it's on the screen, okay. So my colleague is Mirko and I'm Martin and we both work for one of the Hadoop vendors, namely Cloud area. And we both work with customer solution architect, basically means that we go on site with the customer and try to help them to learn Hadoop, to adopt Hadoop and bring more and more workloads to Hadoop. And it's a very general question that we run into customers trying to ask for help to understand the system, that what does it mean that they are used to being one, maybe one monolithic application or used to having a database system and now this is this brave new word of Hadoop technologies where everything is distributed and your data is floating around in the cloud and how do I make sense of that. So one of the ways to make sense of it is actually concentrate on the bottlenecks and network traffic is very usually one of the bottlenecks of Hadoop processing system. So we came up with a couple of ideas of how to visualize this for clients and also to make sure that we ourselves understand these workloads better. So I will give you a little bit of the motivation of why, how Hadoop works and what we would like to visualize here. Then we go into the pack up capturing and how we capture the data. We do a little bit of analytics with the cloud area Hadoop stack, the CDH for short and the Gaffy toolkit which is an open graph visualization platform and produced the figure that you see on the side. So let's have a look at the network load of the Hadoop cluster because we have hundreds and in a couple of use cases thousands of machine in one single cluster simple operations change. When you put a file onto a file system that most of the time because it's a distributed file system DFS for short includes some network operations. Probably these files can be so large that they are bigger than a hard disk in a single machine also you want replication for full tolerance. So just putting something to the file system is a network operation of course getting something from the file system is also a network operation. Then you have to make sure that there is some service discovery parts of the system know about each other. So you need some hard beating going on in the system as such I would say a passive noise going on still network traffic is being generated and of course data is not just sitting around in a cluster you actually do some stuff on the data transactions analytics you name it again generating traffic. And one of the not necessarily nice features that we discovered by trying to explain this to customers if you look at the standard Hadoop monitoring tools today the metrics that you can see for network utilization they are aggregated on the host level and a lot of times it doesn't give you the capability to look at individual services. So instead of I'm using 80% of the bandwidth maybe I'm going to say that okay my DFS is using 20% my Harbids are using 1% and my processing jobs of these three workflows are using an additional 50% maybe that's what I would like to see in a setup like this. And currently that's not what you necessarily get from these systems. Also in this data room today you have heard about cyber security and there's an Apogee project that we are familiar with called Apogee spot which is mostly pushed by Intel and cloud era but it's open source as is an Apogee project for intrusion detection and we'll mention a little bit how we differ from their approach. So let's move to the data that we are using we used standard packet capture data so this is the library it's both implemented for Unix and Windows systems that you are familiar with if you use the tools built on top of it TCP dump, Nmap, Wireshark, Snort you name it they all use this basic implementation we used a Python hook for this but it's the same data a couple of gotchas in the data capturing of course we don't want to create additional noise while capturing the data so we capture it on the local machines and write it to local file system and we only gather it after the capturing has finished and we throw away the data in the packet itself we are only capturing the structure of the graph it's mostly for clustering and for visualization not necessarily learning from the data inside the packet itself. On the contrary what the Apogee spot guys focus on in terms of cyber where security they would like to learn topic analysis on the data itself inside the packets one of their most important approaches include this and they would like to build a cyber security platform for Hadoop using Hadoop technologies instead we focus on the clustering and the visualization as I've mentioned. So we had to come up with a couple of scenarios and we examined five scenarios in detail but here we only present three of those because those three were more important in terms of results so we had a heavy batch workloads called TheraSort where we read input from the distributed file system and we do a big distributed sort on the keys in this input and would like to eventually write the output to the distributed file system again. We also captured the idle cluster so have an idea of this background noise and hard bits and then we had a more interesting approach when we reached out to the Twitter streaming API with Spark streaming and collected data from the outside and we wanted to see how that appears in a network like this and also we wrote to HDFS we could have written to Kafka or any other source of your choice. So the way currently it works it's semi-automatic we collect data in the average format for which you had the average schema in two slides before with this peek-a-pie script. We transform the events into networks using Hive which is a SQL API on top of Hadoop. Now we are migrating these workloads to Spark because then it's easier to register UDFs and makes our life easier but currently it's in Hive and also we use GIFI this open graph visualization platform to do the visualization layer. So let's have some initial results to give you some idea of what we have managed to capture. Both the sides of the nodes and the widths of the edges correspond to the workload and in the Terrasort benchmark what we have seen is the five nodes of our cluster were of course very prominent because we were reading data from the distributed file system and the sorting also involves a big shuffle in between the nodes so of course that communication had to be really prominent but some other nodes were also included with just a little bit of traffic and of course it's important to examine those. In the other case we run the Twitter collection on just one single node and of course that's that's the one of in the center of this Sarsch email and that's the one that is of course also connected to the other four nodes that are in the cluster. However it's connected to a lot of unknown new hosts that we haven't seen in the cluster yet. Those are the ones that are actually sending the Twitter data to our cluster. With that I would like to pass the microphone to Mirko who is going to go deeper with the analysis. So after looking into this very high level initial results we could see what we expected. We see two very typical different topologies of the communication networks. You see communication networks here on a host level which is pretty interesting as a starting point but what can we learn from here? Not much yet. Okay there's a central node because one node speaks to the outside world and delivers data as an ingestion tool. We have an internal operation running on five nodes in parallel. Okay that's cool that's not something which helps us a lot. We have to look deeper. We have to increase the resolution. We must look into more details. This means we add port information not only host information as we did in the very first analysis results. Next thing we add a timestamp. We track the time-dependent graph not as a snapshot for the whole period.  This allows us to connect to a time domain. This means we can switch between graph analytics and time series analytics and in order to show something useful Giffy will be our friend. Giffy allows us to collect data with time resolution and Giffy plots and visualizes graphs with time resolution. You will see an example later. I mentioned to increase the resolution. We did it here. We added more details to the picture. What can we see? Not much at the moment. We see some clusters, some clouds of bubbles with the same color. Everything which is in the same color means here's a port available on a particular host. While at this port some communication happened during our experiments. So we reconstruct first a static graph which only shows which ports and hosts have been involved during a certain data capture period. That's the first start and now we must group the data a little bit, do a better layout and do some statistics. First thing in this cluster or in this graph now we have 1.5 thousand nodes, about 3,000 edges and I realized that Giffy behaved not well anymore. If this number goes up to 100,000 but capturing half an hour or an hour of data will explode the stuff. So we went to Spark and used Spark tools for analytics but finally we do the whole visualization in Giffy. Our whole visualization means we start with a static network which represents our infrastructure. The colors here encode at the moment the hosts. Same color means ports on the same host. The black lines are our real communication channels. Whenever a package traveled from one port to another it gives us such a trace. Okay this looks good for the moment but we can do more. We can now study the topology of this network. This communication network changes over time. So topological properties change over time, very interesting. So different nodes become important depending on what measure we apply. We can now tune the things like we need. This is still a field where research has not delivered final answers yet. With Giffy and with Spark we are able to calculate different topological properties. The interpretation is still an open problem. We don't go deeper here. Our goal is to track the whole system in a more specific way. Our initial question was how much of the network traffic is related to HDFS? How much is related to Spark, shuffle and sort or shuffle services? And all these different questions arise if you want to tune the system so this topology does not answer these questions. But we can see what hosts are very important and what are the dominant ports. Okay that's an interesting aspect but let's go away. Let's make this whole dynamic thing visible here. On the left hand side you see dynamically changing connectivity links between hosts. Okay that's what we have seen. On the right hand side you see a different representation of the same data. It is now not grouped by host anymore. Such a color-spanning area here such a cluster in the same color is now not a host it's a subsystem. It's for example hbase or it could be zookeeper or it could be method use or whatever. Depending on what software you deploy on your Hadoop cluster you may end up with a very different setup of such clusters and you don't usually not know what it is upfront. We collect data without knowing that such clusters even exist. Our analysis procedure highlights this and finds them out automatically. So this is how we do it. We go from a host-centric representation to a layer or subsystem-centric representation. We turn things around. This means we track the graph over time. Now we apply such a component or cluster detection algorithm and this cluster detection algorithm identifies such usually isolated clusters because these subsystems are usually well isolated. In a Hadoop system depending on your real environment this could be different. Such clusters could be interconnected. In our case they are not. So finally if we have such clusters where the different ports can be on very different hosts it makes sense to aggregate along these clusters not along a host anymore. We aggregate now per subsystem and get some time series and with this time series we can say how much is the correlation between HDFS and hbase or reproduce and HDFS and so on. This means if we change the perspective and aggregate along these new dimensions we end up with time series and we can look deeper into the behavior. A final remark here this component-centric view is very helpful. It allows us to look again into the topology but be careful. Absolute numbers in such a graph are really dangerous. These numbers here this high degree or this huge circle which is used here and here they are totally misleading. You cannot conclude from this picture that these both are the most important ones. This is an artificial an artificial effect of the analysis procedure. Just as a warning be careful with this always think about the right normalization usually you must look deeper into your data to get this. But such pictures help you in order to get central and decentralized stuff by using a different layout algorithm. It depends all on your layout algorithm how these pictures look like and if you want to cheat you just have to change the algorithm for layouting. Depending on the layout you can make the graph speaking that's on it's powerful but also dangerous. You must be careful here again. We switch the domain we go from time we go from graph to time series. What we see here is the overall activity during the method use job and during the Twitter and just job. Both jobs we're running and you see what we do inside the cluster has not much heavy traffic but during data ingestion we record a lot of activity but we don't know which subsystem is really causing this activity. This is why we group the stuff by subsystem and we come up with one time series which represents the number of packets per time interval for each individual subsystem starting with hdfs where the name node is here just as an example used. We have the node manager which organizes the workload in the cluster in red and blue, black and yellow are communications related to the job management system. So we can clearly separate this time series and see some effects. For Hadoop behavior analytics you can figure out which kind of application has a lot of internal traffic and which not and if you look in this picture here we see an average the activity on the name node is much smaller we have no blue yellow or black one here this means on this layer there is no activity going on but what a surprise this job is of the same type like this jobs we can see here here a result of system tuning. In the old ways we have a lot of internal traffic after tuning the system in a specific way we could reduce this intermediate traffic and we could avoid the bottleneck that's one result of using this method. So we are in the very first experiments we have two at the moment we have the idle cluster where only the background noise is visible we have heavy ingest activity and we can clearly see our method allows us to segregate this ingestion activity from all the background noise so we can split multiple channels like you would do in an EEG where you use frequency filtering during data collection. So we can observe that we have either one very active component or we can see we have multiple competing component multiple channels yeah and all this depends on your algorithm it depends on your workload a normal ETL cluster would behave different than a graph processing cluster but with this kind of measurement approach we could measure the things and then we have at least some benchmarks and we can start tuning the stuff. So finally we have to think what's coming next first of all we must improve the experiments this means we have to run more experience we have to add more realistic workloads for example flink streaming twitter spark streaming is not done yet impala queries or even heavy hbase use cases or even solar base search use cases have not been tested yet we will add all this to our agenda to our scenario list and then we come up with a different kind of visualization inspired by the talk in the morning where the twitter real-time yeah visualization was shown we concluded okay it makes sense to have this for this measurement as well so we can measure strange behavior or suspicious connections and if we find such if we have a classification for our traffic then we could visualize it using gaffy as well and gaffy together with spark works here with our gaffy hadoop connector and then after we have settled this experiment platform a little bit better we can do much more sophisticated analytics on top of the system. The goal is we want to learn more about the time series we want to model the time series but therefore we must first aggregate the data in the right way and here we have shown our first experiments towards this results in the future that's all for today we have to say thanks to the audience and thanks to a bunch of colleagues which support the project in the background which are not here and that's all from my side it's time for questions yes it is open source it's not published yet we have it still in an internal repository we just clean it up but yes it is intended to be published soon yeah because i can imagine it's really useful also for other use cases like exactly between methods or dc os nodes or something like that right other cloud exactly it's an experimentation and measurement thing like an oscilloscope for it guys yeah but in general this sub clustering by subsystem is done on a port level or a port range level or currently we do it without knowledge and the result is overlapping with port ranges so we can do port range statistics and this is a fuzzy matching with known port ranges if you have such if you know the configuration then you can do this matching otherwise we can just learn from the data that's really cool that's the goal we have the capturing project it's a python script it still needs a little bit documentation to make you fast in doing the experiments and not not wasting your time that's our goal and then we are thinking about contributing the stuff to the patch your spot project and finding the right structure and the right procedure and all the stuff this is going on we are not so fast so far yet but yeah that's the next thing we and also the analytic part is currently a little that's why we decided to converge towards park so if you are new to these tools then all you need to learn one tool not learn three why do that yes please I could not really hear well could you please repeat we can do a mapping to known port ranges in order to see which subsystem it is if you have no information about it we can do a good guess and we can look into the pattern shown here by looking into this pattern into this one and with a little bit knowledge about your algorithm and with some expectations you can say which curve belongs to which subsystem as a system engineer you would probably be able to guess good but let's say we have two jobs running in parallel doing the same stuff from different users such a thing could not be isolated anymore by this algorithm here we need a little bit advanced information maybe we have to combine this approach with log information coming from yarn and if we have this integration then we could hopefully isolate on an per application level this would be in next step to isolate not per technical subsystem but per application right now we cannot do that that's a good question for the package if there's enough information because you have to have like a tag on the package to identify where it belongs to and the raw network package content is not really that helpful right because it's just partial information so to reconstruct the application level you need both the backup data and the yarn resource furniture logs because that's going to tell you that this application has these ports on these machines and then you join that together then you can construct it right when you have to tune some uh a little cluster that was happening uh what do you find the most that you need to tune the network the job or the uh do you want to explain for this example here we did one thing we increased the replication level of the data instead of having three replicas we have now five one per host in a hundred node cluster this would be overwhelming but for our example if we bring maximum data locality into the play the network drops it's an it's an academically example here just for showing the effect or even just to figure out if the effect is visible and result is yes but in order to do a better quantitative analysis we need to figure out what resolution is the best we don't have this yet so depending on time resolution your results could vary so that's also a problem we have to deal with we are interested in looking to graphics to see how it works when the node communicate with each other on the graph level and not on the server level maybe we can also yeah help to tune the graph algorithm for itself because such a graph algorithm is heavy network bound and then we can say okay this could be a helpful tool we will see how it works out yeah so we capture it on on the host and we these jobs weren't really CPU bound so that only part it's captured on each individual host so we are not introducing any new overhead to the network it sounds that better we are trying to capture that that was the the basic setup that that we tried to avoid of course we shouldn't introduce any overhead possibly if we can that you you can adjust this in pcap the way you want to sample currently we try to capture as much data as we could to capture the most of the network but if you find that that overhead is just huge then pcapi has an option to only capture one data packet in a hundred or a thousand so you can go in that direction and you could I've also like a window based as you say okay I just capture how many bytes per second go on this port and windows and start in one windows and then we have like a volume you get a capture kind of which kind of summarizes and then you kind of choose the time range if what it per second per five seconds or something like that and then it would give you been enough information about the communication flows but you don't need the raw data because you actually want to know when did it start and at the end and how much data went over the connection cool any other questions otherwise thanks a lot for your talk it was really really interesting before you leave I want to use the opportunity to thank you all for coming for attending all these talks I hope you got something out of it for me it was quite interesting there's a feedback mechanism on on the foster website so if you have any feedback to any of the talks for the presenters or also for us then please don't hesitate to get feedback and otherwise have a good evening and a good day tomorrow and