Welcome back everyone. We are on the ground at the data artisans user conference for flink It's called flink forward. We're at the kabuki hotel in lower Pacific Heights in San Francisco the conference kicked off this morning with some great talks by uber and Netflix and we have the privilege of having with us Chinmay somon from uber yes, and Welcome Chinmay. Thank you to have you so You gave a really really interesting presentation About the sort of pipelines that you're building and where spark where where flink fits But you've also you've also said there's a large deployment of spark help us understand How flink became a mainstream technology for you right where it fits and why you chose it sure So about one year about an year back when we're starting to evaluate what technology makes sense for The problem space that we are trying to solve which is the near real-time analytics We observe that spark is actually the spark stream processing is actually more resource intensive than than Some of the other technologies we benchmark. It's a more specifically. It was using more memory and CPU at that time so we That's one and I actually come from the apache samsa world It wasn't the link the same LinkedIn team before before I came to uber so I have we had in-house expertise around samsa and I think the reliability was The the key motivation for choosing samsa So so we started building on top of apache samsa for almost The last one and a half years, but then we hit we hit a scale where samsa we felt was lacking So with samsa, it's actually tied into Kafka a lot and then you need to make sure your Kafka scales In order for the for the stream processing the scale in other words the the Topics, yes, I guess and the partitions of those topics you have to keep the physical Layout of those in mind at that. Yes at that message queue level. Yes in line with the With the stream processing. That's right. Yeah, the parallelism and samsa is actually tied into a number of partitions And Kafka and furthermore if you have a multi-stage pipeline where one stage Processes data and sends output to another stage All these intermediate stage stages today is again goes back to Kafka So if you want to do a lot of these use cases You actually end up creating a lot of Kafka topics right and the IO overhead on your cluster shoots up exponentially Oh, and so it when creating creating topics or Creating consumers that do something and then output to producers if you do too many of those things You defeat the purpose of low latency because you're storing everything Yeah, so it the trade-off is it is more robust because if you suddenly get a spike in your traffic Yeah, your systems is going to handle it because Kafka buffers that spike, right? So it gives you a very reliable platform, but it's not cheap. So that's why we are looking at fling Because in fling you can actually build a multi-stage pipeline and have in memory queues instead of writing back to Kafka So it is fast and it gives you you don't have to create multiple topics for pipeline So all right, let me unpack that just a little bit to be clear the in-memory queues give you obviously better IO yes, and I and If I understand correctly that can absorb some of the back pressure Yeah, so back pressure is interesting right with if you have everything in Kafka and no in-memory queues There is no back pressure because Kafka is a big buffer. Yeah, it just keeps going With in-memory queues there is back pressure and now the question is how do you handle this, right? So going back to Some systems they actually degrade and can't recover once they are in back pressure, but fling as you've seen It's it slows down consuming from Kafka, but once the spike is over once you're over that hill It actually recovers quickly. Okay, right? So it is able to sustain Heavy spikes, okay, so this is so this goes to your issues with keeping up with the growth of data That's right that you know the system there's multiple levels of elasticity and then Resource intensity tell us about that and and the desire to get as many jobs as possible Yeah, a certain level of resource right so today We are a platform where people come in and say here's my code or here's my sequel that I want to run on your platform and They in the old days they were telling us oh, I need 10 gigabytes per container and any these many CPUs And and that really limited how many use cases we onboarded and made our hardware footprint pretty expensive And so we need the pipeline the infrastructure to be really memory efficient because what we have seen is Memories the bottleneck in our world more so than CPU Because a lot of applications they consume from Kafka They actually buffer locally in each container and then they do that in the local memory in the JVM memory So we need the memory component to be very efficient and that gives us You know you can pack more jobs on the same cluster if everyone is using lesser memory Okay, that's one motivation. The other thing that for example Fling does and sounds also does is make use of a Rocks DB store, which is a local persistent Oh, that's where it gets the state management. That's right. So you can offload from memory on to the disk Into a proper data into proper database and and you don't have to cross the network to do that because it's sitting locally But that's just to elaborate on what might be a kind of a techie what might seem like a arcane topic if it's residing locally then Anything it it's gonna join with has to also be that's residing local. That's a that's a good point You have to be able to partition your inputs and your state in the same in the same way Otherwise, there's no locality. Okay, and you'd have to shuffle stuff around the network And more than that you need to be able to recover if something happens Because there's no replication to this state, right? If a node with the hard disk on that yarn node crashes you need to Recreate that cache somewhere. So either you go back and read from Kafka or you store that that cache somewhere So fling actually supports this out of the box and it snapshots the Rocks DB state and HDFS got it Okay, so that's more resilient. Yes, and more space resource efficient. That's right. All right. So let me ask one last question Mainstream enterprises they or at least the very largest ones have been trying to wrestle You know their arms around some open source projects, you know very innovative that the pace of innovation is huge But it demands the skill sets that seem to be most resident in large consumer internet companies That's right. What advice do you have for them where they aspire to use the same technologies that you're talking about to achieve You know build new era systems. Yeah, but they might not have the skills, right? So that's that's a very good question. I'll try to answer in the way I can I think the first thing to do is understand your scale, right?  Where do you fit in in your in the industry ecosystem? Yeah, if it turns out, you know, the scale isn't that big and you're using it for internal analytics Then then you can just pick the off-the-shelf Pipelines and make it work. So for example, if you don't care about multi-tenancy, right? If you don't if your hardware spend is not that much actually anything might actually work, right? The real challenge is when You pick a technology and make it work for a large set of use cases and you want to optimize for cost And that that's where you need a huge engineering organization, right? So it's in simpler words If if your use cases extent is not that big Pick something which has a lot of support from the community and then Most more common things just work out of the box and that's good enough, but If you do need if you're doing a lot of complicated thing like machine real-time machine learning or if your scale is in Billions of messages per day or terabytes of data per day then you really need to Make a choice whether you invest in an engineering organization that can really You know understand these use cases or you go to companies like, you know Databricks to get a support from from data bricks or maybe a cloud vendor or a cloud vendor or things like confluent which is giving You know Kafka support and then things like that. So it's I don't think there's one answer If to me our use case for example the way the reason we choose To build an engineering organization around it because our use cases were immensely complicated And not really seen before so we had to invest in this technology All right, Chinmay. We're gonna leave it on that and hopefully keep the dialogue going sure offline so We'll be back shortly. We're at flink forward the data artisan's user conference for flink. We're on the ground at Kabuki hotel in downtown San Francisco, and we'll be right back


The narcissist will always throw you under the bus. Whether the narcissist is your relationship partner, family member or friend, they will always throw you under the bus. They will always betray you. They will always violate your trust and confidence in them. They will always do something harmful to you in order to gain an advantage for themselves. Because they are not loyal, someone who is loyal, remains firm in their friendship or support for a personal thing. They will avoid temptation. They will avoid the desire to do something, especially something wrong or unwise. Because they value you more than any short-term urges for enjoyment that may threaten long-term goals. While the narcissist is very selfish, they lack consideration for other people. They're only concerned with their own personal profit or pleasure, regardless of how it may affect you. They don't entertain any thoughtfulness or sensitivity towards other people. They don't engage in careful thought before deciding or acting. They are very impulsive. They will do things suddenly and without careful thought. Without consideration of the consequences, they will sacrifice people, especially those who are vulnerable, for their own personal gain. They will expose you to an unpleasant fate in order to save themselves. They will exploit your trust for an ulterior purpose, advantage or agenda. They will harm you through deceit. They will deceive you by concealing or misrepresenting the truth. They will betray or sacrifice you for the sake of their own advancement. As a means of safeguarding their own interests, they will condemn you to something by season to take an interest in you, so that they can allow themselves to indulge in a desire or impulse. Their actions will reveal to you that they really don't care about you. They have no consideration for you. They will leave you feeling insufficient and inadequate. They will leave you feeling lacking and wanting. They will leave you in short supply, so that they can indulge themselves, so that they can enjoy a particular pleasure, so that they can spoil and overindulge themselves, so that they can gratify their wishes to an excessive extent. They are self-absorbed and they lack empathy.  They will do anything to get ahead. They will do anything to gain an advantage, because all they really care about is themselves. All they care about is their own selfish needs. All they care about is their own comfort, achieving a state of physical ease and freedom from restraint, easing or alleviating their feelings of grief and distress at the expense of your safety and security. And when they throw you under the bus, they will still return to you. They will give you fake consolation, as though they actually care. They will play the role of someone who is trying to comfort you or have sympathy for you during your experience of loss or disappointment. But their actions will never match their words. They will never take the necessary action to make the situation right, because they're only concerned with their own interests or needs. Regardless of how it may affect you, in some situations, they may act like they're doing the right thing, but it's only so that they can play the victim. It's just another way for them to get supply. It's just another thing that they can hold against you, but in most situations, they're not even going to do that. They're just going to leave you to rot. They're going to cause you to remain behind. They're going to cause you to remain in a specified state where your physical and mental state deteriorates. They're going to leave you to be beaten down, reduced and diminished by an event or place that has made a negative effect on your life, because they have a need to feel superior to you. They have an overly high opinion of themselves. They are excessively proud of themselves. They see it as though doing the right thing would be beneath their dignity or level of importance. And they see it as though they're not going to lower themselves. They have an exaggerated sense of their own abilities and importance. They have a strong sense of entitlement. They believe that they are inherently deserving of privileges of special treatment. They feel like they have a right to something. They believe that their importance, superiority or uniqueness should result in getting special treatment and receiving more resources than other people, which is why they will demand the special treatment or extra resources, which is why they will throw you under the bus. It can be very confusing, abusive and even dangerous with the behavior and demands that you will deal with from the narcissist. They have unrealistic expectations, which comes from their unhealthy interior. Their expectations of special treatment masks the underlying sense of inferiority. Deep down they feel like they're not enough. They feel inadequate, which is why they require so much attention from you, which is why they expect you to drop everything for them, which is why they feel entitled to monitor and control you.  They feel like they're not enough. They feel entitled to act out in ways that they would never tolerate from someone else. They will be passive aggressive towards you. They will take revenge and try to hurt you. An entitled person will always throw you under the bus. They will always step in you to climb up the ladder. Entitlement is about taking. It is about capturing or gaining possession of something. It is about acting out behavior without an understanding or duty of care regarding its impact on other people. They don't have the capacity to care about other people's feelings or needs because they are too preoccupied with masking their feelings of inadequacy and inferiority. Thank you for watching. I hope this video resonates with you. Please like, comment, share and subscribe if you would like to donate. My PayPal link is in the video description. Coaching Inquiries. You can email me at coaching.narksurvivor.uk. Thank you for watching and I'll talk to you soon.


If you're working on a group assignment it may be that you are drafting content in your sandbox before migrating it to the Wikipedia live space. Now at the top of my page you'll see my username and that's my user page, my talk page where people leave me messages and my sandbox page. This is where I draft content before it goes into the Wikipedia live space. So for the assignment you may be drafting content here first, but if you're working with other people in your group what's important is to put the article together in one person's sandbox so that when the article is all together you can then migrate it from their sandbox into the Wikipedia live space. So what I'm going to do here is I'm going to bring up another Wikipedia in a different window and I'm going to look for someone else's username. This is going to be I'm going to imagine this is one of my classmates and they are drafting content in their sandbox number five about the assignment. Okay now there's no content in there just now but let's imagine that we want to migrate content from my sandbox, user Stingelhammer and this line at the top is always what you can search for here and you can have multiple sandboxes and I can show you how to do that. But the main thing is if you want to migrate content from one sandbox here into someone else's sandbox the best way of doing it is through the source editor. The reason being is if you try and copy words like from here from the visual editor to someone else's visual editor it doesn't always copy across exactly like you want so I'm going to hit undo to redo that. What we want is to copy everything and what the edit source editor does is make sure that you copy all the coding elements all the references all the links all the bullet points. So you just copy everything you want to go across hit ctrl C or right click to copy and then you go to their page and you're just copying from source editor to source editor. So I'm clicking edit source so that I can then paste my content into their page and then to complete the save I'm using a short edit summary at the bottom added paragraphs on Mary Symboyd or whatever I'm writing about to complete the save. Normally you wouldn't edit a page that's related to someone's username because it's part of their user account but for the assignment purposes it's perfectly okay to migrate content from one sandbox to another because you've agreed to do so beforehand. So now that you've got the content in there and saved perhaps you want already and you've got all the content in user Emma Candrew's sandbox number five and you want to paste it into a live article so you would bring up another window let's imagine it's the global health article so I'm going to navigate to that article and because we're copying and pasting again we're going to open this article up into the source editor and we're going to open our sandbox up into the source editor and we're just going to migrate the content I'm going to avoid the user sandbox link at the top because that's related to someone else's sandbox I'm just going to take all the information that I want to be in the new article I'm just going to take all of that and paint find the relevant section where it's going to go let's imagine it's in the history section I'm just going to paste it in and then I can do a show preview to make sure it looks okay before I make the edit and you'll see there's some of the my marry sign board information there it's obviously wrong for this article but so I won't complete the edit and then when you're ready and everything looks okay then just put in a little edit summary of added section on for example global health in China and then hit publish changes to complete the edit and that's it migrated across that's all there is to it


Bemidji Middle School students are getting hands-on writing help from an award-winning Minnesota author through a Region II Arts Council grant. The school was able to hold a writer's workshop with William Durbin. The school was also able to host Durbin at the Beltrami County History Center on Wednesday night, where he talked about his young adult historical fiction books and other works. Many of his stories are inspired by real people throughout Northern Minnesota's history. We hope that people will be inspired by Mr. Durbin's writing and how he takes a great interest in the research part and develops his stories from primary sources. Lake of the Woods High School and Red Lake Middle School were also awarded grants through the Region II Arts Council in November. If you've enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland Public Television.


n, and I hit the subscript runs from one to s, the number of species in the pool. And okay, these are just the usual lookable terodynamics. I'm sure you've seen them before with migration from a regional species pool. And of course, even in the simplest of models, still the matrix alpha here, the network of interactions alpha has tons of numbers in it. You can imagine if there's 50 species, then there's more than 200, sorry, 2,000 numbers parameterizing this matrix. And so there's a lot going on, even in this simple framework. And okay, this is it. Later, I'll touch briefly on a few other things on meta-communities, on resource competition, on functional groups, on demographic noise. Of course, very briefly, you're welcome to be happy to discuss more.  And I'll only focus on theory today. Now, I should say at this point, I welcome, please interrupt, please ask questions at any time. I can't see the chat, so Yacopo, if there's anything, feel free to stop me at any time. Okay, any questions so far, anything? Okay. Right. So the basic idea we're working with is that, perhaps, when there are many combined things happening in my system, any interaction mechanisms, many things happening, you know, a cow goes to eat one kind of grass and it steps on another kind of grass and so on, then maybe the interaction matrix of, you know, all species in the pool would look, they won't see any conspicuous order and maybe we can take it as random. This is by now a rather standard step, but it is by no means a trivial one. I do not want to say that this is something that you should believe in just like that. On the other hand, if there's, you know, one dominant interaction mechanism, then you can see this matrix here on the right, it is very order. Okay, so at least there's this very strong contrast between these two cases. I want to emphasize that what I mean when I say this very, this math only random matrix, and this means that all the numbers are independent random numbers, this is for the pool. So once we assemble the community, structure is formed in the assembled, the network, the assembled network, and we will discuss this in some detail. And so one way to think of what we're doing here is that this maximum random interactions in the pool is a null model to understand the filtering that the assembly process applies to your network. Is this a good starting point? At the end of the day, you just, you know, generate predictions and you check and the proof is in the pudding. And I'll say a few words about how to add additional orders such as functional groups, you know, trophic levels and so on to this simplest of models. All right, Yapoko, anything so far? No questions. Okay, go ahead. Great. Or actually, who am I talking to? Just so I... I'm Matel Marsili, I'm chatting this session. Yeah, okay. Sorry, Matel, yeah. Because I was seeing Yapoko in the video, that's why I was, but I heard the voice, it wasn't his. Excellent. Okay, right. So if you run this one simple model and it looks, you know, again, you put very little into it, you already see quite a rich phenomenology of things that can happen. So the first thing is you just run the model and what you see here on the top left is you see an assembly process. So you start with all the species in the system. You put them each line here and with the color is one of the species. The x-axis is time and you can see them going up and down and reach a fixed point. Some of the species, some fraction of the species survive and others go locally extinct. So that's assembly. Now, if you start the same, you run the same equations with different initial conditions, you might reach the same fixed point again and again for the same interaction matrix or you might reach different fixed points and that represents multiple alternative stable states and historical contingency that could come with something like that. For example, dependence on the order in which you inject the species if you do it sequentially. There are other cases, other parameters in the model where you don't see the system reaching any fixed point. Instead, there's dynamical resistance. In fact, it's a very high-dimensional chaos and there's even another behavior which I won't talk about much where you see actually succession patterns, true directionality. But again, I won't say much about that. The question is how is it that you see all these very different things in one very simple model and how do you order and organize things? If you're a physicist, this reminds you immediately of this idea that ice and water, if you've never seen ice turning melt into water, you swear that these are completely different materials, but really they're just the same thing once you change some parameter by a bit parameter called temperature. What you're starting to think is that maybe this is what is happening in these models. The phase transition between a phase of ice and a phase of water would have similar things in community ecology. The phases would be, for example, the single equilibrium behavior to multiple alternative stable states or from a single equilibrium to chaos to this successional behavior. These are just examples of what might happen. All right, so over the years we've done quite a bit of work on this very basic model. We know quite a bit, I should say analytically, properties of the equilibrium, total diversity, biomass, fluctuations. We see that these things are said deterministically when the system size, but sorry when the diversity is large, all kinds of things about the dynamics that I will be touching up a bit. But what is the first important thing that I want to say is that what we find in all these calculations is that the distribution itself of these numbers alpha i j doesn't enter. So it can be a Gaussian, it can be uniform, it can be anything as long as it has a mean and variance. And so what doesn't enter in all the calculations again and again are three parameters, rescaled mean of the interaction. See here's the number of non-zero links. If you like, it's all the species that you interact with. It can be any number as long as it's much larger than one. And then there's the variability in the interactions. And then there's the symmetry of the interactions. How correlated is the interaction of i on j with the interaction of j on i? And so these are the parameters that drive everything. And system-wide properties again, like diversity, fluctuations are fixed deterministically. They're what are called self-averaging quantities by these three parameters. So we have the variables. We have what are the axes that we should start thinking about. And now... So if I may, why don't you consider lambda as a relevant parameter? So the immigration. Right. Sorry. That's an excellent question. So we're looking at very small numbers. So the strength of migration is much smaller than all the other things. It's true that in the chaotic regime, it can enter in some very subtle way. But for fixed points, there's a well-defined, at least for fixed points, there's a well-defined limit of lambda going to zero, but larger than zero, which we can take. But it's true that if it was order one quantity, then it would affect everything. Any other questions in the chat or something? Yes. Mercedes asks whether you can clarify what C is. Yeah. So if we're looking at matrices or interaction networks where some of the interactions are zero, then this counts the number of interactions. We're assuming anyway here that the number of interactions is much larger than one. And so we're also going to assume that the distribution, because it's a large number and we're going to assume the distribution of this number is not very broad, so you can just take one number for that. If you like for simplicity for everything that follows, you can take C to be equal to S to all the, everyone interacting with everyone else. And then it's just a fixed number. It won't change anything in what follows. Okay. Sorry. So in that sense, it's like the degree, right, in the network? Absolutely. Absolutely. Because otherwise I confuse it with the density or the connectance, but it is the degree. It's, yeah. Sorry. That was my questions. Thank you. Absolutely. Okay. It's the number that when you're interacting with everyone, you get S. That's, by the way, it's not the assembled degree. It's the degree in the network before assembly. Yeah. Okay. Anything else? Yeah. And also as a question. Yes. Okay. So I can see it now. Yes. So as I said, you need the mean and the variance to be finite, right? So it's not heavy-tailed in that sense. Yes. Okay. So, right. So you could ask, okay, this is an extremely non-biological model. So my colleagues in Michelot's group, Mathieu Balbié and Jeff Almondie have looked at many other properties of networks that people are, have been using and looking at how much these three numbers, if you just use them blindly, how much you get system-wise properties correctly. And it turns out that many structures that people do use in the literature do not affect the system-wide properties very much. However, there are things that matter. So if you have, you know, plants and pollinators, then this bipartite structure is very important. And then what you need to do is to extend this very simple model to the next simplest model with a few more parameters. So that's basically the idea. I'm not going to go any more into that, but yeah. So that's it. Okay. A question on Monday asked whether the distribution is in the mean or the interaction strength? The distribution is in the interaction strength. So you have a matrix, you have, you sample numbers in it, and they're all sampled from the same distribution, and you fix that matrix and then you run the dynamics. That's how it works. Okay. Right. So the puzzle, so we have the axis for the puzzle. And now here comes the pieces of the puzzle with these axes. So again, so let's put on the x-axis here, the mean of the interaction strength. So more competitive is more to the right. And on the y-axis, this variability that I mentioned before that I call sigma. And we see these different, what we cannot call phases. And I want to start focusing on this unique equilibrium phase where whatever the initial conditions you start with, as long as all the species are there initially, you end up with the same fixed point. Okay. And yeah, so that's a very simple dynamics. But now I want to talk a bit about the assembly process in this case. All right. So let's see what happens to the network during the assembly in this case. And so right. So you have, sorry, the notation here is slightly different. I'm sorry for this.  Okay. And here for it, I slides for something from somewhere else. Here it's S survivors. Okay. And the S species in the assembled coexisting community. So maybe you could call it S star or something. And so for all these species, the important thing is that for all the species that have managed to coexist, for them to coexist in the fixed point, you need, first of all, the final abundances to be positive. That's, that's, otherwise, they're not really coexisting. And then the fixed point has to be dynamically stable. If you push away a bit, like some press perturbation, small press perturbation, for example, then you need to be able to return from that press perturbation to your fixed point. And then there's also the issue that you have to be uninvatable from the outside, but I don't want to talk about that much now because let's, let's just focus on the coexisting species and not on the properties of those that cannot. Okay. And now the question is this, if we look at the matrix or the network of the assembled network, how is it that all these species coexist? So one thing you can do is you can keep reducing the diversity of the assembled community. I mean, if you get to one species, certainly they all coexist, it's fine. But another thing you can do is you can add structure. The assembly itself can generate a matrix which is no longer completely independent random numbers. Okay. And the question is, first of all, does this happen? Does this structure exist? And if it exists, which of these requirements that we've discussed before, these two things having positive abundances and stability, which of these things is affected by the structure and how? Okay. And so it turns out that the assembled matrix is in fact not maximally random. There is structure. For example, you can show that there are correlations along rows and columns. And we can calculate them analytically from some pattern I'll show you next. And so these correlations did not exist in the pool. So assembly does generate structure in the interactions. So that's one thing. The second thing is that these correlations are there, but so... Okay. Let me finish the sentence. The second point is that these correlations are not sufficient to explain coexistence. So if I just have these correlations, or in fact, any enrichment of some local interactions, I do not generate the full coexistence. So I need to... It's just not a full explanation of what's going on. Okay. Let me look at if I can see the chat. Do you also have a stable limit cycle? So we do not see stable limit cycles and the limit of large S, of larger, of highly diverse systems and this asset product limit that doesn't happen. In intermediate values, there is in the transition from a single fixed point to chaos, you do see a limit cycle. Are we taking... Yeah. We're only talking right now about... I'm still in the single equilibrium single fixed point phase and I'm looking at assembly, the effective network structure on assembly on network structure in that case. I'm not even touching dynamics just... Okay. Okay. So those are the... So this is what happens. Now... Okay. So I said that there are these correlations, but they don't explain the coexistence. But okay. First of all, there is structure in the assembly level. Okay. That's one. What is this structure? And importantly, how can we cease to understand what is the structure that is in fact relevant to the coexistence? So for me, the functionality in this case, what this system has to do is to coexist. Okay. So I need to understand what are the... What is the structure that is sufficient for this coexistence? And it turns out that the one way to describe it is the following, that if you look at the assembled matrix or network, it looks very much like a random network as before, but there are subtle biases in the strength of the underlying interactions. In particular... Sorry. In particular, for some reason I can't see. Sorry. Can you tell me what the time is and when did we start? Just so I know how I'm doing? We started five minutes late. So you have 20 minutes more. 20 minutes to the 35... Okay. So it turns out that the structure is... The following looks very random. It's hard to see by eye what's going on, but there's an underlying, as I was saying, bias in the same interactions. So it's my mistake. So it's 12 minutes. 12 minutes. That's also fine. Yeah. I can do that. So this underlying pattern, this bias of the interaction strength can be understood as follows. You organize your species in the assembled network from the least successful one to the most successful one, where by success we mean the ratio of its abundance in the community to its carrying capacity, to its monoculture abundance. And it turns out that the abundant species do not... are sort of specialized. They do not interact... Do not compete as strongly with other abundant species or with other successful species. So it's sort of this mafia or this click type of thing, whereas the unsuccessful species just compete with everyone at the same strength. So that's what you see. It's some form of specialization of the successful species. That's how I think about it. Okay. I don't want to go very much into the details, but I will say that this structure can be proven to guarantee the positive abundances that the interaction network has once assembled. And one last thing I want to say about this is that we've actually derived this same structure for a completely different framework, and we've actually validated, we've seen it in... We've shown it's there in plant competition experiments and biodiversity experiments. Again, this takes us into a different topic, so I won't go into it, but it was just published now. You can take a look if you're interested. All right. So what did we get here? So before we've already seen that there is structured in the assembled network. And so the next question is which of the requirements that we've seen before for an assembled network, namely having positive abundances. Being stable, which of these requirements is affected by this new structure? So we said that abundance has become positive once you use the structure. Stability, it turns out, is unaffected by the assembly process. Okay. And this is an important point. It's a subtle point, but it can be important. I don't want to get into the technicalities, but maybe for the experts, just in one word, you know that there's a spectrum of the matrix which affects its stability. You cannot squeeze this spectrum just by the assembly process. There's a simple reason I'm happy to discuss this later with those interested. Okay. So what the assembly does is it affects feasibility and the abundances, but it cannot push the stability at least for large systems. Okay. So that's all I wanted to say about the unique fixed point phase and how the assembly affects the structure of the network. So anything so far, any questions? There's no question in the chat. Okay. Good. I'll give another second. Okay. Good. So now we step, we increase the variability of the interactions. In fact, we can increase diversity at fixed interaction parameters. It's the same thing here. And what we see is that we lose stability at this unique fixed point and instead we go into something new. Now what happens now depends on the symmetry of the interactions. If the interactions are symmetric, as is often used to model competition, what you get is many, many, many fixed points. You get exponentially many fixed points in the diversity of the system. So you can have, imagine 300 million alternative stable states, alternative equilibria here. This is very different from regime shifts and things like that where you imagine one or two. So it's very many sub-sensitive species that in fact form a stable equilibrium. So I see there's a question, the correlation gamma plays, yeah. So right now we're talking about this correlation gamma equal to one. And this is symmetric interactions. Okay. That's gamma equal to one. And that's what happens in this case. Okay. And right. And as advertised, this transition is indeed sharp, like a phase transition. Namely, you change the interaction parameters by a bit and you jump from one fixed point very, very, very many fixed points. And this, yeah. Okay. So it's as we thought it would be before a phase transition. All right. Okay. I don't have time for this. Right. So now I want to go into the case where the interactions are not symmetric. In fact, let's look at the case where gamma, the correlation is zero. So it's independent effect of i on j and j on i. And then what you get is a transition into a chaotic phase. Okay. Persistent dynamics, very high-dimensional persistent dynamics happens. It starts at a very well-defined strength, variability of interaction strength. But the size of these fluctuations grows continuously as you change the parameters. So at first, it looks almost like a fixed point with small fluctuations around it. As we increase the variability, you get more and more strong and stronger interaction, sorry, strong and stronger abundance fluctuations. Okay. So that's the, that's the phenomenology that we see in this case. All right. So one word about what the mechanism is that's going on here. Basically, the idea is that you have a species pool and invasions try to increase the diversity in the community. But then there's competition, which pushes some of the species out by because of the competition. They have negative growth rates. They leave the system. And so this alone balances, gives it some diversity that would balance these two processes. Now, at this diversity, the assembled community might or might not be stable. If it's stable, you get a single equilibrium. If it's not, then one of two things can happen. You can either say, okay, I cannot go beyond this stability diversity. I stick to the stability bound, and then I get multiple equilibria. Or you can say, I don't care. I'm just going to do dynamical fluctuations, not care about stability at all. And then you get chaos. And it's not an easy question to understand when and which of these scenarios will happen. There's still ongoing work on this question. Right. Okay. Mateo, how do I, with time? Because I can't see my phone. So four minutes. Okay. I'm wrapping up. Okay. So from this, we can generate specific predictions for what we expect from, for example, these chaotic fluctuations that are endogenously created by the system, as opposed to, for example, environmental fluctuations. For example, you expect the sideline fluctuations to grow continuously with diversity. The fluctuations look almost uncorrelated between different species when the diversity is high, as opposed to the things that you happen, you see in environmental fluctuations. And okay. And now the next aim, the next big aim is to look at experiments and data to start looking for these things. Okay. I don't have much time, and I'd rather leave time for questions. Let me just say that a similar, one thing I want to say is that a similar behavior happens if you have a meta-community that with patches connected via migration without external migration. So the system can hold this chaotic behavior without extinctions for a very long time. That's one result. Another thing I want to say is that people think that resource competition models are so very different that none of this will ever be seen there. That is not the case. So one of the issues is that there is this, many of you know, the MacArthur model is a model of resource competition. This is a very, very influential model, and I love it. But it's it's simplicity is sometimes misleading. If you change the model such that, for example, the resource uptake or the dependence of growth on resources is nonlinear, you can in fact see chaos, and you can go beyond the stability bound, which in this case is in fact the competitive exclusion limit. So you can have more species than the competitive exclusion limit using these persistent dynamics, and it's enough to in certain cases it's enough to introduce very little nonlinearity, and you're already there. So the MacArthur model is in a sense a bit single. All right, I won't talk about this directional phase. Let me just wrap up and open to questions. So what characterizes these high-dimensional, high-diversity assembly models and hopefully also systems out there? There's these very few relevant variables. Once you know that you know how to start seeing what start putting your puzzle together, there's these deterministic outcomes for things like dynamics and properties of the assembled community. There's these sharp transitions between very different qualitatively different behaviors. I talked about these interaction patterns. And again, there's extensions to various directions. I'm happy to discuss whatever is interesting in the questions. Yeah, thank you.


In the public revenue mobilization for inclusive development workstream, we cover mostly tax issues. So we look at how the tax system helps to promote economic growth, how it helps to promote employment, and how we can generate potentially more revenues to have higher levels of social expenditure. Hopefully this will lead to lower levels of poverty, lower levels of inequality, and create a more robust democracy. In phase one, all the workstreams that use the tax data invested heavily in preparing the data set, curating the data set for research work, and we are going to continue that. So we want to collaborate closely with the tax authority to understand how different processes, different enforcement interventions impact taxpayer compliance, impact how much revenue ZARS collects, and we think that this empirical evidence that we get out of that can be a very solid basis for ZARS to review their enforcement processes and strategies and to see whether there's potentially some room for improvement. The changes from phase one in this program I think are going to be focused on trying to expand the available use of data. The initial program used basically the low level or the low hanging fruit in terms of what was available. Here we're trying to get into a bit more difficult data but there could be more insights from that data. So in phase two, the tax workstream really wants to explore how much we can help ZARS and of course the tax policy unit in tax administration. So there are various issues there. So one is related to what happens to firms and taxpayers after they have been audited. There can be spillover effects among the network. So that's something that we would like to do basically some sort of cost-benefit analysis about, or it's another enforcement interventions. Then we would like to also do quite some work on international tax. There's been quite a bit of effort in the international tax environment to try to limit international tax avoidance and evasion by multinational companies. So there have been various policy measures that have been undertaken and we would like to follow up what has happened after these policies have been implemented. In the second phase I think we want to look at more of the impacts of particular policies in terms of have they been more successful or not. And you really have to be quite nuanced in how you look at this type of data. We've also got a bit more of a time series with this data so there's more data available over time and we can see whether those changes and policies are having an extended or dynamic impact or not if they weren't just once off. So there should be some deeper insights that hopefully we can look into with the second phase.


All right, welcome, everyone. Thanks for coming to this session. This is a talk that's going to be about accelerating developer adoption of Cloud Foundry. And I'm joined today by Luis Amadeo with Ultimate Software. He's a distinguished architect with Ultimate. And he has some great insights to share about their adoption of Cloud Foundry. And my name is Matt Gunter. I'm a platform architect with Pivotal. And I work with a number of customers that are at different stages in adopting Cloud Foundry. And one of the things that captured my attention recently is what are the factors that help some companies kind of press the gas and just take off with their adoption? And then what helps other or what causes other companies to basically tap the brakes, right? And not be able to get past certain issues and so forth. So I did a white paper on value stream mapping that talked about how Cloud Foundry adds value. And this kind of builds on top of that, which is why does it matter? Who cares about it? And how to make that process go faster. So without any more intro, I'll just jump in here. So we're going to talk about two perspectives here. The developer's perspective of Cloud Foundry and what it means to them, as well as the stakeholder's perspective, and how Cloud Foundry helps them accomplish their goals and mission. We're going to talk about things such as top down versus bottom up. We're going to talk about trust and confidence and alignment and consistency. These are all things that end up kind of getting discussed in the context of culture. And we're going to try to demystify that a little bit in this talk. And as I go through here, I've done some research on a number of Cloud Foundry adopters. And we're going to analyze their adoption of Cloud Foundry and try to understand why they're doing so well, why they might be having some challenges, and then what are some things that we could do to make it go better. So it's pretty clear. Most developers want speed and autonomy. That's not very hard to realize. But even though we can offer all of these capabilities to developers with the platform, there's still resistance. And it's interesting to try to understand that resistance. And one thought about this slide that I chose the Millennium Falcon as an example. If you tried to get Han Solo to trade in the Millennium Falcon, you might have a little bit of a trouble doing that, right? He's got a lot of history with it. Him and Chewy have a lot of shared understanding about the Millennium Falcon. So if you're going to get him to switch ships, it might be a little bit of a challenge. Now let's talk about stakeholders. The stakeholders are interested in outcomes. And they usually tend to translate those outcomes into a mission, right? And it's important to separate what the outcomes you're shooting for versus what the mission is. For example, the graphic on the right, a mission to Mars is not a mission to Mars. You can go into orbit. You can do a flyby. You can land a rover. There's a lot of details about the mission that are important to understand, to really understand what's going on there. And with Cloud Foundry, a lot of times the outcome of developer productivity, operational efficiency, et cetera, those become the mission. And we're going to talk a little bit more about that as we go through this to try to find a higher level mission, something more inspirational, something more compelling to align Cloud Foundry with. And this, I believe, is one of the keys to companies adopting Cloud Foundry in a very successful way. Here's a basic model that I think a lot of Cloud Foundry adopters have in their mind when they look at accelerating adoption or accomplishing their outcomes. They see, OK, I'm going to get Cloud Foundry installed. And then I've got to work with the devs to use it. And then I'm going to accomplish my outcomes, almost like a math equation. And at Pivotal, we have quite a bit of support for companies, even with this mindset, to help them be successful. We help their developers with learning XP and installing the platform and managing the platform as a product, helping them be more customer-focused as a platform team to the developers so that they're listening to the developer needs and trying to anticipate those, et cetera. But even so, with all of this, there are still challenges that emerge. So in looking at this and trying to figure out what the best way to look at it, I came up with this Denison model. This is an organizational change model that was developed by Daniel Denison at University of Michigan, like, I don't know, in the 80s or something like that. It's been around quite some time. But it's based on research that shows that all of these different dimensions in this model are related to company performance in a very concrete way, in a very measurable way. So basically what this model says is that if you're a company and you have strong traits of mission, strong traits of adaptability, involvement, and consistency, you're going to have better profit, better growth, et cetera. And these are all the same factors that I believe affect Cloud Foundry adoption. And if we believe Cloud Foundry is going to be a positive impact on your business, then this is just upstream of that, this notion. So here are the things that we're shooting for in this model. A clear mission, technical consistency, and the systems inside your company, involvement at all levels, and adaptability that's meaningful to your customers or your market. And a lot of times we, and I kind of did this with the value stream, you kind of drill down into the process and break out the magnifying glass and say, hey, what's going on at this level or that level? But sometimes taking a broader perspective is more helpful. So I threw this dolly painting in that thought kind of exemplified that. Now if we take that Denison model and we apply it to Cloud Foundry, it kind of looks like this. So we have, instead of mission, let's just say long-term direction, instead of consistency, let's say technical strategy, instead of involvement, let's just say empowered teams, and then we have growth and adaptability. And all of these points that are inside these columns, these are nothing, there's nothing new here. These are the same things that we would be talking about in the basic model, right? We'll be talking about, let's manage Cloud Foundry as a product, so a platform as product. Let's move towards microservices so we can reduce our development batch size and reduce the sizes by releases so we can release more frequently. Let's reduce technical debt. Let's adopt some agile practices, maybe do some cross-functional teams so that the teams don't have to reach outside of themselves to make decisions. They can be more autonomous. And then if you do all of these things, you're able to support experimentation, learning, et cetera, on the business side. So let's look at how that would work in a little bit more detail. So there's a relationship. If things are working the way they're supposed to, and there's a lot of backwards and back and forth in this model, you can start with the growth and adaptability and learn something, right? That's what learning is, and that can feed back all the way to your mission, so that's a possibility. But let's just start on the left side and say you have a mission to accomplish something for your customers or in your market. You design a technical strategy to implement that. Maybe that includes Cloud Foundry. You want to empower your teams to take advantage of that technical capability and help achieve that mission. And hopefully they make applications and technology that's available to your customers and to your salespeople and to other people in the customer-facing side of your business in order to achieve the mission, right? So that's the idea here. And if you're successful, there's a strong feedback loop that happens from the adaptability and growth pillar back to the mission pillar, and it reinforces that mission. And this is important to make sure that executives stay on board, that the organization feels like they're making progress, removes frustration, et cetera, from things. So it just clarifies exactly what the company's trying to do. Now, if we look on the bottom side of the Denison model, this is focused on internal systems and people. So this is really the interplay between the technical strategy and the empowered teams that are your developers to a large extent here, but obviously your platform team and other people in the enterprise should also be empowered. But what this is really focused on is the fact that if your platform team just throws Cloud Foundry out there and they're not looking at how developers are using it, they're not listening for ways to improve the platform, they're not engaging with the developers to solve their problems, then there's not going to be a good feedback loop. And this inner loop is going to not be reinforcing. It's going to cause problems, and it's going to limit your growth of Cloud Foundry. So let's take a look at a very positive example here. This is a retailer that has done amazing things with Cloud Foundry. It's affected all corners of their company, and they basically want to be on par with Amazon. They want to be leading edge. It's a very bold, daring vision for them. They're going so far with their technical strategy that they want to rewrite their core apps. Their teams are wanting to adopt XP, so not just agile, but go all the way to pair programming and TDD and some of these more less common practices. They're even promoting accountability to the point of giving devs access to production, the ability to release code into production. So that's also a very bold thing for them to do. And all of these bold steps add up to a lot of growth and adaptability improvements. So they're revisiting all of the decisions and practices that they've done in the past and making sure they fit with this new model. And the business is really feeling those benefits. And both of these inner loops and the outer loop are very much reinforcing in this scenario. And they're making great progress with the platform. So let's take a look at a manufacturer that's more cautious. They have a process improvement mindset, and they're trying to implement Cloud Foundry as an incremental improvement, build on success, enable the business. Their technical strategy is also influenced by this mission, so they're trying to prove the tech incrementally. And they believe they, from their perspective, they stood up Cloud Foundry very quickly. However, it took them 80 days to do that, which sounds a little bit on the long side to me. And after it was done, they had to certify it, et cetera. So those are all kind of telltale signs that they are being more cautious, and they haven't changed their practices to a large extent. Now on the empowered team side, they are making strong progress in their digital factory model. Basically, you might have heard this concept before. There's this common pattern of creating a center of excellence around Cloud Foundry and agile development. And in a lot of large organizations, you kind of start with that COE and try to expand across your dev population from there. So outside of that, there's still some complacency. They're looking for more change agents inside of their dev teams to foster more adoption of Cloud Foundry. So the feedback loop could be better on the inner cycle there. However, on the growth and adaptability side, they've done enough to have some impact on sales. Executive support is growing, so they are getting a little bit more executive level support. And so if I'm looking at this objectively, it seems like they need to really leverage that executive support to help transform more in the technical strategy and in the empowering teams to make sure that that doesn't stall, so the progress that they've made continues. Let's look at the flip side. So this is a company that manages prescriptions and prescription drug programs. And they have the flip side problem, right? So they are very strong on the inner loop. So they're automating everything with their platform. Their dev teams are doing a lot of training, a lot of app transformation. There's a strong feedback loop there, and it's very reinforcing for them. So that's a very strong part of their transformation. But they're having trouble getting the business to know what to do with this capability that they've built, and they're having trouble with the executives knowing how to help them get that engagement and that involvement with the business. So in this case, I think on the flip side, they need to also get more executive sponsorship and get more help with the business to leverage what they're doing on the tech side. So again, you can see how this model is helpful in understanding what you're doing well and where you might need to improve. And one thing that that Denison model is pretty good at is it points out ways that you can use one strength of the enterprise to help counterbalance weaknesses in other areas. So here's another kind of perspective that I think is helpful too. This is an idealized adoption scenario. And ideally, you'd have the persistent mission from the beginning, right? You'd have executive sponsorship from the beginning. You'd build capabilities based on that. You'd grow confidence in the new practices following on those capabilities, and then you'd get some early wins and you would start to pick up momentum. That doesn't always happen. A lot of times the dev team start using agile practices before the platform shows up or the mission comes later. I have some examples about that as well. And so here's a global bank that is basically having to refocus their interloop because they delegated Cloud Foundry to a shared ops team that was not able to support their developers as well. They were letting the platform not be upgraded as frequently as it needed to be and so forth. So they're bringing that back into the business unit. In this other scenario, a financial services provider had a bottom-up scenario where the developers brought in Cloud Foundry themselves. They handed over to the ops team. Now the ops team, they're treating it just like any other piece of infrastructure. They're not managing it as a platform. But luckily, the devs have been able to rewrite a core application that's made an impact to the business and it's starting to get the executives aware that there's something big going on with the platform and with this capability. So they have an opportunity here to get that outer loop to support some improvements of the interloop and have their adoption accelerate that way. So a key takeaway here, I think, is that the cultural aspects, this reinforcement that happens around success or failure or what have you. A lot of those boil down to beliefs and fears and things that have happened over years inside of your organization. And if nothing else, hopefully this model will allow you to kind of bring some of those factors up and be able to talk about them explicitly and not just have decisions being made about things based on beliefs, fears, policies, habits, et cetera, that don't move you forward in terms of your technical strategy or the way you're working with the technology. Okay, I'm gonna skip ahead so that Louise can tell the story of Ultimate and their journey of adoption with Cloud Foundry. So Louise, come on up and I'll turn it over. Thank you. Thank you, Matt. So Ultimate Software. So what do we do at Ultimate Software? So Ultimate Software builds software, human capital management software. This is basically, we have a product that's called Ultiprope that enables organizations and employees to manage the complex life cycle from recruiting someone into the organization, onboarding people into the organization, and then living in that organization, getting paid, getting performance reviews, setting your goals, all the way to retirement. So Ultimate Software, the mission has been very strong and consistent for the 28 years that we've been in business. And the mission has actually driven, or the culture of the company has actually driven its mission, and it's basically to put people first. Not only to put people first as the employees of the organization, because we strongly believe if the employees are empowered and we treat people first and put them first, they're gonna take care of the business and eventually the customers will be happy. But also that culture and that mission translates into our product mission, which is simplify people's lives. So that is our customers' lives through our product, as well as our employees who are actually servicing our customers, right? So as you can see in that red graph, in the journey to the cloud, that mission has actually stayed pretty consistent, right? Now that mission has led into many, drive many capabilities that we've actually driven, we've actually created in the cloud. Starting in the late 90s, we moved to a software as a service model. This is a big transformational shift. After we moved to the SaaS model, we realized that obviously you need a lot of automation to be able to live in a SaaS model, especially driven by quality. Once you actually start delivering software in a SaaS model and you're actually trying to do it fast, then the first thing that we did, we need a lot of automation and actually test automation ensuring that the product is actually a quality product. After we went through that, then we realized that we actually wanted to deliver more, more value, right? So what happened was that the cloud capabilities allowed us to actually test and ensure quality, but we weren't actually delivering fast enough. So we started introducing more cloud capabilities that allowed us to actually do infrastructure as code. We introduced an infrastructure as a service platform, and that actually started the DevOps movement on alternate software. We got development teams that felt empowered to actually write all this automation and deliver code to production. But what started happening is that we realized that we had reached a speed and an agility, but that speed and agility needed to improve in order to meet the customer's demands. So what we did after that is that we introduced an appliance framework that actually encapsulated a lot of that infrastructure as code for complex deployments such as databases and message brokers, and that improved the experience for the development teams significantly, but again, trying to meet the customer's demands for people for sculpture meant that some of these teams actually needed even more cloud capabilities. So as we struggled to see how we can actually make this faster in scaling this business, we took a dip on the confidence. I mean, as more and more developers actually needed more from this platform, and we struggled to actually give them more, more capabilities, right? Confidence started to dip, and then we looked at ourselves and said, what is it that we need to do here to deliver more with quality to our customers and ensure that the development teams are empowered and have the tools that empower them to actually deliver? So that's where we introduced Cloud Foundry into our ecosystem or a portfolio of cloud services that significantly started helping the confidence. What we got is from early winds of actually internal tools, even some early micro-services actually start generating that adoption. And once you started getting that adoption in there and other teams started to see the capabilities of this, then the confidence started growing and we're continually going into that, increasing that adoption and that confidence as we speak. The next journey for us is going to be what is going to be next after the Cloud Foundry application platform? So where we started to think and look at what are those other technologies that will enable us to deliver more with quality? So if we look at the forces that play here that have played out through this journey, it's very, very clear to us that our mission, because of our mission has been so strong, people first, simply by people's lives, and that has been so strong that we've always tried to meet the market demands. We've always met our customers by market demands. So that's a very strong relationship. Simplify people's lives, deliver what the customer needs, do not deliver just shiny technology objects. That has been very, very strong for us. Now, what has been also healthy but challenging is that internal reinforcement loop. I mean, in this internal reinforcement loop, what it means is that because of our culture of empowerment, these teams, development teams are encouraged and super empowered to actually innovate and deliver more. So they're constantly innovating, so they're constantly asking more from our Cloud Platform team for more Cloud capabilities, right? So there's this internal loop that's actually constantly innovating. The challenge there is, is because we have that strong, very strong relationship between what the market demands, strong to our mission, and what we deliver to our customers. Well, there's sometimes we believe that we don't have time to focus on these Cloud capabilities because we need to focus on the features of the customers, right? So even though we have these two very healthy relationships right there, or feedback loops, it's still a little bit challenging on how do you actually weigh in what you invest in your Cloud capabilities versus the features that the customers are demanding, right? So is it really a challenge, though? Is it really an opposing force, right? We believe that right now, the really kind of the challenge is making sure that everyone organizations are on the same page, but it's not necessarily an opposing force when you look at it. And what's happening right now is that we have these empowered teams innovating, creating more technologies, asking for more Cloud capabilities, and that's actually feeding our technology strategy. That's actually helping us align the business strategy with what the technology can provide, shape the technology strategy, communicate the technology strategy, and start bringing in more Cloud capabilities. And at the end of the day, what these Cloud capabilities are, are improving the delivery value chain, which is essentially just giving more of these people first features to our customers, right? So at the end of the day, both of these loops are seeking the same goal, right? So our challenge today, just to recap, is that we as technologists and as people on empowered teams and people working on the technology strategies, we have to actually start communicating a little bit more of how these solutions actually empower the business to reach their mission critical goals and their actual cultural mission, which for us is to simplify people's lives. Thank you.


Today, you're going to learn some of the pure power of Hugo to be able to create via scripting very complex sites without any real effort, without any kind of dynamic site-side crap. It's we're going to do, here's what we're going to do in this video. I actually have my personal website pulled up here as an example, and we're going to use this just because I already have pages created for this. What we're going to do here is something that a lot of sites have. Let's say I want on my main page, of all the blog articles I have, let's say on my main page, I want to have a kind of a list, but really like a bunch of previews of all of the blog posts I have that have the title, they have the date, the link, they have a summary of whatever, you know, what the blog post is about, and maybe some other metadata and stuff like that. Because we want that on our front page, that people can kind of look at things, browse it and follow them if they want. We're going to show you how easy that is to do in Hugo, and we'll learn some other stuff in the meantime. So here I actually have, notice I have here, I should probably be clear, we are in my website's directory in this folder right here. I'm going to go into the layouts folder, and you'll see that in the layouts folder there is a short codes subdirectory, and of course if it doesn't exist in your website you can go ahead and make that short codes directory, and in that we are going to create a file that we are going to create the rules, the script to generate all of this. And it's not difficult at all because we're going to look at Hugo variables. So here's what we're going to do, I'm going to create a file, and I'm just going to say, we'll call it 4video.html, and this again is in the short codes directory, and at the same time I'm going to go ahead and get rid of all the stuff that's on my website. So I'm just going to delete all this just for the example. So I'm going to just put some random stuff, random text in this 4video short code, and I'm going to call it here by saying 4video, the name of the file, notice that it's inside two curly brackets, and one carat bracket thing, whatever they're called, greater than or less than sign, and I'm going to save that and you'll see that now my main site has this lolol thing here, lololol, and that of course is it just printing out the content of this short code. I'm actually going to go ahead and we don't need to modify that terminal at all whatsoever. So a couple things about short codes, right off the bat. Even if you're not putting any kind of programming into them or any kind of like for loops or anything like that, short codes can function as macros, right? If you want to just, if you have text that you're repeating on a bunch of different pages, like that you need to put in different places, a short code is a good way of just abbreviating that. So if you just have some information like this highly important information, you don't need anything else, you can just like put this content in your Hugo page without any trouble, right? And of course this is supposed to be an HTML just to be absolutely clear. I know it's technically not right now, but you know, okay, now it's HTML text, right? Well, either way. So, but let's actually start doing what I said we were going to do. So I want to loop through, I basically want to have all the pages on my website and we're going to want to, again, have a little preview of each of them. Now I'm going to go ahead and write some stuff and then I'll explain what this is doing. We are going to say range, site, pages, okay? Then we're going to say in and then we're going to say in the middle here, we'll say title, okay? So actually I'm going to put in some line break there. So let's save and look at that and we'll see now it's printing out actually the names of all the pages on my website. So what this thing is doing on the right, what range does, is you give it something more or less to loop over. Range is basically like a for loop. So it takes all of the elements in site pages and for each of those things it does this. It prints out title in our case and then it prints out a line break, right? If we got rid of this line break just to be clear, it would print everything just without breaking a line. So that's why I put that in. Or we can do something more, I don't know, syntactically correct. We could do something like this. We could say, let's say open a list, close a list and for each of these elements in the range, we're going to have that in a list item, hypothetically, right? So we could do something like that. So that's all this is doing. But let's explain what, so what is this site stop pages for? What is this dot title for? Just to be specific, notice I pulled two pages up, web pages up in the Hugo documentation. And I will go ahead and say this. I recommend if you're on a desktop especially, go ahead and pull these pages up because you're going to want to play around with these. Hugo actually has pretty decent web documentation, which is very rare among a lot of projects. They actually list things out like economically, simply and like tell you basically all the stuff you can do. Now this first thing here is a site variable. It's prefaced by dot site. And this is something that is basically site-wide. So dot site dot pages is going to be all the pages in your website, right? And then this thing here, which does not have site in front of it, this is as we're looping through those pages, we can also call one of these page variables. In fact, we can call all of these page variables if we want. This is all the stuff we have to work with or we can use while we're going through those pages, okay? So site variables and page variables, you can check these out. We'll be using some in this video, okay? All right, so let me save this again. Actually, let's put the line break in. Actually, you know what we're actually going to do? Here's what I'm going to do. I'm going to put the title. I'm going to put that in a, you know, give it kind of a heading or whatever. So notice a couple things about this. Firstly, there's some weird page titles here. So there's like crypto software that isn't capitalized. There's tags. These are not actually pages that I've created on my website. Really what we want, if we look at site variables, there's pages, there's, I'm going to be searching a lot of pages, but there's also a site variable called regular pages, okay? And what that is is, well actually, let's just look what it is. Why find it in the documentation when I know the answer anyway? So if you put in regular pages, you'll see that all those other little tag titles and stuff, they disappeared. So just to be clear about what the difference is, pages is all the sites on your, all the pages on your website that you've created, plus the ones that Hugo auto-generates. So it auto-generates tags and stuff based on the, or pages based on the tags that I've created. The dot pages variable includes all of those, but dot regular pages does not. It is just the files that you have created. So that's really what we want here, just to be clear, okay? So now we're using this, but I'm going to show you some things that you can do in addition to this. So you can pipe this content and you're going to guess what this does. If you pipe this into first five, what do you think that does? Wow, you're so smart. It actually takes that for loop and only takes the first five elements in it, right? So it only takes the first, if you only want a preview of the first five or the most recent five, it goes through those. Or if you want 10, it goes through those, right? Notice also you can do other things. By default, I think, I don't think it lists it here, but let's say list Hugo page by date or something like that. So one thing you'll notice here, well you might not notice because you don't know how my website works. But by default, it's actually looping through these kind of in a date order, right? So this is, these are like the most recent articles that I've made. And if you go back further here, these are like the oldest ones, okay? So by default, it's actually looping through these kind of by dates. But if you look at the Hugo documentation, I want to say, I think it's in this page. I think it's in this page. What it's actually, oh yeah, here. So it's actually looping through by these variables. It's looping through by weight in all my pages. You can set custom weights for pages in Hugo. I don't have any custom weights, so it's kind of ignoring that. Then it sorts it by date, then link, title, then file path, right? But you can change this by putting in things by like by weight or by date specifically. So let's actually do this. Let's say by date at the end of regular pages. And you'll now see what it's actually done is it's now linking them, well now it's looping through them. But it has the things that are most original at the very top, whereas most recent down here. And we can reverse that. We can reverse that saying reverse. We got a kind of long command here. But so that's just, if you want to customize, you can put by date, you can put by publish date, a bunch of other things, length, oh that's an option. Let's do that. So let's say by length. So you have a couple of different options, even before we get into the kind of stuff that you can do. What exact way you want to go through these by, right? So this is by length. These are the shortest. These are the longest. You can flip those around if you want by putting reverse, blah, blah, blah, blah. So I'm going to leave that as it is right now. I'm actually going to make more changes to this, I think, at the end of the video. But let's just get to the content, the meat of the page variables, really. So as I said, now that we've dealt with our range, we're going to say we're happy with this right now, we can use these different page variables to refer to stuff in the page. So for example, title, we already have here, title prints the title. We can also, if we wanted, we can include the entire content of the article with content. And that will list out literally everything. But really, I'm going to put summary. Now what summary is, I talked about this in the previous video. It takes kind of the most, the first content of that page and estimates where you want it to end, basically just gives you a snippet of it. So that's what we're going to use for our preview. So actually already it looks okay, but we need a couple things. Actually we don't have any links to the actual pages, so we got to put those in. So I'm going to probably want, you might want two kinds of links. And we're just using normal HTML syntax here. So I'm going to want to put a link, let's say we're going to make our titles a link here. And what we'll need here is there's a link here that's like permalink, okay, rel permalink, right? Actually we can go ahead and put it in whatever. I don't need to show it to you in the documentation. Rel permalink. And that is going to refer, we can key over these, right? Or we can click on them and see that they're actually working. This is the actual link, the relative link within the site. You can actually see it in the source, let's see. I think if you just put permalink, it's including like the domain name, which is kind of funky or whatever. Actually let me just double check that. It won't make a different in function, but it's kind of ugly on this site. Yeah, so it puts like the full domain link. But so I prefer to link things with rel permalink because honestly it looks trashy if you do otherwise. And I don't know, maybe it depends. I don't know how like browsers or whatever call it, but it might be less efficient. Like in turn, it might be making more DNS queries. I don't actually know about that. Someone who knows more about how that stuff works could probably say. Either way, so we have links here. But let's say we want something at the end that's like read more. That's a pretty common way to have a little link here. So I'm actually going to copy this. Let's put it at the end of the summary. Let's put in read more. So now we have two little links per article. And of course we notice that you can style this in whatever way that you want. So I could put this in article tags. I'm a big fan of semantic tagging. I know I said this. I know it's totally soy dev, but I like it. Or you could put in line breaks, horizontal line, horizontal rules or whatever they are. So you can style this in however, whatever way you want. It doesn't matter. I'll actually get rid of these so it's not left in the way. But here's the tricky thing about the read more thing. Here's something to think about. If we scroll down, here's a really small post. And this is just, now technically speaking, if we click on read more, we're not reading more. We're only looking at the content that is already here. There's no extra information. So if you want to be a perfectionist about this, one nice thing about the page variables is there's this cool little variable called truncated. So what truncated does is, in the page, if the summary is a truncated version of the article, that is if it doesn't contain all of the stuff, this variable returns true, otherwise it returns false. So let's actually use this. So what we can do here, what we can do before this read more thing, we can say if truncated and then after it, we can say end. And what that is going to do is for each article, instead of just printing that stuff, notice that the read more has now disappeared from this page. It was a liar anyway, right? Because we're not actually reading more. But what this is doing here is it's saying for each of these pages, if the summary is a truncated version of the full article, we will have the little read more link, otherwise just end, right? We don't need to go through it. So that's a nice little feature to have as well. So let's start putting some more metadata in other stuff in here. There are a couple of things we can have, for example, a word count. That's kind of nice. People like having those sometimes. I don't know if they're a low attention span and they don't want to read 1,000, 2,000 words or something. At the end of this, we could put another break here and say, let's just say word count. Word count, maybe I should put it in some kind of, I don't know, heading or something. I think it's kind of ill-advised to put that in a heading. Who cares? We're just not gonna say. So we're gonna say word count words. Okay? Oops, what did I do to, oops. It's not two words silly. And it's, yeah, okay. So now you'll see in each case, Hugo, because it's so smart, it has put a preview of all of the different words. However, many words you have in each article. So you can say, oh, this is nearly 1,000 words. This is over 1,000 words. This is five words. Right, nice and simple. Okay? And I wanna say, I know nowadays people, a lot of sites will have like that minutes estimator. I mean, do they have something like that? Oh, reading time. Okay, I didn't even think. I was gonna suggest like, oh, you could probably script this in. But I guess, let's see. I guess Hugo has this. Let's see what that looks like. So one, I guess that's minute. Yeah, okay, so yeah, we can, okay. I didn't even know that. See, I'm learning stuff in this video. Wow, this is fantastic. So if you don't like that, you can say, reading time, minutes, read. Okay, so now one minute read. One minute read. 10 minute, oh, that's cool. Wow, that's cool. 13 minute read. That's, oh, okay. I'm happy that I found that. I'm glad that I thought of that in this video. Let's see what else. Let's see what else. So anyway, let's put dates. Dates are pretty common as well.  So we can actually go up here. Let's go, let's put date at the beginning. Why not? I don't know. Notice that I'm not like, people complained in the last video, well, this site looks bad. Why aren't you styling this? Because this isn't a video about CSS.  You know how to use CSS. Same thing applies here. Don't complain about how, I'm showing you how to do things within a script.  How to beautify it, because I don't care about that. That's your job. You're supposed to be a professional developer, aren't you? So anyway, if you put in date, you will now see that it actually prints out this massive date thing that actually includes the exact time. And specifically, I wanna say this is the exact time that when you first create the article and it puts that date in the metadata. But here's the thing is, you know, this is ugly, obviously. So we'll probably wanna format this in a different way. Actually, let's just make it bigger just for this video. Instead of having VR, yeah, let's bite the bullet and put it in like a heading tag. I don't endorse doing that in real life, but in whatever. For the video, it makes sense. So what you can do with date, actually, is date has this sub command called format. And then you can provide a format for what it looks like. And Hugo is nice and intuitive. I know that when people are using date on the command line, dude, I have to look up what all of these symbols mean. Oh my goodness. I totally forget. Oh, is M like, is lowercase M month? Or is it, you know, whatever. Who cares? You know, who cares? But in this case, what you can do, Hugo. So the thing you have to remember about the date format command is that there's a very special date. And that is January 2nd, 2006. I don't know why they chose that, but what you do is you take that date, let's say January 2nd, 2006, and put it in the format command. And based on how you format that date, it will format all the other dates. So for example, I said Jan, therefore all of these months will be abbreviated. Next I have the day, and it's gonna be after that in the year. I mean, we can move this around if you're in one of the many countries where the day's in front. You can put that in front. Or notice also, so on days that are one digit, if we get rid of this initial zero, they will disappear there. We can say, you know, January, write out the full thing, and it should write out the full date here as well. Notice also, if you use any other date, if you say December, that's not gonna work. It's just gonna interpret that. It's not gonna parse that as a date. It's gonna just put December in there, right? So that is one tricky thing about it. But yeah, that's just how it works. And yeah, I think there's probably some official time to put in there as well, if you wanna include time in there as well. But you know, just know the important thing is like, if you change it to some other number, it's not gonna work. So like in this case, you know, if we change the two to something else, the like days, what looks like the days of the month is actually using the final, you know, of the 2006. Like it's using like the years of the day. So either way, January 6th, or January 2nd, now I'm confusing you, January 2nd, 2006. That is the right date. All right, so now we have actually, like if we look at this file, okay, look how simple this file is, okay? Look at it, all right? We just got a couple of lines, and we have a lot of content. We can already include reading time, word count, the name of the article, what it kinda looks like, give people a preview of it. And I think you can also put, you should, I'm not entirely sure how to call these, actually, you should be able to put tag names as well. Let me actually try. You might have to say params tags, because I think that's how tags work. Yeah, I don't know. There's definitely a way to, oh yeah, okay, yeah, yeah. Okay, here they are, yeah. Let me move them above that line. So you can put params tags, and that can include the tags as well, so we can say like, let me add in some more breaks, tagged with blah, blah, blah. And I know there's a way to change, let's see, how do you get rid of the, there's some like delimiter function, maybe like delim comma, I'm making things up if things break. Okay, I'm gonna look this up, I'll be back in one second. Okay, I see how to do it. I did it in my header file, or I think someone else like PR'd this, because originally it was just like using the ugly format. All right, let me explain what this is doing. I'm just gonna like paste it in, because we haven't talked about with yet, but I guess, again, it's a learning moment. Let me get rid of this, tagged with blah, blah, blah. Okay, so now you'll see it says tagged with technology comma, software comma, crypto. Okay, so what it is doing here, actually what we should do, well, I'll explain what's happening first. So what it's doing here, so with how with works is it gets a variable, and it can be anything, it can be a parameter, it can be like, it can be truncated or something like that. And then it performs the stuff that is before the end. Okay, so it puts this delimited by, there's, you can look up the delimiter command, but it puts those with commas delimiting them. So with how that works is if there is no content there, it just like doesn't print it out, right? So when I, I'm gonna do a video on like giving arguments to a short code, probably the next video, and that is gonna, it's gonna have with all over the place, but it just know that that works. Actually, okay, let me think about this. So this is like looping through, let me think. So in this case, if we don't have a tag, we still have tagged with. So what we wanna do, I think, is let's get rid of this and put it in here. I think, was that gonna work? Yeah, okay, yeah, yeah. So in this case, in the articles that don't have tags, you're not gonna have this because it's gonna do with parameters tags, and if there's no tags, like it's not gonna input this text, in input this text tagged with. I'm running out of juice to say words at this point. Either way, computationally, okay. So our site, if we look at our site, it's actually very ugly right now in terms of like, again, I haven't done any styling, but look at this stuff we're doing. Look at this stuff with this little file that's basically 20 lines, less than 20 lines, and could be slimmed down. We already have this pretty, we have article previews, we have words, we have read counts, all of this stuff that is built in that you go and we can so easily add and extend. Oh, actually, you know what I should do. I'm not gonna do, I'm gonna draw this article to a close, but what you could do is each of these tags has a link to the tag itself. So you just put that, that's another thing for you. Try that yourself, try that at home, kids. So yeah, this is the kind of stuff you can do and you can customize it to your own liking. Like this is, you know, a lot of, if someone is using like some dynamic WordPress site that's taking a million years to load, this is the kind of stuff they're using WordPress with and you don't even need it. Like this is something that Hugo does absolutely automatically, absolutely fantastic. So, oh, I need to fit it. I haven't finished and put up this article. I need to finish that. So yeah, I'm gonna draw this article to a close. I think in the next video, I'm going to talk about short codes again because you can script them, but as I said, let me give you a preview. I kind of gave you a preview actually in the last video. Well, I may have said what I was gonna do, but I might, we might either use this or we might use another one, but I'm gonna talk about how to create, not just have short codes, but have short codes take arguments so that you can do really complex stuff, like make very complex HTML without any kind of effort. You know, you just, when you're writing text, you don't have to worry about it. So that's about it. Hopefully, that's given you a view of the kind of stuff that you can do in Hugo, and I'll see you guys next time.