A film in three minutes, The Reflecting Skin. What ingredients are needed to make a truly great horror film? Is it how scared the story makes you feel, or how many times you jump up in fright? Is it how relatable the characters are when trying to escape from some desperate situation? Or does your definition of horror come from some place deeper, a place where the landscape and people seem a bit off? Because Philip Ridley's 1990 masterpiece The Reflecting Skin could perhaps be the best example of such a terrifying horror recipe. Easily one of the most visually startling and haunting horror films ever to be shot on celluloid. Set in a nameless, isolated prairie community in 1950s America, the story follows eight-year-old Seth, played by Jeremy Cooper, whose parents live a drab, impoverished life operating a remote gas station. When a group of men driving a Cadillac pass through the area, a series of strange killings begins to affect the lives of the locals. Killings that Seth suspects are linked to a foreign widow who exhibits almost vampiric characteristics when she's alone. It's hard to know where to begin when discussing the cinematic artistry and complexity that is The Reflecting Skin. Its narrative, although seemingly basic and quite minimalist upon first viewing, belies a far more complex and thought-provoking beast, making you squirm with increasing discomfort the more you delve deeper into this world. The beautiful cinematography composed by Dick Pope contains a hypnotic mesmerizing quality, which slowly washes over you thanks to the hues of golden light and endless fields of yellow wheat. The heavy use of saturated shadow and contrast is reminiscent of the works of painters such as Caravaggio and Andrew Wyeth, making full stunning compositions in almost every scene. But one of the film's strongest achievements is how claustrophobic the world of the characters feels despite the endless open planes. There's an almost suffocating tension from the moment go as Seth's troubled childhood is slowly revealed to us, with the featureless planes of wheat acting as a character unto themselves by keeping the roster of strange beings seemingly trapped forever in this world of morbid shadow and gold. This feeling too is greatly enhanced by the extraordinarily gripping and oppressively bleak score composed by Nick Baccar, whose string-heavy melodies blend in perfectly with the stark yet alluring visuals director Ridley created. Despite a style-over-substance argument made by some, few film settings come as close to the concept of horror as the terrifying universe presented to us here.  Are you daring enough, like Seth is, to reach out and see what lies beneath the reflecting skin?


When you are in a merry-go-round, you feel this outward force called the centrifugal force. If you mix sand with water and make it rotate just like you rotate in a merry-go-round, the sand particles also feel this centrifugal force. If you keep rotating this mixture of water and sand, the sand particles will eventually settle at the bottom. This principle is used in biology to separate out cells or even biomolecules from a water suspension. For the rotation there is this device called a centrifuge and centrifuges can actually rotate in large speeds so that even smaller particles like biomolecules such as proteins or DNA which are much smaller than sand particles or us in a merry-go-round even they can settle down with such high speeds and this process of taking an acute solution or suspension and rotating it in a centrifuge is called centrifugation. There are different types of centrifugation depending on what we need to do and one of the types is called density gradient centrifugation. In this type of centrifugation what we do is we take a substance and make a concentration gradient of it along this solution in the tube that is being rotated so the substances that we usually make a density gradient of are say sucrose or cesium chloride. So the concentration of this substance increases towards the bottom of the tube so when you take genomic DNA and centrifugate using the density gradient technique what you see is you get a band like this. The DNA settles right here not at the bottom even though DNA is heavier than water it still settles here because there is a density gradient usually the substance used is cesium chloride so there is this gradient and the DNA will go and settle at that point where the density of the solution matches the density of the DNA so since the density of the solution because of cesium chloride is increasing as you go downwards there will come a point where the density of the DNA matches the density of the solution that is where the DNA goes and settles and forms this band. Now you would expect only one band right for DNA but in fact we see a few minor bands and those are called satellite bands or satellite DNA so what is the satellite DNA so when you isolate the genomic DNA from a cell it has a fixed density for that particular cell or for that particular species and that depends on the nitrogen base content so what is the proportion of ATGC adenine thymine guanine and cytosine the density of a particular genomic DNA depends on that that's why you see the main band this band this is the genomic DNA band which has a particular density there are some DNA sequences in our chromosomes which have tandem repeats you can take a look at our video on DNA polymorphism in which we talk about tandem repeats where a sequence of DNA is repeated over and over again in a DNA stretch for example if you say something have something like this and this is repeated again and again this is called a tandem repeat and these stretches of DNA usually have a different density why because they are often more AT rich or more GC rich as compared to the genomic DNA now whenever there is a stretch of DNA that is more GC rich it has higher density I'm not going to the details of the structures but if you take a look at the structures you will see that guanine is the biggest nitrogen base hence it has a higher molecular weight and hence it has a higher density so when you look at the satellite DNA bands when genomic DNA is centrifuged you might see some above the genomic DNA band and some below the genomic DNA band so naturally the ones above the genomic DNA bands will have lower density right so that will be AT rich and the satellite DNA bands that are below the genomic DNA band or the main band are the GC rich sequences which have a higher density satellite DNA can be of different types based on the length of the sequence that is being repeated over and over again the two major types of satellite DNA that we see very often are mini satellites and micro satellites mini as the name suggests is bigger than micro mini satellites have a base pair length of around 10 to 100 and micro satellites they have a base pair of length of less than 10 base pairs usually both the types of satellite DNA have a high degree of polymorphism as you can find in our video on polymorphism this means that there is a high probability of variations in these DNA sequences why because these sequences are not found in the coding regions of DNA they are found in places like telomeres or centromeres since they're not found in the coding regions the probability of them accumulating in subsequent generations is high because they are not really affecting the proteins that we produce from our coding DNA sequences and because of their high degree of polymorphism they result in a lot of variations that we see around us satellite DNA has applications and various fields a popular one is DNA fingerprinting which we will look at in a different video


headline news update. Hi everyone, Basil Chapman here on this 23rd day of December, just about wrap up time. And we're looking at the Dow 115, it was earlier thanks to Boeing, but now we've got about two or three stocks in United Health, it's one of them, CVX, Triple M. It's getting a few and fewer stocks that are already moving the Dow higher, but it is higher, didn't make an all-time high, 28,608 was the high on Friday, and then it closed at the load, it was one of those merbosa candles where there are no wicks. So it's going to be very interesting because it should be slightly higher highs and I'll tell you why, because the S&P, I don't know how it did it, but managed today to squeak to a new high, all-time high of 3227.78, and this is only a leg B, there should be a peak B in the next couple of days, and then a leg C, and then a peak C, and then a leg D, and a peak D, and that takes you to January, well we'll see, but it could take you to the first week of January, we'll know later about that, the QQQ right now, the QQQ in the X100 hasn't made a new high above Friday's high, so this could turn out to be a dojo candle Friday with an inside bar, and maybe a week, half day Tuesday, we'll see, but 212.13 is the high, the high on Friday was 213, I believe, no, yes, no, 212.52, so up in the all-time high area, but it's not yet an all-time high from Friday's high, the IWM, the Russell 2000, down, it's up 10 cents, 166.07, down from Friday's high, I think it will be making, now this is a peak C, and there's no new high, so we're getting close, I mean, we've got the Dow, the IWM, but if you've got the S&P, only a leg B, I don't know how this is going to unfold, I think we're going to have some rogue waves, slightly higher highs, that's all, and then we start to pull back, and meantime we've got gold, at this particular point, gold is up 6, it's having a nice rally, but within a very contained area, and the irony is that this has become a very quick peak A, peak B, peak C, and a leg D, and it still hasn't gone above the high of the 4th of December, the continuous contract, 1489.9, today's high is 1489.1, and the dollar has been doing very nicely lately, today it is unchanged, it's a 79, 97.69, I'll talk more about all these areas plus some key stocks, in a moment when I do my Tiger technicians hour, coming straight up, and don't forget, the opening call is my daily newsletter, I'll be back in a few moments.


This study investigates the use of locally delivered antibiotic impregnated nanohydroxyapatite, NHATB, beads for treating chronic osteomyelitis. The results showed that the success rate was approximately 98% with no reinfections reported. Additionally, the quality of life of all patients improved significantly over time. Furthermore, systemic exposure to vancomycin and gentamycin was limited while high local antibiotics concentrations were detected in wound drainage fluid. No nephrotoxicity or hepatotoxicity was observed and most adverse events were resolved without removing the beads. Overall, these findings suggest that NHATB beads are a promising option for localized antibiotic delivery in chronic osteomyelitis treatment. This article was authored by Chitawee Jayamton, Adinan Apavakarun, Sari Anuramwat and others.


Do you know that you can participate in deciding how things are done at your university? There are various ways in which students can have a voice in decision-making processes at Tilburg University. They can do so actively as a member of Student Party's Front or SAM in the University Council, as a participant in a program committee or school council, or as a student member of a school board. But you can also share your opinion with fellow students who play an active role. For example, what we achieved last year was a regulation on the timeframe between the publishing of the results of a regular exam and the reset of the specific exam. My name is Bob Bokbeer. I'm studying in my third year of the Bachelor of Dutch Law and my function is being a coordinator of the program committee belonging to Tilburg Law School. It's hard to give a concrete example of what we talk about. It's about policy and budgets, but also about new programs, for example. I'm Lawrence Bolte. I'm a master's student supply chain management at the Tilburg School of Economics and Management and currently I'm the student advisor of the faculty management team. Last year we had this case and the management team wanted pre-master students to obtain five out of five courses in their first half year at the university. We have student party, but okay, that's not really fair. And we just said no, we do not agree with that, what about four out of five? My role entails working in a team of students and professors whose objective is guarding, maintaining and improving the quality of our education. It means that I advise the faculty management team on policy issues. Six times a year we come together with the faculty council and we defend the interests of students. I advise on policy issues from a student perspective. Well you have different stakeholders and different interests and these can clash. During these meetings we set goals for the future in order to improve our education. We are with seven and we always make sure that those seven votes are for the same directions. I'm the linking pin between the student community and the faculty management team. Of course you have your main objective of having good and high quality education. I think I've learned a lot from this political playground. It creates an opportunity to develop a lot of skills such as negotiating. By just knowing it is there you can probably understand organizations way better. You want to say you studied a good program at a good faculty at a good university. My creed is practice what you preach so get into action when it's needed. If you want to make a change you should join that's where you can make a change that's where you can help make the faculty better. Do you want to learn more about student participation at Tilburg University? Please visit our website for more information.


The addition of vanadium to AS355 steel grade improved its mechanical properties, but did not significantly affect the formation of residual austenite or the size of vanadium-rich precipitates. This article was authored by Julia Stornelli, Anastasia Tselikova, Dunyele Marabale Gadia, and others.


Hi, welcome to preschool storytime from the Long Beach Public Library. My name is Ms. Shayna and I'll be your storyteller. I thought we'd start with a song. It's probably familiar to you, so feel free to join along. If you're ready for a story, clap your hands. If you're ready for a story, clap your hands. If you're ready for a story, if you're ready, if you're ready. You should be flying a kite. I'm guessing Storytime's about flying kites. Well, thanks, Storytime Fairy. I can't wait. Today's story is called Stuck. It's written by Oliver Jeffers and published by Philomel Books and Imprint of Penguin. Thanks, guys. It all began when Floyd's kite became stuck in a tree. He tried pulling and swinging, but it wouldn't come unstuck. The trouble really began when he threw up his favorite shoe to knock the kite loose. And that got stuck too. So he threw up his other shoe to knock down his favorite one. And unbelievably, that got stuck as well. In order to knock down his other shoe, Floyd fetched Mitch. Who's Mitch? Cats get stuck in trees all the time, but this was getting ridiculous. Floyd fetched a ladder. He was gonna sort this out once and for all. And up he threw it. I'm sure you can guess what happened. The ladder was borrowed from a neighbor and would definitely need to be put back before anyone noticed. And in order to do so, Floyd flung a bucket of paint at it. And wouldn't you know it? The bucket of paint got stuck.  A chair to knock down the duck. His friend's bicycle to knock down the chair. The kitchen sink to knock down his friend's bicycle. His friend's front door to knock down the sink. The family car to knock down their front door. The milkman to knock down the family car. Did you get up here the same way he says to the cat? An orangutan to knock down the milkman, who surely had somewhere else to be. A small boat to knock down the orangutan.  A rhinoceros to knock down the big boat. A long-distance truck to knock down the rhinoceros. The house across the street to knock down the long-distance truck. Floyd. A lighthouse to knock down the house no longer across the street. A curious whale in the wrong place at the wrong time to knock down the lighthouse. Hi, what are you doing? All got stuck. A fire engine was passing and heard all the commotion. The firemen stopped to see if they could help at all. And up they went. First the engine. Followed by the firemen. One by one. And there they stayed. Stuck between the orangutan and one of the boats. Firemen would definitely be noticed missing and Floyd knew he'd be in big trouble. Then he had an idea and he went to find a saw. Brilliant. He lined it up as best he could and hurled it into the tree. That was it. There was no more room left in the tree and the kite came unstuck. Floyd was delighted. He'd forgotten all about his kite and he put it to use immediately enjoying the rest of his day very much. That night Floyd fell asleep exhausted. So before he did, he could have sworn there was something he was forgetting. What could he possibly have been forgetting? Right. And that was stuck by Oliver Jeffers. Will you help me count and say the colors of these kites? One little kite in the sky so blue. What color is it? Red. Along came another. Then there were two. One, two. And this one is two little kites flying high above me. Along came another. And then there were three. One, two, three. And here we have yellow. Three little kites just watched them soar. Along came another. Green. And then there were four. One, two, three. Four little kites so high and alive. Along came another and then there were five. One, two, three, four, five. And this one is purple. Five little kites dancing across the sky. What a sight to see so way up high. I wonder what else they can see way up there in the sky. You can help me guess by listening to the rhyme, okay? Pretty kite, pretty kites almost out of sight. What do you spy in that bright blue sky? It's shining bright. You can play in it and have fun. It keeps us warm. It's the big round sun. That's right. Sun is also up in the sky. Let's see what else might the kites see. Pretty kite, pretty kite almost out of sight. What do you spy in that bright blue sky? It's beautiful song is something you've heard. Says my flying friend, the bird. Very good. We can see birds up in the sky too, can't we? Pretty kite, pretty kite almost out of sight. What do you spy in that big blue sky? Hazy and lazy. It shades a crowd. It's like a big marshmallow puff. That's white. It's a cloud. That's right. We'll see clouds drifting by in the sky too, huh? Pretty kite, pretty kite almost out of sight. What do you spy in that big blue sky? It soars through the air faster than a train. Its pilot is the captain of a fast airplane. That's right. How many of you have seen an airplane up there in the sky? Pretty kite, pretty kite almost out of sight. What do you spy in that big blue sky? Take a wonderful ride on a colorful afternoon. Strings and a basket on a hot air balloon. That's correct. And those are other things you might find soaring way up high in the sky, right? Great job guessing. This story is called Love is in the Air. It's based on this book by Jonathan Fenske. The cake was gone. All the children had gone home. But balloon was still tied to the table. He was all alone. He started to droop until there came a gust of wind and with it a new friend. Come with me said kite and she rose on the breeze. Wait for me said balloon. He pulled and he pulled until snap. He was free. Hello said kite. Hi said balloon. Wait for me said kite. She pulled and pulled and snap. She was free. At last they were side by side. They flipped. They dipped. And they spun in the wind. They played in the clouds. They rested in the warm sun. It was all such fun. Until the wind stopped blowing. On down down drifted kite. Down until she landed on the branches of a tall tree. Balloon did not fall. He kept going up. He was so sad. He missed his friend. He did not see the bird. And the bird did not see him. Down and down went balloon. He tumbled around and around. It was such a long way down. He was getting dizzy. He was scared. At last he landed. And kite was there to catch him. Thanks for joining us today. I hope you enjoyed some of our stories about kites. I hope you get to go fly a kite this spring on a nice windy day. Take care. We'll see you next time.


So today, I'm going to be talking about this class I teach. It's a typeface design class. And specifically, I'm going to be talking about the way that I use testing as part of the course. Really, it was more like quizzing, but testing sounded more academic. So yeah, I've been teaching a class at Type of Cooper West. The program is transitioning into Type West at the letter form archive, which I'm also going to be an instructor at this coming year. The class I teach is called Principles of Typeface Design. It's a 10-week introductory course. And it's for working professionals who are interested in learning how to make typefaces. Most of my students are graphic designers or product designers. Most of them have never designed a typeface before. And a few of them have never worked with Bezier curves before, even in Adobe Illustrator. We do a lot of different exercises in class, but their primary project over the course of the term is to create an original typeface. As a teacher, I place a lot of value on originality and process both. I really want students to work on something that excites them, something that's new and something that they feel really compelled to create. At the same time, typeface design is a hard thing to learn. Forms have a lot of interconnected parts. And it's really easy to get lost when you're starting out. And so I see my role as giving students the tools and the structure and technique that they need to make something original that is exciting to them. So I first learned how to design type from Jesse Reagan at Type of Cooper in New York. Jesse worked for Tobias when he was starting out. And then after I did the Cooper program, I worked for Tobias. So I'm really lucky that my foundational years were all spent under the Friar Jones umbrella. And the way that I work and teach is very much informed by Tobias' methodology. So as you all know, type design kind of consists of these two parts. There's the drawing of letters. And then there's the negative space that surrounds them. And this spacing dictates the flow and pace of a typeface. And so it's this drawing and spacing together that makes type. And I really want students to work on these two things together always at the same time. They start out with a small character set, which they draw in space. And then they continually add more glyphs to their character set over the course of the term as they keep refining their work until they have a full typeface. I expect them to review these things digitally as they go. But each week, we take time for a more structured review of their projects. And this usually takes the form of a printed proof. When we look at these proofs, we ask really big questions, questions about is this design working. But we also ask these smaller questions about the execution of the typeface. And it's these questions that I'm going to talk about today, because these are the kinds of things that I was able to incorporate into the quizzes I've been using in class. So proofs usually have these big characters, which are useful for figuring out what's wrong with certain shapes and what needs to be changed to make the outline smooth and the weight right, all of that stuff. And they also contain text. Sometimes it's spacing strings, which are what's shown here. These are neutral characters with letters embedded in them so that you can evaluate each letter. And sometimes they're paragraphs of running text. And seeing type in these text settings is really important for evaluating spacing and making sure that the rhythm is what you want it to be. So here is the kind of issue that I would sometimes spot when looking at a student's work. If you look at this O, you might notice that it's kind of deflated, that bottom left part. It's kind of deflated right there. And now you might want to make a typeface that does this, but in my class I really try to steer students towards really clean curves and consistent weight, because that's a better learning experience when you're just starting out. And here's the kind of issue you might spot in a proof when you're looking at spacing. And so here you might notice that any time there's a lower case A, it's crashing into the character that precedes it. And that's because there's not enough space on the left side of the A. So type design consists of a lot of this process of drawing and spacing type and then refining your work, proofing it, looking for things to fix, refining those things, proofing it again, and you repeat and repeat until you have something that hopefully doesn't have any more things to fix. And so typeface designers, when they're going through this process, they usually have this kind of mental checklist, or at least I do when I'm looking at work. You think about is the weight right? Is the contrast right? How are the character widths looking? How's the spacing? And I really wanted to find a way to help students also develop this kind of thought process as well when they were looking at work. So both my parents are creatives and have taught at design schools, so I grew up really familiar with kind of the classroom, studio, environment, crits, all of that, and I love it. It's a great way to learn design and it's awesome. But I also had a fairly traditional high school and college experience, so I also am really familiar with standardized tests and multiple choice exams and things like that. And I kind of like both things. I think that each have their roles. And when I was thinking about how I wanted to organize my class, I realized that a lot of the things, a lot of those skills where you're looking at type and you're evaluating it for inconsistencies could kind of be tested and quizzed in a structured way. And I thought this might be informative for me as a teacher and for my students. So to put this in context, the class is three hours long and the quizzes I give are 10 minutes long. I really want them to be pretty short and I want students to go through the questions quickly and not linger too much on any one thing. And I should say that you can't learn type design from taking quizzes. We've been the vast majority of class time kind of discussing big concepts and doing hands-on work, but these quizzes have been really informative for me in certain ways. And so the return on investment, given the time that they take, is pretty big. So what do the quizzes look like? This is like, the first questions on the quiz often have to deal with factual material that we've covered in previous classes. So questions like, what is the primary kind of contrast in this character? Or questions about the reading to make sure that students have done the reading. Or this really cruel question where I make them calculate the coordinates of a point on a Bezier curve. They really don't like this one, it kind of unnerves them. But so these questions are pretty straightforward there. The kind of things that people are used to seeing on any kind of test or quiz. But the majority of the quizzes deal with picking up on those inconsistencies of drawing and spacing that I showed earlier. So let's see some of those. So this is a question that I like to ask when students are beginning to work on their typefaces. And right now we're looking at the baseline overshoots and we have a cap I, V, I, O, I. And one of these characters does not have an overshoot that is quite right. You maybe have noticed it's the V. It really should be a little lower so that it appears even with the baseline. Here's another question that I've asked before. He were looking at weight. And one of these letters doesn't have quite the right weight. You probably picked up on it. It's the cap O was a little bit too light. And so I really try to time these questions with where students are in the design process. So like this question about weight and that previous one about overshoots, those are things that are really important when you're starting out a typeface. You really want to get those things right at the beginning because as you go on, other things will come together more easily. And so I would ask questions like this during the first few weeks of the class when students are starting their projects. Now, here's a question I would ask about drawing. And here, these questions I tend to phrase something like which of these characters is, in which of these characters is the weight most consistent? And here you might have noticed that those two Ms have this weird kind of like tapering and swelling going on that happens nowhere else in the letter. So the left one seems, I would describe that as the most consistent. And I sometimes ask questions about punctuation, like which of these periods is drawn most smoothly? And you can probably see the furthest left and furthest right, both have these kind of lumpy areas there and there. And so that middle one kind of has the smoothest curves. And I try to divide the questions pretty evenly between drawing and spacing. And so for the spacing questions, it's a very similar kind of structure. Here, there's a letter that has not quite right, side bearings. And the students are kind of asked to detect which letter that is. You may have noticed that there's a little too much space around those lowercase n's. This is another kind of question that I would ask at the start of the course because it's really important that students get these letters, the cap H, cap O, lowercase n and lowercase o, spaced correctly because then the rest of the spacing kind of comes together more easily. So I ask variations on this one with this exact set of letters a lot at the beginning of class. Here's another spacing question. Here we're trying to detect which of these letters has something wrong going on with the spacing. And you maybe can see that that lowercase s has kind of shifted to the right on the body. If you look at it between those two o's, it's way too far to the right. And this also raises another interesting question. Some of you might have seen that lowercase v and seen that when it's next to the o's or when it comes after the n, it looks like there's a lot of space there. But that's actually a kerning problem. And when students are starting out working on their typefaces, there is no kerning in them. So it actually is important to me as a teacher that they can differentiate between a spacing issue and an issue that shows up because there's just no kerning in the font yet. So those are, at first they see these as trick questions, but it's a good skill for them to pick up on for sure. And here's a kind of question I might ask towards the end of the course. And here again, there's a spacing issue and they need to pick up on what it is. And if you look at irredeemably in the top right, you might see that these e's just look a little loose, like everything looks a little too open. And so that's the problem there, is there's too much space to the right of the e. So one of the reasons I started doing this is because when students are working on one project over the course of the term, they are seeing one style of type over and over and over again as they work on it. And I wanted to see if, as a teacher, I could check in with how generalizable the skills they were learning are, cause I want them to be able to take the skills that they're learning in the class I teach and then apply them to other styles of type. At first I started making these quizzes like in Adobe Illustrator, but it got to be, it took too long. So in the end I ended up making tools that they kind of build these graphics. And they're pretty straightforward. I basically take an existing type base that I think is well done and I break it in some way. So the script will move points around randomly to make those drawing error questions. And it will generate text with kind of an altered side bearing on one of the letters for the text questions. So it's definitely possible to ask a bad question. It took me a while to get the error amount into the proper range. If the letter is too close to the way it was originally drawn, then the question's really hard and no one can pick up on it. And if it's too different than the question's too easy and it just doesn't really serve much of a purpose to inform me of anything about what the students are learning. This is also complicated by the fact that these quizzes are done on screen. I use Google forms to build them. So all the students have their own computers and it was tricky to figure out the right amount of error to put in the questions. And this is true of spacing as well as drawing questions. And perhaps most interestingly, it's also possible to just ask a question that's not answerable. This is a mistake that I made early on in giving these quizzes. I asked whether in the text here, whether the straight sides were too loose or too tight or whether the round sides were too loose or too tight. And you can see there's clearly some kind of issue going on. The left side of this text looks much tighter and the right side looks much more open. So either the straight sides are too tight or the round sides are too loose. But because there's only straights and rounds in this text, there's just not enough information to answer the question. And what was interesting about this is that almost all of my students answered this and said that the rounds were too loose. But if we're basing this on the way that this typeface is drawn, it's actually the opposite. The streets were made too tight in that question. This is a typeface that's made for text so the spacing is pretty open. And I found this really interesting because one of the challenges of teaching type design is that it's really hard when you're starting out to anticipate how something is gonna look at a small size when you're working on it at a really big size.  But then when it's at a text size, that period is too small. And this question to me was sort of like the negative space equivalent of that punctuation problem because when students are working on a typeface and it's large on their computer, it might seem like that tight spacing is gonna look fine. But when it gets small, it's just gonna, everything's gonna crash into each other. And I had noticed that my students were spacing everything really tight at first, even if they thought it was supposed to be for text. So asking this bad question kind of helped me to think through why that was happening and I worked through it. So I track students' progress over the course of the term. This kind of helps me to see where students need help so that when I meet with them in class, I can kind of tailor my time with them towards what they need help with. And on top of that, I find this to be fun. I don't know if my students feel that way, but to me it's fun. And here are a few of their projects. All right, thank you very much.


Just wait for it to be connected. It should be recording now. OK. Hope the connection is more stable than yesterday. Let's hope for the best. So welcome back, everybody. We start today's class by a quick recap of what we had yesterday. And as always, we start very gently. So the first fraction of this lecture would be devoted to reviewing the material of yesterday and taking a look at one simple example of value iteration. And then we move to the bulk part of today's class, which is proving the Bellman's optimality equation, which you see here in front of you in red, from a different angle, using a entirely different set of techniques and approaches, which are somehow more geometrically intuitive about what is happening with this Bellman's optimality equation. And we also be useful very far in the future. So let's say some 10 lectures from now, and we will have to introduce special kinds of algorithms to deal with very complex situations in robotics that will leverage on what we do today. Of course, in due time, I will refresh your memory about what we are going to do today. But keep in mind that the material that we cover today is going to be used in the following lectures as well, which is usually the norm. OK. So one of the key findings of our last class is that we can write down for the discounted objective. So I recall you that our objective here has become to optimize this quantity here, OK? The expected value of the sum of the rewards from now to infinity with the discount, geometric discount factor, gamma, which I repeat, summarizes the notion of an horizon in a sort of stochastic form in the sense that you can understand gamma as the probability of death for my process at each time step, OK? So at every time step, the process gets killed with probability gamma. That's the same interpretation as in the more economical term of having a discount factor over the value of things that will happen in the future. All right, so given that objective, we introduced the value function, which is here at the top of the screen. And then we ask for maximization. And then after a little set of manipulations, most of them are of formal nature, some inequalities. And then we end up with the Bellman's optimal equation, which is written here. And then we face the question, so how do we solve this? No linear equation. So we have a vector in a finite dimensional space because now we are dealing with a finite set of states, discrete and finite. So it's a vector, our optimal value function v star. It's a vector with components v star s for each state of the system. And therefore, the Bellman's optimal equation is an operator, no linear operator acting on this vector and returning another vector. And the equation is just solved when there is an identity. So when we apply the Bellman operator to our vector, it returns our vector itself. And then, which is the equation that I wrote here, and then the key property that allows us to know that there is a unique solution to the problem is that this Bellman operator is contracting in some norm. And the simplest way to prove it is to use the L infinity norm or the total variation norm, which means that to measure distances, norms of vectors by taking the modulus of each component and the maximum over these components. So after, again, a little tour of inequalities, we were able to prove that the Bellman operator is contracting with a constant, which is gamma, which means that if gamma is small, our algorithm converges very quickly to the algorithm that I will describe in a second, so the contractivity and ensures it to converge very quickly as it should. And when gamma approaches 1, it takes a lot of time because the contraction is very slow. But that's understandable because you have to reproduce the behavior of a very long time. So in the following, the example I want to take will be instructive in this sense. OK, so given that the system is described by the fixed point of a contractive operator, there is a very simple algorithm, which is the fixed point iteration or value iteration in this specific case, which is very straightforward. You start with a guess, and then you apply the recursively the Bellman operator. So you map vectors into new vectors, and then you go on and go forth until you get the variation in these two vectors defined in absolute or relative terms is smaller than some tolerance. And then you are happy, so my algorithm has converged. And then from that value function, you can instruct your policy at the final step. And this is arbitrarily close to your true optimal solution depending on the tolerance. So let's see how this works in a concrete example, which is, I hope, will be somehow enlightening. So value iteration example, we're going to do this for GridWord. So we're going to take a very, very simple instantiation of GridWord.  Not surprisingly, so it's sort of how Picasso would draw a grid probably, but your imagination will rectify everything. So the goal here is to identify a reward which is placed here. So there is a single point in the map which gives some reward R. There are no other rewards around. I'm defining the structure of the problem. So I'm repeating again, sort of going slower here. So the states are each of these points in the grid. Every tile on this map is a state. So what are the actions? Well, the actions are the steps that you can make in every accessible neighbor. Let's say that we move along the verticals and horizontals to diagonal steps. That's just for simplicity. So these green objects here, these are the actions. So for simplicity, again, we consider, in this case, a situation when the system is deterministic, what it does mean. It means that the probability of being in state S prime, given that I start in state S and I take action A, which is an admissible action, is just, sorry, I should write better here. This is just, OK, let me draw this symbol and then explain what that is. 1 of S prime equals S plus A. This one means that every time that I'm writing the following, it's something like 1 of some condition. This means that this object is 1 if condition is true and 0 if false, OK? If you're more familiar with that other notation, I could have written this in terms of a Kronecker delta. This would be a Kronecker delta of S prime S plus A. This is also called the characteristic function of a set, if you wish. Same different names of the same thing, OK? I tend not to use delta because delta, as we discovered in the following, delta has one particular interpretation in reinforcement learning, which is the temporal difference error. So in order to mix up notations, I tend to use this notation for one. So what does it mean? The term is that if I take the action of going east, I will go east by one step. And if I want to go south, I will go south. So there are no errors in the implementation of the actions. There are no execution errors. There is no stochasticity in this part, OK? So given that there is a single reward, OK, this is a situation in which we can basically solve the decision problem just immediately. What's the solution? Well, suppose you start from a certain state here, any point on this matrix. So what is the best outcome for your problem? Remember that our goal is to find the best expected discounted sum of rewards, OK? So this is the general expression. Now, as you've seen here, the structure of the problem is such that, as a matter of fact, this doesn't depend on the actions, and it doesn't depend on the new states. It just depends on where you sit, OK? So it's a very simple structure of the reward function. And as a matter of fact, this is only this object here is, again, 1 if st is equal to this target position, which let me call s, I don't know how to call it, but let's say s hat times r, OK? So this reward object, it's a vector itself. It's a matrix over the space of states. And takes value r, if I am sitting on the target, and takes value 0 elsewhere, OK? But then you easily realize that what is my value of g? Well, it's just given by the time I need to get to the target. So if I take this path here, suppose that sometimes I produce this path. So what is the value here? Well, it's just the number of steps I take to get there. So in short, if I start from a state s node, my g is nothing but gamma to the distance, if you wish, in the sense of the so-called Manhattan distance, so the distance along a squared grid between my target point s hat and my initial point s node. So if I have any sequence of actions, this will be my g. There is no stochasticity here because everything is deterministic. Very good. So there is a r here. Sorry, can you repeat the question? There should be r. Yes, thank you very much. There's an r as a prefab. Thank you for pointing that out. Of course, the key thing is that the best strategy is to minimize d. So this problem, this problem of grid wall with a single point here, is equivalent to finding the shortest path from any point to the target. So this is one other way of rephrasing the problem of shortest path. So we are casting another classical problem of computation, which is finding the shortest path. Of course, I'm writing this for grid wall, but you could write it on any kind of graph. So any problem in a graph of finding the shortest path from point a to point b, given that you know all the ways that the transition probabilities occur, so all the edges and what other problems you can take in different edges, can be cast as a problem which has a Bellman equation and can be solved by value iteration. So this is both simple because geometrical intuitive, but it could live in abstract graph spaces. OK? Very good. So let's see how good value iteration work here. So if you remember, the algorithm starts by choosing a guess for b node. Well, here we are totally agnostic, and let's say that our b node is 0 everywhere. It's a very bad choice. I mean, because you know that if the target is there, you know where the target is because you know the structure of the reward.  So what does it mean? What is the value function? Well, the value function, remember, it's a vector of the state space, which means in this case that you put a real value in every point of this, OK? So for instance, if you have a point here, here there is a value of this state s here. So if this is your state s, this will have value s. So every slot of this array has a real value attached to it, which is your value function. And you start with a guess, which is everything is 0. And then we apply the first step of the value iteration algorithm, which says, well, v1 is going to be the Bellman operator acting on v0. So let's go component by component and write down again what is this meaning. So my next approximation at any given state s on my grid is Bellman operator max over a, sorry, sum over new states s prime, p of s prime s a. Then I have my rewards in state s plus gamma v0. OK. So OK, let me do just one little correction. So because for the logical purposes, it's best if we say that this is a small modification. Let's say that we have r s prime here. So now I'm explaining why I'm doing this just in a second. So let's consider this question where it just depends on the side you lend on, OK? Not from the one you just left, but from the one you lend on, OK? Just one step ahead. It's just because it simplifies the things and the speed of the map, and this will be clear in a second. Sorry for not having realized this earlier. OK, so do we all agree on that? This is the first step of our algorithm, which means that we have an input on the right, which we know well, OK? Inside of the square bracket, we can do this computation. It's a linear combination. So we can do it very fast over our r a. And then we have to take the maximum of all possible a's. So we do this. We repeat this for all possible actions. And we take for every state, we take a sweep through these values and select the a, which is the largest one. And then we obtain the term on the right, OK? So what we need to do in practice? So at the first step, let me have a look at this. So at the first step, the matrix was everywhere 0. So v0 was 0 everywhere. What happens at the step number 1? I'm not painting, I'm not drawing all the grid points, OK? But you know that here around, there is the place where the r stays, where the reward stays. So somewhere here, we have our reward. So what will happen? OK, remember, by choice, this object is 0. So this goes away. And then what we have to do, basically, is consider that only the point where the target is contributing to this sum. To what does it contribute? Well, it contributes. It may contribute to all points as, OK? So this is the point where r is. We can have a contribution to all points or close points which are accessible, from which the target point is accessible, OK? So because remember, this object here is going to be r times s prime equals s hat. So in order to insist this object here is 1 of s prime equals s plus a. The only possibility of having this object not being 0 is that if you combine these two expressions, this requires s to be s hat minus a, OK? Because there are two delta functions here. So both conditions must be verified in order to have 1 and not 0. And if you just combine these two, well, this means that s must be s hat minus a. So this means that the only states that can contribute on the left, the only entries that appear on the left can only be these ones. And these are the ones from which you can get, OK? So at your next step, your v1 will be everywhere 0, except on these points here. Next step, well, let's repeat. v2s is equal to the maximum over a. So yeah, just one thing before we move. So you notice that in this case, what happens here is that we are just propagating the value of r. So I can even put the actual values here. This will be r, this will be r, this will be r, and this will be r. So after my first iteration, I can say that around my target I will get r. This is clear because at the first step before reaching the target and from that point on, of course, you just then get the final word and then nothing else happens later, if you think in time, sort of. This will just get the reward at the next point. So what happens at the next step? Well, we have to repeat this again. And again, taking the sums here, this will be just explicitly 1 over s equals s hat. Sorry, no, this is not correct at this stage. Let me write it in full again. So we are repeating the same argument as before with the same things. But now there is v1 here. And the consequence is that now we have to look at all points from which we could have reached r, any of this. So this propagates back in the sense that now we have it could have come from here. It could have come from here. It could have come from here. It could have come from here. So this allows us to fill at the next stage new values here and here again. And this expands all over. So every time that you make an iteration, you move away from the target. And you fill in with new values for the value function. So at some point, this sort of spreading out of the values, we reach the boundaries in the system that will start to converge towards the final optimal value. So this is an exercise that you can try to do by yourselves. In some form or another, maybe for Gridworld, maybe for another system, we will do explicitly value iteration in a system in the tutorial session, which will take place on April 9. But this is just to give you an idea of how it progresses. Any question on that? Yeah, so one question, if I can. Sure, please. About the notation in the last equation, V2, didn't get the point why R is of S prime, and then V1 is again on S prime and not maybe on another S. Like I'm in another state now. OK, so the argument of V must be always S prime, because that's the kind of recursion thing. So the value now depends on the value at the next state. So this was always there. You see here in the Balmain's equation, it's always S prime, where you have the next value to be evaluated. And here, the only thing that changes is which kind of form you choose for the dependence on the triplet. And here we're saying that the reward depends on the place where you step in, rather than the place from which you left. OK, because we cancelled the S and the action. Yeah, we just assume that it depends on the place you land. Just to be clear, if you assume that it's from the place you start, it doesn't change much. It's just that you have one additional step in the iteration, which makes it a little bit more cumbersome, but doesn't change them. You can think about it. You will realize that it's a minor modification. OK, that was another question. Yeah, no, I just wanted to clear something. So in the first, the V1 in the first step is R on the adjacent tiles of S hat. And zero elsewhere, including S hat. Yes, because there's no way to get there. Because we did not have the action stay there. OK, so we have the action in the same place, then we would update that as well. OK, OK. Any other question? Just a small one. And our goal, we define it as the sum from t equal to 0 to the infinity. But in this case, if we know the environment, so if we know the grid world, it wouldn't be better to impose an horizon like the maximum distance from one corner to the other one. Because maybe the agent could end up inside an infinity loop without never reaching the reward goal. Well, so let me try and understand what your concern is. Because here, this is a problem of planning, OK? So first part of the answer, you can define this problem in different ways, OK? So this one of using the discounted version is one of them, of one of many. So you can define this with a finite horizon if it's long enough, OK? Because if it's shorter than the time it takes to reach the target in the shortest time, then you will not be able to learn anything there, OK? So the choice of a horizon or a finite horizon requires some priori knowledge about which is the longest of the shortest path to the target, OK? But you could define that as well. That's perfectly fine. You can define this shortest path problem in other ways. For instance, you could put a cost every time that you make a step and you are not at the target. And then make the target an absorbing state, a terminal state, OK? This is another way of setting exactly the same problem. The solutions will always be shortest paths. So there are several ways of casting this problem. But the second part of your question is one that I think it's more important to pay attention to. Remember that this is a planning problem. So the agent is not doing anything. It's just looking at the map and saying, OK, just right before I start, I want to compute what's the best thing to do. So it's really like what Google Maps does for you when you ask for an itinerary. Nobody's sitting in the car. Nobody's switching on the engine. You just, beforehand, you say, let's imagine that. I want to go to Milan now. What route should I take given the current conditions? The map, the envelope, the road, et cetera. And this is a way of solving the problem. So there is no such thing as you said, at least maybe I said correctly in the case of please correct me, such as the agent cannot make it in time and can be caught in an infinite loop. Because unless your map has some position which is isolated from the target, if there are barriers which separate one state from the target, that clearly there's no way of learning the last part. Did I answer your question? Do you want to answer? Yeah, no, I got it, I got it. Thank you. Sure, no problem. Any second order question in the sense that what I said elicited some other doubt or question? I'm thinking something that in this case, so we are able to solve the problem because the setting is that we know exactly where the reward is so that we can do the backward path and determine the best path. But a situation in which we are not certain about where the reward is, where we want to go. How would that be presented? Like, we don't have a reward in a single place or we have, I don't know, I can't imagine. I don't know if I explain myself about the. Sure you did, sure you did. So this basically leads us to jump ahead by several lectures in the sense that this will be the focus when we deal with the model free situations. Because knowing the map means knowing the model of the environment, knowing what to do. So your question basically is about what I do if I don't know where the reward is. I don't know, maybe I know where it is, but it's a stochastic object, which may be on and off. There are several variants of this problem. So all of this boils to this finding model free solutions of the optimal optimal strategy. I can anticipate without, of course, giving you any detail because it's too early. But what we will do to deal with this kind of problems where you don't have a map, like I said, qualitatively, the idea is that you have to use experience. So you have to go around by trial and error. And as you move, basically, you get the amount, feedback about the environment. Either you build the map, OK, this is a way of doing it. So you collect information, you say, OK, there was a reward there, let's mark it down. And you construct your model and you use, you sort of can solve the Bellman equation approximately using your approximate map and your approximate model of transitions. This is one way to go. But there is an even more radical approach which bypasses this construction model, which in a nutshell amounts to saying that you can solve this equation here, the Bellman equation, without knowing this. How? Well, let me just use this as a teaser. Well, you can, OK? You can. You will have to replace, of course, the knowledge, the a priori knowledge about these things with some empirical knowledge. So you will have to sample the environment and at the same time, trying to solve the Bellman equation. That's what temporary difference methods do, OK? OK, that is an interesting teaser. This was a sneak peek of something that will come into time. OK, any further question? So if not, since we are going to have take a very long walk in the second half of the lecture, so it's derivation which doesn't require any super special knowledge, but lots of things from Markov processes and linear algebra will come in the way. So my suggestion is that we take the break now, a little bit earlier, and then we reconvene at, say, 5 to 10, 9.55. And then we make one single long stretch until we're done. OK, so go ahead and prepare your coffees or teas or whatever. See you in 15 minutes more or less. OK, thank you. Sure. OK.


So, this new talk is the talk of Patrick Malboer. He will explain how to split a monolithic application with the repository and the rest, okay? So, good luck. Thanks. Yeah, hi, welcome to my talk. I'm Patrick. I'm working as a software developer at Blyanda. And if you came here because you only read the title without the abstract and like I do sometimes and are now expecting I will show you how to split a monolithic application into a microservice architecture, then I have to disappoint you. That's not what I'm talking about. I want to talk about what we did with our application which was, which consists of various Python packages and we had all Python packages in one repository. And at one point we decided, yeah, we want to have one repository for every package we developed. And yeah. So, let's start with this easy example how a Python package usually looks like. You have, for example, this structure. You have the actual package, my super library with this standard init module in it. Maybe you have a requirements file. You probably have a setup.py file. And of course you will have tests. And for us it looked like something like this. So we had lots of these packages in one repository. What we also had was this one requirements file for the whole application. And this was probably the worst design decision at the beginning. So even for the unit tests of every of these libraries we just installed everything and used that in our virtual inf to run the tests. And we had actually a lot more than four of these packages in it. Yes, so why do we even want to split it? So there were various reasons. One of the reasons was other teams of our company started using some of the libraries in their own projects and they always complained about, okay, we want to contribute, but every time I have to look into this big repository it's such a pain to get it just running whatever. So yeah, this was one reason. Another reason was, okay, I called it spaghetti code here. It's this, what I mean here is cross dependencies. So you, for example, in library one you import from library two even if you don't actually want it at the end because it's not well structured then. Yeah, so as other teams started to use our libraries we also had to release them. That's also easier if they have their own Git repository. And we also wanted to use something like setup tools SCM to get automatic versioning. So if you don't know what setup tools SCM is it creates versions for your Python package out of your Git versions or Macular versions. So usually I set a py file, look something like this and you have this version keyword argument where you have to manually adapt the version string every time you want to do a new release. And setup tools does this for you automatically. So here you would just say, yeah, my setup requires to setup tools SCM to be available. And I want to use this SCM version, that's here. And what such a version then looks like is something like that. So here we have one commit after the latest tag. So before I set a tag 0.0.1 and this step one then says, okay, you are one commit after the latest one. And at the end you have after plus and G the start of the current Git commit hash. Okay, so talking about Git commit hashes. So we now decided we want to split our monolithic Git repository, but how do we do it? If you just move certain sub packages somewhere else and then initialize the repository from the start we would lose all the history we already have. But there's, if you use Git, I don't know if there's something similar in Mercurial or any other SCM, but with Git you have subtree and subtree has another sub command split and split creates a new history of commits for a specified prefix. So here I have as prefix library three and if you also specify a branch name it will create a new branch which exactly has this newly generated history. What you can then do is create your new package and initialize it with Git and then you can pull this branch you created before from the monolithic repository. Now you have a new repository for your new library three and with all the history which affected library three. So okay, this has to be done for all packages in our monolithic Git repository. What then changed for us is how do we do our continuous integration workflow because before we had just this one repository it didn't matter in which package we made changes for our next feature as we used Jenkins. Jenkins just checked out the latest commit, every change in every of our packages were available and yeah, that was easy. That's one of the advantages if you have monolithic applications, architectures, whatever. What we then did is okay, now we have lots of Git repositories. Let's just check out every Git repository at the beginning and create our application artifact out of this. This ended up in a really messy Jenkins job. So if you don't know Jenkins, Jenkins has lots of plugins. One of those plugins is the multiple SCM plugin. There you can specify multiple repositories which should be checked out at the beginning of a job and yeah, we did this for all of our extracted libraries and had this huge list and this was horrible. It was really horrible when we had to do bug fix releases. I mean, can you imagine how hard it is if you have to configure all this? Okay, so in Jenkins you have to specify which tag branch or whatever has to be checked out. So if we wanted to do a bug fix release we had to specify the tags or commit hashes which were used back for our release so that we only changed the repository where we had to do the bug fix release. And I think you can imagine that this was really horrible so don't do that ever. What you actually want to do of course is just use your libraries in your application just like any other library. And the problem with that was if you add your libraries to your application's requirements, TXT, every time you change something you would have to raise the version in your requirements file because you pin your requirements of course. And yeah, that's also not a good workflow cause it happened quite often that we implemented new features in our library packages then the unit tests for this library passed then we thought, okay, we can do a release but when we actually used the new version in our application we saw, oh, no, that's not working at all and you want to have this feedback a lot faster. So what can we do here? We came up with this workflow so we run unit tests of our libraries. If they pass we let Jenkins upload a reel to our internal DevPy server and at the beginning of our application job we just install those with the minus minus pre-option of PIP install. With this option you can install pre-releases so beta releases, alpha releases or those Dev releases you saw earlier created by setup tools SCM. So we always had the newest versions in our continuous integration pipeline of all libraries, very new, at least the unit test pass. Then we created another job for actually doing application releases. So if you want to do a release of our application we now have to do releases of all the libraries where we know, okay, with this version it works and can then run this extra release job where this minus minus pre-option is not used. Okay, I mentioned DevPy, who of you knows DevPy or uses DevPy? Okay, not many hands. So DevPy server is a PIPY server. We use it at Bluioner for our internal packages but you can also use it on your laptops. So it's also just a mirror for PIPY and for example, if you are on the train and want to hack but don't have internet connection and have to install a package you could do it offline if it's already cached. Yeah, you can whitelist, blacklist that package and do lots of more things. Another thing is requirements pinning. Who of you ever had this version conflict exception? Okay, my colleagues and a few others. So what we agreed on, we said we don't pin the requirements in the setup py file. So you have this install requires cause if for example, library A requires requests greater 1.0 and smaller than 2.0 and another library wants to use the next feature of which came with request 2.0 for example, then you get this annoying version conflict error and most often it does not make sense even giving a saying, okay, I want to have smaller than 2.0 and yeah, the application is then responsible to use the correct requirements and this will avoid lots of these exceptions. One other comment, so we don't pin the requirements in the setup py file, but we pin them to have a special set on requirements for running the tests. So another developer can then check out tech 5.0 for example and see with which requirements the tests actually passed back then. Okay, so now we have all those repositories. What did we actually gain? So we can use setup tools SCM for every repository. Don't, we have happy library contributors. But on the other hand, for us application developers, it got a lot more complex. So now if you develop a new feature, sometimes you have to do changes in three, four, five repositories, you have to keep them updated all the time and that can be really annoying sometimes, but the quality and the structure of our code really improved. So now we have defined the requirements which every library needs, just this minimal set. So it does not happen that easy that we have this ugly cross imports which you don't actually want and so on. And that's actually a point. Now that you have a cleaner structure, it might be easier to see, okay, which components change with the same speed or something like that and where might be an extra service, where might it be good to introduce a new service with only that library packages. So maybe that, so I think what we did up to now is a step before actually getting to microservices. So, yeah. And I think that's all for now. If you have any questions. Okay, so thank you for your talk. If you have some question for another friend. So thank you for your talk, it was a nice experience. We have a pretty the same problems in our project. And so you mentioned that during the deployment, your Jenkins job, your Jenkins deployment is a simple peep install and it always installs the latest versions, right? So, I mean, it is the latest development versions for CI and the latest stable versions for the production release. What if you deployed some broken package to the production and you need to roll back to the previous version? How do you deal with that? Since I don't think you pin versions during your deployment. So how do you do roll back to the previous version? No, no, that does not happen. So we don't pin what the libraries want, but we pin in our application and we also then install with minus, minus node apps so that we don't get any recursive requirements. I'm not sure if I understood your question correctly. So do you pin versions of your libraries somewhere during the production or deployment? Yes, of course. So, like I said, in the libraries we don't pin in the install requires section set up pie but we pin our, we have an extra requirements file where we pin the versions and we use this versions to run the tests for the library and in our application we then pin, we want library is like 5.0 and so it cannot happen that something else comes in. Okay, we can talk later, maybe. Thank you. You mentioned that you have dependencies between libraries, so library A imports library B and each other. So how did you resolve it? That depends on how, it depends. Sometimes code duplication is better than having this cross dependencies. Yeah, that's basically it. It depends on how this cross reference looks like so I don't have a special example right now, sorry. How about depending one library on the other? So for the library A you require library B. Okay, sometimes it's not a problem at all but we noticed that for some, in some cases we had libraries which depended another library and this one again required lots of others and we had again this big unstructured thing. So, yeah, I don't have the... Did you consider using Git sub-modules for your problem like for pinpointing versions and then being able to have different kind of release branches for that? We said from the beginning we don't want to use Git sub-modules. You tried?  Okay, because we are doing that. That's the way how we can pinpoint different versions and do like even, yeah, hotfix branches on a super repository, so maybe we can talk. If it works out for you, great. Hi, you mentioned something about the code quality like by following this refactored code structure you had some improvement in the code quality. I'm just curious, like how did you measure that? Do you measure some kind of cyclomatic complexity or something else? Like what were your metrics to say that your code quality was improved? Okay, I have to admit that's just a feeling I had. No, that came, it's difficult to explain. Maybe we can talk after that. Another one? No? Okay, thank you so much, Patrick. Very well.