Continuing with the testing hypothesis, continuing within French statistics, we are moving forward from t-test to analysis of variants. In t-test, we were comparing two groups. But in analysis of variants, we then compare more than two groups and more than one independent variable. Analysis of variants is a hypothesis testing procedure that is used to evaluate mean differences between two or more treatments or populations. In psychology and in social sciences, there are many conditions when we have to compare more than two groups. For example, we are comparing treatment effects of different therapeutic techniques on depression. We can compare different teaching methodologies and their effect on performance. We want to compare performance of employees in organization working in different groups. So, there are many conditions when we have to compare more than two groups or more than two conditions. Inova helps us there. While running Inova, we must decide between the two interpretations. Number one, whether the differences between the populations exist or are there or the differences are not there. When you are comparing three or four groups and you are running Inova, you will be interpreting whether the differences exist between those groups and they are because of the treatment or if the differences exist, they are not because of the treatment, they are because of the other factors or maybe random error or within factor variables. When our researcher manipulates a variable to create the treatment condition in an experiment, the variable is called the independent variable. Now, we have to compare our groups. For example, let us say that driving conditions affect your mobile phone in driving performance. So, the researcher is manipulating the variable which is the use of the telephone during the driving. So, here we did not have three conditions, the researcher and I have taken this example from the Gravator. In one, they are not using any phone, they are driving in a driving simulator and the researcher wants to see the driving performance. In second condition, they are using phone but they are using hands free or in third condition they are actually using phone and they want to see how that affects the driving performance. So, when we manipulate the variable, it is called basically independent variable and it is categorical and we have manipulated it by ourselves. But there are many conditions when we compare the groups but actually we have not manipulated the variable by ourselves. So, non-manipulated variable to designate groups, the variable is called a quasi-independent variable. This example can be seen that we want to see that if I want to study something, then what is the effect of the learning of the age level? So, I am comparing three groups of different ages, in which I have eight-year-old children in a group, in one group I have eight-year-old children and in one group I have ten-year-old children. So, I have three groups of six years, eight years and ten years and I want to see the learning performance or the learning curve of these three different groups while I am teaching something to them. So, now this variable is actually non-manipulated variable across different groups. So, this will be called a quasi-independent variable. You can always remember that for ANOVA or for T-Test, for quasi or for experimental groups, independent variable is always categorical, that is, it has different levels, like we compared male and female in a team, so gender was variable, which was categorical with two levels. And here we are comparing driving use of phone, that there are three conditions, three groups and we have manipulated them, so they are categorical, that is, group one where they are not using any phone, group two where they are hands-free using group three, which is they are actually directly using the phone. So, using a phone is an IV, independent variable, which is categorical, which is manipulated variable. And dependent variable is always our continuous and our score is like we want to see driving performance, or we want to see the learning curve of the children through a test, or we want to see the running score and we can compare it. ANOVA is a very robust test, we will go ahead and talk about how many probes there are and how many options there are and how many different conditions ANOVA is the solution for us to test our hypothesis. So, in ANOVA we can compare more than one factor as well, just to first say that in this driving condition example, I want to also compare the three conditions, what is the role of gender, how use of phone is affecting driving performance in different two groups of gender, that is males and females. Now, I have two independent variables, number one, driving, which has three levels and number two, my independent variable, gender, because I want to see the effect of use of phone and gender on driving performance. So, if there is one independent variable or one factor, we call it one way analysis of variance and if we have more than one factor, which is two factors, we will call it two way analysis of variance. But our independent variable, we also call it factor, we also call it independent variable and we will go ahead and talk about it. A study that combines two factors called two way analysis of variance or factorial design or two way ANOVA, a study that examines only one independent variable is called single factor design or one way ANOVA. ANOVA can the testing hypothesis may be the same assumptions or the same logics as we do in T that mu one is equal to mu two or there is no difference in the groups a priori, that is our three groups, they are equal and the alternative hypothesis is that at least one group is different from the other two. We can also say that mu one is not equal to mu two and mu two is not equal to mu three. ANOVA we use variance to measure sample, mean difference is just a team we compare means and we do mean one minus mean two and difference. But in ANOVA, we are actually comparing means but we are using variance to compare means. So the test statistic for ANOVA uses this fact to compute the F ratio. What we do is we calculate variance in it. I will tell you shortly what is the logic behind ANOVA. But we calculate two types of variance in it. Number one, which is because of our treatment variability. We manipulated a factor and we wanted to see what effect it has. What variability it takes in driving performance use of phone. And one is our error variance or within group variance. For example, if a group in which there are ten people and they are using all of them, their performance rating will be different. So that is called within variance or error variance. So we calculate the F ratio ahead and divide between variance with within variance or error variance.


We move on to our final panelist for this afternoon, final paper, Alistair Sutherland. He recently completed his PhD at the University of Leicester, looking at non-armorial personal seals. And he's now working on the Ark Hives Project at Copenhagen, Cambridge, that I mentioned earlier. But today he's going to be talking to us. Alistair, I can't actually see you. Oh, there we are, good. He's going to be talking to us today about widows and their seals. So a nice follow-on from the previous paper. Over to you, thank you. Thanks very much, Elizabeth. Okay, hopefully you can all see that. Oh, hang on, where's the start gone? That's much better. Yeah, hopefully you can all see that now. Thanks very much, everyone, for tuning in today. This paper stems from some of the work that I did draw my PhD in which I explored non-armorial personal seals and the ways that they were used to express personal identity in rural English communities between the late 12th century and the middle of the 14th. Non-armorial simply means something that's devoid of heraldic motifs and coats of arms. So we're looking at seals that many of the seals we've already seen today that have like plants and animals, crosses, stars and figures on them. And it means the same as other terms you might have seen in previous publications on the subjects such as non-armorialists and non-heraldic. Traditionally, studies of seal impressionists and seal matrices have concentrated on the seals of government, the pure, essentially, ecclesiastical elites and corporations. In spite of the fact that they make up possibly only around about 20% of all surviving seals from the medieval period. However, over the last few decades there's been a welcome shift in the study of seals to focus on the much greater numbers of non-armorial personal seals that survive in archives throughout the British Isles. This may be about 80% of the total. Scholars such as Elizabeth New, Fitch, Gofield and John McEwen have used these seals to refocus attention towards the seals and sealing practices of ordinary men and women. Groups such as peasants, craftsmen, artisans, small landholders and merchants are examples of where this research has been concentrated. But one group of non-armorial seals that have hardly entered the discussion are those that were used by widows. Therefore, the aim of my paper is to draw attention to this overlooked group by looking at their seals, the ways in which they use them and what this can tell us about the widow's sense of personal identity. To do this I'll be drawing upon the seals from two rural settlements. These are salt fleet being Lincolnshire, which is just up here. And then Tamworth and Morrickshire, which is just down here. There's a couple of more focused maps there showing that position within their respective counties, salt fleet be just here and Tamworth just over here. Both of these localities are rich and surviving deeds recording small land transactions between their inhabitants. And a great many of them retain their original seal impressions, excluding duplicate impressions made by the same matrix. There are 285 surviving impressions from salt fleet be another 184 from Tamworth. And the case of the former, they've run throughout the late 12th century until 1300. And the case of the latter again from the late 12th century until about 1348. That's where I drew the line on the time period I was looking at. I will first provide a general overview of the numbers of widow seals from my case studies and the contents of their legends. And I will then move on to the main part of my paper beginning with an examination of seals that identified their owners as widows. Specifically, before proceeding to look at a seal that may identify its owner as a wife. Having dealt with marital status, the next part of my paper will focus on the more common nonmarital means of identification in seal legends. And finally, I will investigate whether there are any motifs associated with widows from either of my case studies. But widows were an important subgroup of non-marital seal users common to both salt fleet be in Tamworth. Of the 45 surviving women seal impressions from salt fleet be 13 were made by widows. You can see them here. This is the orange section for widows, blue section for non-widows, which amounts to nearly 30%. Looking at Tamworth now of 30 women seal impressions that survive from here, Tam belong to widows. So exactly one third of the total. From this evidence, it is clear that widows made up as a substantial proportion of seal users in both localities. They should not come as much of a surprise given that widows were major landholders in the medieval period when it was common for the holdings of the deceased husbands to pass to them. This table shows you how salt fleet these widows identified themselves through their legends. They did so in a variety of ways but not normally as widows as you can see. By far the most common legend involved the use of a patronym which appears on nearly half of the impressions. So A daughter of B for example and nearly a quarter used locative by names. These are names that identify their owners by a place. This can be in a landscape. This can be a place name or a feature such as a hill. We then have a couple used by a couple that include widow in their legend, one possible example of a wife. And then we have an example of another person's seal being used by someone, possibly Pory. Tamworth, there are only five impressions with legends, partly legible. But two appear to use a locative by name. Two use another person's name. So indicating that they're using another person's seal. One possible example of a family name. And unfortunately five of the 10 have legends that are unidentifiable. So we can't do an awful lot with those unfortunately. Seal matrices and impressions which identified their owners as widows are rare as the following national figures reveal. Of the two and a half thousand plus seal impressions in the DL25 collection at the National Archives, just four have legends identifying their owner as a widow. This is amounts to about 0.16% the total and two of these are the ones from Salt Fleet that I'm going to talk about next. Of the 6,394 medial seal matrices recorded on a Portland ticker scheme database as of about last week, I found there were 12 that have identified their owners as a widow. Helen said that were there were 11 but either way it's around about 0.2% of the total. This probably just relates to like the kind of search terms that I used. In comparison, Salt Fleet is a high proportion of women who use matrices with legends describing them as widows, even when considered as a percentage of all the men's and women's seal impressions and matrices from the locality. This is, it's nearly 1% for Salt Fleet. The evidence suggests that widowhood was not something that the majority of widows actually included in their seal legends. Those that did, there were three principle ways of doing so. You could use the words, the first two mean widow and the third means formally wife. So basically it means widow as well. And it was the latter that was used by Genevieve, widow of William, son of Robert of Salt Fleetby here on the left, and Matilda, widow of William Smith of Salt Fleetby, as shown by their legends. I've transcribed the legends down here. So we have seal of Genevieve. And then this whole thing that looks like a nine is a symbol for a woman's seal. A symbol for this little symbol on the legend here, which looks a bit like a nine when it's viewed from the correct side. It's obviously upside down here, it's a particular nature of the impression. But it means con. And then we have following on from this dam, the con dam, and over here we have a B and an X, an apostrophe indicating a wife. So we've got seal of Genevieve, formerly wife of William. And we have a similar situation over here with Matilda, except it's formatted slightly differently, but it continues to use a little symbol indicating con. The phrase con dam, or con dam, is normal to find in the deeds. This seems to be unusual for seals. And the only other example I am aware of is from Reston, which is about six or seven miles away from Salt Fleetby to the Southwest. Raylite appears to be the most common throughout the collections I've looked at, such as Porto Mantic with this scheme. And Fidgetway appears to be quite rare. There's only one example from Norfolk with this legend that simply says seal of Constance Whitter, also found on the Porto Mantic with this scheme. As suggested by Elizabeth New, seals were generally the choice of their owner and provide a brief insight into the ways in which people saw themselves or the ways that they wanted to be seen. Seals are therefore objects that provide an exceptional insight into personal identity, into the personal identity of the seal owner. Moreover, Brigitte Bedos Rezac largely looking at seals in a more elite context, stated that seals were intended for repressive use in a wide variety of circumstances as representatives of individuals who often had multiple titles and sexual roles. These conditions of usage and the small size of seals necessitated economy and legends, which included the most essential elements of a sealer's identity. On non-armorial seals, which tend to be a bit smaller than noble seals, space was even greater premium because of their typically small size. Therefore legends on non-armorial seals which diverge from conventional formulas, specifying filial relationships and locative by names, for example, could be argued to be an even more potent expression of an individual's personal identity. Thus widows like Genevieve and Matilda identify themselves as widows through their seal legends because it was key to their personal sense of identity. Widow had changed the status of women in a way that did not apply to men. Judith Bennett argued that it brought about an expansion in the availability and exercise of public power, which surpassed that of other women. In her study of the land market at Briggstock, Northamptonshire, she revealed that widows more than daughters and wives emulated the participation of men by independently trading, exchanging and selling parcels or property. She observed that the widows were distinguished from other women by their legal actions. Like daughters, they owed soup to the court, answered complaints and pursued litigation without the coverage of a male. And like husbands, they were legally liable for the actions and problems of their dependence. Bennett also noted that the social experiences of widows revealed a similar breadth of social activity and power to men. The court associations of most women were characterized by strong focus on kin, but associations of widows more closely matched those of men with a wide and varied reliance, not just on kin, but on neighbors as well. Consequently, as a widow, a woman in early 14th century Briggstock faced many more public opportunities and responsibilities than she encounters a married woman. So too would have Genevieve and Matilda in salt fleeting. And indeed, clauses and widows' charters would stress their legitimacy and power to engage in land transactions and their ability to formulate property-related agreements further attested this. So the addition of a seal, which identified the sealer as a widow to a deed, was not only an extension of their legal independence as specified in the document, but a forceful statement or assertion even that was represented in material form and visible to others. Condemn Luxor followed by the name of the husband stressed the marital relationship that had existed between the two, that the husband was now dead and the sealer was sealing independently as a widow. These two seal impressions therefore represent a stark shift in their identity from dependent spouse to independent widow, a point reinforced by the fact that the seal must have been made after the husband had died. The object itself and the legend therefore embodied a new phase in the life cycle of these women that they wished to demonstrate to others. Most widows, however, did not identify as such on their seals. Alice, widow of William Stinnell of Saltfleet Me, identified herself as a wife in her seal legend. This is the seal impression, by the way. Owing to the illegible nature of the seal legend, we are dependent on the partial transcription in TNA's catalog, which are provided down here. This was taken a long time ago, the seal has since deteriorated considerably since while I'm working with this transcription. But the letters FEM, Fem, likely indicate feminine meaning wife. If this reading is correct, then it is almost certain that Alice would have acquired her matrix prior to the death of her husband William. And the choice of wording here tied her identity to him. Whether this was initiated by Alice herself or William even cannot be said. But what is interesting is that it was appropriate for her to carry on using the seal after his death, even though the legend did not accord with her new marital status. Her impression suggests that she continued to define herself in a sealing context by her marriage and her legal dancing on William alive or dead. Evidently, this did not impede her ability to quick claim her right in the Dower land of one Toft and one Croft, as well as all of that land that William had sold to one Andrew Gore. The connection of the land to William that she was quick claiming may have been significant in her continued use of the seal. And it also indicates that widowhood was perhaps not as important to her sense of identity as it was to Genevieve and Matilda in the sealing context. I mean, Alice may well have expressed her widowhood in other ways that are now invisible to us. Indeed, she may also have been stressing that this relationship was particularly valuable or important to her. But I'll touch on this a little bit more in a moment. It's much more normal for widow's legends to identify their owner by means other than as a widow or a wife. At Salt Fleet Bee, we've got six widow seals identified them through their patrilineal descent, so as daughters of their fathers, which is nearly half of the total surviving widow's impressions from the locality. For instance, a seal impressionant Agnes, daughter of Richard Legerefter, has the legend seal of Agnes to Richard. So this is the seal right here on the right-hand side. Transcriptions just provided down below as well. And patronymes were very common among men and women in the medieval period, and it comes as no surprise that they would make it onto seals as well. One explanation for why they would appear on so many seals might be because these women were heirs to land. Season John's noted that aristocratic female heirs tended to be identified through the patronymic because that name was the most prestigious and identified the connection of the family land. But at the level of non-lead women, the situation may have been similar in some respects, with many deities continuing to identify widows by their relationship to their father and potentially linking them to land that they might one day inherit from them. Again, to touch on the point made by Helen that maybe these legends emphasised a particularly valuable and important relationship, this is also another possibility in this situation. Agnes may have felt that her relationship with her father was much more valuable, whether emotionally, personally, or legally, and thought to emphasise that. The reasons why widows did not identify themselves as widows on their seals may also have been one of pragmatism or economics. It is very apparent that it was not compulsory for a woman to acquire a new matrix when she entered widowhood as the example of Alice, wife of William, that I've just spoken about, demonstrates. This is apparent also in the case of Horwissia, daughter of Walter, who used the same seal matrix for an impression appended to an acknowledgement of a gift made in conjunction with her husband William Mason to one Philip ghoul. After William's death, she used the same matrix to make a gift of six acres of land on her own to the same Philip. Clearly, some of these widows already had a matrix on which was an engraved an element of their personal identity that was most essential to them at the time they acquired it. And it was unnecessary to acquire a new one when personal circumstances changed. An ordinary seal matrix was not particularly costly as it was made of base metals, but many may not have wished to spend money on another matrix if it was not necessary, especially if it was for single use, as has already been pointed out, many of the seal matrices for people of low status may well have been. This point is reinforced by the fact that there are three impressions, one from Salt Fleetby and two from Tanworth. They have legends that do not match the name of the individual in the document and were almost certainly borrowed or acquired second hand. So here on the left is rather fantastic looking seal. It says on it that it's the seal of Gnil de Burrell. But in the document that it's attached to, the sealer is actually called Alice, daughter of William Burrell. Now if we move over to this example from Tanworth, we have a partial survival of the legend and it says seal of Mary.  But the name of the person in the document is actually identified as a Marjorie. Having focused on legends for most of this paper, I would now like to consider the motifs on widow seals that fall drawing into a conclusion. By far the most common motif used by widows at Salt Fleetby was the stylized Lee. A motif that looks like a flirt Lee because it is found on seven of the impressions from the locality, which is over 50%. You can see them here. These are quite kind of crude, almost angular style ones. And I saw a comment earlier asking about the stylized Lee being upside down. This is an example of that that I've got from my source material. And there are multiple other examples as well. That's of interest. To see if the popularity of the stylized Lee was significant, we can compare it with the number of non-widows using it. So altogether there are 24 stylized lilies at Salt Fleetby on women's seal impressions, seven belonging to widows and 17 belonging to non-widows. And looking at the proportion that survive for widows, we get a figure of 53.2%. And then if we compare it with non-widows, we get a figure of 53.8%. So only a 0.6% difference indicating that the use of the stylized Lee by widows at Salt Fleetby was not significant in the context of widowhood. Additional types of motif on widows seals include two different types of plant or flower. Two of a radial motif that resemble flower heads, as well as an occupational motif and a fish. My research suggests that none were associated with widows in particular, but the very small numbers of the other examples of motifs used by widows makes it very difficult to actually make a statistically meaningful calculation in this respect. At Tamworth, we have one widow who used a stylized Lee. Two that use motifs that may or may not have been stylized lilies. One using a radial device to also look like a flower, in this case we've got six petals. One a bust facing right with the hands in the prior position. One a figure, maybe the Virgin Mary, and two with some blank shields. Another two of the motifs are sadly unidentifiable. This suggests that there was no one motif that was particularly associated with widows at Tamworth either, even when compared with non-widows motifs. However, a clear understanding of the situation at Tamworth is somewhat compromised by the small numbers of each motif type available. The study on sealed impressions and the damaged condition of the impressions making accurate description problematic. The seals from Tamworth do not survive as well as the seals from Salt Fleetby at all. Nonetheless, the variety and the types of motif that they used are in keeping with the popularity of motifs of all of Tamworth's women, widows or not. It therefore appears that no one motif was associated with widows from Salt Fleetby at Tamworth. I'm just gonna draw my paper together into conclusion now. Widowhood redefined a woman's identity because it led to a number of important changes in her life, most notably the increased degree of personal and legal freedom that she could exercise. But for most widows, it was clear this was not an important aspect of their personal identity, specifically with their sealing context, because it was rare for a widow to identify itself as such on her seal through the legend. Much more common were legends emphasizing aspects like patrilineal descent, perhaps they've owned the fact that there was some potential heirs to their father's lands. Even so, just as viable explanation is that they were trying to emphasize particularly personal relationships as well, or that fundamentally it was not necessary for a widow to acquire a new seal if she already had one. So on the odd occasion where a widow did identify as one with a seal legend, we can conclude that the widowhood was particularly important to her sense of personal identity. Seals like those of Genevieve and Matilda are a rare moment showing the point in time at which a new matrix was commissioned and the reason for it. These seals embodied their own as new sense of personal and legal freedom and were a public display of it. Having looked at legends, you can see they're associated with widowhood, but it seems that motifs based on my research cannot be associated with widowhood. And thank you very much for listening. Right, thank you very much indeed, Ulster. As you say, so little work's been done on seals and sealing patterns of widows, but that's just so interesting to have this information. There's a question from John Cititon to start with. At Salt Fleetby, you've got around 70% of women who are not widows. Of that 70%, do you know the split between wives and spinsters? Wives and unmarried women. So within the 70%, what's the split between wives and... Sorry to put you on the spot. Sorry, can you repeat the question for me? Of the 70% who are not widows, how many of those are single women rather than married? Very hard to say. We only have what the documents can tell us, and they don't always give us the full impression. There are a number of married women who appear. The exact proportion of them I do not know off the top of my head. But by the nature of the way that the documents were written, it is not always easy to tell whether someone's married or not. Quite a lot of them are also used, and they didn't specify whether they were married. But we don't know if that means they were or not. Yes, no, I'll tell you that. I hope that answers the questions. Yes, I think so. I can't see any more specific questions at the moment in the chat, but I have one for you in terms of... Widows using seals that call themselves widows who then remarry. Have you come across this and what happens in those cases? I have come across no examples of remarrying that I can discern from the documents. Again, because of the nature of the documents, we wouldn't necessarily be told that information. But you've not been able to say, yes, I'm pretty sure this is the same individual who calls herself widow here, and then in a couple of years' time, she uses a different matrix. No, I can't... There are no obvious examples that I can see, which say that. Right, thank you. A good question from Caroline Barham. To what sort of documents are these widow seals attached? Is this different from the documents to which other women's seals are attached? No, it's almost all deeds recording little land transactions. Basically, my entire source base was these little deeds. Basically, I had no other source material on which seals were attached to them, so very deed heavy. Again, I'm going to take chairs prerogative and ask you another question. It's a bit unfair. Do you have any thoughts on what we call upside-down lilies in terms of if you're taking the start point to the legend as 12 o'clock, they look as though they're upside-down. Any thoughts on this? I did look at this in a little bit more detail. In some cases, there doesn't seem to be any particularly obvious reason for it, but I have an example of a widow, actually, and her two daughters, and all three of them use stylized lilies, and all three of them are upside-down. I don't know whether there's an awful lot in that. It may be that they all decided that when they got their matrices commissioned that they all wanted them to be upside-down for some particular reason, or maybe that was the way that a particular craftsman liked to do his in particular. That's very interesting. Helen's geeks just commented that we have mirrored lilies too, so symmetrical around a horizontal line, which is interesting. A final question from Caroline, because we're very nearly at the end now. One from Stephen Freeth. From Caroline, do you have any wills to which seals are attached? From Stephen Freeth, does the law help us to identify single women? In answer to Caroline's question, no, no wills whatsoever with seals attached. I mean, I had no wills in the first place anyway. What was the second question? Will there be more helpers to identify single women for a sealing? What's meant by that, sorry? If they're married, could they have any property to sell independently of their husbands? Or are single women indeed always going to be either heiresses or widows? Generally married women were at least required some sort of permission from their husbands to engage in land transactions. A lot of the deeds that I've got show that the women and like a man and a wife doing it together as such. So there's quite a few examples of joint ownership specifically. No, thank you very much. If we don't mind, I'll just finish by a follow up Stephen's point that I actually think and I'm sure lots of others will agree that it's a very interesting point that married women sealed and used seal matrices that identified them as individuals anyway, because after all unless it was specifically their land or property in legal terms, they actually there was no need for them and really they shouldn't have done. So I think that's interesting and from the imprint project we have very strong circumstantial evidence from the prints, hand prints that's in at least a couple of cases women were actually including married women were holding the wax into which their matrix was impressed so they were physically active in that process. Right, thank you. I see we've gone just a couple of minutes over but I'm sure everyone will have borne with us.


As you're watching this, some of the last old growth trees in rainforests across British Columbia are falling. Some of them are over a thousand years old. Premier Horgan, you promised to protect them. Less than one percent of forests in British Columbia still have big, old growth trees. These forest giants live on indigenous land and have been taken care of since time immemorial. To keep the air we breathe and the water we drink and clean. They are essential in our fight against climate change. Minister Conroy, you promised too. While you're dragging your feet elders, teachers, scientists and many more are going to jail. And still the trees are falling. If you're committed to work with indigenous peoples, stop the logging of old growth immediately. Do the right thing, defer logging in all at risk. Over a forest. Fulfill the problems that you've made to people. Not in three years. Not in three months. Today, right now, the world is watching.


Thanks for sticking around this late. I'm not sure if you're actually sticking around to see me or if that's because this is the room where they're serving the happy hour after my talk. Hopefully it's to see me, so thank you. My name is Brad Urani. I work at Procore. We make software for the construction industry. It's a huge suite of tools. Procore is a giant suite of tools. So many tools and so many features that I don't think one person knows at all. It's one of the oldest, largest, most mature Rails apps on Earth. I think it's about two million lines of code and it has, I think, 90 or 100 people working on it every single day. We deploy it four or five times a day. And we get a lot of errors and that is not because we're not careful or anything. It's just because we release a lot of really, really advanced business tools and we're constantly shocked at the way our customers use them. They use them in all sorts of really interesting, creative ways that we can't often predict. So we often find ourselves surprised by some of the exceptions that pop up in our logs and stuff when customers do things that we're not expecting them to do and we have to react. We have to react very quickly because we love our customers and our goal is to make our customers happy. Quite frankly, they pay a lot of money for this software so they expect us to be there right there on the phone answering their support questions and we are. And a lot of the content of this talk came out of some techniques that I've learned and that our team has learned over the years in order to make errors more useful in order to create error architectures that make it easier to troubleshoot potential problems and just make our lives a little easier. I really like this subject. I'm kind of weird. I get really excited about error handling. Maybe it's because when I go to sleep at night, at least I know if something goes wrong, at least we'll be able to troubleshoot it easily and figure out kind of what happened. By the way, so this is RailsConf, right? And everyone kind of makes their presentations kind of funny and puts like lots of memes in their talk and I'm not very good at picking out memes. I don't know, like the technical content comes naturally to me but not like finding all the humor and putting like the funny cat photos in so I put some funny like memes in here and then I didn't like them so I took them out. So I just put them all at the end. So stick around to the end and there's just like a random selection of completely irrelevant memes. But like I said, actually I left a couple in and hopefully I don't blow the timing. Anyway, let's see. This talk starts off, it's kind of about fundamentals and it starts off with some basics, some real simple things but it gets advanced pretty quickly so if you're here for some more advanced architectures and stuff stick around because it ramps up. Okay, so a little word of caution like before I dive into this is that when you start talking about what you can do with errors, exceptions and I'm gonna explain the difference in a second, it's really easy to overdo it. I'm gonna show you a lot of kind of like tools and things that you can add to your application that quite frankly add complexity but you want to make sure that that is necessary complexity that you're not doing this for no reason. If you're doing vanilla Rails like MVC and a lot of crud, don't go using the things I'm showing you here. These are architectures that are best done sort of reactively when you discover gaps in your knowledge or that your airports are coming through and you don't have the context that you need to go and add them after that becomes a problem. This is a good thing to do reactive architecture because inevitably I've seen it from junior engineers, senior engineers who were like, wow, I've got this new power from the stuff Brad taught me and they go and create something really elaborate that's not necessary. So don't overdo it. And by the way, most of these techniques apply to any language that supports exceptions. In fact, a lot of the best literature on this if you want to study deeper into this comes from the Java world. It really originates all the way back in the C++ days. And so here's just like a simple just to get started. We've got raise, oh no, and this is how we raise an error in Ruby. We can just pass a string, right? That is the same as this runtime error.new, right? Runtime is a class just like any errors are just a class in Ruby. So those are equivalent really and we actually have three syntaxes. So all of these three do exactly the same thing. And then, oh, where's my clicker? No, I lost it, nevermind. So anyway, in errors in Ruby are a hierarchy, right? And you can see up at the top of the hierarchy we have exception, right? So exception is the root class and everything below it kind of extends exception including what you see there is standard error. Standard error is the class we're mostly gonna be dealing with today. So I mentioned exception versus errors. The difference is that error is a subclass of exception. So for the more visually inclined among you, right? This is a hierarchy. This is a tree. So and spoiler alert, we're gonna be creating our own tree. We're gonna be creating our own tree exception hierarchy a little bit later, but you can envision it like this, right? Standard error extends exception and a lot of those like subclasses, you know a lot of those other errors that are in the framework like zero division error, they extend standard error. Okay, so when we rescue them, we can rescue just like this and that is actually equivalent to rescuing standard error. Now we said standard error is a subclass of exception, right? That will not rescue, for instance, if you just raised exception that would not rescue it. It only rescues standard error and subclasses of standard error. So just like recap, right? When you raise, you raise runtime error, right? But when you rescue the default level of standard error and since runtime error extends standard error, just using rescue will catch runtime error and anything that extends either of those classes. Okay, never do this. Notice I've rescued exception, which is the root case, right? Which is the root at the top of that hierarchy because you will rescue all kinds of crazy things like system stack error, which is what happens when you get a stack overflow. If you really wanna see what happens when your program runs out of memory and you continue executing, right? That's what happens with no memory error, signal exception, that's the one where you press control C, that's what gets raised, so it will prevent you from actually quitting your program and script errors actually gets thrown if there's a syntax error in your program. So don't rescue exception because it includes all of those where a standard error does not. You can make our own, right? Like a really serious error, like out of nachos error, that would be terrible, right? Just by extending standard error. Okay, and it's probably worth mentioning that like inside the framework, inside all the popular gems you use and inside rails itself, most of them have their own sort of exception hierarchy. So this is what the exception hierarchy, the error hierarchy from ActiveRecord looks like. Okay, so let's talk about rails. Oh, this is one place where like I had a meme and it was actually a good meme, right? So yeah, check this out. I didn't choose the chug life, the chug life chose me. By the way, if any of you are thinking of having children, don't do it because you'll have to watch this insufferable cartoon called Thomas the Tank Engine which will drive you bonkers, right? But if you do have children, make sure they watch it because it's about these trains on the island of Sodor and they cooperate and I swear to God it is the perfect parable for like a good functioning dev team. Like if you want your child to be a software developer, Thomas the Tank Engine will like teach them all the proper principles about like cooperation working together. So it's a good deal. So anyway, here's like a typical rails controller. This is just like straight from the scaffold, right? No surprises here. We're calling user.save which triggers the validations. Will this ever throw an error? Turns out it can, right? For some very, very unexpected things. Like what if you just can't play and can't connect to your database, right? There's some completely predictable error. It can happen, even though .save, it doesn't have the bang, that won't throw errors, right? It is still possible.  This is an example of like not good air handling and I'm gonna get into, I'm gonna slowly explain like as I go on the kind of why, what the problems of this are. But you might be tempted to do it. For instance, like what if there is like something in your save process there for a new user that does occasionally air? Like what if like in an after save callback, you're contacting a third party service and occasionally that third party service isn't available and you kind of want to recover it from it. You might be tempted to do this, but this is lacking in a lot of ways and I'm gonna get to why. Okay, so rule number one here then is don't rescue just because you can. In some cases, right? What I'm showing here is a lot of what I'm showing is what happens and where and when and why you might rescue airs and when it's good. But the first rule here is that sometimes you don't want to do that at all. So just remember that then in a lot of cases, if totally unexpected airs comes up, the best chance is just to let them come up and show the air page, right? That generic 500 air page. But when we talk about all this, right, we have to like focus on who are audiences and we really have three audiences, right? We've got our users and we want to sort of direct them in the right way. We want to show proper air messages, redirect to the right pages if possible. We got developers and computers, right? So on the users, like I said, right, we've got messages we have to worry about. We have control flow, whether they get redirected to the right pages. For developers, we want developer facing descriptive air messages so we can figure out what's going on. Additional metadata which is I'm gonna get into, right? We wanna make sure that those are reported to our air logging system. We wanna make sure that the airs show up in the logs so that we can troubleshoot them. And for computers, right, those are like they could be consumers of our API, in which case we wanna use errors to properly return the correct status codes from our APIs, et cetera, so people can program against our APIs. Or in the case of your own program, we're gonna show class hierarchies for showing control flow. So we have a few goals here, right? Just like I mentioned, user facing air messages. And then we want to control status codes, air messages, add contextual data. And we're gonna talk a little bit about notifications, how you can make sure that emails and SMS messages in Slack go to your team in proper ways given the proper things. We wanna make sure that it ends up in logging and we're gonna talk about like how to change that on HTML versus JSON. So, here's our method again, right? And I said this has problems. Well, what's one of them already, right? We can already see one of them which is that one of our audiences is developers. And as developers we rely on tools, air reporting tools and logs and you'll notice that this does not end up in any logs. So if I do this and I just rescue it, I never know what happened. This gets swallowed as we say, right? Because there's never any evidence that occurs. So here's another rule, do not swallow airs ever, right? Make sure that if something bad happens that there's always a trace of it somewhere in your system. I didn't think this like snake was like kind of scary enough. So like, oh, there's a really scary snake, right? Just to rub it in, do not swallow airs. And honestly just to like belabor the point, the scariest like kind of snake at all, right? Do not swallow airs. Not sure where I was going with that one. Anyway, but like under reporting, right? So one of the things, one of these awesome great tools we have is reporting and there are all these wonderful solutions these days, these third party systems. And the old days before these, like we used to have to build this stuff ourselves and it was always awful. Now you can just buy one of these. These are third party SaaS services that you can integrate into your program, right? They're awesome and they do really, really wonderful things. And I'm gonna show you how to take advantage of these a little bit more than you might already be doing. So here's like an example of like an air report. This is bug snag. We use bug snag and it's been good and reliable for us, we like it. They really should be paying me to do this talk, but that's another story. So this is like an example of an air message that comes up when something goes wrong. And you can see that they've got these great graphs and charts and wonderful tools. They have also like a really, a lot of nice search features and filtering features and things like that. Out of the box, they're really configurable. They know whether you're in staging or development or production and behave differently. Most of them you just drop the gem in your gem file and it just kind of works and you don't have to do much config. And like for your hobby projects and things like that, there's usually a really nice free tier. So take advantage. But there are a couple power user features that I think most people don't realize are there and people don't utilize quite enough that can really fine tune your usage of these products in ways that you might not be aware of. Severity level and custom metadata, I'm gonna show you both. So back to this again, right? We know that this is not going to our air reporting solution, but there are some cases, this may not be one of them, but there are some cases where we want to rescue an air but still get it into our reporting system, right? So what I recommend first is you start with a class specifically for that. So whenever I start like a new project, right? One of the first things I do is create sort of like a utility class which just has a class method called handle. And just to start off, it's just gonna look like this, which is just calling out to that gem for whatever your air reporting solution is. It calls that gem. We're gonna build on this. We're gonna put more in here in a bit, but just to start, right? All we're doing is using it to notify our reporting solution, okay? We've improved that a little, but let's go like a little bit further. As I said, most of these systems report some kind of like severity level. So we've added an optional argument here to pass in severity, okay? This one is a particular, this error, right? It's on a user create. Some of these errors might be predictable. Like in the example, like I said, we're a third party services down. We know that's gonna happen occasionally. So it's kind of a error, but a slightly predictable one. We may not wanna be bothered every time it happens. So the default severity level is error. We can turn it down a notch down to warn. And what's nice is that in those air reporting systems, right, they have all kinds of configurable filters on like what shows up in your email, right? Some of them have like SMS and Slack alerts, but you can generally filter them out by severity level. So we're already using this tool in a more powerful way that gives us a little more control. We pass that down to the gem. All your usage may vary, but almost all of them support something like this. But the other thing is it never showed up in the log, right? And sometimes we depend on log files. Sometimes we have to get in there and grab the logs to really try to see what happens. We wanna make sure that we end up with our error in our log too. So we're gonna add to this class just some logging. And you can get a lot more sophisticated with that. You can, you know, based on severity levels and other things, channel it to different logs. And there's no end to what you can do, but we're already starting to build sort of a powerful framework here. Now we've got it in our air handling and our logs. So that's nice. Okay, now I've added something else here. How many people work on a big department with more than one team, more than one squad, right? We have, I think, 20, and like I said, like 80 developers, and guess what?  When we pull up our, you know, our bug snag in the morning, it is a nice way to filter on these. So I've added something else. You notice I've added a metadata property and I've said tool onboarding. Well, this gives me another really convenient filtering option. This is something that I can use to tailor alerts, to tailor emails, and various other things. It makes working on a team a lot more powerful. And metadata is like an arbitrary property. It's a free for all kind of hash where you can pass in whatever you want. But believe it or not, almost all air reporting tools handle this, right? It will actually like show up in your report, and some of them get crazy, like in bug snag, you could actually like use the metadata, like add new tabs to the report, and doing all kinds of things. But check the docs for yours, because you can probably do this, and it's probably searchable, filterable, and can be used to tailor alerts and things like that. So there's a very powerful feature for you. So whatever you accomplished already, we're adding contextual data, and we've given us flexibility notifications, just with a few lines of code. So getting a little more advanced here, right? I mentioned we wanna build our own hierarchies. You don't wanna do this for no reason at all, but there's some cases where creating custom air classes and things like that can make your whole architecture a lot more powerful. The most common case, not the only case, the most common case where I see this comes useful is when you have applications with very deep call stacks. So like in this example, like imagine like a fictional online shoe store. You might have like, you know, if you're doing MVC, this is not, if it's just like the controller calls a model and that's it, you don't need this. But in more advanced flows with like a lot of business logic, and Procore certainly has a lot of business logic, you might have this kind of scenario where you have like a purchase class which like, you know, calls a few methods, it calls a method which reaches out to an item class, which reaches out to a warehouse class, and like all the way down at the bottom there, you've got this like check available. So this is a shoe store, we're checking to see if like the shoe is an inventory, and there's a lot that could go wrong there, and we're way deep in the call stack and we're way far away from the controller. So it's possible that someone checks out and wants to buy a shoe and the shoe does not exist. So we know way down at the bottom of that call stack that we need to display an error message now, we need to tell the user what's going on, but the problem is we need to show it way up in the controller is where we have control over that and we can actually affect change and show the user the message, right? Now, what's nice about this is like, it's either a feature or something a detriment is that all those layers in the middle don't need to know about that error. Some say that's good, some say that's bad, right? That's what exceptions allow you to do is allow you to raise an error way at the bottom of the call stack, ignore it in the middle, and catch it all the way in the top. Other languages don't work like that. For instance, Go takes it to the extreme, it does not support exceptions. Every function that might have a possible error in it returns two values, a result and an error, and then there's like a conditional, and you would have to pass errors like all the way up the call stack, which is a lot of extra boilerplate, right? But also a lot less magic. So it's just a trade-off, I'm not gonna say one is better than the other. The good thing about errors is convenience and less boilerplate, the bad thing about them is that sometimes they come out of nowhere when you're not expecting them. Elixir takes a slightly different take on this where they encourage this sort of pattern-matching take, which is similar to the Go approach in which you have to pass the error all the way up the call stack, but it's got this nice pattern-matching syntax that makes it a lot more pleasurable to work with. Weirdly, it also supports exceptions, I don't know, maybe they couldn't make up their minds about what's better, I wouldn't be surprised to see them take them out, honestly, but in Ruby we have exceptions, regardless of what you think about them, we have them. So what we can do is raise here and rescue way up in the controller. Okay, so for this, we're gonna create an error hierarchy and an example one might look like this. You can see that I've started with just a generic store error, right? I've got a purchase error, which is anything related to the checkout process on my shopping cart, and I've got certain inventory errors, inadequate inventory or like an out-of-season error, I just made that up, this is my example error hierarchy, and you can see that also extending purchase error, we've got a coupon error that might include, that might also have subclasses, for instance, coupon expired or invalid coupon, right? Okay, so here's a controller for our purchase, so pretend this is the controller from when the user clicks purchase shoes, right? We've created this error handler class, like I said, we've changed the severity level down to warn, why is that? Well, actually some of these are kind of expected, we would expect sometimes to someone to enter an invalid coupon, we would expect sometimes someone to try to buy something that's out of stock, ideally they might, they would probably know it's out of stock before they click purchase, maybe there's a race condition, maybe two customers checked out at the same time, right? And one of them got the shoe and one of them came just a little too late, so these are sort of unlikely, but these are unlikely errors, but unlikely things happen and we know that, right? Now, I want to show you something else, right? Which is cpe.userMessage, notice that I'm not using it's user message and not message, this is a little trick I learned over the years, a little bit of wisdom, which is don't use error.message for users, user facing errors, it is a lot more convenient to have, because they're really not the same thing, right? error messages are for developers, those are computer language things for us software engineers, whereas messages that are supposed to face users are completely different, so we're using user message instead of message and that's something I implemented myself, which I'm going to show you in a second. Okay, now we've got a lot of flexibility in this controller, because notice, let me go back, notice what we caught here, we caught purchase error, and if you remember purchase error is only at the second level of that hierarchy, that includes inventory errors and coupon errors, but that might not be enough control for you, because what if you want to do two different things, depending on what kind of error it is? If it's an inventory error, you might want to redirect back to the shopping cart path, if it's a coupon error, you want to redirect to a coupon code page, so now I've changed the one I'm rescuing, I'm rescuing a subclass of purchase error, in this case inventory or coupon error, and now I can take different actions. You can already see that the error hierarchy I created is giving us more power to control flow and do slightly different things depending on what kind of error occurred. Like I said, user message, right? Don't use error dot message. In fact, it might even be a security alert, you could potentially leak information about the internals of your program if you expose user dot mess or exception message, right? The standard just message, it's unlikely, but it's possible. It's better to have that piece of totally separate thing. Okay, here's another slightly different example. Also part of our error hierarchy, if we're rolling a lot of our own, like authentication, or not authentication, but authorization really features and things like that to allow permission to certain pages, we might have to create an error hierarchy for that. This is a little different in that it could occur on any page where those purchase errors are only on checkout, right? This could occur on any page. So what we can do here in this case is up at the top of the controller we can rescue from store unauthorized error, which is up near the middle of the hierarchy. And in this way, we can capture errors no matter what controller action they're on. In fact, we might even take this out of this controller and move it all the way up into the application controller so it applies to all pages. So now we have the power to not only control on a per action basis, like what the result is when something unexpected occurs, we can now apply this application wide. And you notice where it says status unauthorized, we're also changing the HTTP status code when this occurs. That is awesome for our API responses, right? I talked about having one of the audiences we have is not only us as developers and users, but also computers, we're now affecting the status code too. And it's worth saying that the framework does this quite a bit, right? Okay, so the framework does this quite a bit. Active record, for instance, automatically changes the status code for you. So if you like find a nacho that's not there, right? Because they're not a billion nachos in the world, unfortunately, it will automatically change the status code for you to a 404. And if you do like cheesiness to a million, but the max cheesiness level is only 100, it raises record invalid, which gives us a 422. So the framework's doing this, we're just doing the same thing that the framework is doing, which is using errors to control HTTP status code responses. And if you're doing JSON APIs, it creates this nice, it does this for you, which is pretty slick. Or if you want something different, you have the ability to control this now. Okay, and you'll find things, like this is source code from Rack, for instance, where you can see that Ruby on Rails or Rack at least is automatically matching up error classes to status codes. Okay, so I mentioned this hierarchy, this is back to our purchaser one. I'm gonna show you what these classes actually look like. I'm gonna show you down the left side, their error purchaser inventory error and inadequate inventory error, what an implementation of these might look like. So here's the top of the hierarchy, right? This is the generic store error. You can see that I'm passing in a message. I have that generic metadata, a hash called metadata and a symbol called severity. Wait a minute, we've seen this before, haven't we? These are the same parameters that I passed into that custom handler class I made, only now I'm passing them in and construct your arguments to the error. Dun, dun, dun. So what we're doing here actually is now we have the ability to not only add these things at the point where I rescue the error, but also where I raise them. Also remember, I said use user message for customer facing errors. Not only have I put that into effect, I've also got the ability to internationalize it for non-English speaking people. So we've already got new abilities here and then see how I've got adder accessor at the top with metadata and severity. Believe it or not, for whatever error reporting solution you're using, at least most of them, just the fact that I have those adder accessors up there will automatically add them to my error reports no coding required. It's usually built into the gems of these things, of these solutions you're probably using is the automatic ability to just have that just work. So that's really cool. Here's the next one in the hierarchy, purchase error. There's no code in it. It's basically an abstract class, right? It's just inheriting, but that doesn't mean it's not useful. We already saw an example of me rescuing this in my controller because I wanted to treat purchase errors different from other types of random who knows what errors. And then that had a subclass which was inventory error, which is also completely empty. This is also useful even though it's completely empty because remember I had an example where I rescued inventory error separately from coupon error and do two different things. So just because the class does nothing but extend my purchase error doesn't mean it's not useful. It ends up very useful. And I may never actually instantiate this class and use it by itself. It may just be an abstract class, but it's still a very handy tool. So here's down in the bottom of this tree, inadequate inventory error I've passed in some actual useful data here, things I want in my report. Product name, requested quantity, and available quantity. And I've added them to that metadata hash. So now it's gonna automatically show up in my airport which is cool because when this happens I might wonder what's going on. Like it shouldn't be possible for someone to order something that's not in inventory, right? Usually they should get warned before they check out but in certain cases, race conditions among them it might happen and I kinda wanna know what's going on because I might get a support call about it and this will help me now that this is in the airport and I've overridden user message for a better more descriptive message. Okay, so back to our controller here. I'm catching purchaser, I'm rescuing purchase error and displaying user message. Okay, so what have I done here? Remember, down where I raise I added some metadata. I added product name and I had requested quantity, realized that that's all the way down at the bottom of this stack and I did that with constructor arguments in my error. User ID, actually that's out of date, but user ID or whatever else I want I can pass it up at the top in my air handler so I've created an architecture now with an air handler class and a custom error to add additional metadata, additional things I want whether it's at the site of the raise or at the site of the rescue. What if I want something from the middle, right? Maybe there's a coupon code being applied in there and it's somewhere in the middle of the stack in my business logic. I can basically rescue it and re-raise it. I didn't show you this but presumably I've added an add metadata field to that error class, right? I can always rescue and re-raise if I want to add additional context on the way up. So why rescue and re-raise to add metadata? Worth pointing out there's something else going on here, right? In addition, like in this purchase process, I may be removing items from inventory, applying a coupon code, clearing the cart and processing the payment. All of those involve database inserts, all of them can fail. What if I get through the first three and done a process payment, it fails? I've got new inserted rows in my database, right? That are not valid. What do I want to do? Roll back, right? Realize that the errors are affecting this. By having those errors bubble out from the base class, I don't have to do anything special except for use of transaction that I'm also getting that rollback behavior based on my airflow. Okay. There are other times, I showed you raising and rescuing the controller. There are some places where we want to raise an error but then also continue the program execution where something unexpected happened but we can recover and show the user a page as normal. Here we have a shipping price, right? And I'm getting a quote from UPS. Guess what? The UPS API goes down. I used to work in e-commerce, it happens. Do we still want to take the user's money and purchase their transaction and send them their shoes if UPS is down? Yes, right? So I'm returning $6.99, just blanket charging them $6.99 for no reason than that's just a random number. So this is a case because I'm just going to charge them $7 for shipping because I want the order, you know what I mean? So here's a good case where we're going to rescue and you might continue. We have that special handler class we made because we also want to know what happened. So when someone orders like 50 pounds of shoes and it's like $100 and we're charging them $7 for shipping, we want to know what went on. So we want that in our air log, don't we? So we've got that custom handler class that gives that to us. Okay, so we raise and forget, right? We've got three basic behaviors. Raise and forget, which are unknown, system errors, connectivity errors. Raise and rescue and controller that we do that so we can change the status code, customize the error message, soft error messages which I'm going to show you. And sometimes we raise, rescue and continue which is anything that's a problem but not enough to stop us from continuing which is my shipping price example. So three kind of distinct ways to sort of handle these things. Let them fly and just show the 500 page, recover and show something nice or continue as if nothing went wrong. Okay, wrapping errors, right? Oh wait, I think I have a meme for this. Yeah, oh, wrapping error, get it? Because it would be an error to wrap a cat in wrapping paper. Like RailsConf, right, cat memes. Cool. So, all right, so here's our call stack again. Check available down at the bottom. This is exactly the same what we saw but imagine instead of doing like querying our own database this hit a third-party API. Maybe it's one of our own microservices or maybe it's a third-party service, right? Things are gonna be a little different because we might have something like this where we're using REST Client to make an API request to something else. Well, that is from a gem, right? REST Client's from a gem. We're gonna get, we might get back in some cases request timeout. We could just let that one fly all the way up to the top but if we caught request timeout in our controller we're not really sure what to do with it. We're not really sure where it came from and we don't have a lot of data to go with it, right? And we might not be able to direct the user to where they should be going. So what we're gonna do is rescue that one and raise our own inventory unknown error, right? Okay, if we rescue that, right? Ruby, by default, has this great property called e.cause without us doing anything the cause in that case will be the original exception the REST Client request timeout exception. It does that automatically. The Ruby runtime gives you that automatically and most of these error reporting solutions also will show that. So we've got a call stack on top which is the call stack of our own exception and then it says caused by and there's the inner exception and we have two call stacks for each error and these reporting solutions already show this. So now we can really track things down. So that's really useful and most you don't really have to do anything to get that to work. Okay, so why wrap exceptions, adding metadata. We might choose to catch one error and raise another so that in the case of if this is an API for instance we can change the HTTP status code by using logic on our controllers we get better control flow. Here's a real world example of a case where we're doing something like this like using some pretty advanced error logic. So this is from Procore. This is a construction budget. This is like one of the world's most complicated webpage. Any button or click you make on this page sends this cascade of business logic. You know, it calls a hundred different methods from a hundred different classes. It's really, really complex to calculate someone's budget. See they're over on the left. There are two orange buttons in between one. There's a gray button. Just in order to know if that button is enabled whether it's orange or gray we have to first run a query that joins across six tables. Then based on the result hit a third party API service in which case is an integration partner get the result back from that and then decide whether the button should be orange or green. This is a real world example. Guess what? We're not doing all that in the controller. It's 25 levels deep down in the call stack and like way deep in our libraries and guess what? Third party services can go down. Can't they, right? So what we do in this case is we do kind of what I showed you which is capture the rest air, right? Wrap it on our own air which is like third party service unavailable air and then we have special logic way up in our controllers in the application controller in fact so that it affects every page so that instead of just showing a blanket 500 blow up we can give someone someone nice. This is like a soft air page, right? Where it actually gives people like a little nudge on where to go, a link so they can check their settings. You know what I mean? So this is a good real world example of why you use custom air hierarchies in deep call stacks. Little extras, these are things that I didn't have time to add but I'm just gonna give you like a little teaser for this stuff. Some people like to add a custom, like a unique identifier at the raise site. So if you just add like a randomly generated string or some kind of identifier that allows like easy searching through logs, you know? You could like correlate certain airs with pages in your documentation or maybe you could add a unique identifier and have some kind of mapping file that maps a unique ID to an air in your documentation so you could provide someone in an automatic way links to, you know, pages that help them figure out what went on. And we've accomplished quite a bit here, haven't we? Let's just go over what we did. We've changed the user-facing message and we now have complete control over what we say to the users in a way that's internationalized and totally configurable. We've better program or facing messages by raising our own hierarchies, by creating our own hierarchies, by creating our own air classes. We make our logs more intelligible to us so that we can actually comb through there and more quickly figure out if this is actually a real problem that we need to address or if it's something we can safely ignore. We've changed HTTP status code for our computer consumers of these APIs, right? They need to act on proper status codes. We now have full ability to control those. We've added contextual metadata which is everything related to what was going on when the air was raised. We now have the ability, both at the site where we raise the air through passing things to our exception and the part where we rescue it through our custom air handler class to add anything we want to that air report because we've added severity levels and various metadata that we can filter on. We now have completely configurable notifications so that we can make sure that the right team gets the right message and thank God we don't have to build all that ourselves that you get that with these solutions. We've got it in the logs, we have full control over the logging and we have this control flow. We've gained these control flow abilities where we can either continue execution, we can decide whether we just wanna fail and let the page blow up or we can show some kind of soft message or some more appropriate message to get to the users. You might wanna, if you get really complex, generate some kind of diagram like this because it gets complicated and hard to keep track of. This is a fictional one but if you really get into this, if you need it, right, and I'm gonna reiterate my original warning which was don't do this stuff if you don't need it. It's only in really complicated big apps that this stuff becomes important and that you need it but you might wanna diagram it like this just to show, just to keep track of it all because it's hard to keep track of sometimes. A little bit of further reading, there's some great posts online that I've linked to here. I'll tweet these slides where you can read more about this stuff and learn a little bit more about some of the capabilities of these tools and how you can build this stuff yourself. I'm Brad Urani, I love Twitter, I'm kind of an addict, follow me, I'll follow you back. I'd love to connect with you on LinkedIn. If you're into Mastodon, which is like this new hip, cool, decentralized, farm to table, social network, feel free to connect with me on that. Like I said, I work in Santa Barbara at Procore, we make construction software. We have one of the coolest, awesomeness teams of all time. I love my job and we're hiring like crazy, growing like crazy and I'd love to talk to you. This is the view from the office, it's tropical paradise, it's sunny all the time, it's gorgeous, come visit us. And oh, also check, my coworker Derek is giving a talk tomorrow about how we built out a really complex and really powerful API structure. Be sure to catch that one. And then here's some memes, like I said, they're totally irrelevant, right? Oh, there's some pigeons with sunglasses, yeah, great. Oh, live longer prosper, oh, it's Spog with a cat, right? This is, uh-oh, Brad's getting political, be careful.  And I think that's all I've got. And really that was just a ruse so that I wouldn't have time for questions. So, thank you, but I'd love to take questions afterwards.