So before I begin, I do want to acknowledge that many of these slides were adapted from the National Center for Home Food Preservation, which is led by Dr. Elizabeth Andrus, and that is a great place to go for additional information as well. And as Bob mentioned, this whole series and a lot of the materials that were developed for Field to Fork, and we have a lot, all were made possible in part with funding from the USDA. So today, what are we going to look at? We're going to take a look at the ingredients and their functions and also potentially some challenges related to making jams, jellies and preserves. And then the second part I'll be looking at with you, pickling, pickling products. So as I mentioned last week in our introductory seminar to the overall food preservation, when you preserve food, you really are a scientist and it's really important to be accurate and to know where to find accurate information. So that's what I'll be walking you through today and giving you a little bit of background, especially on jams, jellies and pickles. So before I get started, I'd like you to click yes or raise the hand icon on the participant list if you like jams and jellies. So who likes jams and jellies? Okay, I think I knew the answer to that already. You came to a webinar after all, but now secondly, you can take your hand down or you can respond in the chat to some of you are doing. How many of you have preserved jelly or pickles in the past? Okay, so we have some jelly makers or picklers out there. So I hope that I provide you with some new or updated information so that you expand your knowledge. It's really fun to take the wonderful produce that we have available and we've had an excellent growing season and save it for next winter. Maybe we won't have winter. It's a really odd year, so maybe we're just going to have nice weather forever. Okay, first I'd like to teach you some sweet spread terminology. I've mentioned jellies, jams and preserves, but believe it or not, there are a lot of additional terms related to sweet spreads, which is the overall umbrella term. So first of all, if we look at jellies, technically that means it's made of fruit juice and sugar and it's typically clear and if you were to drop it out of the jar, it would hold its shape. It's flavorful, it's sweet, it's fruity, and it has a fresh taste. Jam, on the other hand, has a different texture because it's made of crushed or chopped fruit and again sugar and it's thicker and of course it's sweet, but it's less firm than jelly. So if you would pour out a container of jam, it's not going to be a jiggling mass that holds its shape. Now let's go on to preserves. What are preserves? And you can certainly buy these in the grocery store as well, but technically, by definition, a preserve includes small pieces of fruit in a clear gelled syrup and that fruit is tender and it's plump and it's plump because of how it's made and so on. Now we have another term and you may or may not be familiar with this, but this is a whole other category conserves. These are jam-like, so kind of like a jam, but they're made with fruit and also nuts, raisins, or coconut. So lots of different things and a different texture compared to the others. So again, you could probably go to the grocery store wherever you like to shop if it's a larger one and you might see a whole variety of different sweet spreads. But wait, there's more! Syrups are another category of sweet spread, technically. You might like having choked cherry syrup or another type of syrup and how you make this is to cook the fruit juice or the pulp with sugar and it should attain a honey-like consistency. And yet another sweet spread is a marmalade. And a marmalade is different because it's usually made with citrus fruits. So you might have some pieces of the fruit or peel and it's transparent. So if you look at the picture on the screen, that's about what a marmalade would look like, but typically it would have some orange peel or orange zest in it. So now you can impress all your friends and family by calling things different names. All right, so let's get back to ingredients. So a successful sweet spread product will be made using a proper ratio of these typical four elements. So you need fruit, sugar, and we'll talk about some alternates for sweetener later, pectin and acid. So in the next slides, I'm going to go through some of the basics and why these products are present and maybe a little bit of the science, I guess, behind making these sweet spreads. So whenever you're preserving food, one thing I really want you to think about whether you're making pickles or jams and jellies or canning something else, you always want to preserve the best. You don't want to take your subpar fruits or vegetables and try to make them better by preserving them. They won't get better, trust me. So preserve the best. Look at the color, for example, in the fruits. Consider the flavor. You never want to preserve something that looks like it's spoiling or super overripe. Consider how much acid is present, you know, the sourness, and how much pectin. And I have some little tips and tricks on how to know how much acid and pectin are present. So let's keep going. So one thing that people typically ask me about is making apple butter. And apples are notoriously good sources of pectin. So if anyone has an apple tree, as I do in my backyard that looks like it's going to have a bumper crop of baby apples, this is what you should do. So you take that fruit pulp, mix it with your sugar and spices, stir frequently. And to test for doneness, you take a spoonful out, hold it away from the steam area for a couple minutes. And if it's mounted on the spoon, it's done. Or you could take a small quantity of that apple butter, put it on a plate. And if you don't see a rim of liquid that separates out around the edge of the apple butter, then it's ready for canning. So that's just one of the tricks to think about that our grandmas and great-grandmas probably knew well. But other than recently, people have drifted away from home food preservation, but now we're drifting back and that's a good thing. It's a fun thing to do and it's very rewarding. So if you have an apple tree, consider making some apple butter. So let's talk more about fruits. So first of all, anytime you're making jellies or jams or any of those other products, we talked about working small batches. You can't make a giant batch of jelly or jam or you probably will not be happy with the consequences. You might have a whole bunch of syrup in the end. So we work in small batches, sort through really well and discard any damaged parts, wash them underwater. We never use soap, just plain running water. Leave the skin on if you're going to make juice out of it because directly under the skin and actually throughout some of the fruits you're going to find your pectin, which is the thickening agent that's naturally in fruits and various amounts. Then cut them to the desired size. If you decide to make a berry sweet spread, strawberry or raspberry, in the case of strawberries or any berry, you need to rinse it first, wash it under running water. And for example, with strawberries, you'd remove those green caps and also any stems. We don't want those as part of the mix. That's not the texture that we want. Okay, let's talk a bit about sugar. Well, sugar is certainly a prime ingredient in sweet spreads. That's how they get their name and it has several functions. So it serves as a preservative and what sugar does is hangs onto the water. So it affects what we call in science the water activity. So it helps prevent organisms, those bacteria, other organisms from being able to use that moisture or water to grow and reproduce. So sugar kind of helps prevent that but not completely prevent because we all have seen moldy jam, I'm sure. So some organisms will still grow. Sugar, of course, also gives us the taste or the flavor that we look for. And it assists with making a gel. It's all part of a three-dimensional chemistry that occurs when you mix these ingredients to make our jams and jellies. And when we talk about sugar, we're talking about granulated or table sugar, as shown on the screen. You also, in the recipes that say it's okay to do this, you can use light corn syrup or mild honey. But keep in mind that honey will affect the flavor. So there are some substitutes that you can have but you might have to make a slight change in how much to add. And that kind of information is included on the National Center for Home Food Preservation and I believe I included some of that on our website. On the other hand, we have some stronger flavored sugars that are not recommended. Brown sugar and molasses will certainly affect the flavor and the color and probably also the preservative factors of the sugar. So those two are not recommended for those reasons. Okay, what if you want to adjust your diet? You don't want to eat as much sugar as what you might like if you have a sweet tooth like a lot of us do. There are some specially developed recipes for using less sugar. And last summer, I put together a handout that you can find on our food preservation website that show a number of different lower sugar recipes. So check those out. I took those all from the USDA National Center site. So those have all been tested and they're safe. Many of these lower sugar jellies do use a special modified pectin that were specially formulated and chemically modified, I guess, in order to work with less sugar. So it will say on the package, you don't want to substitute some artificial flavoring for sugar. And if you are using a lower sugar recipe, some will call for an artificial sugar called Splenda. And that measures out just like regular sugar, but you need the special recipe. You can't just put it into any recipe that you like just because you want a lower sugar recipe. So only use recipes that call for that particular product. Okay, let's talk about pectin and throw around words once in a while. And sometimes we wonder what does that mean? So pectin is the substance that's in plant cell walls naturally. It serves a structural function for the plant. But for us, it causes the fruit to gel and make these delicious spreads for our toast and pancakes and however we like to use them. Pectin is naturally in some fruits, but most recipes suggest that we use about three-fourths fully ripe fruit and one-fourth slightly under ripe fruit.  We can purchase pectin at the grocery store or at discount stores almost anywhere. I've seen so many displays set up about with canning information that's kind of exciting. But you can certainly buy pectin in various forms including liquid and powder. And you can also buy it online so it's readily available depending on where you shop. If you use commercial pectin instead of the naturally available pectin that's in many of our fruits, you will get a more consistent product because you'll know exactly how much pectin you're adding to your recipe. But some people like to try things on their own and I do have a little tip, a little science trick you can try at home. The other thing I want to mention is that you should use the pectin by the date on the package. It will have a use by date on it. It won't make you sick if you use it beyond that date. It's not really that type of a food safety issue. It's more of a quality. You can have food including pectin degrade over time without necessarily becoming unsafe. So look at the date. The same hole is true that there are dates on most packages of canning lids. All right, so here is a slide that shows the relative amount of pectin in various fruits. And I see someone has asked a question about what can I use instead of pectin. Well in some cases you don't have to add pectin at all, a commercial pectin. You can, if you can eat the fruit safely and you choose fruits that are naturally high in pectin, you can get away with using any kind of commercial pectin at all. So if we look in group one, you'll note it says sour apples and sour blackberries. That's for a reason. It's, these fruits are higher in pectin when they're not fully ripe. So that's the whole reason behind using, you know, part fully ripe and part fruit that isn't ripe. So apples, blackberries, crab apples are naturally sour, pretty high in pectin, cranberries, currants, gooseberries, grapes, specifically the eastern concord, lemons, loganberries, plums, but not Italian, and quinces. We, to my knowledge, don't grow all of these types of fruits in our area, but just so you have a list. Group one has enough natural pectin and you'd only have to add the sugar and cook it and so on. Group two, if you start with ripe apples, ripe blackberries, sour cherries, choked cherries, elderberries, grapefruit, grape juice that's bottled, grapes, loquats, and oranges. Of course, we don't grow in North Dakota. Those are lower in natural acid or pectin, or they may need an addition of acid or pectin.  But this is just to give you an idea. Group three, all of the ones on this list, and I won't read them through, need added acid, pectin, or both. So check that list out and you'll see plums and pears and so on. So this again comes from research from the U.S. Department of Agriculture and they were the ones that initially did all the canning research and a lot of this was done around World War II as people were growing victory gardens and you might still have some recipes from your mom or grandma or great-grandma. Unfortunately, we really don't recommend using those super old recipes because they've done a lot of research since those early times and those aren't, and I'm not so worried about jams and jellies, but other types of home food preservation recipes that are quite old. You want to retire those and use the most recent ones you can find from a good source, not necessarily Pinterest and Facebook. All right, so here's your little science experiment you can try. So if you want to test for the amount of pectin that is in fruit, here's all you do. So you make your juice and that usually involves cooking of the fruit with water and then you get a tablespoon of the juice after it's been strained, for example, and you add it to a tablespoon of the natured alcohol, basically rubbing alcohol, and if you see small particles of jelly-like materials sort of, I don't even know how to describe it, it looks kind of gummy. That means there's not enough pectin. On the other hand, if there's a solid mass that forms when you combine these two things because you're basically precipitating out the carbohydrate technically, then you have enough pectin. You note that in red I have discard when done. You don't want anyone in the kitchen coming and eating that two tablespoons serving of poison, so don't have them get rid of that, where no one, including your animals, would eat it, because that's a dangerous toxic mixture there. Okay, so let's get back to pectin. So let's say you don't have enough pectin and you want to make some jam or jelly. I do need you to look at the recipe. These are not interchangeable. You can't say, well, I have some liquid pectin in my cupboard. It hasn't expired. This recipe calls for powdered pectin. You got to go to the grocery store and get the right one or find another recipe that calls for the one that you have. So you'll notice on our jam and jelly guide, we sometimes provide two recipes. I know for sure we have a choked cherry recipe, one that calls for liquid pectin and one that calls for powdered, so depending on what people have available. The difference here with liquid pectin that comes in the pouch, you see this over pouch, so it's liquid, it's kind of gummy. You add that to the hot cooked fruit sugar mixture, so that's when you add it, whereas if you have powdered pectin like the sure gel it's shown, you add that to the unheated fruit or juice and then you cook it. So basically the liquid pectin's been kind of pre-cooked for you, whereas the other one, you have to have it go through that full process. And I'm not endorsing either of these brands by any means. There are other brands available, but this is the one that was easy to access as we look for pictures to show you. So use them according to their directions. Most jellies and jams will come with tested recipes as well. And these companies do not want you to get sick or have a bad experience, so I always say when you find a recipe on a box or package, some food specialist has tested that out to make sure that you have a good experience. Once in a while they mess up, but you know, I have always had good experiences by using what's on the container. Okay, well the other piece of making jellies and jams is to be sure that you have enough acid, because again you have a chemical reaction going on with either the, well certainly with the fruit and the acid in the fruit along with the natural or added pectin and the sugar. So you're basically forming this cross-linked gel. It's like a matrix, and that's what gives your product its final form. So it's not going to be a syrup, it's going to be a jam, jelly, conserve or any of those other things. So here's how you estimate whether your fruit is acidic enough. So you have to have a good taster. So either you or someone in your house can help you decide if your product has the comparable tartness of a tart apple.  So that's a little experiment again you can do.  You can also add to your fruit some bottle lemon juice or citric acid to increase tartness. And as I mentioned last week, and I will again next week, all of our recipes in food preservation use bottled lemon juice or powdered citric acid. I'm showing lemons on the screen, but none of our recipes use fresh lemons, and the reason there is that there can be such a variability in the acidity of fresh citrus fruits. So we want, again, we're doing science, we want it to be consistent. Okay, now we're ready to extract some juice. So we've got our acidic fruit, we've got our sugar lined up, we figured out our pectin. So the method will vary depending on the type of fruit. So if you have a bumper crop of apples, you cook with a cup of water per pound of apples. And again, I would invest in a scale, it's always a, they're not very expensive, and many of the recipes actually include weights, as well as measures, or sometimes they only have weights. So it's, you know, for best consistency, if you're going to make these jelly or jam products and sell as a cottage food, for sure I would suggest getting a scale so that you're not wasting your time and having an unsatisfactory product. Keep in mind that when you extract juice, you know, it can take a long time to have this juice go through a jelly bag, which is basically a cloth bag or through a cheesecloth, for example, to get out that pulp. So you have to be very patient and just let gravity do its job. If you squeeze it, you're going to get cloudy jelly, because you're going to extract some additional things out of the fruit. So just be patient. With soft berries, you only have to add enough water to prevent scorching. And here's a good tip. Let's say it's a beautiful day like it is today, and you extracted juice with the intent that you were going to make jelly today, and now you're thinking, no, I want to go outside. You can actually refrigerate the juice for up to a week, or you can freeze it for several months.  It's anything that's solidly frozen is going to stay safe. So you can do some work ahead of time, but keep in mind that refrigeration and freezing have different shelf lives, of course. Okay, so what equipment do you need? Well, you need a heavy metal pan with a cover, the jelly bags I mentioned, and you can buy all of these things easily in many hardware stores, discount stores, and so on. Knives, bowls and mixing spoons, you have all these in your kitchen, I'm sure, measuring cups and spoons. A thermometer is a really good thing, especially a candy thermometer, for example, is really good, because often they have a little clip that you can clip to the side of the pan, so you don't have to hold it. Have a good timer, and if you have a phone nowadays there, it's easy to have a timer. There also are some timer apps available. I believe Oregon State University Extension has a timer and a food preservation app that's free. So something to explore a little bit. Canning jars and lids, a funnel, ladle, rubber spatula, jar lifter, and I'll show you pictures of these in a while, potholders, a canner, and a rack. So it's a little bit of an investment, but the good news is once you have these things, you can keep using them over and over, including the canning jars, but not the lids. You can use the screw bands over and over, but once you use a metal lid, you're done. That's over for that one. There are some reusable lids that are available. I think Tatler is the name, they're white. Those have not been fully tested by the National Center for Home Food Preservation, so they just say that you can expect some seal issues and not as successful as using the regular two-piece canning lids. No, I'll have more to say about those as well. Typically when we're making jellies and jams, they do recommend smaller jars. You can make a pint if you want, but half pints or a cup is typically recommended because as I mentioned earlier, you can have mold form. Mold will grow in the refrigerator. It won't grow in a nicely sealed jar, but as soon as you open it, you could have the mold growth happening. So that's why it's just best to open up your jelly jam. Enjoy it in about a week or maybe a little longer than that. It would be fine, but if you have a large jar and you're not going to use it all up, you could end up with moldy jelly or jam, and you have to throw that away. Sorry to say. Check those jar tops for any cracks or chips. You don't want to use those because those could lead to bad seals. You want to wash and rinse them, of course. You can do that in the dishwasher. We do recommend a sterilizing process for jams and jellies. So a nice big pot of boiling water, and you put the jars in and let them boil for 10 minutes, keep them hot in that liquid, and then you're ready to go. Keep them in the hot water. Of course, at this point, I want to make follow the manufacturer's instructions for treating lids. I've had many people call me and let me know that their lids came unsealed. I also had a call not too long ago that someone's raspberry jam had literally blown up. The lids flew off, intercovered. She had many, many jars, and what we found out or what we figured out was she hadn't put those jars of filled jelly into a water bath canner, so she hadn't killed some of the natural microorganisms that were present.  So that's why. That's why we tell you these things. As far as processing, this is how to do it. Some of you may be very familiar with how to do it, but place the jars in a canner with one to two inches of water over the tops, and so you have your hot jars. You want your simmering liquid. You set those warm jars, hot jars, into the fairly hot water. Otherwise, if you put it in cold water, you could crack the jars, and then once that water starts boiling is when you time, and typically the times are five minutes for zero to a thousand feet and 10 minutes for a thousand one to six thousand, and then don't disturb for 12 hours. You're just going to let those stand. So after this processing time, you got to be patient again, and in some cases it might take a while for for the product to set up.   It's nice, and it's exactly the texture it should be. It can take a while for that cross-linking and that chemistry to happen with the gel formation. So give it a little time. If you think that you had a total failure, you do have with any canned good 24 hours to redo it. So there's a couple tips I hope help you. Okay, I also have been asked what about just flipping the jars upside down. Technically that's not recommended by the USDA. It's what they do in the food industry. They invert the jars, but they have very, very carefully controlled temperatures, and scientists monitoring this process, and the temperature is typically higher than the temperature that we would usually attain in our own kitchen. So it isn't recommended. If you do it, you could probably expect more failures of the seals. You could find those coming unsealed. All right, well I grew up, many of you probably did as well, with paraffin wax sealed jars. Not recommended. Like many things that maybe our moms or grandmas did, and what they found with their research testing and all the ongoing tests that happen in food preservation is that they found pinholes, shrinking cracks occurring, and this could let moles and other microorganisms enter the jar. So again, the only thing we're recommending currently would be the two-piece lids. So if you see people doing these sorts of things, I would ask a special favor and let them know the information that you're learning today. It may be hard to convince people to change what they're doing. It happens to us in extension all the time, but I do ask that you share this with them because this is based on the latest information that we found on the national website. Again, the two-piece lids, we do recommend the water bath canning, and the reason again is to inactivate those molds or yeats and produce a good seal. I do want you to read those boxes that the lids came in because I've had a number of people that grew up with boiling lids, and that hasn't been a common practice or a recommended practice for many years. In fact, some of these lids don't even require you to do anything with them. You might want to wash them, rinse them. Some might have you put the lids in simmering water, but not boil them because what can happen is the lids could come off. They just got the plasticizer that plastic inside is too hot and the seal fails. So read the directions. When in doubt, read the directions on the package. Okay, let's take a look at nutrition. As Bob mentioned, I'm also a dietitian, and I just want you to be aware that these aren't calorie-free items, so savor the flavor. There's about 50 calories in a tablespoon serving, so half the amount of butter or oil, so that's good. There's no fat and about 13 grams of carbs in case you're watching your carbohydrates for various reasons. So they're giving you quite a bit of flavor for not a lot of calories. I think I mentioned this already, but a little bit more on the moldy jams and jellies. You may have grown up scraping it away, but we do recommend throwing it away because it could have what's called mycotoxin, Michael means mold, toxin is poison, could have mold poisons present. So toss it away, and again, that's why we suggest making the smaller jars so that you use this within a shorter amount of time. So this is the publication that Bob nicely linked for us on the Field to Fork website. We have more jams, jellies, and spreads. I have one that's called from the orchard to the table. Those are about growing different fruit trees and so on. I wrote with a horticulture specialist and we also have one about wild berries and fruits and using them to make jellies, jams, and spreads. So again, all these resources are free. Please use them, please share them, and so on. Now we're going to talk a bit about pickles. I'll ask the same question. I don't have a slide about it, but raise your hand if you like pickles, any kind of pickles. All right, so we've got some pickle lovers. So my favorite kind of pickles, I will tell you, and it's on our Pickling Guide, are green bean pickles. If you've never made green bean pickles or dilly pickles or dilly beans, I guess they're called try them out. If you like, whether or not you like green beans, they are wonderful. So what are the ingredients in pickles? Well, it can be almost anything. We're going to talk about cucumber pickles, first of all. So cucumbers, salt. If you're making fermented pickles, you never want to reduce the amount because salt is playing a preservative role as well as a flavoring role. Vinegar. We want the vinegar acidity to be at least five percent. So that means five percent acetic acid. That's the acid in vinegar. I've had people ask me, well, can I make my own vinegar? Well, you'd have to be able to be a really good chemist and figure out if it's five percent acetic acid. It would be much cheaper and easier to just buy white distilled or sider vinegar in the grocery store. Quite inexpensive. The other ingredients, there's usually sugar, typically white granulated or sometimes brown. If you decide that you want to use corn syrup or honey, it will alter the flavor. So again, I would suggest finding a recipe that's been tested that uses the ingredients that you want to use. Water. Typically, we want it to be soft water. Soft water is naturally higher in salt. So if you have soft water at home or there are ways to soften hard water. So if you want more information, I can just drop me an email and I will send you that process. Spices. Typically, we want those to be fresh spices. You don't want to use up some old spices you've had for a few years and you want them to be whole spices. If you use powdered spices, it will tend to darken your pickled product. So it won't look the way you want it. We don't use firming agents like limes and those sorts of things in our recipes. There are some really old recipes that use things that we probably would wonder if they're safe anymore for a lot of reasons. So we don't have any recipes with those sorts of firming agents. When it comes to your cucumbers, and here's a question for your horticulture specialist. I'm not a horticulture specialist, despite my last name being Garden. I guess I should have been. Um, but you want to use a pickling variety like burpee, pickler, hybrid, Parisian, bush pickle, harmony. Those are examples of pickling varieties. You could also check on the information that came with your cucumber plants to see if it's a pickling variety. Usually those will state on there. If you use table cucumbers or slicing cucumbers, you will have a poor quality pickle. It will not have the type of texture that you might want. And you want to try to get these pickled as soon as possible. In fact, best within a day of picking them. So leave them in your garden until you're about ready to start pickling. And you also want cucumbers to be firm and free of any spoilage. And the spoilage, you might see it, or it might be indicated by softening, for example. So if you're wondering how much a bushel is, for whatever reason, that's 48 pounds. And that would make 16 to 24 quarts of pickles. And there's about two pounds per court. So think about, you need a lot of cucumbers in order to make a fairly large amount of quarts. So two pounds in a quart. Whenever you're canning pickle products, you don't want to alter the vinegar, the food that's the ingredients, or the water proportions, or use a vinegar where you don't know the acidity. And you want to follow the tested proportions in recipes. In the end, what we're looking for is a minimum uniform level of acid throughout the mixed product. And we're trying to prevent that most serious of foodborne illness called botulism. That is our most deadly form of foodborne illness. A tiny amount of the botulism toxin can be very fatal. There's anti toxin, but it's a real scary one. You don't often think about it associated with pickles, because most people use vinegar, but it can be associated with other low acid foods that haven't been canned correctly. So some equipment you need, and some tips about it. When you're heating your pickling liquid, you always want unchipped enamelware. Because of the acidity, you could actually extract some minerals from certain containers. Stainless steel is great. That's a wonderful thing to use. And often if you go to a grocery store or a place where they're selling equipment, a discount store, hardware store, they'll often have all of this equipment all together in a canning section. You can heat in aluminum or glass, but you never want to use copper, brass, iron, or certainly not galvanized utensils. In that case, any of those, we could have the extraction of those minerals, which are actually toxic.  If you decide to brine or ferment, you would want a gallon size container, and it should be able to hold five pounds of fresh cucumbers. Or you can use glass or food grade plastic. For example, if you go to a, even a restaurant or a deli, they often sell those food grade plastic containers for a dollar or two. So I have some of those pickle buckets. They're green or white, and they're very inexpensive, and you can reuse them. You could also use stone cracks, but again, if they're chipped or cracked or otherwise damaged, you don't want to use those. So here's what you do. Again, similar with what we talked about with jams and jellies, you don't want any imperfections in the jars. You see the cracks, chips, dents, use those jars for something else, where they're not going to hurt someone. You might use it for a craft project, something else. Wash and rinse them, sanitize again, because you don't want to add any microorganisms to what you're doing, because that could affect your final product. And there's my reminder about preparing lids according to manufacturer's directions. In this case, we were just simmering these just to slightly warm them. So here's how to make the pickles, and we didn't have pickling pictures, so imagine. But you fill the jars with cucumbers. You always want to remove the air bubbles. Always wipe those sealing edges clean, because anything that's on the top, whether it's some spice or in the case of jams and jellies, some of that syrupy mixture, that is going to affect your seal. So you want to be sure those are nice and clean. And you add your lids, add your screw bands, and take a look at your hand right now.  Those are the two fingers, finger and a thumb that you're going to use to tighten the jar lid. It's fingertip tight with those two fingers. Don't get your super strong person into the kitchen to really tighten those up, because that's how we have seal failures. And you're not going to be happy if you've gone through all this work and your lids don't stay sealed. We do recommend a water bath canner. You preheat the water to 180 degrees for a hot pack. That means you're starting with hot ingredients. You preheat it to 140 if you're starting with raw ingredients. Load your jars. And again, you want the water an inch above the jars. So it's a good idea to have some extra water in a teakettle or something in case you don't have enough water. Cover it.  And at the point you have that boil, that's when you begin timing. So if you have a 15 minute process time, it's 15 minutes from the time you had that gentle boil, not from the time you put it into the container or the pot. And then look at the recipe. I just added the slide. So if you printed out the handout or looking at the handout, it won't be on it. I wanted to just quickly mention that there is a process called low temperature pasteurization. They did research on in the last several years, and this was developed so that the pickles would be crisper. But there are some guidelines.  You heat the water so it maintains that 180 to 185 degree temperature for 30 minutes. And you repeatedly check with a candy or jelly thermometer to make sure that that water temperature is at 180 during the entire 30 minutes. So it's more of a scientifically controlled, you're actually controlling the temperature of the water. You're not boiling the pickles. But again, you know, go to that national website where they describe this in greater detail. I just wanted to mention that this is something that's been tested by USDA. And if you do it correctly, you can have a very successful end product. Depending on where you live, and I see some people were in Bismarck and somewhere in Fargo, we have slightly different altitudes even throughout North Dakota, Minnesota, I even put altitudes for mountainous areas on most of our publications now because of the web anyone can access your materials regardless of where they live. So keep in mind that higher altitudes will impact the pressure and also the boiling point. So you want to know the altitude you are and process according to that altitude. So if you're wondering where you live, I don't know what you do if you're right along the line, I'd probably process the longer time if you couldn't quite figure out where you are on our line. But this is something we put on almost every publication that we have. So you can find your approximate altitude. And if you're from another state, you go to your extension service, they will have similar maps for your state. I'm 99% sure. Okay, you're all done now. And again, gotta be patient. You have to, once you take it out, set it on a nice soft area, maybe some dish towels, towels, and then you'll be patient. And you get the wonderful sound of the ping that indicates that your jars are sealed. We'll let them sit there for at least 12 hours, and that should say 24 hours, sorry about that, 12 to 24 hours. After they're all sealed, cooled down, you remove the screw bands to take those off before you put them in a cool dry place. And the reason there is you want to know that those seals are still intact. And you can't really tell if there's a screw band on top. We also could have the screw bands could become rusty, depending on how humid it is in the area you're storing your jars. If the lid's unsealed, examine, replace the lid, reprocess. There's a whole way of doing this. And we do describe how to reprocess your pickles or whatever products. That's included in our materials. So this is where you can learn more, the making pickled products. And again, Bob listed that it's linked on the Field to Fork site, and it's always on our food preservation information. And now I am happy to answer any questions or attempt to. Thanks, Julie. If you have a question for Julie, go ahead and put it into the chat box and we'll get to as many of those as we can. Julie, you were talking about using artificial sugars or low sugar without that. Are there a lot of recipes out there for that? And what do you recommend? Is it a good idea to try it or not? Well, my first inclination with lower sugar artificially sweetened would be to ask people why. Because one thing you can always do is eat less. Well, maybe that's not so easy, but you can certainly have less of that full sugar product. But there are recipes and maybe, Bob, if you could also link the reduced sugar spread on the Field to Fork site, that would be a good one. Some of those include Splenda. You really can't use Nutrisweet, you know, as you might find in a soda pop because with the heating process that would become inactivated. So there are some recipes, there aren't a ton, but there are special recipes that come with the modified pectins, like the high methoxy pectins that you can purchase. So a couple of ways to do it, artificial sweetener or use the special pectins and less sugar. Thanks. Yeah, you mentioned pectin and I know you kind of touched on this as you're going through your presentation. Are there any pectin replacements or you just have to use, if you don't want to use added pectin, you have to use just those fruits that, you know, will have enough pectin. Right. If you can't or don't use or, you know, the commercial pectins, then you would have to rely on a little kitchen chemistry and try making it without the added pectins. Okay, Becky, I see Becky's coming. Yeah, you want to jump in that one? Sure. I recently saw a video that recommended not boiling the jars before filling with jams and jellies. Call for jars to be heated in the canner, removed and filled, and then return to the canner and processed. Well, you could do that. Certainly, some of the recipes that I've also seen suggest that you don't have to boil the jars, just have to have them hot so they don't crack, as long as you're going to boil the jars with the filled jelly for at least 10 minutes. So that 10-minute boiling time to make sure that it seals will counter that time for the jellies and jams. So there are different ways to do these things, but these are the things I showed you are what would be located on the national canning site. I don't know if you saw Joyce's question there about, can you heat the water in the canner prior to putting the jars in? Yes. That's a good idea to begin heating the water because it takes a long time on most stoves, but you don't want to have it fully boiling. Otherwise, you could crack the jars when you put even the hot jars in. So you can certainly heat it. And then Diane's asking about any concerns about making habanero or jalapeno pepper jellies. I would suggest if you want to make like a pepper jelly that you find a tested recipe. We work with a lot of canners or people who are selling their products, and so I've worked with some of our food scientists on campus and we've tested the pH and some of the jellies and jams do not meet the standards for what the pH should be for a safe canned product. In some cases, those were invented recipes instead of using tested recipes. So, you know, find a tested recipe and I know there's a hot pepper jelly that's available on the national site. So, you know, use those tested recipes. And Linda's asking, she said she tried boiling apple peel and used the water in place of pectin for jam. She said it's okay, not as thick as she'd like. Is it worth the time to try that? Well, if you have the time, most of us have more time than we used to have. You know, it certainly is something you could try. But probably, you know, try some of those combinations of fruits, like I mentioned, with that are very high-impact and naturally. And apples are among those. And then Jean is asking about the vinegar to sugar ratio in refrigerator onions. Do you have any comments on just like refrigerator pickles in general or is that we don't consider that food preservation or? Well, it's short-term food preservation. In fact, we're going to have a brand new, we just tested it last week. It's like a quick pickle recipe with various vegetables. So that will be appearing soon. It is important to use vinegar sugar ratio that is recommended. But anytime you have something in the refrigerator, you know, that that is a form of preservation as well. The thing I want to caution people against is finding a recipe like a refrigerator pickle and decide to can it. That isn't necessarily safe. But you know, with the general short-term use refrigerator pickles, you can find those all over the web. I'm not too worried about those. It's the canned ones that kind of make me nervous. So thanks for all the questions, everyone. We are at three o'clock. If you have additional questions, please email Julie or contact your local county extension agent. You should very shortly have an email message in your inbox regarding the survey for today's presentation. I really hope that you'll take the time to fill that out. It really helps us improve this series and let our funder, the U.S. Department of Agriculture, know how people are taking these presentations. Thanks again, everybody. The recording will be posted soon on the Field of Fork website. Have an awesome afternoon. Thanks, Julie. Great presentation.


Alright you guys what is going on? Welcome back to another video. In today's video we are going to finally about a week late conduct the Major Key Physiques YouTube subscriber giveaway number two today's date is the 24th of December 2018 that means that tomorrow is Christmas day so I'm assuming that this video will come out on Christmas day and hopefully it'll be a nice Christmas present for three of you because there's going to be three winners we're going to get the screen recorder going we're going to get the YouTube video link from the Major Key Physiques giveaway video we're going to put that into the random comment generator and we're going to pick some winners so stay tuned guys have a sip of coffee if you've got it I've got mine of course and I'll see you back here in just a second okay so here is the channel and if we go across here no we're going to have to go to videos here it is I'm going to copy that link we're going to go to the random comment generator I'm going to put it in here we have 75 unique comments we're going to go down here we're going to click start and this will be our third place winner okay third place let's go and obviously you need a type done oh there you go great look at that comment it's set out fantastically winner Alex L says done first black track pants second gray and white track pants third black jumper so if you are third place then well we're going to have to take the black track pants and give them to you and that's exactly what we're going to do I don't know what size you are I assume you're probably not an extra small but regardless one pair of major key for Zeke's track pants are all yours Alex thank you for entering my friends let's get a second winner pick another winner let's go second place Brendan Graham says done great first item medium black jumper second item medium black track pants third item the black track shorts okay so we got a black jumper and black track pants although we've just given away the black track pants so we're going to go jumper and track shorts here are the track shorts and here is the jumper okay Brendan is done now last one first place let's hope I've actually got these items left I would have assumed someone would want a single it although they haven't said it yet so let's go first place who's it gonna be Chris see this is a terrible comment Christopher try rock Christopher try on done shorts pants top well which ones do you know what it actually makes it easy for me because now I can give them anything any shorts any pants any top a winner's a winner first place you said done you've come and followed the the instagram page which is good and now I'm going to give you I'm going to give you which ones probably the biggest size we've got an extra small which is going to be too small and we've got a size small so you my friend Christopher you're going to get a white on gray pair of track pants and you are going to get a black jumper and unfortunately the shorts are gone so I'll make it up to you with some socks okay major keeps the socks these were an early prototype an early sample and they're the last pair so they're yours last but not least we've got a kid's jumper for each so each winner is going to get a bonus kids jumper a black and white you know it's probably going to suit like a four four year old maybe five year old and here's one for the second place so we've got three of these one for each winner okay guys so third place whoever you were I've actually forgotten the names I'm going to have to look back at the video third place we've got a pair of track pants black and white and a black and white kids jumper second place we have a pair of black and white track shorts we've got a black and white jumper and we have a black and white kids jumper and first place merry christmas we have a pair of gray white and gray track pants we've got a black and white jumper we've got a pair of socks and we have a kids black and white jumper so there you go all done christmas giveaway done and dusted and I would assume looking at the pile of stuff which is pretty much non-existent now that's that's honestly that's like all my samples gone although there will be one more there'll be one more giveaway and I'll tell you exactly what you're going to win hold on next minute in the next giveaway we're going to have a choice of a singlet white and black we've got gray sorry gray and gray track pants and we have two of the black and white kids jumpers so you know it's not a not a huge amount left but that's what I wanted I wanted to get all this stuff out to you guys I mean that's that's been my strategy this whole entire time with these sample garments is to just offer them in giveaways get them out there you know you know I've always said I've got no expectations any any pictures you want to take and send them through you'll get an automatic repost on the page so that's good for you and it's good for me but if you don't want to take any pictures then at least just wear it possibly tell someone about it and that's all I can ask so I hope you've enjoyed the three winners I will be in touch very shortly and as always guys have a fantastic day if you have liked this video hit the like button if you want to subscribe please do so that was a merry Christmas from me and major physics to you have a good one


Hello everyone, my name is Gotti Delorme and I'm a software engineer at Apple. Today we are going to talk about how to run cash efficient builds in Kubernetes using Billkit. So first we'll discuss our goals. Then we will go a bit deeper into Billkit itself and what kind of features it offers. And we'll finish by discussing how to schedule Billkit jobs efficiently in Kubernetes. So our main goal here is to duplicate build system bare metal infrastructure. Many organizations already did the transition from bare metal to Kubernetes for application deployment, but a lot of them are still relying on bare metal for build systems, CI systems or any other kind of automation systems. As part of this move, we want to make sure we keep running secure fast and cash efficient builds. By fast, I mean at least as fast as on bare metal. We are also looking to provide a way for developers to run CI builds locally, like in the same environment. And we also want to maintain like a single way to run builds, a single build system that's not specific to stack, like not specific to Gradle, Go or Restful. It should work with all of them. So Billkit is a builder toolkit created and maintained by the Mobi community. It's also part of Docker since 2018, but it's disabled by default. Compared to Docker, it supports way more features like concurrent multi-stage builds, persistent caches, build time secrets, outputs others and Docker images, and frontend others and Docker file, and also rootless and Daemonless support, which is super convenient if you want to run Billkit in a non-privileged container. So if you want to try Billkit, you can use it directly from Docker. To enable Billkit, you just have to set the Docker Billkit environment variable to 1, or you can also rely on the BillDix command. Or if you don't want to install Docker or you don't want any Docker dependency, you can use BillCTL directly to run builds using Billkit. So let's talk about multi-stage builds. On the left side of this slide, you can see a very simple Docker file containing three different stages. You can also notice the last stage depends on the first one and the second one, but the second one doesn't depend on the first one. When you run a classic Docker build, it will execute each step sequentially, as you can see on the right side. So first stage one, then stage two, and then stage three. When you enable Billkit and you look at the Billkit log, you have the build log, you can see it runs stages in a different order. In the log, we see step number four, then step number seven, then step number five. So what's actually happening here is Billkit will build a graph representation of your Docker file, and will make sure to run concurrently stages that don't depend on each other. So in that example, since the second stage doesn't depend on the first one, we can run both of them in parallel, and then we can finish by the stage two. It also means if you make a code change in the first stage, you will not have to rebuild the second one because it doesn't depend on it. While with classic Docker, you will have to rebuild stage one if you change stage zero because it executes the build sequentially. So Billkit offers a better layer caching than Docker. Another cool feature of Billkit is cache mounts. So cache mounts are super useful when you want to persist caches from compilers or package managers. For example, like when you add or delete or upgrade the dependency, you don't want to have all of them to be read and loaded because the cache layer has been invalidated. In this example, I'm showing how to use cache mount with bundler. So you can see like the first run took the same amount of time with the cache and without the cache. But then when we add, upgrade or remove the dependency, you can see it's way faster if you rely on a cache because it doesn't have to read and load all the dependency. It just has to download the new one or the ones that change. Another cool feature is built-in secret. So with Billkit, you can share secrets with build without linking them into the produced Docker image or its metadata. You can share secret from environment variables or from files and you can expose secret to just specific instruction in your Docker file. So in that example, I'm mounting like a secret client key to be used in a code command as part of my build. Billkit also supports other type of mounts like bind mounts, for example. So bind mounts are useful when you need to mount some directories from your build context, different stage in the build or also different Docker image. And for example, like you want to mount some data or some binary and but you don't want to persist it into your produce image. You can also use SSH mount. It's pretty useful if you need to run some SSH command within your build. For example, like git clone command using SSH URL. And you also have tempfs mount. Tempfs are useful when you need to mount a tempfs directory into your build because you don't care of some data or you don't want to pollute your resulting image with some logs or something like that. Billkit also supports output others and Docker images. So you can produce Docker images with Billkit obviously, but you can also use what they call local output. So local outputs are a way to report a specific set of files from your build environment into your host where you run the build. So that's the output we will use to produce any kind of artifacts like binaries for example. You can also produce tarbols with Billkit. So it supports three different types of tarbols. The first one is tar. So tar is similar to the local output but archived. And then you have the Docker and the OCI type to produce Docker images, tarbols or OCI images, tarbols. So let's see how we can build a simple binary using Billkit and a Docker build command. So on the left side of the slide, you can see a simple Docker file that runs a go build command and then copy the binary into a scratch image. So because we use a scratch image, we know like the image is empty except for the binary we just copied over. So when we run the Docker build command and we specify the output local type, we can see at the end of the log, instead of producing a Docker image, it actually copies some files from the build environment to the current directory. And when we try to run the produce binary we just copied over, we can see it works as expected and we get the hello world. So that's one way to produce artifact other than Docker images using Billkit. Billkit also supports exporting layer caches. It's pretty useful when you have different Billkit instances running on different hosts and you want to share caches across them. Just note it's only export layer caches. It doesn't export mount caches. And also, depending on the size of your cache and your Docker registry, it might be slow to export and import caches. So you will have to decide by yourself if it has a good or a bad impact on your build. So you can use custom front end with Billkit. So front end is a component that converts a build definition into LLB. LLB is a low level builder. It's the graph language used by Billkit. Of course, if you build a Docker file, you don't have to provide a custom front end because Billkit has built-in support for it. So how it worked is basically you have your build definition. So for example, a Docker file that consumed by a custom front end. That custom front end produces some LLB and this LLB is consumed by the Billkit daemon that will run your build and produce the output of the build. So front ends are usually defined as Docker images and specified at the top of your build definition using a specific syntax containing the Docker image name. So in this example, because it's KubeCon and I'm sure we all love to write the ML here. I'm just giving like an example to see how we could replace like a classic Docker file with some ML syntax. So at the top of the build definition, you can see I provide a custom front end. So Billkit will know how to convert this ML into LLB and will be able to use it to build. Because of this, I can actually build this ML directly using a Docker build command if I have Billkit enabled. So it's pretty powerful. So front ends are typically created using the official Billkit Go library. But the Rust community created a crate to be able to write front ends using Rust. Billkit also supports running as non-root users and in non-privileged containers, which is what we are going to use in Kubernetes. Just note you have to disable app armor and sync comp. So for Kubernetes, it might not be a big deal because usually it's disabled by default. But depending of your cluster configuration, you might have to disable it explicitly. So you can do that by adding an annotation and a new field to your security context in your pod specification. And you can also run Billkit as Daemonless. The trick here is to start BillkitD and BillCTL within the same containers. So for example, you start the Daemon first, then you wait for the Daemon to be ready, and then you can start your build using a BillCTL command. So now that we know more about Billkit, you probably wonder how can I run Billkit jobs in Kubernetes.  Which one is more secure? How do I leverage caches? And all of those questions are totally valid. So to answer them, what we are going to do is for each of the type, we are going to list the point counts, and then we will decide which one works the best for us. Let's start with deployments.  So you will most likely have a BillCTD deployment containing multiple replicas, so multiple pods. Because of how deployment work, it's very hard to know on which pod your build will land, so you will be unlikely to hit a local cache. Also, because like the deployment Daemon is decoupled from the actual build submission, you cannot really specify how much resources the specific build requires. So you will most likely have like over, you will most likely like over request, or maybe you won't request enough resources ahead of time, so it will lead to poor resources utilization in your cluster. So with stateful sets, we are in a better place for caches because pod names are consistent in stateful sets, so we can rely on something like consistent hashing to hit local caches reliably. So consistent caching is a way to make sure like same builds will always land on the same pod within your stateful sets. But it still suffers from the same limitation as deployment about Daemon lifecycle, you still need to manage your stateful set separately from your actual build. And you also cannot like provide how much resources your actual build needs, so it means you will have like the same resources utilization limitation as with deployment. But with job, it's actually better because we don't need to manage Daemon lifecycle separately. We will start the Daemon within the same pod, so the same containers as the build. So because of this we can also specify exactly how much resources the build needs. And there are ways also to leverage local caches with job, and it requires a bit of build kit knowledge, so we talk more about it in the next slide. As a quick recap, here is a table listing like all the supported features per resource. You can see job is probably the best fit for us. So we are now going to discuss how to leverage caches with jobs. So the first thing we want to do is find a way to identify similar builds, build from the same project, because that's the builds that should leverage the same caches. So what we are looking for here is basically our cache key. So usually the cache key is like a project name or like a git repository, maybe like the git repository and the context path, something like that, basically any key that makes sense to you. Once you have the key, you need to know how to mount the corresponding cache into the job, into your build. If you use rootless build kit and you should use rootless build kit, you will have to mount the cache at that path, so slash home slash user dot local share build kit. Also, you will probably noticed you will have better performances with hospice volumes, rather than distributed or volume or remote volume storage. I also want to use local volumes, but because I don't support dynamic provisioning, depending on your setup, it might not work well for you. So in this example, I'm showing how to mount a hospice volume at the build kit rootless path using the specific key we decided in the slide before that one. Then we want to know how to make sure the build will be scheduled on the node where the cache is. And to do that, we are looking to rely on the default Kubernetes scheduler and we will use labels to identify pods and matching nodes. So we will use preferred node affinity to always attempt to schedule a build where a cache is available. But we also don't want to block a build to be scheduled if the node containing the cache is, for example, full of other jobs and so doesn't have enough resources to handle the new build. So in that case, the build will be scheduled to a different node. Also, we want to make sure we prevent cache collision and to do that we will rely on required pod entire affinity to make sure two pods relying on the same cache. So with the same cache key are never scheduled on the same node at the same time. So here is a slide to show you how to define your node affinity and pod entire affinity. You can use that slide as a reference later if you want to try it out. Okay, so we just saw how to build our job specification to make sure we can leverage the Kubernetes scheduler to schedule our builds and mount the corresponding caches. But it's pretty manual and we probably don't want to have like complex job definition for all of our builds. So we are looking for a way to automate all this logic for us so we can focus on like the actual job specification. So do that one solution might be to rely on the Kubernetes operator pattern. So if you are not familiar with the Kubernetes operator pattern, it's a pattern composed of two components, CRD, custom resource definition, which is a way for you to register your own custom resource into a Kubernetes cluster. And then a custom controller and the role of this custom controller will be to watch and update your custom resources and make sure all the Kubernetes resources required by your custom resource are always created and up to date. And all their status changes are reflected into your custom resources into your custom resource status. In our case, our custom resource might look like this. Basically, it's very similar to a Kubernetes job, except it comes with built-in support for caches. So in that example, I created like a cached job resource, and its spec is like a Kubernetes job spec, except it has a caches key. That key is a list of caches to mount to the job, and each cache is identified by a key and has a mount path where to mount the cache. So let's see how it works in practice. So we have like a custom resource entering the cluster. The custom controller will create the corresponding Kubernetes job will add the proper label keys. And then like the pod will be created, and we will delegate the scheduling to the Kubernetes scheduler. Because none of the node has any caches yet, the Kubernetes scheduler will schedule the pod to one of the node. So in that case, the first node. Now we have a second custom resource entering the cluster, still with the same cache key. The controller will create the corresponding job and pod, and we'll make sure they have the proper labels. Because we have a job running on the first node with the same cache key, and we rely on pod entire affinity. We know the Kubernetes scheduler will make sure that new job is not scheduled on the first node because we will have a cache conflict. So it will schedule the new job on the second one. Then we have a third custom resource entering the cluster. The controller will create the corresponding job at the required levels. We can also notice the first job running on the first node just finished. And the first node was labeled by the controller with the cache it has on it. So the cache built by the previous job. Because the new custom resource requires the same cache as the previous job, it will have the same label. And because we use node preferred affinity, we know the Kubernetes scheduler will schedule the pod on the first node so the new pod can leverage the cache built by the previous job. And now we have again a new custom resource entering the cluster, but this time it has a different cache key. The controller will create the corresponding job at the expected levels and delegate the scheduling to the Kubernetes scheduler. Because none of the node has any cache matching this cache key, it doesn't really matter where this pod is being scheduled. So at that point we have two build, so two jobs with the same cache key running on the first node and the second node, and one job with a different cache key running on the second node. When all of them are finished, the controller will make sure the nodes are correctly level. And you can see the second node has two labels because it has two caches on his disk. So you probably wonder how to clean up caches at that point. So one solution might be to rely on diamond sets to have like one pod schedule on each node. And the role of that pod would be to watch the age of the cache to clean up the odd one or maybe watch the available disk space to make sure like odd caches are cleaned up to make space for the new caches. So what did we achieve here? So we saw how to run secure, rootless, diamond-less, fast, cache-efficient builds by relying on build kits, mount caches, and layer caches. We also saw how to run like the same build kit builds in Kubernetes and locally using Docker build or build CTL. And we also saw how to use build kit to produce Docker images, but also any other kind of artifact like a GoBinary, for example. So it seems like we have a pretty good way to deprecate build systems by a metal infrastructure and move them to Kubernetes. That's pretty cool. And Apple is looking for Kubernetes engineers, so please don't hesitate to reach out and go to jobs.apple.com to see all our open positions. Thanks everyone for your attention and see you later.