and we've been talking a lot about bark, and we kind of coined this phrase barkophile because we love bark. And like Tom said, I mean, there is a lot of value to that for an ornamental feature. So, what's so great about bark? We have just that. It's an ornamental feature. Not only just fact that, you know, leaves are not on a lot of these trees for a good part of the season, but they do enhance the quality of the actual ornamental plants as well. I don't do handouts, so I do apologize. So, and we're going to go through quite a few trees. But with bark in itself, it is very diverse. We have different colors. We have different textures. We have exfoliation bark, and that's what gets really fun. We have early exfoliation when they're young, and then that we may lose that. We have some species that don't have exfoliation when they're young and they get it later in their life, so there's a lot of variability, a lot of exciting. But ultimately adds multi-seasonal interest because this is what we're used to, unfortunately. And this is a pretty typical Fargo landscape, which is absolutely horrible. So, we want to get beyond our despair, and so why do we want multi-seasonal interest? Because we want to have happy trees. So, just like Bob Ross, we're going to have happy trees. I wanted to start with one that will never ever grow, but this is a zone 10 to 11. I apologize, at least for those of you in this room in Colorado. This projector is absolutely horrible. So, what is this? Not sure. That's not at all. This is a rainbow. You can use this to get rid of the chance to go in. Okay. Close right. On the line or to the right. You have an opportunity to just touch a bark. And, again, this projector is horrible. This should have some nice reds, orange, greens, yellows, all in it. So, I'm going to apologize, at least for here in Fargo. This presentation is going to be absolutely horrible. Menture and stride maple. So, I'm going to start with some maples. And, I just want to talk in general about them, and then we're going to look at the bark. So, this is a smaller tree. It has a really nice form. The fall color is yellow, so nothing too exciting there. But, what's neat about it is the bark itself. It has this green-white vertical striping. And, it maintains that for a good long time. Here, you can see the fall color in contrast with that kind of green and white striping. And, so it's a really, really striking plant, you know, in the landscape. Acer trifluorum, this is one that is reliable for a good fall color. But, it is propagated by seed. And, so, it's variable with the colors you're going to get. You can get some yellows, orange, or reds. And, yes, you have selected one that's red. So, you get a really nice fall color, good high-quality foliage in the summer, too, with the trifoliate maple. So, it has three leaflets, not a single leaf. A smaller tree, 25 by 20. And, what's great about it is it has a vertical, fissuring, peeling bark. And, it is quite attractive. And, it's actually seen very early on in its growth. So, it's a very attractive tree, very early on. Here, you can see the three leaflets. And, it actually, you know, gets a really nice kind of pumpkin, at least this selection is orange or glow. And, this has been selected more for that orange-red color. Here's just another couple shots of what you can see. And, unfortunately, with a lot of these trees, people don't properly lign them up. So, you can actually see the bark. And, here's a good example where it has been properly pruned to enhance the bark feature. What's nice about the triflorums is that we don't really see the damage with sunsculls in the winter because of this type of bark. This is a new one for the NDSU release. It's the costata birch, the dwarf Korean birch. That's what I'm really excited about because we have river birch, which we're going to mention here in just a moment. But, there is one that's similar to this, it's a dwarf. But, it's not reliably hardy for us in our region. And, this one is extremely reliable. It's also a dwarf. We got a picture of a full-sized costata when we were in the Morton Arboretum down in Chicago. And, as you can see, it will be about 25, 30 foot tall. Ours is only about 8 foot tall. And, the name is Cinnamon Curls because the inside of the bark is actually cinnamon. On the outside, it's more of a creamy white. But, a really beautiful natural form. And, again, this projector is absolutely horrific. But, the background of the snow on this plant, and when you see it, it's just absolutely gorgeous. Great profile, outstanding color. You know, this is one that would make a great accent plant into any entryway, whether it's a residential or a commercial type setting. Which goes into then river birch. River birch for us. Most of the cultivars are not extremely hardy for us. We hit about that zone 4. Once we get into zone 3, we see a lot of tip dieback. Even in a zone 4, we can see some tip dieback. Also, with river birch, we see a lot of problems with chlorosis. Just what was mentioned earlier with the soil. So, we can have some of those issues. So, we got to be mindful of our soil type. But, with river birch in general though, we have that really nice, creamy peeling bark. And, this is more of a later. It doesn't start to peel until it starts aging a little bit. What's nice about river birch compared to all the other birches is that it is completely resistant to bronze birch borer. Which is the number one problem that we have with birches and ultimately with decline of them. And, this doesn't have that. We do have a release out of NDSU that is resistant also. It's tolerant to a pH, high pH. So, we don't have the chlorosis issue with Northern Tribute. And, that's going to be on the market here in a few years. So, don't get too excited about running to the nurseries and trying to find this or asking for it. It will come eventually. But, no winter kill, no chlorosis. And, then the outstanding, just beautiful peeling bark.  You know, that's why we have these barco files.  Heritage river birch is really common. But, this is one that you'll find in the nursery. And, I don't really recommend it because they do get some winter kill. Especially if you're into a true zone 3 somewhere in the state. So, you do have to be careful with this one. Paper birch is one that does very well for us. But, again, it is susceptible to bronze birch borer. So, you'd never want to allow your birches to be stressed. But, we get that nice, peely bark. Again, it's being washed out for us here. But, you know, really a beautiful, beautiful bark in the winter time. Even during the summer because it's just, you know, nice vivid white really stands out of the landscape. But, we want to make sure these birches stay moist, cool, with our soil. And, that helps with the bronze birch borer component. If they get stressed, then you can have serious issues. Prairie dream paper birch is the one I recommend. Not just only because it came out of MDSU. But, it is bronze birch borer resistant. Also, within drought we had with 2012, all of the birches out at our research farm were in pretty poor condition. A handful of them were doing well. That's worth Korean birch I mentioned. As well as prairie dream. And, this is readily available in the trade. You can find it in many nurseries locally. But, it just really performs well. And, it's the brightest white bark out of all of the cultivars of paper birch. So, quite striking, you know, even with the yellow fall color. You know, we need to have diversity. But, it's nice sometimes even to have the yellow. But, that yellow and white is quite striking to have that bark color. I'm a big fan of horn beam. And, so, carpina, the older branches develop a fluted muscle-like appearance. So, just like what you see, you know, with your arms and your legs with, you know, with our muscles, that's the type of wood that this develops. So, it's kind of creepy at first. But, it kind of, you get used to it. But, it's very unique. But, it is a smaller tree, 25 foot. One of the big problems is it is two of those variable fall colors. So, I know we're talking bark, but it's also nice to have these other ornamental features. But, it's pretty much what's propagated by seed as well. So, you just didn't know what you were going to quite get. It's tolerance of periodic flooding, which is nice. So, if you are in an area where you do receive flooding, it's a good one for that. It's, that's its entire environment that I really enjoy. But, what's great, and part of what this talk is hopefully going to come across is not only is bark exciting, but it's just another feature to focus on and a different way to look at these trees. But, also, here we have one that's pest-free. And, we have the big problem that we're facing now, and we'll ask for, you know, not a matter of if, but when it's going to hit North Dakota. We have Asian longhorn beetles coming as well. And, so, we need to get diversity.  And, so, we need to get variety. We need to have diversity. And, so, here, this is what this is all about, is talking about diversity and ornamental features. So, here's how that fluidic look to it. Traditionally, the fall color, again, is more of a yellow. Out of Johnson's Nursery in Southeast Wisconsin at Menominee Falls, they released a, what's called, fire spire. And, it's a very tight upright of a horned beam with really nice fall colors, reliably red fall color. And, it's a quite striking plant. It's one I'm definitely recommending. I'm a big fan of hackberry too. You know, hackberry is just a beautiful tree. Used a lot on boulevards because it can withstand just about any condition, compacted soil, poor pH, you know, whatever, extremely urban tolerant, great tree of, you know, again, we're looking more to yellow fall color. But, with the corky bark extensions on them. You know, I kind of enjoy that. Some people don't really like it, but I find it, you know, to be intriguing. It's, again, a little bit of variety than just a standard smooth bark. Ketsura tree, this is a tree I really love. It doesn't maybe have necessarily the best bark, but it has a little bit of a feature to it. But, it's the overall package that I really love about this tree. And, not too many people are familiar with Ketsura tree. But, it has a, kind of more of a shaggy bark where it attaches in the middle and it'll split on either end at the top and the bottom. And, I'll show you a picture of that here in just a moment. But, here's the form. It's 40 by 25, so it's still a bigger tree now. But, it has a, you know, a little tighter, kind of width to it. Again, it has variable fall color, but that's actually one of the better features. But, the leaves, really, and again, you know, we're talking bark, but now we're going to talk about leaves. The leaves are kind of cool, because when they first come out they have a citrus fragrance to them.  If you have a bench underneath there, you can really enjoy that. And then, when you end up when they synapse in the fall, they actually smell like vanilla and cinnamon. And, this is quite striking. You can smell this 50 feet away. And, I really enjoy it. But, here's the bark, where it gets this kind of peely bark when it's younger. When it gets older, it kind of makes this little zig-zag pattern. So, you know, there's, you know, not the most exciting bark, but it's the whole package, I think, that I really like about it. And, again, this is, I think, projector is unbelievable. Because this is not neon leaves. So, but it is a mix of these apricots, kind of red, apricot, orange, yellow. It does this real mix of a fall color. It's not just one solid color, which I find quite attractive. And, again, so you can, they can be kind of variable, but, you know, this should be, this is a really nice apricot kind of color, but it's not really coming across. But, whenever I go into, like, an arboretum in the fall and there's a katsura tree, I mean, you can just smell, like I said, from 50 feet away and you're looking for it. Where is the katsura? Because it's quite, quite fragrant. A Kentucky coffee tree, again, this is one that is awesome for the urban environment. It does have that interesting bark pattern. Again, these are not ones that are, you know, jumping out saying, wow, amazing bark. But, it is cool bark in general, but it's the whole package. Again, very large tree, 70 by 40, maybe a little bit smaller, again, depending on where it's located, but it's an urban tolerant tree, coming up on a lot of the Boulevard lists throughout the state. No serious pests or disease issues, at least at this point, you know, so, again, it's about diversity. But, the bark itself, it's a very stiff bark, you know, and when you look at this at first, it's like, you want to go ahead and start peeling it like you do, like a river birch, but you really can't. It's a pretty stiff bark, but it has a lot of good textural component to it. American Hophorn Beam is also another one that is gaining value, and I'm really excited about Emerald Ash Forest. I think nothing better could have happened, because now we're forced to do diversity. You know, it's great that ash, ash are great trees. Don't get me wrong, but 80% in the town is not a good idea. And so, it's great that Emerald Ash Forest is coming across, so we will be forced to diversify. Because that is what we need. We never learned, you know, Dutch Elm Disease, right, you know, we learned from that, not really. But, here, the same thing, we have that vertical fissuring that exfoliates at the end, similar to that young cassera. But then, again, the whole package, we get this hoplite fruit, again, a little bit larger tree, 30 by 20, but no serious pest or disease. You know, so these are the things that we should be looking at. One big liability, though, is not tolerance of salt. So, you know, I do see them on the boulevards, and I don't think that's a really good idea. But the bark itself, you know, has a really neat texture to it. You know, here, you can see where it is doing the long fissuring of peeling. Here, on the older bark, you get more of a plating type of an effect, but it is a nice visual. But the fruit is cool, because it does look like hops. And the fruit itself actually has, it's blown up with air. And it is kind of cool. It will turn brown in the fall, fades from a kind of yellowish green to a brown. Here, in this picture, you can see how it's dotted with the light green. That's all the fruit all over it. So, just adds a neat little feature to it. But again, diversity, really no pests, really nice tree. Amber cork tree, this one, too, no pests, you know. And I don't understand why we're not promoting these more. Why, and that's, again, why I'm so glad that emerald ash borer is coming through. Bark is really heavy and quirky. This is one where you want to touch it. You don't realize you want to touch it until you finally do touch it, and then you don't want to take your hands off from it, because it is a spongy cork. And it's maybe not necessarily the most attractive bark, but it's spongy, and it's really weird. And as soon as you experience it, you just got to touch it. But it's wider than it is tall. So, this is one that will never end up on a boulevard. It should never end up on a boulevard. But it's, you know, it's a nice park tree. But, again, no pests. So, we're going to look at the one on the left first, then look at the one on the right. So, this is the pattern. So, it's a really irregular look to it. And, again, it has a spongy cork to it. They don't make cork from this bark. It's from other species. But it just has a really neat feel to it. Now, it does have the inner bark as yellow. It's one of the few species that have the inner bark as yellow, but, again, that's not a good thing to test and find. Not very helpful to the tree. We have sycamore. And this is one that's out of NDSU, but they are growing across the state. It's one that I think we definitely should be utilizing a lot more. It is a bigger tree, though. So, again, you look in 40, 50 foot tall, 20, 30 foot wide. It's a big tree. But it starts to exfoliate in these pieces that will shed very early on in its young branching. And then the older growth continues to peel into this kind of a gray type of appealing. And you get kind of a model of the fact with multiple colors. So, it's really, really attractive. I am a choke cherry. You get a really nice cinnamon brown bark to it, 30 by 30. You know, again, we do have the flowering effect as well. You know, so we do get a really nice color. We also get the exfoliating bark on the problems that we do have as splitting with snow loads. So, that can be problematic with this tree. You know, this is typical to see a nice crack going through it. So, it's not necessarily one I would recommend, but it does have on a mental bark that's quite attractive. Small white oak is definitely one we should be utilizing a lot more. 50 by 50, it's a big tree, drought tolerant, you know, that's a great feature for it. But the young branching, it starts out, again, with this exfoliating bark, you know, which is not very typical of oak. So, really good urban tree for us. Here you can see another shot where the bark is just splitting. I mean, it looks like it has a disease or something where it's kind of split and shed the bark, but there it is. And then it does hold on to its foliage into the fall or into the winter. So, you got kind of a neat effect there. Burrow is highly variable in its bark. You know, it's one that, again, is a good urban tree for us. Now, we do have diseases moving in on that too. So, in diversity, there's a new one called burroke light that's going to decimate the burrokes. 70 by 70, pH adaptable. Again, it's variable in the amount of cork. Some of you can find are quite corky. What you should, though, if you're looking to grow a burroke, when you go to the nursery, find one that has extensive amount of cork, that will be a characteristic that holds on into its maturity. If it's light on cork, these corky extensions, it will never develop more cork. It will always be light on the corky extension. So, there are some now that are being selected for the sixth corkiness. So, cobblestone oak, that's a cultivar from Jay Frank Schmidt, where it's been selected early on for that type of corkiness. Peek and lilac. You know, Japanese tree lilacs are really popular. It's one of the most popular trees that's being planted now. I recommend peek and lilac, because Japanese tree lilac does not have exfoliating bark, whereas peek and lilac does. And you still have everything, because it's a subspecies of Japanese tree lilac. And so, you still get the creamy, panicle flower, but then you get exfoliating bark. And there's several cultivars that have been selected for this coppery color, one being copper curls from NDSU. And it maintains this peeliness all the way into its maturity. The color does fade a little bit, but it just is quite a striking plant. There is China snow out of the Morin Arboretum, and that's one that's also been selected for a high amount of this coppery kind of color and the exfoliating bark. So, beautiful, beautiful tree. Again, the peek and elastics, they tend to be a little bit more multi-stemmed, as you can see here. So, that's one of the disadvantages compared to straight, reticulata of the Japanese tree lilac. But just gorgeous bark. Just a couple of evergreens, and we'll be done. This is not one of my favorite trees, Layspark Pine. You know, I don't understand why people are not planting it more. I found it when I was in Inner Mongolia, which is the exact climate that we have. And so, we're working on this one now. Thirty by thirty. One problem though, it can have some limb breakage issues with the crotch angles that it has, but I think it's definitely worth trying. This is one though, I don't know about a zone three. If you're in a true zone three, I would be hesitant in using this. So, those of us in zone four here now, this is one, you know, definitely, you know, Fargo, Bismarck, and across the zone four range. But it does have this exfoliating bark, and I call it the camo tree, because once it starts dropping that bark, it ends up looking like this. So, you get all these different shades of green and all these different patterns. And, you know, it ranges from some yellows, some greens, and again, this is kind of washed out. This is all pink all through here. So, some can be turned into more of a pinky color, so it's quite attractive. Last is our Scotch pine. I think a lot of us are familiar with Scotch pine. Really nice pine for us. It tolerates drier sites, it's pH-adaptable.  And it does exfoliate quite readily, so it's quite attractive. But this past summer, I was in Chicago, and I saw one of the coolest things, where in a Japanese garden, they're using Scotch pine and pruning it to get this Japanese look. And I think, wow, that would look awesome in everybody's backyard. Pruning it that way, and I'm like, that's how all Scotch pines should look.  Because that's what Scotch pine does. They like that horizontal branching. So, just gorgeous, gorgeous specimen. So, I'm going to get some and start doing this to them. And even out at our research farm, if you ever come out, they're all going to look like this. So, with that, happy trees is with all the different ornamental qualities, but more importantly, diversity, so we can combatter past. Thank you, guys. I'll give you a quick question, and guys, if you could please remember to repeat the question. First question, how much of what this bark is genetic? How much is environmental? Is there anything we can do to carry for the trees to create more bark? Okay, so the question was, how much is genetic? How much is environmental with this bark? And for the most part, it is primarily all genetic. And that's what breeders and plants improvement people are actually looking at, like that cobblestone oak, is actually selecting for bark color, such as like with copper corals, pecan lilac. Because here on campus, we have copper corals next to a straight pecan lilac. The straight pecan lilac has a dark brown, not very exciting bark, whereas the copper corals has that coppery bark. They both exfoliate the same, but the copper corals is copper, whereas this other one is not. And so, that's been propagated asexually, plants it everywhere, and it maintains that bark. There's not a whole lot we can do to necessarily enhance that. Like with the late bark pine, you could, and, but again, you know, maybe as us as avid gardeners, because we're a little crazy, we could go out there and actually peel.  You can actually pull the bark off and create these patterns. There's another tree that's not suitable for our zone, but it's a Japanese Duarte. And I actually recommend to people, they actually scrub their bark. Because as you get the plated pieces come off, they discolor over time at different rates, and that's how you get that weird coloring. But there's really not a whole lot we can do. It's just properly selecting trees that have these. And the idea, again, is like with the fir oak. You know, if you're selecting with the nursery, and a lot of them are propagated by seed, if they're not necessarily a true cultivar, it's select ones that have good bark right from the beginning. Okay, the next question has to do with the hardening. Especially in northern counties, there's some skepticism out here about the hardening of the leaf trees to highlight some of the leaf trees that may not survive. Some of the leaf hardening ones we talked about tonight. Yeah, the lay spark pine. So the question was, you know, hardiness, especially in the northern counties, you know, there's some skepticism on survivability of these.  I would definitely put a solid four untested three. Lay spark pine is definitely a four. Questionable three, you know, those are the things that I'm going to be working on in the coming years to test them at some different various sites in the north. You know, we're going to be planting at Langdon. We're going to be planting at Williston. Just have some of these and hopefully we can find out. But definitely with that lay spark pine and the capture, I'd be very questionable on using those. A lot of the other ones that wouldn't have any problem get into a zone three. And because I really, that's what I try to focus with this talk is to focus on zone three or four plants. How about you talk about finding the tree in North Dakota nursery? Are there any tips on finding some either relatively unknown and most new? Yeah, so the question was finding these, especially in North Dakota. You know, that's one thing that can be difficult. You just have to work with your nurseries. You know, and that's the key is not only educating ourselves, but educating them. You know, the reason that there's so much ash in North Dakota is because it's a tree that is extremely easy to grow in the nursery industry. And it does very well in any environment. That's why we ended up with ash because it was easy. And hopefully everybody's learning now that easy is not a good thing. And so we do need to go to the nurseries and start asking for them. You know, I guarantee you're not going to walk to any nursery in North Dakota and find a cat surroach tree. But if you tell them that I would like to catch a surroach tree, there's likelihood in the future that they would be looking to get that. I know nurseries here around the Fargo area are very proactive. They're willing to extend beyond their own general knowledge and explore. But it really is an educational thing for us and them. Next question, Anthony, you would tell me your selection on saline soil. And specifically mentioned orange, Anglo-Maple, Northern Tribute River Birch, the very green cake and birch, cat surroach tree, can they grow in West North Dakota with the same answer? Cat surroach tree, I would say no. That may be a difficulty. With the others, they should be fairly okay. And the river birch, that river birch, I recommend the Northern Tribute. Again, it's not going to be one that's available for another 5, 10 years. That's the big problem with our program here in North Dakota. Is that it's a full process to get a plant out. But for a river birch itself, it has this, the Northern Tribute, it has all the characteristics that a river birch does not. It really shouldn't be a river birch, even though it is. It likes compacted soils. It likes high pH. It's tolerant to drought.  But this one was, because it was actually selected from Dickinson. So this river birch does very well in the Southwest part of the state. How about if you talk about, your sites are beautiful, but how about till we see these with our own eyes some place? Yeah. If there are rains in North Dakota, is there a field day to answer that? So the question is, can we see these with our eyes? Where can we see these in North Dakota? And the answer is yes, you can see these with your eyes. Unfortunately, very limited location. One being at Apsaraca, that's the NDSU Horticulture Research Farm, located just east of Apsaraca by a few miles. And we do have a field day, generally in August. We haven't picked the date yet, but we will. And then when we do that, we'll be giving that information to all the county extension agents. So hopefully, that information will be available to everyone. But we have most of those out there. We are working on the Lace Park pine now. We're building up a population, so I don't have those to show you. But generally, everything else that you've seen here are already in your community or can be seen where we're at. Because it's amazing how much is actually out there, because there's a lot of people like us that are willing to try things. So be surprised that there isn't most of this somewhere in North Dakota. Which of these are the trees roll faster? Oh. Well, obviously, like the burrow, if you're looking to have a slower growth rate, but it's worth it over time. Which grows the fastest? I mean, they're all variable in growth. That's a hard question to answer. None. What do you mean, not none? Yeah, you've got to be patient with them. Yeah, for even maple's luck. Not a fan. Question here. So I have a question. We just bought a house that has an existing tree up on the street. So the previous owner has banged up the bark around the base of the lawnmower. Oh, yeah. But is there a point where it's not worth trying to save the tree? So the question was, gentlemen, bought a house. The street tree has been pruned or trimmed with a lawnmower. Mechanical damage, what do you do? Is there a way to save it or is it just an end cause? At that point, yeah, really, if there's enough damage around, it's been pretty much girdled and it's just on a downward spiral. There's nothing really you can do to it. Now's the time to diversify. Get something else in its place. Because I'm assuming it's ash or maple. I don't know what it is. It most likely is. It's sad that, again, we haven't learned, but that's, you know, most likely. Or even a linden, you know, that's the other thing now is we got Japanese beetles coming into North Dakota, too, and they decimate lindens, you know, and we have lots of lindens. So, again, we've got to diversify. Question. Can you actually say on this slide if you get on a stress kick and you start to lean? Okay, so the question is, if you get on a stress kick and you start peeling the bark off, can you damage the tree? You know, to a point, you know, because you don't want to act cause any wounding to it. You know, with a lot of those peeling barks, I mean, that's what they do naturally.  Because you're going to cause stress to the tree and potentially wounding. But, you know, it would take a lot of effort. But it's definitely not encouraged. Hit a stress ball. No, never peel a bark. No, never peel a bark. Yeah, never peel a bark. Question. Regarding the swamp. Yeah, swamp, so the swamp white oak, question is it a Zone 4? Yes, it's a Zone 4. Yeah, does it cover the pests and disease? The question now is, pests and disease, are there any issues with that? With the swamp white oak, really not so much. You know, there are some oak wilt that we could potentially worry about. But for the most part, not too bad. Again, it's more of the diversity. And maintaining that, you know, we don't want a full bowl of art of swamp white oaks. But, that adds a diversity in. How about the hardiness of it? Northern is an American plain tree. Yeah. So, the question is, what about the hardiness of the Northern Advanced Plain Tree? I would be comfortable on a Zone 4. It really hasn't been tested in a true Zone 3. So, that's still to be determined. We are propagating it now. And we will have, hopefully, plants in Williston and in Langdon this year. So, that's going to be to be determined. Last question, another, there are a lot of hardiness questions. I'm sure. American hornbeam, American hornbeam, American hornbeam. I would be hesitant with the American hornbeam. Maybe in a protected spot. You know, none of these trees in general.  You know, we're looking at a lot of these being protected into your city-type parks and residents and, you know, not on a farmstead for a windbreak. I'm not a windbreak person. I don't promote any windbreak trees at all. I am an ornamental specialist. So, these are not windbreak trees. Do not put any of them in a windbreak. Do not plant them out into an exposed acreage. You know, that's not what these are for. But, you know, AmeriCorps tree, I would be questionable in a zone three. American hornbeam, I wouldn't be so concerned with. But, again, it's not one that I would put out in the open, you know, just to be left to die. Okay, let's say five. Okay, we got a five minute break.


In 97, I was just moving on the road with three children. Something just bombed us like a landmine was down and we don't know. So one of the children was the one who was standing on it. This is where I got injured. Two of them died. I was taken by people. I didn't know the time that I was awake. I was asked whether the leg should be cut or not. When I refused first, I said, how will I work if my leg is removed? They said if we leave it like that, I may die. So I accept it. Yeah, I think I encountered Mary the first time I met her was in Cuba. She had the landmine when she was still young. During the war, the long civil war, she was lucky but then she was taken for amputation. So she started using devices since she was still young, around seven, eight years old. So in 2012, there was protrusion of the bone. So she got the re-amputation. When we came here, she was complaining of working. The device was a bit short and there was pain. So we had to change. So the current one that we saw yesterday she was using is the device that we made from here. When I see them doing things like something that I'm not able to feel, but like running, we will not run again. What community do I am not able to? But some I can do. Like now I can do a job that I'm able to do. But if this one is not there, I'm not able. I'm feeling good because I can work and dance. There are various forms of injuries, ranging from landmines, gunshots, other diabetic cases, and then polio cases. When a person comes, he or she will be registered. After that, I did the physiotherapies part assessment. Then we take the cars. After taking the cars, then that's the part of the technician. Then he will modify after modifying. Then we send to Juba. When I hear the sound of the gun, I fear. If people fight again, someone can be shot. Or they may put the mines again on the road. And they say maybe they will become like me again, which is not good.


One of the beautiful things about looting and rioting is that it just immediately makes people's lives better. They get things that they would otherwise have to work for weeks or months to get. They get them in an instant. The writer and activist Vicki Osterweil is the latest and most strident defender of the violence and destruction that's accompanied more than three months of protests following the death of George Floyd. Osterweil argues that in the last years of Martin Luther King Jr.'s life, as his focus moved beyond desegregation and voting rights and towards promoting socialism, he had a change of heart about looting and destruction as a tool for social change. Though he wasn't calling for violent revolution, she writes, neither was he chastising or rejecting rioters anymore. Others have pointed to the famous King Line. The riot is the language of the unheard. In reality, King was unwavering in his commitment to non-violence. This summer, we are going to have this kind of vigorous protest. My hope is that it will be non-violent. I would hope that we can avoid riots because riots are self-defeating and socially destructive. He tried to make his position crystal clear in that same 1966 interview with 60 Minutes Mike Wallace. I will never change in my basic idea that non-violence is the most potent weapon available to the Negro in his struggle for freedom and justice. And less than a year before his death, King delivered a lecture addressing skeptics of non-violence following the 1967 riots. Many people feel that non-violence as a strategy for social change was cremated in the flames of the urban riots of the last two years. Yet King doubled down, calling for massive, sustained, non-violent civil disobedience. In this world, non-violence is no longer an option for intellectual analysis. It is an imperative for action. King considered violent protests to be self-defeating because as he argued in 1968, every time a riot develops, it makes a right-wing takeover more likely and helps the segregationist, George Wallace. Vicki Osterweil writes admiringly of the nationwide riots that followed King's assassination as an act of pure mourning, grief, and rage that perhaps felt so natural, so immediate, so appropriate, even those who would normally marvel at their scale fell quiet. These uprisings are widely understudied, theorized, or historicized, she writes, and the silence on the part of historians, scholars, and activists has been deafening. But the Princeton political scientist Omer Wassau has studied the 68 riots in detail, finding that they allowed Richard Nixon to beat Democrat Hubert Humphrey in that year's presidential race. As King understood, media coverage was why looting and violence backfired. When protesters engaged in violence, that predicted a headline about riots tomorrow, a headline that was much more likely to focus on crime and disorder, and public opinion moves very closely with that. So when the newspapers are talking about civil rights, civil rights becomes the most important problem in public opinion surveys. And when newspapers are heavily covering crime and riots, crime and riots become the thing that is the most important problem. And so the tactics that are employed by protesters can really make a powerful impact on how the media tell the story, which in turn shapes public opinion. Some media outlets have opted to downplay the violence and destruction. But King's warning and Wassau's research still resonate today, as polls show a sharp downward turn in approval ratings of the Black Lives Matter movement in recent weeks, especially in swing states like Wisconsin, even before riots hit Kenosha. And Democratic presidential candidate Joe Biden seems to fundamentally understand that. Riding is not protesting. Looting is not protesting. Which might help explain how he's maintained a lead so far against an incumbent who's often fanned the flames with his rhetoric. If a city or state refuses to take the actions that are necessary to defend the life and property of their residents, then I will deploy the United States military and quickly solve the problem for them. A riot is the language of the unheard. When King said this, he wasn't excusing riots, but imploring observers to acknowledge the unjust conditions that contributed to their emergence in the first place. Certain conditions continue to exist in our society, which must be condemned as vigorously as we condemn riots. The paths forward to social change couldn't be more clear. Persistent, nonviolent resistance as practiced by Martin Luther King Jr., who helped achieve historic civil rights legislation, or rioting and looting promoted by activists who believe we need to argue and defend every tactic that might help us overturn this miserable world of white supremacy, anti-blackness, sysheteropatriarchy, capitalism, empire, and property.


It's significant because I was talking to an environment minister from one of the countries and this person was in Glasgow for the COP26 where people are debating and discussing climate change in a big way and she said, I was there for one whole week, nobody… I never heard the word soil anywhere. How? How can you address climate change without addressing soil? How? If you do not understand this, in the month of July, when it's really hot, stand outside for one hour in hot sun, then move under the tree, you will understand what is climate change. If more soil comes under shade, some kind of vegetation, it will easily reverse climate change and soil is the most active exchange of carbon. Carbon is the best way to bring back carbon into the soil from the atmosphere.


Hello. Welcome to the next presentation for the 101 Essentials Cloud Track. This presentation is by Sangram Rath, a multi-cloud professional bringing 15 years of experience to his presentation on container security. After the presentation, there will be some time for Q&A. So feel free to type any questions you may have into the question box that's part of Intrado. Without further ado, here is Sangram. Sangram, we can't hear you, but perhaps we can use the phone bridge instead. Am I audible now? Okay. All right. So hello everyone, and sorry for the technical glitch. Good afternoon, good night, good morning depending on where you are, and welcome to this track on container security 101. Are you sure? Are you sure? Am I audible now? Okay. So again, apologies for that. Hello everyone, and welcome again. Good afternoon, good night, and good morning depending on where you are, and welcome to this track on container security 101. The talk includes some slides and a demo, and we will have some time for Q&A post that. My name is Sangram Rath, and I'm an independent cloud architect and technology advisor currently working with a couple of companies, both are startups, and lately I've been focusing on security, especially assessing security posture of infrastructure, application and data, implementing recommendations and remediation logic, et cetera. I also do a lot of trainings.  This session will provide an overview of cloud native security, but focus on container security, covering the challenges and risks that come with using containers. We will then see some common threats and scope and highlight some of the container vulnerabilities that are out there. The talk will finally introduce various tools, mostly open source, and they can be used to provide a layer of protection for your containers. Some of these open source tools also have a paid version that provide additional capabilities. Cloud native computing is the hottest trend in the cloud ecosystem right now and is increasingly becoming the new norm for application development and deployment. At the core of cloud native computing infrastructure are containers, and with the increasing adoption of it, because of the scale of deployment, especially the large scale deployments that we see out there, security has become an important aspect of the adoption strategy. So one of the best practices around security in general, even before you can say, cloud became popular is the layered approach to security, otherwise called as defense in depth approach. The four Cs of cloud native security are basically based around that, and these four layers of cloud native security are code, container, cluster, and cloud. This session will be particularly focused on container security, but just understand that cloud native security cannot be managed by container security alone. So you need to ensure that there are other layers like code, cluster, and cloud, and you're following the recommended security practices around them as well. For example, secure coding practices, checking for code vulnerabilities during development, maintaining security for cluster orchestration, like Kubernetes, Docker swarm, or any other platform that you use, right? And of course, don't forget to secure your hosting platform. The hosting platform could be on cloud, very common nowadays, but make sure your platform is secured as well. In this session, we'll be focusing on the container part of it, but make sure that the other areas are not ignored, right? So you've got the four Cs of cloud native security there. Okay, I still see some chat messages on some people not being able to hear me. Can someone confirm if I'm still audible? Just wanna make sure that it's not a problem for my end. All right, great. So let's move on to quickly look into what containers are before we look into container security, right? Especially for those among us that are new to this term and what the container adoption market looks like today. Why is it that we are looking at container security and why is it so important? In this definition from Docker, a container is a standard unit of software packages of code and all its dependencies. So the application runs quickly and reliably from one computing environment to another, right? The idea of containerization or earlier called as process isolation, right? It's not new and goes back, you can say a decade or more. However, the way we use it today, you find it in every application, mobile applications, you find it in IoT devices, you find it everywhere, right? It's definitely a game changer. And because of this huge adoption of containers in every aspect of technology today, we are looking at security, right? So the problem statement that we're trying to solve with containers is the reliable movement of code or software between compute, storage, network, security environments, while at the same time keeping it light, fast, modular, right? So that's the problem statement we're trying to solve. And we do this by packaging the code and its environment, which includes dependencies, libraries, binaries, configuration files into a single unit, minus the operating system. That's the key here. So containers involve virtualizing the OS and they're popular, there's a growing community and the enterprise adoption of computers is also spiraled over the years. We'll see some numbers here quickly to kind of really prove you that that's true, right? So this report by 451 Research, which was in 2018 states that containers will be a $4.3 billion market by 2020, right? Container adoption is mostly driven by maturity over time, growing community, which I mentioned a couple of times, resources, the amount of learning resources and implementation resources that are available and many enterprise products that are getting built around these containers as well. And not to forget tech giants and high growth startups advocating it, which also builds a sense of trust and confidence for others in similar industries to adopt it. Got some more numbers here. Let's take a look. This is a similar 2019 report by Portworx and it shows that container adoption continues to grow in large enterprises, especially in production environments. So here almost 87% of the people getting surveyed said that they are using containers and of the 87% you have almost 90% people saying they are running container technologies in production, right? And while we are at it, not to forget a survey by CNCF as well that shows that containers, container use in production has increased significantly and you had almost 84% of the people responding to the survey mentioning that they're using containers in production. And this is up roughly 15% from 2018. This is a result of organizations having trust in containers and using them more for production workloads, user-facing applications, customer-facing applications. Yeah, I mean, there has been absolutely a growth in container adoption of lately. At least ever since you had Docker coming in and making container as much more easier to use, I would say that, right? Now on the same lines, when we talk about growing adoption of containers, right? Particularly technologies such as Docker, Kubernetes, OpenShift, this is great and its benefits like faster deployment, scalability, portability has fueled the growth of cloud-native technologies. But this also means that security, management, and monitoring these three areas continue to be an area of concern for many of these large enterprises, especially people running containers in production. Again, looking at a few surveys, not really a lot, but just a couple of them, you will see that container adoption is not without its challenges. And this survey by, again, Portworx along with Aqua Security on over 500 IT professionals shows that security is at 51% is a top challenge for that. Similarly, the CNCF survey also includes challenges that companies have around container security. And you will find that security, challenges around container adoption, I'm sorry, and you'll find that security is on the list. And monitoring is also on the list on FES. So this adoption rate means security is going to be more important than ever. And while it may seem, if you go by the container architecture that they are inherently secure because they provide isolation, how and where you use them introduces a lot of security concerns. And this could be at the OS level, this could be at the application level, this could be at the data level. So as containers are deployed by more and more companies, they are becoming a hot target for hackers. And some may say that container security may not be that much of a concern at the moment because maybe they feel they are running it in a private environment or they feel that their current security is enough. But trust me, there is an increase in the number of threats and vulnerabilities that we have around containers today. And more and more bad exploiters will continue to focus on targeting containers, create container focused exploits and we have to look at security very seriously. So it's a growing priority in container adoption. And if you do it well, it can help you react to security issues faster as well. So with containers being used everywhere today, such as mobile applications, IoT, machine learning, data science, and even databases, it's important to know what kind of container threats are possible. And there's been an increase in the attack on container environments such as Docker and Kubernetes lately. Some of the popular container threads are listed on the slide deck here and I'm going to talk quickly a bit about them. Container malware is gaining popularity. An example of this threat is the Kinseng malware, which runs on Ubuntu. And it's provisioned in a container environment, typically through a misconfigured Docker API. And the malware container, once it is provisioned inside your environment, can actually work as a crypto miner and move laterally and spread the malware misconfigured or other vulnerable containers and hosts as well. Another threat that you will see out there if you search for container threats is cryptojacking. And this involves fraudulent cryptocurrency mining using your computer sources.  This is also a type of malware and the main work it does is uses your computing capacity. Of course, figuring out less secure machines and infecting them with a container that can mine cryptocurrencies around Ethereum or Monero, for example, and then send the currencies to the wallet of the attackers. Another one that you will probably hear quite a bit from security professionals is the Dirty Cow Container Expert. In this threat, basically the exploit is on the kernel bug to overwrite a program called Set UID in the system. And this allows the user to temporarily elevate the privilege in order to perform a certain task. So by replacing the Set UID program, the attacker can gain root access privilege when the program is executed and basically may be able to do anything. And if you think that just by deleting the containers or removing the infected containers, your threat is removed. That's not true either. Sometimes these threats result in container immutability being broken. So an example that I can talk about is the PASSWD Set UID program in Linux system. Then you've got the escape vulnerability. Again, you know, something that is pretty... You can say that it's one of the top threats. And this involves, you know, Run C, which is a runtime that ties container engines to Linux kernel. And in this threat, an attacker with root-level access inside a malicious container will gain root-level access on the host as well. And they might, you know, be able to perform... They might be able to do that by performing certain privileged tasks as container or execute certain privileged escalation bugs or do certain misconfiguration. There could be various approaches to gaining access to your host, root-level access to host. Docker engine is an example of a container engine that was affected by this and has been patched since. So this is just an example. However, there could be certain container engines that still have this. And if you're using Docker, of course, provided that you've upgraded the Docker engine, you're safe. But again, you know, in a container, the threat vector can be compromised, image compromised, you know, container at runtime or a compromised process. And a lot of these threats kind of are around that. Similarly, if we take a quick look at the types of container vulnerabilities, here's a list. And we'll only talk about a few of them in the subsequent slides in the interest of time and the level of content. But just to quickly introduce them to you, considering this is a one-on-one track, container images are building blocks of containers, right? They are layered, they are made up of other images such as base images. And hence, they are a common source for injecting vulnerabilities, right? If you're using a base image that has vulnerabilities, they tend to be a part of your image as well. Further, it's a common development approach to reuse images, right? Especially open source developments. And hence, you know, it becomes very important that you, you know, check for container image vulnerabilities. Many images may also provide root access within the container that too without a password. So this is a type of misconfiguration scenario. And this can expose containers to vulnerabilities both within the container and in some cases to the host as well. An example of this is the official Alpine Linux docker image that had a blank root password for quite some time. This was found by Cisco Talos researchers in 2019. And this might not be a direct thread depending on what's running inside and how you've configured it, right? But it is definitely a vulnerability, it's definitely a way in which some other vulnerability might be introduced into the container, right? So it's a concern. And there are many such images out there that come with blank passwords, especially blank root passwords, right? Some other vulnerabilities include less secure container engine, less secure orchestrator APIs like Kubernetes APIs, allowing attackers to get into your systems to provision containers, misconfigure, take control of your cluster, right? Or your container environment. Another example that I can talk about here is an incident involving, you know, Tesla. Where hackers gained access to unprotected Kubernetes console. And in a classic example of lateral movement, they found out the S3, Amazon Web Services S3 access credentials to some telemetry data, which was, you know, apparently sensitive data, right? They also, at the same time, while they were at it, they apparently also performed crypto mining. And according to the report, they employed some really sophisticated, evasive maneuvers, you know, to perform the attack. Similarly, privilege escalation involves exploiting vulnerabilities during container runtime. The run C exploit that we spoke about earlier is an example of this. Within a container environment, remember containers can talk to each other, right? Over the overlay network. So a compromised container can actually move laterally and in fact, spread the threat or spread the vulnerability to other containers as well. Right? This is a situation that could lead to, you know, DDoS or SQL injection or cross-site scripting, especially in case of internet exposed containers. Insecure or less secure configurations result in vulnerabilities. This could lead to stealing data and consuming computer sources to do crypto mining, et cetera. So while the value provided by containers is immense, you realize that, you know, there are a lot of threat vectors that are there with containers as well. And we have to have a different approach to securing containers. We cannot use a standard approach that we have for typical infrastructure, right? So reducing attack surface area is very, very important to reduce the possibility of exploits through vulnerabilities and this starts at every layer of the four C's of cloud native security. So it starts from the code. Of course, we are focusing more on containers here, but remember that your aim should be to, you know, reduce the attack surface area. Right? Some examples of common container vulnerabilities. Here you have, you know, some CV numbers just for your reference. And they again are around the different threats and vulnerabilities that we spoke about. Run C root access, remote execution, blank root password, local users gaining privileges by placing a Trojan horse, executing arbitrary code, code injection possibility, right? Attack vulnerability resulting in a read write access to the host file system with root privileges. These are some, you know, common container vulnerabilities. Some are old. Some have a very high threat number in the CV database. While you might be already following certain best practices and many of these container vulnerabilities might not really apply to you, just make sure that you are aware of all of these and you are implementing them and following their recommended security best practices. Within the container security challenges, if we want to just focus on container security challenges, right? Again, taking a look at a look at a survey, I want to point out two things here. One is a vulnerability management and the second is runtime protection. And when we talk about vulnerability management, we're talking about container image vulnerability, right? So these are the top security challenges. This is again a survey by Portworx and Echo security. And, you know, we have those, we have the survey revealing that vulnerability management and runtime protection is in the top three security challenges that come with using containers. So focusing on those two aspects in the next set of slides, let's start with container image security. One of the most important components and, you know, or you can say elements in container security is securing the container images. And because container images are commonly available, they are very popular in reuse. So they focus on reusability and, you know, many popular open source line of distributions are used as a base for these container images. They are an important vector that brings in, you know, vulnerabilities into your application, vulnerabilities into your system. They may result in you using vulnerable system libraries, vulnerable base images and all of that. So a few questions you should be asking yourself in this step in securing container images are, are your container images free from vulnerabilities? So what do you do? You start by using latest images. You run container image vulnerability scans and this should include base images. Prefer certified images published by the vendor itself. Do not pull images without validating authenticity.  Fix vulnerabilities before using them in production. Let's say you have to use a base image and there are some vulnerabilities. It's your responsibility to fix them before you actually run your code in production. So this should be a part of your entire container lifecycle process. Do you trust these images? Right? So this is where signing the images and verifying that the images, you know, are signed by a trusted, you know, a publisher is important. Use signed images, preferably pushed by the publisher and look at the history to see if it has been modified. So that brings me to my next question. Is your image repository or registry secure? Compromising registries is also a very popular, you know, threat. People gain access to your registry and they might be able to change the images or modify the images and add vulnerability to it. So make sure that you secure your registry with username, password, use TLS, use private registry wherever possible, right? These are some things that you can do to ensure that your image repository, your registry is secure. Are you following the least privileged access model? Very important. A container is usually executed as a root user if you don't provide a user value while creating the Docker image. And this could lead to a security issue. So if you really want your container to run as a root user, make sure that, you know, that's actually required. You don't always need a container to run in a, you know, a root user mode, right? When that namespace is mapped to the root user in the running container, the container could potentially result provided there are some misconfigurations or some other vulnerabilities. It could result in, you know, providing root access to the Docker host as well or to the container host as well, right? So don't use a root user unless you really have to. So these are some questions we ask ourselves, especially, you know, while we are talking about when you're thinking about container image security. Again, a quick report by, you know, a company called Snipe. I'm sorry about the typo there. The company provides, you know, fixing and monitoring vulnerabilities, detecting and all of that. And according to them, the top 10 most popular Docker images contain at least 30 vulnerabilities. Similarly, there are posts and I would like to quote a post here by Kenna Security, which is also a company that secure the focus is on data science, you know, to measure and predict cyber risk and all of that. And they state that 20% of the 1,000 most popular Docker containers have no root password. Of course, these are slightly older, you know, you know, surveys, but I want to, you know, stress on the importance of these things, right? And that's why these numbers. And these numbers are alarming. If you have 20% of the, you know, 1,000 most popular Docker images having no root password, you can understand that there is always, and if you have just used them as it is, you have a threat vector there, right? So it shows that security is not to be taken lightly and you got to watch out for these kind of things. Again, make sure that there might be certain use cases for running containers with root privileges, but don't do it unless you really have to. Let's talk about, you know, container runtime security index, right? So we spoke about two things, container image security and container runtime security. They broadly cover almost every security threat or vulnerabilities that we, you know, took some examples earlier. So fresh vulnerabilities always appear, right? You might have done everything. You might have used an image that is free of vulnerabilities. You might have done everything at the core level to ensure that your image is, you know, as per the security recommendations, you followed all the best practices and all of that. But vulnerabilities are not static. You always have fresh vulnerabilities appearing and hackers are always on the lookout to, you know, exploit. And while we may have taken every possible step to ensure our containers are created securely and, you know, there is always, you know, a new vulnerability or threat that could come up and this is where making sure that your container runtime security is also addressed is important. So that you have a better chance of remediating any exploitation that might arise, you know, during container runtime. So some of the things that you should be looking at at this, at this level, you know, and again, this is not a very, you know, fixed list, just some of the things that, you know, out of my experience, I feel should be highlighted here. So this is a list that includes, but it is not limited to, right? Knowing your application behavior, very, very important. Today's applications are modular. Today's applications are decoupled, highly decoupled. You have, you know, serverless and whatnot in today's applications and it's very important to understand, you know, what your application does. If your application is only supposed to make, let's say, a get call to another component, you need to monitor that and any deviation from only the get call should be detected, because that could mean that there might be some security issue during the runtime, right? So knowing your standard application behavior, right, standardizing it and then detecting any deviations through monitoring is an important step here, right? So monitoring plays a key role in container runtime security. Secure your container API access, very important. Orchestrator APIs, container APIs, or today service mesh is very popular. So even service mesh APIs should be secured, right? Because of the complex nature of orchestrators and service mesh, especially, they bring in additional complexity and introduce new attack vectors that might not be there when you just talk about running Docker containers. That's a monitor and secure them. Follow best practices, secure container engine platforms as well. For example, disabled capabilities like, you know, set UID to prevent processes from changing their GID resulting in escalation privilege, right? Use a separate user instead of a privileged user for Docker commands. Use C groups, use namespaces. They also help in avoiding privileged attacks.  Secure your communication between client and server using TLS. Don't forget network security. It's very important. As I mentioned earlier in the talk, containers can actually talk to each other by default over the overlay network. So if you don't want certain containers to talk to each other, use network namespace, use network security policies, right? They are available in many orchestrator platforms like Kubernetes. You may also force network traffic through an IDS IP system for added security, right? So do these things to ensure network within your container platform is secure as well. You don't have containers just talking to each other for no apparent reason. Because today's, you know, hackers and attackers actually look at lateral movement as one of the, you know, approaches. So if they find one vulnerable container and if that container can talk to other containers in your network, that's a very easy, you know, that's an easy thing for them.  There could be, they might get access to something that you probably would not even have thought of. Or just use your compute resources for their computing needs. On that point, I would also like to mention that do not put sensitive information like username, passwords, tokens, or any similar things, configuration information into your containers as well. Use solutions that can help you retrieve these details programmatically on runtime and encrypt them during transit as well. So you have solutions like Vault from HashiCorp that can be leveraged to dynamically retrieve these secure information certificates, SSH keys, username password and all of that, right? There are similarly other solutions out there as well. Use a multi-stage build and create runtime container images that have a smaller footprint.  Have only the required modules and dependencies. If you can use container-specific operating systems such as CoreOS, you know, Red Hat Atomic, PrancherOS, there are many out there that can help you, you know, help you implement a container host solution that has a lower footprint. Lower the footprint, lower administration, better security, lower, you know, attack surface area, less number of attack vectors to, you know, address and so on. Keep your host OS up to date, apply latest security patches, all of that. Unauthorized access should be prevented. Use solutions, and these solutions are in-built in Linux. You know, so you can use SE Linux or some containers and orchestrator solutions provide their own security policies or profiling, security profiling as we call it. Use them, right? They don't cost. They're part of most of these products, actually, today. App Armor is another example of a solution that can be used to prevent unauthorized access to containers. So container runtime security could involve a lot of things. It all depends on what your container does, but these are some top things that come to my mind when we talk about container runtime security. Right? So with that, we are kind of, you know, nearing towards the end of the slides. We quickly take a look at some tools that are available to help you protect your containers. Most of these are open source tools, or at least they have an open source or a free version that you can leverage. Of course, for added capabilities, you might want to, you might have to buy subscriptions or something like that. But I would like to make a point here. Security is not cheap and shouldn't be looked at it that way as well. So there might be sometimes a cost to implementing these security solutions. And the more complex your application architecture is and if you require that kind of security at every layer, yes, this adds up to cost. But considering today's threat scenario around technology and considering everything is driven by technology today, especially if you are using containers, it makes sense that you invest in them if it comes to it. So here are some container image security tools. And when you're looking at container image security tools, give a thought to, or when you are analyzing and selecting tools, make sure you see what is the scanning ability? Are they able to scan into the container images? Are they able to scan inside containers? What's their update frequency? How frequently they update their database with the latest vulnerabilities? Do they provide automation for DevOps? We won't really get into a deep dive into DevOps or even talk about it much. But in the end, I will like to just introduce that security should be a part of your entire lifecycle, right from development all the way till release. So use DevOps wherever you can. It helps you in automating your security scanning and these kind of things. It takes off the burden from you, right? And you can rely on the fact that you have a process in place that checks for these things whenever there is a code that is deployed into production. See for tools that provide policy support, policies help you create various security policies. Different containers might require different kinds of security policies, so it's a good idea if you can create various policies and apply them to respective containers, depending on what they do and what kind of security is required for them. Clare, for example, is built by CoreOS and performs static analysis of container vulnerabilities. Anchor Engine provides CVE-based vulnerability reporting. It's DevOps compatible and supports policy-based security. And there is an open-source version, which is the Anchor Engine, of course, that you can use. This app is, again, a benchmark guide and configuration baseline-providing solution. It helps you in ensuring that you follow certain guides and implement security as per certain standard baselines. Similarly, around container runtime security, things that you should be looking at, one very important is real-time monitoring, right? Although there are inbuilt protection mechanisms like control groups and namespace capabilities in container engines like Docker, like we discussed earlier, if you are looking at a tool for container runtime security, look at real-time monitoring capability, access control capability, look at networking monitoring capabilities. Do they provide forensics, which might be important for you to do root cause analysis in some cases? Do they provide compliance and audit? A lot of the companies that I work with, they need tools that provide compliance and audit. They eventually, let's say, build products that are around PCI DSS or HIPAA or maybe conforming to GDPR regulations, whatever the need may be. Sometimes they want these tools to provide some sort of compliance and audit report as well. So SysTig is an example that provides deep system-level capturing and real-time analysis capabilities. And Falco is also an open-source, cloud-native runtime security project by SysTig, which is the de facto threat engine you will find on Kubernetes. So these are some tools that you can look at. There are many out there. I have just picked out some of them that are very popular out there. Again, depending on my work, you might have worked with other tools. And of course, whatever the tool is, just make sure that you look at some of these things and select a tool that works out best for your needs. Some other container security tools. You've got Docker Bench for security. Very good for auditing Docker containers against security benchmarks. They've got the Docker Content Trust, which we use for image signing. You enable this and you will not be able to download unsigned images. You've got Silium Calico that are around network security. They both provide network security policies, by the way, which can help you isolate containers and enforce networking policies for a group of containers, normally a pod or something like that. Then you've got Notary, which is for image signing and separation of roles. That's also an interesting tool that can be used. In fact, all of these have some sort of a free or open source version available. You are good to start with some of these at least initially and doesn't cost you time to look at the security posture of your container engine, both at runtime as well as container image security. With this, I'm at the point where I'll quickly show you a couple of these tools in action. I've got an environment app and a couple of environments up just to save time and show that we're quickly able to see how these tools give you reports and give you vulnerability reports and all of that. We'll take a look at Docker Bench for security for auditing Docker containers. We will also take a look at a demo on how to use Anchor Engine and maybe if time permits, look at a couple of more demos. Pretty close and then there's just one more slide. Then I'll be happy to take any questions you might have. All right. Getting some messages in the chat about some of you not being able to hear me. Again, it'll be great if someone else from the group can quickly confirm if that's the case or maybe you can hear me just quickly. Let me know. I hope I'm still audible to the majority of the audience. All right. I'm getting some all good. It looks like I'll move on to the demo here. We'll be sharing my screen here in a second. Quickly connect some of the sessions here. The first one that we will take a look at is how to run the Docker Bench. I have a machine here that I'm connected to. This is a Docker Swarm cluster. I've got some containers running here and quickly run a container that's available for doing Docker Bench for security auditing. These commands are easily available to you. I just want to quickly show how you can do some things without any cost and look at your security posture. We run a container that scans your host and the containers for... It checks your Docker runtime engine as well and checks your... If you have Docker Swarm, it checks into that as well. It checks your Docker Swarm configuration, Docker security configurations. Container runtime, like I said. Image and build file auditing. Docker Demon configuration file auditing. It does a lot of things. It's very similar to PCI DSS.  It does all of that based on the Docker Bench for security standard practices, you can say. This is a quick run of the Docker Bench for security. This is version 134 and it checks for a lot of common best practices around deploying Docker containers in production and it is inspired by the CIS Docker Community Edition benchmark, which is version 110. That's your Docker Bench for security. Similarly, if you want to take a look at the Encore engine, right? We'll quickly run some commands there as well. I know I might be running out of time, so I hope I can... If it's still here and looking at this, I hope I can quickly... I've got some containers running here as well and I've got the Encore engine running. There's a command to quickly bring this up. There is an associated Docker compose file as well and then I've got a couple of Docker images, like Node 10, for example.  This is roughly around 911 MB. If I had to run the analysis using Encore CLI, I run something like this. This is the URL of the Encore engine running as a container here. You first add the image, wait for the analysis to complete by running the wait command. Your URL, username, password, image, wait, and then the image name and the tag. Then you can look at the vulnerabilities after the analysis is done. The command here is IAM AG VULN. This takes a few seconds and you can see a lot of them. There is severity level as well. If I start with more, you can see some high severity vulnerabilities associated and their equivalent CV numbers as well. You get all the details when you do the analysis through Encore engine. It should be coming in anytime. You can see some high severity vulnerabilities as well. Another example that I was talking about to perform as part of Docker security, container security in general is using Docker content trust. For this, you do export Docker content trust equals to 1. Basically saying that I want to download only trusted images. Then if you try to pull untrusted image, it says that the remote trusted does not exist for the container image. You can either disable content trust and download it if you really have to or you can just, you will only be able to download images that are trusted. These are some examples of how you can check for vulnerabilities or audit your container environment for known vulnerabilities. Then of course fix the problem and rerun them again. These are some of the best practices using Docker Bench for security as part of your container adoption in your enterprises. With that, a quick summary of some of the container best practices. Again, ensure that you use latest images, scan your image for vulnerabilities, use images from trusted repositories, least privileged access, which is very important. Your aim should always be to reduce your attack service area in every way possible and use a gated approach to security. The more layers you have when it comes to security, you have some time before you can actually, before the attackers can actually get through your data. More number of gates or defense in depth approach and container security approach helps a lot, especially when you are attacked. Please make sure you try and implement some of these. A lot of these can be also experimented using sandbox environments. Please play around with it. I want to end by introducing two terms or two words you can say. One is container security must be embedded into your container life cycle. It takes a few times and should not be an afterthought. Hence take a look at how you can leverage DevOps as part of your container security practices. It will make things easier for you in the long run. And second, shift left when it comes to security. It's very important that you do security steps as often as possible and as early as possible in your life cycle. The sooner you do, the less you will have the less threats and less problems you will have later on in your cycle. So thank you for your time and if there are any questions, I'll be happy to take them. I'll quickly take a look at the chat window and see if there are any questions and then read out and address some of them. A couple of questions that I want to point out is are there tools to scan images? We saw some tools. Another question by Unikrishnan was can you please share the slides? Yes, they will be a part of the recorded session. Saravanan had a question, isn't some of the threats addressed by SE Linux? Yes, hardening on the host level. Remember we had the four levels of security and one of the levels was cloud security and basically your host security comes at that level. So any host that you're using to run orchestration platforms or just standalone container engines yes, we follow the hardening approach and SE Linux is definitely something that can help you address some of those issues. I think there are any more questions in the chat window. Do any of you have any more questions? Please feel free to post them in the chat window. I saw this message now. Yes, I ran some commands to run the Docker Bench for security. Some questions by Sven Troy saying you were not able to see anything on the console but sorry about that if you missed them out I felt my screen was being shared so yeah, again there might have been some there might have been some latency when it comes to the refreshing so you might have seen them later. So again just to address some of these concerns here latency between my slides and the demo. Maybe the demo is still running for some of you. Is it good to have production containers and virtual machines? Anders had this question. Yes, in fact, using containers in virtual machines and not running them directly on host actually gives you an added layer of security. So yes, it is good to have production containers and virtual machines. Lawn had a question do you have a recommendation for which container platforms tend to be more secure Kubernetes versus Docker versus anything else? Well, I've had experience working with Kubernetes and a bit on CoreOS and some years back with the Mesos as well platform as well. When you talk about maturity and when you talk about the number of tools available and the compatibility and the amount of you know ease of integration you will find that Kubernetes stands out. So in spite of having inbuilt security options, yes, you have got other tools as well that can be easily integrated or they can be made a part of your container data center whatever you want to call it. Having said this, that's my experience some people might have a different opinion around it, but yes I would say securing pods and all are relatively much simpler and easier and you have a lot of resources around that. Can you point us to a collection of links regarding best practices and tools don't have it right now bookmarked but I would one link forward from Docker that I can quickly share so I'm going to put it on the answer here and there's another one as well, which I have bookmarked from Docker. I'm going to share that. What helps containers run smoothly? That's an interesting question and many things help run containers smoothly but I think the first thing there are a couple of things that I would like to point out one is very important that you choose an orchestration software when you have hundreds and thousands and tens of thousands of containers you need a good orchestrator. Number two, monitoring what monitoring solutions are available and sometimes these orchestration systems have some really good integration for example Kubernetes with Prometheus is a good example of a monitoring solution that would work out really good for you and you need monitoring to run things smoothly for that matter understanding the security posture and where you stand in terms of security at the moment. So monitoring orchestration these are some things that I would say help containers run smoothly. Some people who run complex architectures also might add networking to it but these are some things that I would say do that. Raimar had a question it seems that I will micromanage many of these containers especially the list of CVs is as long as we have seen any advice on practical approach of fixing all these vulnerabilities one of the things that I do is I use latest images and I have deliberately used some older images to show you vulnerabilities so use latest images and use minimal images start something very minimal and add your own layers and of course when you make the final image remove anything else that is not required during the process of so use a multi-stage build process but use latest images and make sure that your application security and data security is well taken care of so even if sometimes you have certain vulnerabilities that get introduced see vulnerabilities will come in and you know even if you have taken an image that has zero vulnerabilities there is no guarantee that these will not appear again in the future so there might be new vulnerabilities but if you have designed your application and data to be secure enough and the communication between them and application seekers and data encryption and transit at rest and all of those you will have a better protection especially when it comes to data breach and you know protecting your data and applications but security is something that you cannot say that you fixed it you know it is always something new around security okay moving on to some more questions yes Christian I hope you saw some of those examples so one way to enforce that you are using only signed images is to enable docker content trust in case you know you are using docker I use you know docker images often or it is a private registry and you use for example notary to sign your images so use docker content trust or use notary to sign your images and that will be your way to enforce okay what is the performance hit for having containers within VMs good question and unfortunately don't have numbers in my head right now it would depend on the virtualization if you have virtualization using platforms that are purpose built for containers running within VMs maybe lightweight once or it depends if you are taking a core OS that does not have a lot of a lot of additional bells and whistles and you know you have a good amount of and the provider is good and your virtualization capabilities your hypervisor all the things come into play I guess whenever I have had this question I have always my answer has always been give me some you know hypervisors that you choose and I am going to do some tests and tell you what it is because compute changes platforms change and you never know they might have been a performance bottleneck you know earlier for example I have seen again this is from my personal experience running containers in VMs in Google is not performance it does not have a lot of performance impact but then you trade off performance versus security so certain containers that require that kind of added gate you might use them in containers in VMs some containers you do not need that added security maybe just that minus the VM security is okay for them you know you can probably benefit from performance like UIs and presentation layers and things like that but I have seen that Google platform is relatively better when it comes to running containers in VMs compared to other platforms I have not done a comparison between Google, AWS and Azure although I have worked on all three of them but that is a good point that you brought in right so moving on to the next question looks like I still have some time recommendation for running Docker daemon as a non-root user yes please you know Michelle had this question try and run Docker daemon as a non-root user and I think Docker has a new feature you know it is called a rootless mode and that is still in preview though please take a look at that I guess that is something it is still in preview though and rootless mode executes the Docker daemon containers inside a user namespace very similar to you know user NS remap mode yes why not and Anders has a question can you use tools like POPET to keep the container update of course you know use DevOps approaches and you can have POPET scripts that are written that regularly you know map or benchmark against a database or something like that and also maybe you know recreate your container images with any added vulnerability fixes and all of that sure yes in fact like I said you know I ended up mentioning that use DevOps as your approach you cannot just remember everything and ensure that security has been checked it is very difficult to do that alright I guess I'm gonna move on back to the front I have hopefully answered all the questions feel free to check me out anywhere you feel like and I'll be happy to answer any more questions you might have alright alright I guess we have no more questions so I thank you for your time especially if you are in one of those time zones where it's either early morning or maybe you know middle of the night like it is for me and you know I'll hopefully talk to you sometime in some other session thank you and all of you have a good day or good night good evening whatever