Hi, I'm Romoreira, I work for Intel, and I'm here to talk a little bit about FineIBT, which is a compiler optimization that enhances the guarantees provided by Intel's control enforcement technology, CET, specifically when it comes to indirect forward edges. Here are some important disclaimers, and if you ever check out our source code, just be aware that some bugs may still be there. Of course, we are not really using lasers to improve CET. This was just a bad joke. Let's start our talk with a metaphorical example. Imagine you have a function full indirectly calling a function bar. This should be your control flow. Yet, when you start executing, first FBTR, which is a function pointer is assigned at the address of bar. Then a memory unsafe function is executed, allowing a hacker to overwrite the contents of FBTR with the address of bar incremented within offset. Then FBTR is invoked and the control flow goes directly to the middle of the function bar, is keeping an important user ID check that should have been executed. Because of situations like this, Intel introduced indirect branch tracking, IBT, which is part of CET. What IBT does is that it requires every indirect call and indirect jump to target and end branch instruction, just like the one highlighted in green in the function bar. Such policy blocks attackers from freely redirecting the forward branches, preventing situations like the one just described it. With something like this in place, you can have your compiler emitting and branch instructions in the prologue of every function, creating an ABI-like scheme that forces indirect calls to always target the first instruction of functions. This is actually what we call coarse-grained control flow integrity. The result of this is that now any function can be reached from any indirect call or indirect jump. This guarantees to relaxed, and it still allows functions to be used out of context to exploit the system. In that regard, coarse-grained control flow integrity does not fully mitigate the control flow hijacking problem. Now let's talk about a not so metaphorical example. The sudo heap-basic buffer flow recently described in an advisory released by QALYS. If you look at the advisory, you will notice that there are three different methods for exploiting the bug there. Here, I'll be focusing on the first one. In this method, the attacker corrupts the pointer fngth, which is in the heap-basic through sudo hook entry, and now the corrupted pointer targets either the exact V or exact V PLT entries in the sudoers.sl file. If you look at the assembly dumps present in the QALYS advisory, you will notice that the PLT entries there display in branch instructions, showing that IBT is in place and yet was not sufficient to prevent the control flow hijacking attack from taking place. Because of situations like this, researchers came with this idea of fine-grained control flow integrity, which tightens the control flow graph of the application even further by enforcing additional rules for the ABI-like scheme. Ideally, an indirect branch should only be allowed to execute if its target is the supposed to be target. But in practice, well, you can't really define this rule during compilation time, as this is a statically undecidable problem. Because of that, we need to use heuristics to reach practical implementation. The most common use heuristics is clustering functions and pointers by their prototypes. Functions have prototypes, function pointers also have prototypes. Thus, whenever indirect call is taking place, the prototype of the function pointer should match the prototype of the invoked functions. Back to the pseudo-example, if you look at the fngat-env pointer prototype, it is charpointer, charpointer, pointer, pointpointer, which differs from execvs and execvs prototypes, but matches the prototype of sudoers hook-get-env, which is the function that should be legally invoked. Some software CFI schemes out there successfully implement prototype matching policies. Some of them are PECS GRCQ with RAP, Clang CFI, and Microsoft XFG. These schemes are implemented in software and materialized in the form of pseudo-ABIs being wired in different ways internally. RAP uses tags hard-coded in binary to mark valid branch targets. Clang CFI uses intermediate jump tables with the sets of valid targets, and XFG passes a hash for a register to an intermediate dispatcher that then also checks for hard-coded tags. Finally, other schemes for forward edge CFI, like pointer authentication for example, do exist, but are beyond the scope of this talk. Given the scenario, we came with these two hypotheses. First, can we enhance CTIBT, possibly reaching a hybrid approach that combines the benefits of having a swift, harder foundation with the flexibility of a software instrumentation in a way to make it fine-grained, and if yes, how much of the perks implicit to the hardware nature would it retain? Now we are ready to start exploring FinIBT. FinIBT is also an ABI-like scheme, which uses end-branch instructions to anchor the control flow through the beginning of functions. Then by doing that, it enforces additional policies in the functions prologue. These checks consist in instrumentation, which is emitted in the binary by the compiler. In the instrumented binary, indirect calls are argumented with hash set operations. Function prologues are argumented with hash check operations. Direct calls have their targets incremented with an offset ensuring that whenever they are executed, they skip the prologue hash checks in the colleagues. The hash is used in these operations are generated based on function and pointer prototypes. This is a regular assembly code as normally emitted by an ordinary compiler, and you can see in direct call and a direct call over there. This is the assembly code generated with the basic IBT instrumentation. You can see an end-branch instruction there. Well, this is the fine IBT instrumentation. First, this is the hash set operation. It places the hash respect to the function pointer used in the indirect call into the register R11. These are the regular end-branch and the hash check operation. Since the end-branch is in place, the control flow will go through the hash check, which then exhausts the contents of R11 with the expected hash. If the result of this operation is zero, then the following jump is taken. Otherwise, it is not taken and the execution reaches the halt instruction. In this snippet, we use an extra operation because it will consequently destroy the contents of R11. What can be important to prevent reuse attacks if you are ever mixing fine IBT-enabled code with non-fine IBT-enabled code. Finally, the direct call now targets the address right after the hash check snippet. As described, this instrumentation would work properly with statically linked binaries and possibly with kernels. Yet, in some scenarios, your applications will be interacting with libraries and sometimes they won't be fine IBT-enabled. In situations like this, we must ensure that fine IBT-dynamic shared objects do not break non-fine IBT-dynamic shared objects. If you look at the x8664AVI, you'll notice that the basic CT form solves this problem by disabling the policy enforcement whenever a non-CT DSO is loaded. We'll follow the very same approach. First, we have a bit in the DSO health header that flags if it is a fine IBT-enabled DSO or not. Then the loader is augmented to check if the DSOs are fine IBT-enabled during the load time. Finally, if all the loaded DSOs are fine IBT-enabled, then a memory flag is set informing that the policy should be enforced. In this assembly dump, you can see in green the previously described fine IBT pieces, and in red, a new snippet which will check for the fine IBT policy enforcement memory flag. The bits verify at 0x1 and 0x10 are respective to the IBT and fine IBT flags because it doesn't make sense to enforce fine IBT without IBT. The FS0x48 is the memory address where the loader sets the fine IBT global flags. Fine IBT also requires special PLT entries that have two slots. The first is a 32-byte slot, and the second is a 16-byte slot. The first slot is reached by indirect calls, and it checks for the proper hashes and then jumps to the target function. The second slot is reached by direct calls. Since an indirect branch is about to take place in the PLT, it then sets the hash and then jumps to the target. This scheme requires early binding because we don't want symbols to be resolved while the control flow is going through the PLT. This would require the hashes to be saved in memory and this would be unsafe. Here is the assembly snippet showing a PLT entry and the control flows from an indirect call and from a direct call. And here you can see how the fine IBT piece is placed into the PLT entry with the indirect call reaching the first slot, which does the hash check, then jumps to the second slot, which goes into the target. The complex PLT wiring is needed because it references pointers from the God PLT, which can be hijacked. Yet, the existing model already requires early binding. Considering that this is already in place, pushing the binaries into a full railroad model isn't a huge hassle. And with it in place, the PLT indirect branches become implicitly safe. This opens pace for optimizing the PLT by combining railroad and indirect jumps with the CET no-track prefix, what relaxes the requirement of it landing on end branch instructions, allowing the branch to jump over the hash check prologue in the target function. This optimization remains as future work, yet to be explored. The previously described scheme actually requires all DSOs to be fine IBT enabled for the policy to be enforced. Because this can be very limiting, we started exploring different models to enable the cross-DSO compatibility. This method I'm about to describe is currently under development, and it actually uses the shadow stack also provided by CET, but with a different purpose. The shadow stack is in place with the goal of preventing control flow hijacking in the backward edges. When a shadow stack is in place, every construction will copy the return address pushed on the stack to the shadow stack. When a return happens, this copied address is verified for a match with the address on the regular stack. By using the instruction RDSSP, we can read the shadow stack pointer and then retrieve the caller's addresses from the top of the shadow stack. By doing that, we can identify the function which is calling into a fine IBT enabled function, verify if it belongs to a fine IBT enabled object, and decide to enforce the policy based on that. Here's how the policy enforcement piece looks like. After the hash check failed, we run our DSSP and get the shadow stack pointer. We deference it and get the caller's address. Then we compare it to the object's boundaries, identify if the call is coming from the same DSO or not. If yes, we know that this is a fine IBT enabled DSO, thus the policy should be enforced. This method as described enforces a self-contained intro DSO policy. I believe it can be extended to support multiple DSOs, although with more overhead than the global bid scheme previously described. Yet it is meaningful that the overhead is only paid when indirect calls from coarse-grained DSOs into fine-grained DSOs happen. As calls between fine-grained objects should have matching hashes and calls into coarse-grained objects are not at all checked. Another important perspective on fine IBT is observing how it behaves when it comes to transient execution attacks. If you read the Intel's SDM, you will find the following interesting snippet that says, when CT tracker is in the wait for end-branche state, instruction execution will be limited or block it, even speculatively, if the next instruction is not an end-branche. In practice, what this means is that the speculation after indirect forward branches is confined into the coarse-grained control flow graph, limiting transient execution attacks. Yet, if you look at a fine IBT's instrumentation, you will notice that it has a conditional branch right after the hash check. Whenever enforced in a complete ABI-like scheme, the decision regarding the target of this branch will depend only on the instructions move and XOR. Because the instructions only use immediate and register operands, they are retired with variable latency. This not only makes it hard to attack these specific branches, but also limits the speculation to proceed further down if the hashes are not matching. We tried to use this conditional branch as a transient execution gadget, but we were not able to. And this led us into assuming that in a full fine IBT process, the speculation remains confined to the refined control flow graph. As said, we assume here the statically-linked scenario where no policy enforcement checks are made. I actually would like to explicitly thank Kisan, who is also a member of the Intel storm team, for doing this analysis. When generating PLTs, the fine IBT linker needs information regarding the hashes with which it needs to instrument the entries. Thus, for enabling that without depending on LTO, the fine IBT compiler embeds the required data in a special section on each generated object. Then when this data is consumed by the linker, it discards the sections with no impact to the final DSO size. Fine IBT was implemented on top of LLVM and LLD12.0. We implemented cross-DSO support on top of the Google runtime environment branch of GDBC. We also wrote a very basic IBT support on Muscle 1.2.0, as we wanted to use it for some performance tests. The source code of fine IBT is available in these repositories, being the first one, the compiler and linker implementation, the second, a claim-enabled version of GDBC with fine IBT cross-DSO support, and the third, a few scripts use it for testing. Okay, now let's talk a little bit about performance. We used two different test sets to evaluate the performance of fine IBT. The first test set consists of a custom synthetic benchmark, which was written as an attempt to take the worst out of forward edge CFI implementations. It is composed by three different applications, which are very dense when it comes to the number of forward and direct branches. The first application is a dummy loop that indirectly invokes an empty function over and over. The second is a Fibonacci sequence calculator that does its recursion indirectly. And the third is a bubble sort implementation whose swap function is invoked indirectly. Each of these applications was compiled in two different versions. The first uses a global function pointer for its indirect calls. The second uses a local function pointer for the very same indirect calls. The reason for having the second version is because in our early test with the first version, fine IBT displayed overheads much smaller than those introduced by Clang-CFI. Under a deeper analysis, we noticed that Clang-CFI's instrumentation would always force the control flow through the PLT table, while all the other analyzed binaries would go directly into the targeted function. While it remains a bit unclear to me what are the reasons why Clang-CFI misses this optimization opportunity, I felt like it would be good to provide numbers which are intrinsic to the CFI instrumentation only, trying to rule out overheads introduced by major side effects. The numbers presented for these test set are the average of 10 brands and these are computed by Perf. The second test set is composed by two applications from the SPEC CPU 2017 benchmark. SPEC has in its source code some typecasts what causes some false positive policy violations when these are executed under CFI instrumentation. To overcome this problem, I use the modified version of fine IBT which uses the very same tag for every prototype, creating a coarse-grained like policy enforcement, but with the same instruction sequences of the fine-grained scheme. To run these applications compiled under the Clang-CFI instrumentation, I had to add a few violating functions to an ignore list, preventing them from being instrumented, thus avoiding the false positives, but also giving Clang-CFI some performance advantage. The first application was 600 per bench which is suggested by the Clang-CFI documentation as a good application for testing CFI. I had to add these four functions to the ignore list to make it work. The second application was 625x264, which was picked at random. I had to add these two functions to the ignore list to make it work. The numbers regarding the SPEC runs were picked by the benchmark itself out of tree runs. The testbed usage in these experiments was this machine running federatory tree with a kernel patched to support CET. IBT was verified to be enabled and enforced. Turbo Boost, ASLR, and SMT were off. Each application was linked to a setup equivalent muscle and compiled using the same compiler and arguments, except of course for the CFI scheme that we are trying to evaluate. The fine IBT global check was hard-coded as knobs, ensuring that the policy was always enforced. The compilation arguments are in the backup slides. The tests were executed under these five different setups. No CFI, which is the regular binary fault CFI. Course, which is the binary compiled with the regular IBT feature. Fine, which is a binary compiled with the regular IBT feature, plus fine IBT compiler instrumentation. Clang CFI, which is the binary compiled with the Clang CFI instrumentation, and Clang CFI NC, which is the binary compiled with Clang CFI instrumentation, but without cross-DSO support. This last one was only used for comparing space overheads. These are the numbers for the runs when using global function pointers. As you can see, we have a huge overhead of 57.42% for the dummy application when compiled with Clang CFI. While this overhead remained at below 2% for the fine IBT version. Similarly, the Fibonacci application presented a 32.78% overhead for Clang CFI, with a significant but still smaller overhead of 13.77% with fine IBT. BubbleSort, which is the last indirect branch dense application of the mall, displayed an overhead of 5.12% for Clang CFI, while remaining on 1.78% with fine IBT. When control flow through the PLT was enforced under all setups, the overhead data became less massive, but still evident, and with fine IBT presenting better performance than Clang CFI on null tests. We believe we can improve these numbers even further after optimizing the PLT on the railroad. For the spec applications, 600 per bench under fine IBT presented a 1.66% overhead while Clang CFI overhead reached 3.56%. The second application in the test set presented the same result for both fine IBT and Clang CFI. Despite the tie, it is important to mention that Clang CFI in both applications had functions added to its ignore list and thus not instrumented. The space overhead of Clang CFI with cross DSO support is very high, especially in the tiny synthetic applications. I assume that this comes from all the machinery that Clang needs to link in to provide the cross DSO support. Thus, again, trying to provide more approachable numbers, we did the space overheads comparison using applications compiled without Clang CFI's cross DSO support. In this comparison, Clang CFI presented smaller overheads than fine IBT, yet at the high price of not having cross DSO support enabled. This comparison was done using full binary sizes of each compiled application. Now some conclusions. If you remember, at the beginning of the talk, we brought up these two hypotheses, which are if we can enhance the T-IBT in a way to make it fine grained, and if yes, how much of the hardware-related perks would it retain? I guess that the answer for the first question is yes, and regarding the hardware-related perks, fine IBT presents good performance, especially when compared to Clang CFI. It improves the transient execution mitigations which are provided by the regular IBT feature. It also presents reasonable space overheads. And then in my pipeline, here are a few tasks I'm willing to work on. The first thing is doing a deeper security validation of the fine IBT design. Most CFI implementations will check the policy in the caller before the branch executes. But in fine IBT, this check happens in the callee. Is this a problem? Is this better? If there are problems, can they be fixed? Also, what about mixing fine IBT and non-fine IBT DSOs? It certainly adds flexibility, but should also introduce weak spots. How can we make this solid? Then I want to improve the benchmarking of fine IBT. I do believe that benchmarking is rocket science, and I understand that the quality of the presented numbers can and should be improved. Thus, if anyone has any suggestions on how to do it, I'm willing to hear about it. Besides performance, I'm not super into the academic average indexes for measuring CFI effectiveness. But what better options do we have out there? How can we really measure the restrictiveness added by CFI schemes? I also want to move all the cross-DSO schemes into Clang's compiler runtime library, removing the Jalipsi dependency, and who knows, maybe making fine IBT streamable. Then I want to explore further the cross-DSO compatibility schemes, looking for more flexible approaches and for designs tailored for specific use cases. CT support was not yet merged into the upstream kernel. Thus, how can I help with this task? Once it is there, what is the landscape for running the kernel with fine IBT? Some anticipated challenges are, for example, interactions with assembly files that should require some rewiring, but shouldn't be as complicated as the ongoing effort to make it support Clang CFI. Currently, fine IBT is a C-only thing. Is the general idea useful in the C++ V-tables context? If yes, what is the best general design and how do we support polymorphism there? The performance numbers show us that the current PLT design is expensive. As we described earlier, fine IBT PLTs may be optimized under railroad. What should be the best design for this? Also, what if we take a step back? Can we remove the dependency on early binding? And these are some very important people to this research. I would like to thank each of them. Professor Vasilius Camerlis and Alex Guedes from Brown University, with whom I have been working very closely in this research. Michael Lee Mai, H.J. Lu from Intel, Kisum and Hiki Kawakami and Alissa Milburn from the Intel Storm team, Jared Candelaria and Vervias Chamborghi, who recently left Intel but left behind meaningful contributions to this research. Thank you all. This has been such a fun ride. Thank you for your time and I'm happy to answer all the questions about fine IBT. Be here for the conference chat or later for email. Thank you so much and bye-bye.


I want you to observe what happens when light travels through a different medium. I'm going to place this lapel pin in the bottom of a cup and over near the side. What you're going to do is you're going to observe what happens as I pour water into that cup. Now what happens is light strikes the surface and it's going at the speed of light in air, but when it hits the water it slows down and as it does it bends. And that's what allows you to see the pin in the bottom that you couldn't see before. It's because the light is bent in the water.


Hey, Danny Kay. Oh, well, it's my average radio listener. Yeah, the man in the street. I want to talk to you about your program. Okay, what about my program? That's what drove me out in the street. Startin' already, eh? Look, what makes you think you're qualified to criticize my program anyway? Oh, I make a regular study of comedy programs, Kay. You do? When any other comedian comes out of the air, I sit in the living room and wait for him to get off a joke. Oh. And when I come on, I sit in the park and wait for you to get off the air. Did you ever stop to think that maybe the reception on your radio isn't so good? Uh, what kind of current do you use? DC. But when you're on, I'd rather have AC. Alternating current? Yeah, Abboton Costello. Look, Chum, you know, in spite of your attitude, I really appreciate your criticism, and I'd like to do something nice for you. Oh, yeah? Like, for instance, what? Well, I'd like to send you on an ocean voyage. Yeah. Mm-hmm. Yeah, an all-expense cruise on a torpedo. I'm surprised that you're talking like that, Kay. And after all the work I've done to try and improve your program. What have you done to improve my program? I've been trying to get you off it, ain't I? That's enough, fella. Good evening, ladies and gentlemen. This is Dick Joy, introducing the Danny Kay Show with Butterfly McQueen, Dave Terry and his orchestra, and our special guest this evening, Mr. Arthur Treacher. And now, here's the star of our show. Danny Kay! Danny starts the show off tonight with a number from the Samuel Goldwyn production, Wonder Man, the Bally Boogie. He's here. But they hadn't seen a new step since Muthuselah did the two-step on their isolated island. They were way off beat from gyve. They were closeted to a star, deposited on the beach at gyve, and jacked from lower basin. Sweet, he went up and got a pin, and he said, hey, what's cooking? This here join is jumping buck with melancholy. So he took their native movements, added in the groove improvements, and here's the way it's played on the ballet history. Bataka boy, boyak bataka, zoo, zoo. Bataka boy, boyak bataka, green bean bataka boy, boyak bataka. Live, dive, ahaha, kutu, chattanoogie, barley, boogie. Oh, walka song, stick lick. Oh, walka song, song ahaha, walka head step. Oh, walka song, song ahaha, walka. Live, dive, ahaha, kutu, chattanoogie, barley, boogie. Now if you can't dig the chatter with the greatest of these, it really doesn't matter. Here's the translation, please. Bataka boy, boyak, ahaha, walka song, song ahaha, shoot the pinky to me stinky. Bataka boy, stick lick. Bataka boy, ahaha, head step. Bataka boy, boyak bataka. Live, dive, ahaha, kutu, chattanoogie, barley, boogie.  Danny, what are you talking about? Well, I'm talking about my favorite comic strip character, Dick. You know you haven't heard of Little Abner. Oh, sure, but what about him? Well, he's offered to marry Lena the hyena, rather than permit Lester Gooch, creator of Fearless Fosdick, to languish in a Slobovian dungeon. Well, yes, but what has that got to do with you, Danny? Let it. His girlfriend, Daisy May, stopped me on the street yesterday and begged me to make this plea. So, Little Abner, wherever you are, listen. Lena the hyena, stay away. Drop that drip from Lower Slobovia. Little Abner, what's come over you? Think of Daisy May, Little Abner, that gal. Even Fearless Fosdick, he would say. Lester Gooch is just a moochie. If he lets you spoil your future with no Daisy May.  They wanted her to star in that picture, the Morg the Marrier. But instead, she went into the Lost Weekend. The end wasn't lost. It was hiding. Everybody knows it. The only girl who can eat a sandwich through a Venetian fly. She's engaged. She should live. She wants grabbin' her. That's a Yokum son named... The interest of Mr. Basil Raffbeau. A legal firm of Peabody, Hammermill, Forsythe, Glendoning, Throbyshire, Pepperham, Breckenridge, Witherspoon, Parkhurst and Fabbisham. And what was your name again? Driscoll. However, sir, I should like you to know that I have seven connections with the firm of Peabody, Hammermill, Forsythe, Glendoning, Throbyshire, Pepperham, Breckenridge, Witherspoon, Parkhurst and Fabbisham. Oh, and business for yourself, Driscoll? In a manner of speaking, yes. I now make my offices with the legal firm of Glenroy Gates, where the Orkin Lake, Partridge, Swinburne, Featherstone, Marblehead and Gates. What happened with Peabody, Hammermill, Forsythe, Glendoning, Throbyshire, Pepperham, Breckenridge, Witherspoon, Parkhurst and Fabbisham? A bit of bitterness, Mr. K. I'm afraid I'm suing them. Oh, and who's handling the case? Glenroy Gates, where the Orkin Lake, Partridge, Swinburne, Featherstone, Marblehead and Gates? Oh, no, no, no, that wouldn't be ethical. I've called in the firm of Peabody, Heatherington, Allenby, Gill, Martin, Primrose, Scroffing's Way, Clay, Pool, Underwood, Kimbler and Surrey. And which member of the firm is handling your case? My brother-in-law. O'Reilly. Yes, my sister, Marit and Irishman. I'm thinking it in backwards. Look, Mr. Driscoll, just what is your business with me? I, sir, am here on behalf of my client, Mr. Arthur Preacher. There are certain things you must not discuss with Mr. Preacher when he appears with you on the wireless. On the wireless, I see. And radio, that is, you know. What things are those? You must not say anything about his British brain. Nothing about his cinema career. Nothing about the way he dresses. Nothing about his brain, a butler. And you must not say anything about his personal life. Well, if the fellow sneezes, can I say gizunt heith? Uh, bless you, we'll be sufficient. And now with your permission, sir, I shall buzz off. Permission granted. And I'd better get my secretary to jot down some of these taboos. Oh, Miss McQueen. Butterfly McQueen. Here I am.  How have you been? Good as a whistle, I'm ready to blow. Well, so long. Wait, wait a minute, Miss McQueen. I want you to jot down some of the topics that a taboo for our guest are.  Glad who's our guest? Taboo, the elephant boy. The elephant boy is not our guest, Miss McQueen. It's Arthur Treacher.  Shall I jot them down in shorthand or longhand with the case? Oh, which is faster? Longhand, naturally. What do you mean naturally? I never learned shorthand. Well, just take this down, Miss McQueen. You ready? Ready. Oh, we must not mention A, Mr. Treacher's English cohorts. How do you spell that, Mr. A? How do you spell what? A. You'll find it in the dictionary under T. Thank you. Now, let's continue. We must not mention A, Mr. Treacher's English cohorts. B, his career as a butler. C, the way he dresses. D, his motion pictures. And E, his personal life. You got that, Miss McQueen? Not quite all of it, Mr. K. I just got A, B, C, and D. Oh, well, then read that back. OK. A, B, C, D. But, Miss McQueen, what comes after those letters? E. You'll find it in the dictionary under S. Well, I tell you what you do, Miss McQueen. Just type up what you've got, Miss McQueen. Make three clean copies. And then throw them away. Well, what is that? A. Let's try that once again, huh? I'll make one extra copy. You throw away for your files. Now you go. OK. Let's just once more. I think we can make it this time. I really do. Good. I'll tell you what you do. Just type up what you've got, and then make the three up and down. Now you say. Throw away for your files. Start with all these taboos now. Now I got Miss McQueen on my mind. All I can do is try and make the dear old chap feel at home. It's the treacher. You know, I've got... Oh, do enter, won't you? Hello, Danny. Well, bless my meter's marmalade. It's Arthur treacher. Nice of you to ask me over, Danny. How have you been? Oh, top. Oh, lobo. Oh, dear old tin of fruit. Simply ripping. What? What? Hmm? Oh, what's the difference so long as it rips? I haven't felt so jolly well quick since I left dear old London, you know. Oh, the tight little isle. I know it well. The rams get to graven. From Brighton to Bathside. From Soho to Limehouse. And from Limehouse to Blues.  Danny, I never knew you'd been to England. Oh, yes, yes, Arthur. Oh, boy, I shall never forget the summer I spent in a charming village called Easy on the Butter. Oh, the days I spent strolling through the moors. The dank dim and dinty moors. And the heather. The hot summer heather. Of course, I always wore a pair of heavy scones to keep my crumpets from getting too tippant. This kid is positively snapping his cap. Can't help it, old chap. I'm simply stuffed with memories of quaint old England. Are your petro details? Yeah, they'll be here when we call them gassy. And they're jolly old lids. Over here we call them elevators. Ah, dear old England, Arthur, old Trees, where we used to ride to the hunt. I can hear myself now calling to the hounds. On Donna, on Blitzen, off Dancer, off Dancer.  Quite. On my last hunt, there were ten of us chaps. How we did telly her. I'll bet you did. You and those nine other yikes. Look, Danny, what are you trying to prove with this dialect of yours? I was just trying to make it feel at home, Arthur. Why, what's wrong with my English accent? You sound like Nigel Bruce talking through Basil Rathbone's nose. Well, I had to do something after all those taboos your lawyer told me about. Oh, it's a lot of nonsense, Danny. I never paid any attention to him. Wait a minute, you mean he was just here wasting our time? Certainly not. He got a few lasts, didn't he? Yes, Mark Wood. Then you mean you'll do anything? Of course. You know, I even do that opera at her. You had me rehearsing all day. The one about the Countess of Coney Island. Oh, very good, sir. Dave, hand me up a handkerchief. My coronation here. I feel I ought to say something to you there. Straight from the heart here, there. I'd like to say something to you straight from his heart. My good people. And that's the quickest touch you's ever made. I want you all to know that as long as I'm the king of this here land, I wouldn't do for any of my subjects here, here, here, hey. And now if I've said anything to offend any of you, any decay, royal dispenser of justice will commence the dispense with justice. Speak up, countrymen. A guile. No dames, please.  I can see just by her face that she's got a problem. Next case. Well, king, this is a rather touchy something. Your late father, Ernest Ellipse, alias King Charlie de Chump, true to his title, mogies are ballparked to the hilt. The hilt, you say? And this mogie has fallen into the hands of Queen Lee of Manhattan. Dad, Zooks, that's true. What will we do? Everybody else dares to do dare. Get that dialect, will you? Queen of Manhattan under ballpark is our word. What's that you said? Me, the king of Brooklyn? The cultural center of the universe? Marry a barbarian? Never. Her and her 32 teeth. What do you mean? What do you mean a 32 teeth? Everyone has 32 teeth. Four on top and 28 on the bottom. I hate to say this, but unless you marry Queen Lee of Manhattan, we are licked. But I can't marry Lee. I love Aindada. Who? In Brooklyn we say whom. All right. Whom do you love? A little knucklewurst of Coney Island. Little knucklewurst? If you are a royalty and she is a commoner. Yeah, but what a built-on. But nevertheless she is a commoner and cannot marry above her station. What is her station? Brighton Beach. Oh, really? No ocean parkway.  Whom do my marry, whom do my not? The king of Manhattan I should be. I could not no more say Satanee. Oh, woe is he and woe is me. He could not no more say Satanee. You got a loin to say learn in New York. Unless you're learning to say loin of car. And when they say you're oily, it don't mean you've a greasy plate. And when they say you're oily, it don't mean that you're not late. What you've got to do is learn a new language. To say sandwich and never say sandwich. You got to. And besides it ain't true. You mean it's a lie? No, it's true. Oh. Remember the ages. It ain't trust. It's thrust. It ain't tread. It's thread. It ain't trite. It's thrite. It ain't thrill. It's thrill. Are you trying to make trouble for me? In a minute I'll change my mind one, two, three. Your Highness, in New York it's not tree, it's three. Imagine that. A three grows in Brooklyn. Okay, okay. But you got to go. Farewell, King of T.D. I'm going to Columbus Circle. There's so long a view right there. I'm going to Broadway there. To save the Dodgers clients. I'm making this a lion. No more will you defiance at the Yankees and the Giants. I've got to save the mortgage. You'd better pack my mortgage. Go on till June or August. Start wishing me bond mortgage. Here comes the finest King Danny the K of Brooklyn is here. I'm not certain that John Bolet is King Danny of Brooklyn. Oh, yeah, verily. Enter King Danny. Thank yous. Boy, what a filth on him. Oh, here beside me. Hey, turn off that charm now, Queenie. I'm here to talk to you. Oh, hi, I'm Turkey. Let us remember our royal station. First we'll have tea. And I will pour cream. AIDS, Vibble. Sugar. A lump. Lemon. A squeeze. Tea. Never touch it. Now listen, Queenie, by some skull dunglery, you got the evidence field mortgage away from my father, and I want it back. You can have it back after we're married. No marriage. No mortgage.  This is bliss.   I almost missed that one. Your Majesty, I bring news from Brooklyn. Where did you get it? The Daily Eagle just flew by. Moical, knockover, lies in Cooney Island, grievously illinois. That's grievously ill. Oh, yeah. What happened, bud? She was eating one of those foot-long hot dogs, and she got her head caught in the Frankfurter Bun. Cad Zooks was the bun empty? No, there was a hot dog in it. Thank heavens she ain't trapped in there without food. Come, no-nose preacher. We must save her. But the mortgage. I'll attend to that later. There's a crowd around the hot dog stand, no-nose. Follow me, make way, stand aside. Look, preach. Mightle's still in there. Hello? Look it. Look at those crumbs. She's all covered with dough. That's terrible. That's great. With all that dough, we can pay off the mortgage on a ballpark. But last, Mightle and I can be married.  Ah, I'll fix that. Hey, Mightle, can you hear me, Might? With this mustard ladle, I've dubbed the Countess of Cooney Island. Now no more deadlock concerning the wedlock. I'm Countess of Cooney Island.  There's a question I'd like to be popping in. One, I don't see nothing stopping you. Well, Mightle, my title, I'll ask you a real taste. Will you take me for better? A race as you chase? Coindly answer no way, Yates. You know the answer. Even if I was a married to real love, we'd all want the head. Music a second. Will you, Dave Terry? Will you please? Thank you, thank you, thank you. I just wanted to say a word to our armed forces radio listeners around the world. I speak. Good night. Dick Joy speaking. This is the Armed Forces Radio Service.


We'll start off going through the existing lightweight threat model we did for Flux. Flux is a Githops tool. Let's see if I can multitask. It's the original Githops tool. We're talking about taking static configuration twfl concerted cubonetans or manifests. Putting them into git. That's a good start. What's on the calendar for today. Yes, GitOps, taking a declarative configuration, pushing it into Git, and having a deployment tool, none of that's relevant, and having a deployment tool reconcile cluster states with the state of Git. So at a very high level is what GitOps is. Flux is a tool from Weaveworks that was the first tool to educate the market if you like. The first instance of a tool that does this kind of thing. And, Marko, would you ping me the link? Thank you very much, you'll weigh ahead of me as usual. And we will whiz through some introductory slides to talk about how this all works. So the goal with the lightweight threat modeling in general is to facilitate the advancement of software through the CNCF through the graduation process in particular. So when a project is submitted to the CNCF at the beginning of the process, if the project has security side effects, so it has cluster admin credentials, or it's used to enforce something with a guardrail, or it's an observability tool, again, with heightened privileges, tag security is asked to have a look at it, to model what could go wrong, and it is a wide, not necessarily deep look to make sure that there are no foot guns, people have all their limbs at the end of the deployment process, and in order to achieve that, tag security has a detailed self-assessment. This self-assessment goes to the maintainers and asks them a specific set of questions, how do people use this, what do you think could go wrong, and then we collaborate with people in tag security working groups and people from the project to build out this self-assessment, which then goes back to the TOC and is used to help determine the maturity of the project. Sometimes a project can't graduate from one stage to the next without making remediations as based upon the recommendations that come through from these things. This is quite a lengthy process. The assurance required to make sure that these projects are doing what's described on the tin and nothing egregious because we are a fully volunteer-led organisation can take some time. The goal of this new lightweight threat modelling process is to reduce the latency for these projects. We're still doing the self-assessments because they're incredibly valuable, they're deep, and they give us enough of an understanding to actually advance and recommend these projects. The lightweight process is to quickly and efficiently get feedback to the project and help to guide from a high level, again looking for breadth and not depth, and ideally to do this within a single hour-long session. Those are lofty and potentially unachievable goals, but it is a moonshot. This began in Tag Security issue 903. Let's pop that as well. The process is based on the Mozilla Rapid Risk Assessment framework. We had some working sessions where collaborators from Tag Security got together and collaborated balancing the different types of threat modelling approaches. The way control planes do this, and in fact just to sort of the issue details, a lot of people contributing some really useful different perspectives at light speed. We democratised this. We wanted an approach that would attract contributors. We didn't want to do something esoteric and unusual that nobody would then want to be involved with. We looked at various different approaches. We have, for example, the stride approach, which comes out of Microsoft, this Adam Shostack's typical approach. We also looked at this Mozilla Rapid Risk Assessment. There are other permutations. There's a pasta way of threat modelling. Ultimately, all of these things have the same goal, which is understand the system, ask a set of pertinent questions related to its security, identify a path through mediation, and then do the whole thing again. Add in for an item, add nauseam in some cases, and identify when we want to repeat that process. Is it to do with a graduation event in this case? Is it to do with a new feature release? Is it to do with a dependency tree shake-up? Whatever it is, it can be arbitrarily decided. So, that's the 50,000-foot view, lightweight framework, sandbox incubation. We looked at all the different approaches that people suggested on the issue, and Trail of Bits, who are a security company out of New York, who did the first Kubernetes code-level security review, put together this questionnaire format based on the Mozilla Rapid Risk Assessment. It didn't entirely fit our requirements, so we modulated it very slightly, but broadly it was excellent and it was fantastic to build on. What is the Rapid Risk Assessment framework? It comes out of Mozilla, and it's used to build Firefox. So specifically, I'm a Rust developer doing my thing in Firefox, and I want to ship a new feature. I fill out a Rapid Risk Assessment, and I go to security and I say, this is what I want to do, can you help me threat model it? So, for example, let's say we're going to ship secure tabs. So, Mozilla shipped cookie isolation for tabs a long time ago, I guess, now. There are privacy concerns. There are potentially information leakage concerns. There are functionality concerns. Actually, we just block access to everything, and while we can't be tracked, we also can't authenticate anywhere or persist the session. And so, the assessment asks, what's the feature, what are the side effects, and ultimately democratizes the process, again, of understanding what could go wrong and how will we fix it. The risk there, of course, that that is specific to feature delivery in a very specific context, which is a browser, which is complexity in itself. So, this is the foundation that we've built this on. Again, the modulations and changes are to focus on a component of a cloud-native system rather than a browser. That, again, that is the 50,000-foot view in Toto, and there we go, that's the slide I perhaps should have advanced to before waffling on it. The rapidity is such that we aim to do it very quickly. Keeping things at a high level gives us enough of a macro view to fix the foot guns without, for example, performing a code review. So, there is no code review in this process, concise and readable, easy to update. As software is constantly under development, so should the threat model be updated in tandem with it. This risk impact in data dictionary is helpful to a new maintainer to come and say, oh, it's been threat modeled. How do I understand how things are classified and what the sensitivities are? And yes, then it finishes with the recommendation. So, I think that means we're ready to have a look at what we did. We started with Flux, as they say. I can't actually read the screen from here. I think it's probably that link that I've mis-pasted in. So, Flux is a deployment tool for Kubernetes. This concept of GitOps is take a static state defined in, that's not the right one, apologies, a static state defined in Git. Use something with cluster admin credentials or some highly privileged deployment credentials. I'm really struggling to click the right link. I'm so sorry. That is the one that we want. I'm sorry. I'm struggling to see it. This guy. Do you know what? I'm just going to move it to this screen. There we go. Far less than that. Okay. Thanks for your help, Marco. Much appreciated. All right. So, just to refocus. Flux allows us to deploy stuff into a cluster in an automated fashion. So, instead of a developer having cluster admin credentials and doing a Qt control deploy, instead we've moved to a pull model. It's an inversion of control. It takes credentials out of a build server and instead gives them Git pull credentials to something inside a cluster. So, the security boundary ceases to be rights access to production. Instead, it's right access to a Git repo. Then we use branch protection. I wrote a white paper called Hardening Git for GitOps for Weave a number of years ago and the model is incredible. I really appreciate it a lot. So, Flux was the first project that we chose to assess in this way. I'll just pull this over here once more. So, what do we do in order to run through this? This threat modeling rapid risk assessment is basically a script. So, we've scripted a standardized threat modeling notes in introduction and then we sit with the team on an hour long call and we run through the documentation and we try and understand exactly what their intent and intention for the project is. So, in this case, and again, I feel like I should probably just maximize this guy. Okay. I've got a project. It's a GitOps deployment tool. Project data classification. Why is it critical? Because it holds cluster admin keys and what's the worst that could happen. Maybe someone gets remote code execution into the container. They have cluster admin keys and then they can enumerate secrets, deploy whatever they want to run previous containers and pop the nodes, all that good stuff. Okay. So, first of all, introducing the project with connections and protocols. These are threat modeling notes up here. So, this is the first stage of any threat modeling process gathering as much metadata, information and documentation about the project so that everybody has a common security lexicon and knows how to describe the various components of the system. We already had the genesis of this in the self-assessment and so what we can see here is reasonably we can just about see here is fluxes runtime behaviour and model. So, we can see at the top the internal and external interfaces we have a bucket so an S3 bucket, we have a Git repository communication services container registry and external event selectors. The reason that there, so this is a data flow diagram, the red boxes around things is because those are trust boundaries. Trust boundaries are something in threat modeling that are slightly intangible. First of all, what's trust? It means that we like something sufficiently to give it some level of I'm going to make a self-reflective recursive definition trust. So, when data is travelling between two different parts of a system a trust boundary might be a step up so if we're lower to a higher data classification could be the reciprocal step down. It could be the fact that two components are written by the same team. So, if somebody ships a malicious update into one well it's the same team anyway so they're probably in the same trust boundary it could be an external interface from outside of the system even though the data is still at the same classification we're passing through uncharted waters almost. So defining those trust boundaries is a little bit ethereal and slightly subjective but it's a required first step because again this is a very abstract process and it gives us a common way of discussing and describing things. So, in this case it's relatively self-evident the external surface there is where Flux says where is this Git config stored? Probably in Git sorry, the YAML config probably in Git, bit of course it could also be in a bucket etc etc other things we have in there and then the set of Flux controllers in the middle so it's just about readable those controllers are what reconcile the states of the repository or the YAML and the observed state of the cluster so it's a reconciliation loop that is constantly if I can remember observe, act I feel like there's three parts to it there's definitely observation and acting actually remember what the third one is but nevertheless it's observing the state and reconciling the expected state with the observed state and it's eventually consistent of course because the cluster is a distributed system which is constantly changing so at a very high level we can see obviously there is a lot of complexity to Flux so moving through this in an hour is inherently its own challenge so what do we do next how do people use this project okay we're deploying with GitOps to Kubernetes it's the only component making changes to a cluster this is a super important GitOps principle it helps us to reason about the security of the system because there's a limited number of people who can do anything to mutate the state and there is some new functionality in Flux which is this Terraform experimental controller the important part here is that we've identified something in the threat modelling process that relates to potentially relates to the threat model but it's out of scope scoping from a threat modelling perspective is how we attain an outcome in a reasonable amount of time that's because if we're going to threat model Flux do we also threat model the container runtime do we threat model the node that Kubernetes is running on do we threat model the data center that the node et cetera et cetera ad infinitum so in order to actually achieve an outcome from a threat modelling process we down scope as much as possible and that laser focus gets us to a conclusion that we can do something with as opposed to just sort of threats everywhere in a buzz lightyear panscape style so in this case Terraform experimental controller it wasn't anything that was in the headspace of the team it's extraneous to some extent to the core functionality so we de-scoped it, easy decision so how does the project work and at this point you can see with these are just really simple heading, question and answer it's a aquatic threat modelling ooh maybe that's what we should call it how does the project work yes well we're looking to get that state and pull it from somewhere and apply it into the cluster are there any subcomponents or shared boundaries this yielded some interesting results so we have these controllers the flux is actually a V2 the V1 of flux I'd used extensively I discovered a whole lot of things about the developments and the changes to flux that I had no idea about it can pull in customized code so it can run ad hoc transforms as customized does it can do helm so it can just pull those helm charts render and apply them in process helm also has a load of deployment hooks which showed extensibility potentially places that malicious software could run the source controller of course is where the source is being pulled from we saw those external interfaces and the image controllers so there is also this concept of flux deploying things and writing back a git tag to the repository that it's pulling the code from and that tag works as a pointer sort of a deployment offset so in case of disaster recovery if the whole cluster is a smoldering hole in the ground it can just redeploy it from the same repo and it's a reasonably deterministic thing those credentials back to the source repository that's otherwise read only the security side effects of the small functionality decisions of what we're hoping to elucidate with this process so what communications protocols does it use at this point it's worth deviating slightly to talk about what we do at control plane which is we perform these threat models and assessments for every activity we engage in we do it for all our customers it is the way that we rationalize quantify and justify the security controls that are going to cost money and that people would like some assurance actually work and are optimal for some some degree of requirements the way we do this is with a large a large matrix and in fact Marker would you mind just pulling up or just linking me through to the threat modeling training and we do have should have thought about this ahead of time, I apologize we do this in a data classification matrix so we look at all these flows we classify them and again it's a common taxonomy and language that makes it easier to visualize security let's say so what we've done here for the sake of rapidity is just list out a load of bullet points but there are other there are other quicker there are other less quick ways of achieving this as well I will also mention that control plane have a free threat modeling Kubernetes course that we do for O'Reilly so if you're on the O'Reilly platform I think we're streaming the next one at some point in the next couple of months we will make some noise when we do it and all of the collateral that goes with that is all open source and free as well so for the communication protocol taxonomy we care about what these specific parts of the system are doing and how they communicate why does that matter? because there are different potential problems with each different style of communication if we're just using TCP well hopefully it's actually bundling some sort of encryption around the thing it's doing is it packet and packet is it TLS are we using insecure ciphers are we using old SSH versions et cetera et cetera so that's what we're looking to determine with this reasonably exhaustive classification exercise so just whizzing through those source, helm, customize notification the point of doing this again is to build this kind of abstract problem space in the minds of the people undertaking the exercise so that when it comes to what could potentially the problems be then we're thinking actually there is a metrics end point I wonder if that's authenticated if it's Prometheus is someone going to check that it's not leaking some sense of information people dumping environment variables into those things that kind of nefarious attacker driven concept, yes I waffle a bit more here than we do there we make a fine point it's an excellent point so the question is how can we be confident in the completeness of the model when balancing that with the speed and compression of one hours time ultimately that is the risk balance we can't is the answer the compromise is such that and it's not an apple to the oranges well sorry it is it's an unfair comparison but for the point of demonstration tax security are also going through an Argo threat model assessment Argo is decomposed into four different parts we've got four different work streams and it's all volunteer led and it's taking us a lot longer than we'd like to do it so we are performing those exhaustive deep introspection on the thing as well but this is really meant to just say what's the most effective and lightweight thing just kind of referring back to the title it's not helpful but yes we're intentionally balancing that risk specifically to your point of how do we know that these are correct the project did fill in the self assessment beforehand they shared the documentation with us in advance so I'm being maybe a bit overly I'm glossing over the fact that there was preparatory work done by the people who attended we also had I mean highly skilled maintainers who have been on the project for a long time so they were just bang bang bang bang but as we go through this it is a question and answer exercise so there were points where things were missed and we said oh but what about this and then it was expounded on slightly so it is definitely fallibility introduced by process which is not in any way meant that it's funny that you say that yeah there's um I think as security people on the hook like the buck stops with security you can't really if you make a mistake it might have PII related implications yeah that's a really good point this is done in a very I think the spirit of the thing is like a blameless pre-mortem idea and certainly going through that with the I mean one of the things that I try and prefix to to any of the kind of the hacking demos or those kind of things that I do thank you to all the maintainers who've put all the effort in to give us all this incredible tooling yeah it's a really good point there is there can be a motion involved for the people involved and actually being the subject of one of these things can feel a bit like an audit sometimes for people and they're like yeah yeah but I mean it's uh yeah I suppose I associate audit as more of a sort of looming dragon of compliance whereas this is like we're trying to help guys that's really interesting yeah I'm taking yeah I'm taking notes for things to blog about yes okay so running through and hopefully getting as close to a reasonable degree of certainty as we can with these communications protocols so then we start getting into the meat of the problem thinking about and actually decomposing things down in my head to Linux namespaces is quite a useful way for me to think about this so what's in the process namespace well that's the runtime, that's behaviour what's in the mount namespace well maybe the secrets are at the disk those secrets might cross over if some of the mount are actually pulled into the application for usage maybe they're decrypted on disk and so they're slightly safer sorry maybe they're encrypted on disk and so they'll be decryption keys somewhere else that we need to use they've got network namespace what are the inputs and outputs coming into this back to the communications so I haven't scrolled down so the question is where does the application store data so we're thinking about the data that we've classified previously when it exists on disk what's its form is it breachable in the CIA triad so it doesn't need to be confidential doesn't need to maintain its integrity doesn't need to be available all the time sometimes the answer is no, strangely enough do we need certain types of data all the time well not if we've got a fallback sometimes metrics for example you can just fire them or fire UDP if you've got a saturated network connection you probably want the metrics to be dropped before you want application traffic system that kind of thing in this case what does flux actually do well it doesn't persist anything which is useful so immediately we can de-scope some of those confidentiality pieces but some data is stored locally where would I tamper with the data and at this point we're into that nefarious attack of mind set and this really is the value of this process because as developers we like to find path stuff we like to get the functionality out and security can be seen as a roadblock in these terms so saying to a maintainer if I was going to try and cause trouble for your users what would I do well man in the middle on the source controller because it doesn't serve TLS we know this because we have seen here that we've got an inbound connection serving artifacts on HTTP so that process has triggered some of the initial thinking to start building out the actual model the team were aware of this already there's an RFC in flight but we look to capture everything so that we have as holistic a view I hesitate to say the word but as wide and expansive a view as possible within the constraints of the scoping that we've chosen of course sorry I cursed me I'm just going up and down anything else here of note that Kubernetes CVE is a do not fit or is it won't fix so again we raise it here actually do we want to scope it if it's not going to get fixed upstream seems a bit redundant we could generate a whole document of things that are there bugs or features from that kind of perspective but we do have a control a suggested control there and again another man in the middle the CRD's explosion here is really quite significant but we're not storing sense of information and then we've got a final question about hard multi-tenancy if we've got a flux cluster controlling multiple other clusters well then potentially the threats there are around organisational responsibilities and who's actually managing those things so we are just whizzing through and let's just see what else we've got so sense of data and where it's stored we've got credentials here and if those credentials are exfiltrated we've got escalations potentially not only for things within the cluster but any workload identity integrations where we can exchange those credentials for cloud credentials of some description may lead to what ultimately becomes an account compromise or an organisational compromise just from someone breaking out of a container these routes to escalation are often possible they do have as we can see some useful sequence integrations with SOPs etc etc okay data storage with the CRD's do we have encryption on the things that we value no so as a recommendation of course we would ask the project to seriously consider encrypting things because it protects against somebody compromising a node sniffing network traffic sniffing and reduces reliance on an encrypted CNI of some description okay then we get into the data dictionary and I think we might be pressed for time if I don't go slightly faster so yes the data, the classification and any thoughts so at this point we've gathered enough information to start addressing some of these high level risks again because of the speed at which we are going the depth of these is not guaranteed so again this is scripted areas of controls that we're interested in based on what the audit working group selected when we say controls we mean a logical section of an application or system that handles a security requirement system may have authorization requirements etc we then detail the control families that we're interested in and these are again they're very similar to what the cloud native controls for the trail of bits order sorry for Kubernetes look like but with a focus on the usage of the system and not the code that authors it and this really is the instructive difference between most traditional threat modelling which is feature based looks at code considers how things are built and what we look to achieve specifically with this framework how control plane does this slightly differently which is the end user usage runtime behaviour and integrations and infrastructure usage of the application so we're looking to secure something that's more in the realms of a cloud misconfiguration than a kernel busting CVE gives a slightly different kind of value and they're both entirely useful in complementary processes but just worth noting that they're both called threat modelling hi Nicola okay so deployment architecture what does our runtime look like sorry I should probably just close signal for the purposes of demonstration there we go how do we deploy these things at runtime why is that important because a vulnerable application can be deployed with a lockdown security configuration that makes it safe to run you may not want to do that by default but if the cost if the choice is between running a trading system in production with a vulnerability or not trading then probably the answer is obvious we'll still have to move through this quickly networking, cryptography, multi-tenancy isolation, secrets management storage or N and Z audit logging and security tests and then we can skip entire groups of families if we so desire and so we want to know for each control what does the project do what's the data classification and then we get into the attacker mindset what does the attacker want to do here how would they attack it are there specific mitigations for example can we just stick some sort of firewall or inspection in between are there availability concerns and have there been similar vulnerabilities in the past it is a case of history repeating it's again deeply instructive to go back and look at how these systems have been attacked previously and help to derive controls there so and I'm not sure I think we've probably almost hit time I will just whizz through the end of this into our threat scenarios these are the things that we found in the assessments so theoretical threats what about deploying into the flux system namespace what would happen if someone gained access to those credentials what about image pool policy this is an old school issue basically can we retrieve other tenants images from the local container cache valib docker whatever it used to be and what about shared secure multi-tenancy multi-tenancy in Kubernetes is really difficult the DNS for example is just defined to show you everything you can see all the services environment variables pointing to everything else that's deployed as a service in the cluster anyway what about network policies okay we get into the multi-tenancy lockdown and it goes a little bit deep but again kind of to the point these are things that the security engineers in an hour ideated and it's just bang what do you think about this yeah that's potentially a problem gets captured and we move on to the next the important thing generally when threat modeling is not to conflate the threats and the control in the same breath because the stride process delineates the what are we building what could go wrong and what are we going to do about it is three distinct parts of the process and this is because it can be difficult to keep the high level view while also thinking oh but one of my favorite controls is X for example so we throw that entire approach out with the rapid risk assessment everything really is done in such an intense period of time that controls are suggested in flights so anything else this is all public of course so everyone is welcome to jump in and have a read in more detail and we went through and recommended our controls as I say these kind of conflate a little bit here in places yeah so for example here we're just going straight into recommending a specific admission controller where we wouldn't typically do that and then with a view to actually and we kind of as you can see jammed some of those in further above and then we go through and actually recommend the controls that we think the project should should well apply to remediate the NS poisoning that's the do not fix but we do have a miscation for it and then write out official recommendations and that was an hour's work huge thanks to everyone who is involved with it it provided enough value to corroborate some of the security decisions the project had already made and to generate a few extra issues I really went on for much longer than I have tended to going through that so rather than doing anything in flights we will draw it to a close any questions or comments before we do so the question is would specific questions on container security context or best practices be useful potentially I would put those under the deployment architecture pod and namespace configuration section probably see it on the left hand side if I did yes so we do have a section for it that was they said everything is fine we don't think there is any problems with the deployment there so I guess yes potentially the quality of the output is really based upon who turns up to contribute at the time so if there is somebody who has specific kernel API level knowledge they are going to be asking the questions that lead to those answers so yes potentially but I think balancing the yeah I hope that is alright but I am interested in what specific questions you might add there as well yes so I guess it would potentially be a question in the previous yeah I see what you mean now so to actually put it into sensitive data that's a great shout yeah I think that is worth taking back to the group and suggesting thank you any more okay thank you very much for your attention everybody that was a light speed, lightweight version


The next item is this jelly cat. I always pick up jelly cat. It's one of those brands where I see it and I'm just like so excited to find it. They're just, I feel like they're really good quality and they typically sell well. Now are all of them a big money Bolo? No, but some of them can do pretty well. I ended up selling this one for a best offer of $50. The buyer was all in for $60203 and I got it at a garage sale for 50 cents. So you gotta love that. This is what the tag looks like if you are not familiar with it, but definitely look them up if you see them. Hey, Bolo buddies, thanks for watching. It's from my little pony. Amazing, right? All right, let's get started. Hey, Bolo buddies, thanks for watching. Okay, I've got 22 big money Bolo's coming your way, items that I bought low and sold high. I'm gonna tell you where I got it, what I paid for it and what it sold for. One item went for over $400 and there's a little bit of a story behind that one. So we're gonna get to that. So let's get started with some big money Bolo's. If you're new here, my name is Courtney and Bolo stands for Be On Lookout. So items to be on the lookout for, items you can hopefully buy low and sell high. All right, so the first item we're gonna talk about is this Sonic the Hedgehog Build-A-Bear. I picked this up at a garage sale for $2 and sold him for $60. The buyer was all in for $78.50. So if you are not familiar with Build-A-Bear, you should definitely be looking them up because some of them can go for hundreds of dollars. I actually have a video on the top 10 Build-A-Bear to be on the lookout for. So if you missed that one, definitely go over to my channel and check it out. This one even had a little flaw right here and right here and still sold for $60. All right, the next item I sold is this Vintage. It's like a model kit, a military model kit. Let me show you, actually this one, I'm trying to remember if it shows photos or not. Nope, this one does not, another one did. But you can see here that it is new old stock but the seal is broken. It's ripped. And this ended up selling for $52.50. The buyer was all in for $59.55. And I got this at a garage sale for $3. When you're listing these model kits, you wanna make sure that you put the model kit number if it has a kit number on it. Trying to remember where it was on this one. I think it was, it's kind of been a weird spot. Right there, 143 and this is made in France. And I did put France in the title as well. All right, the next item I sold is this Dandy Bunny. And it is animated, it is super cute. It has lights and sound, it's ears like flip-flop around. I love picking up animated plush, it usually does well. I typically price my animated plush on the higher side and wait for the right buyer. I did take a best offer of, or actually I think I had it on sale and it sold for $60. And the buyer was all in for $69.25 on this. And again, this came from a garage sale for $3. The next item is this Goldstar HP90 cassette blank tapes. So whenever I find blank cassette tapes, I will pick them up. I decided, I had four of these and I decided to list them individually with a quantity choice. And this actually sold two separate times and it was a non-paying buyer. The feedback was low on both buyers, but I don't always worry about that because I've had a lot of people with new accounts buy from me and they pay. This was the third time I had relisted it and somebody bought all four of them again. And I'm like, oh, here we go again because I think they had zero feedback. And they ended up paying, so that was fantastic. I ended up taking for all four of them $40 and then the buyer paid the combined shipping. So they were all in for $51.17. And I picked these up for 50 cents each at a garage sale. The next item is this Vintage Briar Palomino horse.  And I just, for me, Briar horses, I know some of them can go for crazy money, but they're so hard for me to identify. I feel like they look so similar. But anyway, I believe this was model number 57. If you can figure out the model number, definitely put that in the title, that's gonna help. But I'm wondering if me doing that video over on that channel drove people to my store to look for these horses because I have almost sold out. I think I have two, maybe three left. So I don't know, that's just something I've been kind of thinking about and anybody can do a YouTube channel with their products. You know what I'm saying? To drive business to their eBay store, not necessarily like a channel like this where I'm teaching and helping you guys learn and I'm learning with you, not that type of, this is literally me testing products. And I just did a video saying, oh, I got these Briar horses and I showed them. And then people put in the comments if they knew which horse it was. And that's actually how I identified some of my horses. So anyway, just a little tip, maybe something you're interested in. I don't think I'm actually in that video. I think it was just screenshots, I can't remember. But anyway, sold this guy for $59.20 and the buyer paid shipping. They were all in for 76.27.  It was around the same timeframe. And I just kind of sat them in my garage and forgot about them because I didn't wanna look them up. And I wanna say this one was probably $3 to $5. All right, this Star Wars figure lot. I picked this up at the Goodwill in a bag of Star Wars items. And I actually did this. I listed these on my live listing show over on my reseller testing Bolo products which I have not done forever. And I apologize for you guys that come and watch that and hang out and list with me. I have just been crazy busy. And I keep like, I'm gonna do it tonight. I'm gonna do it tonight.  I need the list, just always a million things to do. But I am gonna get back to that because I do need to list. But anyway, these Star Wars figures, I can't remember. I wanna say the whole bag was like seven bucks, maybe eight, I have a video on it. It's the video where I'm like reaching up for the bag and it's on this channel. And I ended up selling these for $45 best offer. And I'm gonna say my cost of goods because I parted out the bag was anywhere from $1 to $3. And the buyer was all in for $60 to $81 on these. And they are pretty big. So this is what I like to do in my photos is I take a measuring tape and I actually put it in the photo to show people the measurements. So they know exactly what they're getting because sometimes photos can be misleading. I listed a blaze in the monster machine. It was a little car. And I'm like, if I was looking at that, I wouldn't be sure how big that car was. So all right, my husband picked this one up from an estate sale. He went to an estate sale and they said, do you want all this jewelry for one price? I can't remember what the price was, but it was a bunch of stuff. And I was looking over his shoulder and I picked out 10, 15 things. And I went ahead and listed those for him. This was one of the items. It's a Sabona Western bracelet, horseshoe links magnetic accents. And I don't know, it just kind of caught my eye. And I think if I remember correctly, I kind of priced it on the higher end. And this ended up selling for $37 plus shipping. The buyer was all in for $45.80. The next item, I picked this up at the Goodwill. I did pay up a little for her. I paid $7 for her. And I ended up selling her for $50. The buyer was all in for $64.24. I just saw her and I thought she was beautiful.  Let me see, I thought I showed a picture. Here's the tag on the inside. It's dolls by Pauline. Yeah, here we go. How cute is that? I don't know, it just seemed like a good quality doll. And I just, I loved it. And I'm like, I'm getting it. I'm gonna find a home for that doll. But it's by applause and it's a mom and a baby. Super cute there. The next item is this jelly cat. I always pick up jelly cat. It's one of those brands where I see it and I'm just like so excited to find it. They're just, I feel like they're really good quality and they typically sell well. Now are all of them a big money Bolo? No, but some of them can do pretty well. I ended up selling this one for a best offer of $50. The buyer was all in for $60.203 and I got it at a garage sale for 50 cents. So you gotta love that. This is what the tag looks like if you are not familiar with it. But definitely look them up if you see them. Okay, this came from a garage sale. I wanna say I paid a dollar each for these. Another item that sat and sat and sat in my garage and I listed these during a live show. And they're his and her mugs. And Belique, B-E-L-L-E-E-K, Irish porcelain mugs, his and hers. And these actually sold pretty quickly once I listed them, once I finally listed them. Let's see, $45. And the buyer was all in for $57.51 on those. And if you guys are new to the channel, I'm not big on breakables. I prefer to ship things that do not break just because I feel like I overpack and I spend so much time on it. And then I always think, is it gonna make it? So I just kind of stay away from breakables. So I was patting myself on the back for getting these listed. This next item is this vintage storyteller brooch, early Eskimo Turnbull Canada pen copper signed. I got this at a thrift store. I paid 50 cents for it. It was amazing. And it's signed on the back. I just kind of wrote what it said. And I love the caliper. I use this to show the measurements. And these are really inexpensive. I got it on eBay. I wanna say it was like $10 or $12. Back when I bought it, not expensive at all and a really great tool for jewelry. Let's see, I paid 50 cents for it. I sold it for a best offer of 75. The buyer was all in for 88.75. The next item I sold are these calico critters. I was at an estate sale.  I don't know. I'm trying to think where you get them.  Well, it was full of these calico critters. And these were not the new ones. These were vintage. And I've sold quite a few, but I ended up parting them out. So I'm guessing I have a dollar or less than these. Sold these for $37.50. The buyer was all in for 45.72. And this is the mouse family. The next item I sold are these Santa Claus punch mug cup ceramic Christmas mid-century moda mugs from 1976. This is another item that I just put off listing. I think I listed this around the same time I listed those Irish mugs. Because again, breakables, putting it off. And when I do breakables, when I list them, I typically package them right then. That way they're packaged and boxed. And when it's time to ship, all I have to do is put a label on it. I ended up selling these for $65. The buyer was all in for 95.24. And I got those at a garage sale for $5. This item right here is absolutely amazing. And it had, let me see, the original packaging. So I was at a thrift store and this was a mystery. And I want to say it had $10 on it, but you couldn't open it. So I didn't know what I was getting. And yeah, definitely a bolo. Danbury Mint, it's the Bird Watchers Christmas Reef. I think I still have the Snowman left. And then I sold another one that was in this series. I ended up getting three of them. So $150 on this. And the buyer was all in for 195.38, but really, really detailed. I mean, if I didn't have that styrofoam, I don't know how I would have shipped this because these birds are so delicate and the wings and the beaks. And it's just amazing. It's a beautiful piece. The next item is this Coda Pillar. I have been selling these left and right. Not really, but I think this is my third or fourth one that I've sold and I just listed another one. And I actually found another one recently at a garage sale, but it was not functioning properly. So I ended up getting rid of this front part of the body, the head, and I parted out the pieces on the back. So you can definitely part these out and still make good money. I got this one at the Goodwill Benz and I sold it for $40.70 and the buyer was all in for $50.60. This is the second, this is the first one I found at the Benz and then I recently found another one at the Benz. So I don't know if it's the weight of the item that's scaring people off or they just don't know, but this is definitely a good item to pick up. The next item I sold is this 1995 Spiegel Barbie. I got this at a garage sale for $3. I bought all the Barbies at the sale, paid one price for everything, broke it down to, like divided it out by the amount and the number of items and it came to about three bucks. I ended up selling this for $36.99. The buyer was all in for $46.24. So that one's not really big money, but it was over my $35 bread and butter. So we went ahead and included that. This grapevine owl, I almost did not list this. I got this in a mystery box. I believe it was a Halloween mystery box and it was just really, really cute, but I was like, oh, I gotta find a box for that. I am sometimes very lazy about shipping things that are bigger or harder to ship. I just like the easy stuff, but I decided, you know what, this is too cool. I am definitely gonna list this. I ended up selling this for $40. The buyer was all in for $5205 and I got that in a thrift store mystery box. The next item are these vintage hard plastic happy Easter floral picks. And I believe I listed these during a live show also. These are definitely something to be on the lookout for. I got these at a garage sale for $1 and I put them up for auction and they sold for $39.99. Plus shipping, the buyer was all in for $5309. The next item is this Ann Klein Gold Tone Leaf Link Necklace. And I put in the title or in the description here that it does need to be cleaned. I did not wanna take the time to do that. I probably could have gotten more, but you can see kind of like right here in between. It's just got some like dirt and stuff. There were also some light scuffs. You can see here with kind of the, what do you call it, plate wear maybe? Is that what it's called? And then if you look at the back, it's just, it could be cleaned up. So I just mentioned that, but really, really cool vintage piece. I ended up selling this for a best offer of $40. The buyer was all in for $47.65 and this one was from my original 344 pounds of jewelry that I purchased. My bulk buy of jewelry. So that was, I still have so much of that to list. All right. This item is a vintage Royal Dalton Brambley Hedge Winter Plate by Jill Barclam. And again, you guys, it's a breakable. So I did not source this. I was actually gifted this. Terry from Mindful Recycle, she sent me a box, a mystery box and it had like clues and I had to try to guess what it was. It was super fun video. It's over on, it's actually on this channel. And this was one of the items in the box. And when I tell you this box was amazing, she sent me a box of Bolo's. This ended up selling for $37.50 plus shipping. And again, cost of goods was zero. So Mindful Recycle, Terry, if you were watching, thank you so much again for the awesome gift. And I recently featured her channel. I did 50 Bolo's that sold for $100 or more. So go check out that video. She is a major Bolo finder. Check out her channel if you're not already subscribed. You will be happy you checked her out because you will learn so much. All right, this is it. This is the one I want to talk to you guys about. Oh my goodness. So I have had this listed for probably over a year and I had it on local pickup. And I was like, I am not shipping this. And it had the original box. It had some of the original packaging as seen, but I just did not want to ship it. I just didn't feel comfortable. And I know that how much money they go for. And I was like, no, I'm not doing it. I probably had 10 different people message me and ask me to ship this thing. And I'm like, no, not doing it. Finally, somebody's like, will you please ship this? I've been looking for this item. I really, really want it. And I'm like, oh gosh, I don't want to do it. And they're like, I'll pay you. Actually, I think they told me they would pay me 500 plus shipping, but then they ended up, anyway, we ended up working it out to the $431, which I'm very happy with. I picked it up at a thrift store. I don't recall what I paid for it. I wanna say it was under $50. So I spent an hour, I had bubble wrap them lightly, knowing that I was probably gonna have someone pick it up. And I just wanted to protect them a little bit. But you can see right here, like I did not do a great job. It was just a thin layer, just to keep them from banging around. They were mostly protected by the cardboard, okay? I spent an hour putting extra bubble wrap around these things, adding paper. Oh my goodness, the box almost didn't fit into the box. Then I took it to a place, a local shipping place. And what they did was, they put packing peanuts all around it and double-boxed it for me. This camel right here, I was like so scared it was not gonna make it because those legs, and I was like, oh, the tail and the neck. I'm happy to say that the item made it safely. And that was a super, super big sale for me. Really excited about that one. Shipping was a lot, $350, but I had to take it and get it professionally packaged and that's not cheap. And I forget where it was going, but shipping was really high for this. It was a big box.  These are, let me see if I wrote it in here. 12 inches, so they're big ones. And it was really, really cool. All right, the next item is this, the Singamajigs Royal Blue Toy Sing Skin and Meringue Song. If you've never seen these, you squeeze their belly and their mouth, I go, anyway, they're really, really cool. I have a video over on my reseller testing Bolo products. If you wanna see how annoying this guy is, you can go check that out. Links to my other channels are down in the description. I have Sourcing with Bolo Buddies, and reseller testing Bolo products. Thank you guys so much for being here. Thank you for watching. If you're new here, I would love it if you would subscribe to the channel, hit that like button or the thumbs up, and there's gonna be some videos popping up here and here and a subscribe and another video down below. So check those out. Check out another video. I also have videos on items not to look forward. There are, I think, four out right now. It's a four-part series as of now. I'm gonna keep adding to it as I find more items that you should not be sourcing for because you can either get a zero, you can get your item removed from eBay or a suspension. So just bad, bad things. And I typically, what I did originally is I posted in my Facebook group for members of my Facebook group to share things that violations that they've gotten. So it's like things that are currently happening. So I really like that. And again, there's four of those available. So go check out one of those and thank you for being here. Thank you for watching and I'll see you the next one.