Goodness knows, that's a hard task. Thanks for staying for the last presentation of the day. I appreciate the endurance. I'll do my best to make it worthwhile. What we're talking about here is full-text search, and we're going to show two paths. One is your start time or your time elapsed from right now to getting it up and running with all default options is really, really short. Then we're going to show, but you have a whole bunch of configuration options. There are two, so some razzle-dazzle on the end. This is just so we're on the same page. The level set, what exactly is full-text search? Right there, it provides natural language search capability, really natural language-ish, and the ability to sort documents by relevance, not just existence. We'll show why that relevance piece is really, really important. Here's the business case. Why do I care so much about full-text search? What makes it so useful? The first thing is it allows, when we write applications, we can focus on the semantics, not the syntax. In other words, when I look for something, I don't want to get boggled up in uppercase, lowercase, like, I like functions and everything like that. I want to focus on the meaning of it, and we'll show you how full-text search really allows the application developer and the person writing the query to focus on what exactly it means. The other thing, it's a crowded space full-text search. There's Apache, Solar, Lucerne, Sphinx, there's other pieces out there, great products in their own niche. But one thing none of them can do is leave all your data right where it is, which is to say in Postgres. So the preferable solution, independent of your platform, it's always faster to do nothing than to do something. Well, if you can get that capability within Postgres, it's better than having to do it outside. Then we'll compare it, if the Naples Ultra or these dedicated machines, the other thing we're comparing it to is just standard SQL searches, and we're going to say almost no effort, you're about 20 times performance improvement. Anyway, and it's fast enough for nearly everything. That's how to get results. Then let's really focus on the ranking. What makes it so neat is again, we're used to focusing on the existence. All of our thought processes as database developers and administrator is, do I have a record? Change our focus here. We're now focusing on relevance.  So we have a full suite of tools to address that side of it. We have parsers, stop word list, synonyms, don't worry about the definition, we'll get to that through the presentation, but we have a sort of a whole suite of ways to customize it, and it's at the statement level. What that means is the same set of data, you can write different applications to make it look and return results differently. So anyway, and like everything else in this conference, Postgres or full tech searches, extensible. All right, we have a full packed agenda ahead of time. The slides, I'll show you where they're posted. We're gonna have a lot of live action and there's some SQL statements in there. Please don't focus on the syntax. If you're interested, go back and see it later. What I wanna focus on is the functionality. Sort of the idea is you can say, if this is cool for my business application, I'll learn the syntax, but not necessarily the other way around. We're gonna, I did some data wrangling to get the set, to get the data set we're using here and that's also available. Now we're gonna show you how to use triggers, how to manually create the full tech search piece, compare some things and there's really. Also, it is likely that we will not get through all the slides in this presentation. Again, all the materials up there on the website.  I didn't know it could do that. I want my app to do it. Dad, stuff about me. All right, all this stuff is running 941, the one click installer from EDB, CentOS VM. I have two sets of documents in the world of full tech search. A set of documents is called a corpus. So for the purposes of this 40, 50 minutes, our corpus is two things. From the Library of Congress, there's 18 documents in American history. Sort of didn't get the gist that we're such an international conference. So for those of you outside the states, learn some American history. And 38 articles from a blog post from Data Science Central from Data Science Girl. Anyway, interesting domain to me. So those are the pieces that we'll be searching. PG Admin for everything, just a few Linux commands. I cut and pasted all this stuff from the web. The Data Science articles all started out as PDF files. And I used the Linux function PDF to text. What I wanted to focus on here is if you're gonna use full tech search, it includes a bit of this data wrangling. There's no two ways about it. In other words, you're not gonna get a... Fully populated tables aren't stillborn. All right, enough of this setup. How the heck does this work? So full tech search takes a text document, busts it up into pieces, tokenizes it, standard, it breaks it up on white space and on printing characters. It takes each of those pieces and gets what they call a lexeme. Which is you can think of, it's the root of the word. It takes away tense, it takes away singular, plural, it takes away all those pieces. So smarter people than I said, the lexemes, it's the basic unit of meaning. Of the word. What this allows you to do, this is that natural language piece. I can focus on what the word means. I don't care what tense it is in my document. As an example, look at the lexemes, or excuse me, look at all the words that lead to the lexeme jump. And also they're case insensitive. We're gonna show some examples here. But that's in your mind, all these searches, we're looking at the root of the word. Also all this will be done in English, but that's just a configuration option. All right, so I have my document, I tokenize it, I look at those tokens and I take all the endings off of them and then I put them into a vector. And that vector is where I get my speed. The vector is a sorted list of all the lexemes with just a number for what place they are in the document. So we'll show you some of the examples of that. So we don't, in other words, we're not searching on the full document, we're searching on the TS vector. And then against that I apply a query called a TS query. And that is also these lexemes, these word roots, if you will. Only in the TS query I add Boolean operators. All the stuff that you would expect. And or not, and stuff like that. Postgres has the ampersand to ampersand operator that matches, here's a TS query that I built against the TS vector, that gives me my result set. We'll show you some examples, it'll make more sense when you see it. All right, so we use built-in function to TS vector that turns a text document into this TS vector. That's the sequence of events. All the pieces are configurable for our purposes. I really want to focus on, by default, they work pretty darn well. So applying Occam's razor, see if it works by default. If it doesn't, add some razzle-dazzle. All right, one thing I forgot to add. There is the capability in a TS vector, so I have my document to add weights. What that means is, if I have a match in, for instance, the title of a document, it matters more than if I have a match in the abstract of a document, which in turn matters more if I have a match in the body of the document. We're gonna build some weighted TS vectors. All right, and this whole thing that the last two slides in Postgres are called a text search dictionary. That's really, think of it as a collection of all the rules you're gonna use to set up and use full text search. All right, and we talked about this. We have the ampersand operator. Getting too much stuff. Let's look at some examples. We know what time it is. And it never works as planned. All right, I have a database, FTS. And so here's my TS vector. Let's look at here. Just, oh, look at this. All those words have the same lexeme underneath. That's your power. Whether enumerate, enumerated, enumerate. You'll see for American history students the enumeration clauses twice in the Constitution and the Bill of Rights. That's what we're gonna look for. But that's the semantics piece. Don't care about the syntax, it's the semantics. And it also checked this out. In this simple statement, I did a couple things automatically. One, I threw away stop words. Stop words in this lexicon language here are words that show up in all your documents. That means they can't be discriminators. In other words, the word the, since it shows up in everywhere, it will never add value to your search. So full text search has a list of these stop words that's configurable, it tosses them out. So you're filtering noise out automatically. You can also see the roots at the bottom there and it shows what position they're in. That's a TS vector. It's a much more compact representation of your text document. For fans of Samuel Taylor Coleridge, it's from the rhyme of the ancient mariner, what we can see is repeated words only show up once. So there's our TS vector at the bottom. The word alone is the first, second, fifth and sixth position. And this is in our mind, this is what we're building with TS vector. Because I'm a fan of Aretha Franklin and who isn't. That is what the first line to chain of fools comes down to. What a wonderfully compact representation of that piece. All right, just wanna show you one quick example. Remember, we're focusing on tokens that are words. There's 23 different kinds of tokens beyond that scope of class. What I wanted to show you here, we're parsing Microsoft owns a class, you can read it. What I wanted to show you though is that full text search understands URLs, understands IP addresses, understands domains. So Microsoft.com got tokenized into, oh, there's a word, Microsoft, that's a lexeme, and a domain or a name, Microsoft.com. Those other pieces are beyond the scope of this. But you have the ability, as a teaser example, if you have numbers in there, to say store numbers to three significant digits. So if I search in my full text search document, if they match to three significant digits, that's a match. It does that by considering other data types. Okay, so here is a TS query. So that liberty and justice for all, you see and and for are stop words. Don't care, they're not the good discriminators. You see the two roots and the Boolean operator between those two. But that's a little awkward. So that's the same answer. The function is plain to TS query. The limitation is it's all and, so the Boolean operator in all those terms is an and. The advantage is, holy cow, is that a heck of a lot easier to write than some gosh forsaken SQL statement. That's all you need to do. And let's, I'll show you that, I'm sure you would trust me, but you can trust but verify. These two are functional equivalents. Four square and seven years ago, they produced the same result. And again, through the whole thread, what we're after are ways to reduce the space between what the person, the querier has in their mind and what I have to type. All right, let's jump back. We did all these, this is part of it. And then we return. All right, so you say, I'm sold, I want them, but I want TS vectors bigger than one line. So how do we create them? I'm a consultant, so I have to say it depends. Simple way, you can create them at run time. In other words, don't create them ahead of time. When you run it, use the TS, two TS vector function. It's simple, doesn't require any storage, but it's slower. You can create ahead of time with two TS vector.  That is fast, it's flexible, it doesn't slow ingestion as a trigger would, but the limitation of that is it is possible if you manually update your indexes to have the document and the TS vector out of sync. You can create them ahead of time with a trigger. Fast and painless, but the trigger slows up ingest. This is your decision on there. There are two triggers to maintain TS vectors that come with Postgres. TS vector update trigger and the other one associated column. We're gonna show those, and then I'll write a custom one for weighted functions. But this is really, this is the business case. One thing that doesn't happen if you're using Lucerne or Solar or something, they never say it's easy and invisible to you. There's always, whatever solution they have has to have some effort in it. But finding it isn't good enough. I want them wicked fast. I have two types of indexes, as in everything else, gin and gist. There are advantages and disadvantages. We're not gonna get into the particulars, but the gin is a much faster index. It's also much bigger, takes slower, takes longer to create, and the gist is just the opposite. I gotta give a shout out, never met him, but Thomas Vandra has a blog post on Planet Postgres. He has a lot of details on high performance full-text search. I'm more into the functionality. But anyway, keep in mind there are options. The documentation's quite good at this piece. All right, so let's build some tables. All right, search there. I still struggle. I told my boss I knew I'd struggle with this piece going up or down. All right, so here's our first table. American History. What I wanted to point out here is TSVector. I'm creating a data type TSVector document. If anyone hasn't seen that other data type CI text, it's awesome case and sensitive text. It works in certain circumstances. If you don't want to use the upper or lower, just keep it as case and sensitive text. There are some performance implications, but it's generally, I'm not that correct. Comment my tables. We're gonna use a GIN index here, and this is what I wanted to show you here. I am creating, and again, all the syntax, don't worry about the particular syntax that's all available in the white screen. What I wanted to show you here is I am using the built-in TSVector Update Trigger function, so that's my trigger function. Again, everything we're seeing here right out of the box. There's the reference in the documentation section. Now, let's get a table with a bit more razzle-dazzle. Mother, and I'll show you. This is a table, the data science articles, and one thing I wanted to tease in for a couple slides over here. I have a field for the abstract, a field for the body, and a field for the title. Looking ahead, I want to wait. I want to use that in my waiting. Does that make sense? In other words, when I build my query, I want to say, hey, if my lexeme is found in the title, it means more. Well, to do that, you have to have the data separated. That's on you. Not hard, but it is on you. And then, I'll create an index. Here, let me show you. All right, what I wanted to show you in this, so here's my custom function. The first one was created right out of the box, nothing, here's my custom function. And what I wanted to show you is I take the first field, title, and I give it a weight of A. Second field, you can't see, but it's a weight of B. And just the whole body, the default weight, which is D. I only use the defaults. They work great. Details are in the documentation and in the presentation. But what I wanted to show you is you only have to do this once. From here forward, I have a weighted vector, a weighted TS vector to work with. Let me see if I got anything else. All right. And then here's my trigger. I think I had loading the documents as well. So now we have our document set up. We have triggers to populate it automatically. I'm gonna load my documents. I use pgread file for the text, pgread binary file for the PDFs. I could envision that it would be useful to wanna have the PDF. Some data wrangling limitations on that. They have to be under pgdata, but symbolic links work, so it's not such an onerous restriction. There's an example of the syntax. We talked about how the weighted document required you to have the sources bundled up. And, because we can't help ourselves, I couldn't help myself, show you a little razzle-dazzle on the loading with a Postgres function to look at the operating system, do an LS command on the operating system. All right. So, boy, I wish I had more resolution, but I don't. So, I will use this SQL statement to generate these SQL statements to load my files. You can see how it works. I wanted to point this out because it really, my objective, it makes it really easy. So, this is the American History documents. It's only one file for each. I synced everything up with Dropbox. Control-A, delete. Pardon me for this. Let's do this one. Let's do this one more for the data science articles. Slightly more complicated SQL statement. But, like most DBAs, I feel, crap, oh, I feel more comfortable in SQL than anything else. So, that was just SQL statements. It goes to the operating system, does an LS, finds the files there, builds these for you. Gotta live where you're comfortable. What we just saw, by the way, was loading files from the operating system, creating TS vectors, in one case, creating weighted TS vectors in another, and loading the PDF files. And I did that in front of an audience with text that was too big. Pointing here, it really isn't difficult. And I have, I have one more piece that I have to do. In here, to load the titles of each document and some ancillary information, that I created that separately. So, anyway, what we're doing here is, I want this, I have the source of each document, the title of each document, and it's just based on the filing. So, that would be, if I have a list of files on my operating system, you need to know some information about them. That updates that piece. All right, so let's jump back to a presentation. All right, so again, all the files were in PG data. I use a symbolic link. It's actually to my Dropbox location, which is pretty darn well actually. There's my syntax for read file. Read binary file works the same way. And that pglsdir, it's a pretty cool function. All right, so we loaded that. We loaded that. Couldn't help myself with the SQL, but again, we're database folks. SQL's our second home. Gotta go with the one you know. All right, holy crap, that was a lot of setup. So, let's look at American history documents with some full text search. All right, load. Our corpus, to use their terms, is 18 documents. Anyway, don't worry about it. There's 18 documents in here.  All right, here's a traditional search. If anyone's not familiar with the operator I like, its case insensitive like also works great. So, there are documents with the word welfare in them. Turns out, four of the 18 documents have welfare, but that's not what we came here for. This is what we came here for. Oh, you know what I should do before? Sorry about that. Just because we can. So there's, we got our documents back just for giggles. 4.06 milliseconds. Keep that in the back of your mind before. Same four documents, 0.215. Again, no special anything. This is default configuration of the VM, of the database, of all the pieces. We've just increased our search time by approximately a ton. It's round numbers. All right, all right, but there are limitations to I like search or, and by I like that really any sequel, any traditional sequel piece. Now let's look where the word enumerate. For those of you who sort of into American history, there's an enumeration clause in the constitution which talks about our census. We need to enumerate our population. And in the Bill of Rights, there's a right that says, hey, anything that's not explicitly there is in, anything not explicitly enumerated ain't in there. But it didn't show up. And that's all I got. So, same piece, that would be really hard to get those answers in sequel. It looks simple here, but that would be a hard thing to do. That's, again, I focused on the root of the word. All right, so let's look to mother pieces. Liberty and justice for all. So there we have, trust me, there's eight of them, but we got a bunch of documents there. Very simple. I got the same results with just that. If someone wants to say, hey, how do I get documents with Liberty and justice for all? You just write Liberty and justice for all. You can't get any simpler than that. That's one of the other, so we saw two really powerful things. One, I didn't know what form of the word it had and I guess the wrong one in my sequel statement, so I didn't even come back. Two, that's a lot easier to get a user to write. And again, the whole point of it is, people who are interested in my corpus of documents, I don't want to teach them anything. I want to take what's in their head and capture it without modification, all right? So anyway, this is a traditional way that would be really hard to explain. The other thing is, it only returned seven. I could never figure out why there was one document that was missing, so even knowing ahead of time, it's complicated to do it in a traditional way. And you don't have to trust me, it's like 20 times slower, it's a whole bunch of stuff. All right, so let's look, so that was all searches. Now let's look at some Boolean operators, because the default was everything and. But that ain't good enough for us. So let's look for the things with the word colony. I have six things with the word colony, but what if I want to say colony, but not a foreign colony? Probably need to highlight the right sequel. All right, that's, see the Boolean logic there? Everything with the word colony, but not the word foreign. Simple, very simple to do. Right now, let's look at another one, all right? A lot of French, a lot of British in our history. Thank you guys. So if I have French and British, I only get one document back. Some doesn't smell right with that, so let's look some more. So British could also be English, and I don't really know. So here's a way to say French and either British or English. So what I'm showing is all that you normally do with Boolean operators, except for I guess that four value Boolean that they were talking about at that end. But all the other standard things are in there. Make sense? All right, let me jump back. Did that. And again, the I like is a case in sensitive like. It's really nice. All right, that was vanilla. Still cool, but it was vanilla, and I promised you rank the results. What we have before is just the existence of the document. Now we're gonna shift gears, and we're gonna focus on the relevance of the document. So the rank, it's a arbitrary value between zero and one. Zero means it's not found, one is a perfect match. We don't care about the number, we only care about the relative number. One thing to keep in mind with ranked results, just it does more work than a typical limit command. So in other words, if I wanna rank, say 100 documents, I have to look through all 100. If I have a limit five, I just stop at five. So it's a more expensive operation than limit. I got two flavors available, rank and rank CD. Both of them consider how often a search term appears. The second one, rank CD, considers how close they are together in the document. We're not gonna use it, but there's another parameter that weights it to the log of the size. Don't worry about all the details. What it says is, if you find your words in a small document, that means more than if you find them in a big document. Google does this before, it's available here. You're not gonna see it in this presentation. But anyway, it works, without using that, it works well. Now, as part of the ranked results, I gotta set up this weighting piece. Remember, we built the weighted piece with just our trigger. So all this happened magically. It's already been done. You might have blinked and missed it, but it's already in there. There is the excerpt from our trigger. The title is an A, the abstract is a B, and the body is a D. One thing to very keep in mind, that second bullet, that weighting has nothing at all to do with, does a result come back? Unrelated, nothing at all. It only affects the rank of the result, if that makes any sense. So if your objective doesn't include ranked results, I'll wake you up at the end of the rating piece, or the weighting piece. Makes sense? I'm gonna show you how, but we're not gonna do it. This is gonna follow up, I could, if I wanted to, build these weights at query time. There's the bottom, because TS vector has the concatenation operation works for them. The reason that matters is, if you think concatenating a TS vector isn't just sticking one aside the end, it has to reorder them all and put the place numbers in the resulting concatenated document. Anyway, that would be the logical equivalent of how we do it there. All right, what does all this get us? I wanna focus on document relevance, not existence. My argument, customers expectations have increased. They demand ranked results, because everything outside of my world's custom build has ranked results. And the data and the logic are inside Postgres, which I learned at the keynote was pretty cool. Anyway, and they're available to whatever application you use. All right, so let's go look at some of these things, shall we? What we're looking at now, these are 38 documents related to data science that the data science girl said are really important for us in her blog. They're really cool, I was reading them on the train up the way here. All right, so if I don't do anything, so this is what I wanted to show you here. This is not weighted, so this is unweighted, just a vanilla, and I'm looking, hey, correlation, big data. Those are my limit five results. We could just look at the top one, 10 types of regression. It looks perfectly reasonable, but it isn't, and I'll show you why. There's our answers. Now, let's look at rank. Ranked results returns a much different data set. This is, again, my unweighted document, so it doesn't matter where the lexeme, where the term is, they're all weighted the same. Now, my number one answer is the curse of big data. I don't know if it's a curse, but whenever. So I wanted to show you got unrelated results. Your stuff was not quantitatively, it was a qualitatively different result because it's ranked. Again, knowing the back of your mind, this was more expensive than the limit five, but that's it. So let's take it one more and say, hey, remember, we built a weighted TS vector. And how hard did we work to build it? We wrote a trigger. We didn't do anything else. Now, look at this. The number one weighted result is correlation in R squared for big data. Seemingly a pretty relevant title. Also noticed that the top five weighted result wasn't in the top of either. So again, we don't have our user's attention for very long. I got four kids, I understand about short attention spans. So I wanna show them something deliberately relevant. This allows you to do that. Thank you much. All right. Oops. All right. Again, no more work. That's the, as a father of four, all about the lazy. There we go. All right, we got that. All right, so let's just take a quick pause. This whole thing that we did, the set is called a text search dictionary. The default one is PG Catalog English. You can, there's how to see yours. We created weighted and unweighted vectors and we either did it manually or with a trigger, but wait, there's more. Because we've been thinking of a DBA so far. Someone's gotta present this stuff. There's a function called TS headline that'll take our result set and give it a little razzle. Because if you think about when I search for document, I don't just wanna know that it matches. I wanna know where it matches. I wanna know what's matching it. So this is the one function we're gonna customize fairly extensively here. TS headline returns a text fragment, so piece with HTML text. The default are just bold and unbold. There is the result of that Liberty and Justice for All. You see, that's the file you get, their highlight. And that is fine in all senses of the word. But we're not here to be fine. So here's some options that we'll look in and I'll show you how this will give you some pretty cool result. How many fragments do you want? Configure that. What comes between my fragments? How many surrounding words do I have in the fragments? And I need something better than just bold. So let's make it red. 10 fragments between five and 10 words and I wanna break between each fragment. Let's see what this gets us. So here is what we're gonna show the answer to. I wanna know what American history documents contain Liberty, Justice, happy. I want the title in my result set and you can read that. I'll show you what it does. That is, because I ran them here. So there's our default. It returns actually four results. This came right out of Postgres. Did nothing to it. It's a pretty useful result set right here. Again, all I did was save the, I pasted it in a file in the operating system and I'm just reading it locally. There you see the pieces, the document title. Here is more fragments. How many times it, George Washington's farewell address and the Northwest Ordinance. So again, what I wanted to show you again, the emphasis is I always wanna do less. The only thing I customized here was my TS headline and the only place I customized it was in the query itself. And these are some pretty useful results I would argue. Let me, whoops, you're probably not supposed to see that yet. TS headline, those were the results from this query. Those were the results. Again, all I did was, you can see it, comes right out of Postgres. I didn't do anything to it. That, no customization except one SQL, has gotten us directly loaded documents from our operating system, automatically indexed, automatically weighted and maintained the index with no effort on our side, supports natural language-ish. It's not true natural language, but it's a lot closer than SQL, a query with ranked results and a well-formatted output. That's a lot. That really is useful. And again, once you have this set up once, you don't do it again. All right, but since we're at a conference and everything's extensible, that's still not enough. Up to this point, what I was arguing or sort of espousing at this point is, if that's good enough, run with that. If it isn't, we can start customizing those pieces that make up our tech search dictionary. So we're gonna add some razzle-dazzle. We're gonna create a custom tech search dictionary. That's the word that describes all this stuff. We can say, hey, what if my documents have different stop words, non-default stop words? What if I have synonyms? Again, what if I wanted my query to say English? Well, English means British or English. And we won't get to it, but it's in the slide presentation. We can even use this to suggest miscorrections for misspelled words in the query. So a little bit of sprint to the end. It won't cover all the pieces. They're all in the slide there, but it's pretty neat. All right. So now I have to say, all right, how do we do this? We're gonna focus on stop words, because that's a pretty common and simple one. There's a file, comes with Postgres lists of all your stop words in English. They have them in other languages. We're gonna customize it. It is in the PG-shareder, T-search-data, option stop words equals English, means useEnglish.stop. So that's what we're gonna create. One note, and you'll see it in there, is every time you touch the source file, in other words, every time I viEnglish.stop, I have to touch the text search dictionary. Don't have to change it. I just have to, in other words, alter it to itself, because that forces it to relook. You'll see what I mean. All right. We have a function to say, hey, that's how I add stop words, but what stop words do I have? How do I know? We have something called TSstat, which gives us statistics on our TSvector. It says, hey, what words are most common? Because if it's, pardon me, if it's most common, it's not discriminating in my results, and which ones show in most, pardon me, which ones show up in most of the documents in my corpus. All right, let's do that, because that's pretty darn easy. All right, so what we're gonna do, we're looking at the data science articles. No surprise, what word shows up in data science articles a lot? Data, and use data. No surprise to us, but you can imagine in your corpus, it's gonna be nice to throw out noise. So there's our noise in our data science article. I wanna get rid of the word data, because it doesn't discriminate in my documents. And again, the term discriminate in this context means it doesn't help me identify ones that should be distinguished from ones that shouldn't be in my result set. All right, here's a variation on it. I may also ask, hey, how many documents does it show up in? It actually shows up in every single document. So it's absolutely worthless as a term. So what I'm gonna do is create this custom dictionary called Stop or Dict. But first, let's look at it. So PWD, share dir is actually, sorry, PG Home, share, PostgreSQL, we're in T-search today. This is also in yours. You'll see I have an English dot stop, created a backup of it. So VI, English dot stop, go to the end, put my name so I know where they are, data, write quit. So I just, sorry about that noise. All right, so let's do one quick thing here. Create that database. So here's another function here called TSLexSize. That's like TSVector, only it just checks one word at a time. So what I'm doing there is saying, hey, give me the lexeme of data. And it says, that's a stop word. Now we know it wasn't before because it showed up in every single document. So see this sequence of events. I'm identifying what things aren't useful. Just add them to the end of the file. It doesn't matter where. It doesn't matter how. And they go in there. If you added another word, you would have to rerun, excuse me, you'd run an ultra dictionary. You'd rerun this. So in other words, it doesn't change anything. It's just, that's the example of touching something. All right, so we're five minutes from the end, so we're gonna stop here. I told you there's more material coming on. You can, so that if you think of how you can adapt a full text search, this section right here just focused on I can change the TS vector. So in other words, change what's stored in the database. I could also switch that and say no, I wanna change my TS query, which is to say if someone submits a query for the term data, don't let it show up. So let's try that. That is a function called TS query rewrite, and I'm sorry, we'll get to this and then we're gonna stop. It, the full text search function checks a table that we're gonna create, FTS alias, and it says hey, instead of this, use the alias. Well you put a null in the second column and it effectively becomes a stop word. So let us show, we're gonna create this table, full text search alias, boom. We're gonna insert, and what I wanted to show you. So that, there's my cast, I'm casting it directly to TS query as opposed to using the two TS query function. You'll see my warning or my notice down at the bottom. It says hey, the null, the empty string isn't developed. It's a warning or a notice. It's not actually an error. So now, it wasn't gonna work. So now let's look at this. I have TS rewrite. Now, remember we just saw in the previous by adding the word data to stop, English dot stop, I changed my TS vector. That stores it in the database. Same or equivalent functionality, I created a table and I tell my two TS query side, hey, look in this table to see what synonym I should use for data. Now the good news is that table actually has a lot of other capabilities. You can put anything on the right side. What it's saying is if the query has the term on the left, substitute the term on the right. In ours we used it for a stop word. That, what we did, little focus on how to get the files in there because you can't look at anything like this without a little bit of data wrangly. You saw it wasn't difficult. That's the piece I wanted to get out there. Maintain your full text search TS vectors three different ways. Create them with your queries. Create them manually with two TS vector or create them with a trigger function when you load them. Hey, you get two trigger functions built in and it's not hard to create a custom one. Compared full text and SQL, and what I wanted to show you, remember that term enumerate when we searched for enumerate and SQL, it doesn't show up. So you get functional differences on the existence and then we showed you can rank your results. Remember in our top five query, the unranked results, the number one answer there didn't show up in the weighted results. So in other words, if you're assuming your user is a squirrel looking at a shiny ball, you got no attention, you want the number one result to be relevant. No work and you got it there. You can also generate really good looking output or decent looking output suitable for presentation output. The title, how it's presented, very simple to customize. You can also add your stop words. In that directory so that shared or T-search data, you also have a thesaurus, a synonym dictionary, so you have a whole lot of power. Those files there, the mechanics are roughly equivalent. You're gonna go into that file and edit those. Go to the presentation afterwards, you'll see to rewrite queries at runtime and suggest spelling options. Go ahead. One, the problem is hard, so there's not gonna be a just do this and it'll solve. If storage space is not a problem, so it really depends on the bounds of your corpus, I would put them all in one place. I would separate the TS vector piece from the document store itself. Absent that, I'd create a view that presents them all as one because what you really have is the same SQL tools or the same Postgres tools you have if you said, hey, I have data in a bunch of different tables and I want it to look like it's in one place. But the killer is gonna be, can you afford the storage to duplicate your record? That's really what drives it. Unfortunately, there isn't a magic piece.  The end of that whole sort of crap is Lucerne or Solar or one of those other pieces because what they say, what they deal with that problem is, take your data from everywhere and give it to them. Does that make any sense? So it's not a great answer, but it's no worse than the external tool answer. We're done? Any other questions? Oh, there's the presentation is there and the data files and everything like that. Go ahead. No, I haven't. So one, no, because those are sort of beyond the scope of what this has. There are limitations, so I'm not arguing that the full text search has the full capability. Sort of what we're seeing is it is a 90% solution for 10% of the work. And applying Occam's razor, we have a lot of people or we run into people who are getting dedicated solutions who like me don't know what, don't know what all the pieces are. But again, those are dedicated tools, but you're right. These are some of the limitations it's inherent in. Yeah? No. Wish there was, but there isn't gonna be embarrassed to say I don't know. I know word proximity isn't there. The TS weighting functions have some of it built into their magic. The documentation says you can create your own. I sort of think that defeats the purpose. If you're gonna get that, I'd buy a tool that already did that for me. Word proximity is not there, in other words. And that's if you explicitly want, I want this word within five words of this other term. Anything else? Go ahead. Pardon? Oh, absolutely, absolutely. And let me give you as an example, we have a help desk file and the title of a ticket is a small document versus the whole body of a ticket. It absolutely, the beauty of it is, use it for a small piece, you get all the functionality and really almost no overhead. If you have a small document, you don't have to store the TS vector, converting it at runtime is roughly as fast as using a like operator. So in other words, you can no extra storage, no extra work. You can convert to full text search for small fragments and it doesn't really break anything. Very useful for that. And the beauty is not much work. Go ahead. Yes. No, there isn't other than clarity for this piece. So yeah, you're absolutely right. The whole point of it was to have an indexed TS vector. I did it in two stages. There's no magic in doing it in two stages other than to show you what it is. Anything else? Rock on, use full text search. Thank you.


In this era, everybody's completely obsessed with productivity techniques, whether it is a method of journaling or it's a new app you've discovered or it's just the way you structure your day. It seems to be all the rage, but I don't think people are focusing on what is truly most essential to really be productive and also fulfilled in your life. And I want to share what I think that is in today's video. What's up guys? Alex Hein, author of the book, Milk the Pigeon, if you've ever wondered what to do with your life or you feel lost and meaningless, check out this book on Amazon because I think it can really help you. Now something that's very important is that my online business building program is reopening in a couple weeks. The first link below this video is for a free 30 minute podcast on how you can design the life you want through the online business that is your dream business. So if building a business or building a YouTube channel is your thing for this year, check out the link below this video and join the wait list there. Now I think most people don't see the results they want in life because we are expecting to be handed this map. This is how you should live, Jimmy. This is what you do from ages seven to 12. And then from 21 to 24, this is your roadmap and your playbook. But then wait because at 25 to 28, this is what you have to do differently. Sorry guys, newsflash, that doesn't work for human lives, right? It doesn't work because it only works if you want to live a conventional status quo life, which is fine. But if you don't want to live that, if you want to live an unconventional life and there is no map, there is no playbook, you have to design the map. You know, I've shared this idea that when most of us don't know what to do in our lives, we do one of two things. The first is that we conform, you know, we want to be a lawyer. So we go to school, we pass the bar exam, then we practice a hundred hours a week and that's just what being a lawyer is.  And then that's human life for us because we're following the status quo. It's okay. And then the other half is that we rebel. If we're like, no, screw the status quo, that bullshit's not for me. I'm going to be the rebel. I'm going to be a hippie backpacker in Thailand. I'm going to get dreadlocks. I'm going to go do ayahuasca every year in Peru. I'm going to go be the rebel. But very rarely in between is there someone who's just like, you know what, elements of both of those lives appeal to me. But I don't really fit into any of these boxes. So what's me? And that can be very confusing because it's like a kind of identity crisis that so many millennials and people in their twenties have where it's natural if you question things. So you have to design your own map for your own life. But what if you don't know what you really want? You know, when I bring this up to people, they very often follow up with, well, what if I'm not passionate about anything? Like, I don't know anything that really excites me. I don't know what I would really do if I won the lottery. And I think this is that kind of perfectionism that comes in thinking that your next decision has to be the decision for the next 20 years, but it doesn't.  I got the best job that I could that was mildly interesting. After a year, I quit it. I bought a one way ticket to China. I came back. I started a business. The business was not my passion. I became a personal trainer. It was not my passion. I then worked on building that business every day. It still was not my passion, but it was something that was the best thing at that moment. Will Smith shares the story about his dad in an interview that I think will be really helpful here. So this interview on the Charlie Rose show, he said, one summer, his dad tore down a brick wall in front of his business and told 12 year old Will and his brother to rebuild it. It took them a year and a half and they actually ended up doing it. And he said, don't you ever tell me that there's something you can't do. You don't try to build a wall. You don't say, I'm going to build the biggest, baddest, greatest wall that's ever been built. You don't start there.  You do that every single day and soon you will have a wall. If you don't know what to do, you just lay the first brick, which is basically whatever seems to be the most exciting and growth oriented thing right now in your life. So if there is no map for your life, you don't know what you're the most excited about. You can pick what you're 10% excited about, move in that direction. That's obviously where the flow is. And then if that changes or when it changes, then move into another direction. Because when the flow changes, you need to pay attention to that and then lay that kind of brick. For me, the first brick was going to China. That was the first brick and listening to my gut. And then the second brick was starting to build a business. The third brick was writing a book. The fourth was the YouTube channel. The fifth was getting my doctor in Chinese medicine. The sixth is what I'm building now. Every brick reveals the next brick, but you'll never know unless you lay that first brick. So if you don't know what to do, just pick that first brick and lay it perfectly. You know, my China might be your welcome to my channel video. My China might be your decision to take that job as an au pair in France or in Spain. In my China, that may be your decision that you're going to take a sabbatical for a couple months and go figure out what your best, most exciting next steps may be. Just lay that first brick perfectly because you are designing your own map for your life. Now again, for me, one of the most important aspects of building my dream life was building a business where I can control how I spend my time. Now I'm opening enrollment again for my course, Alive Business Mastery. The first link below this video is a free 30 minute podcast on how to design your dream life through an online business. So if that's a priority for you this year, you want to build a brand like mine, you want to build an online business, add your name to that list, listen to that podcast, and I'll let you know my business course opens. All right, you guys, that's all I have for today. There's two videos on this topic right here. Check it out.


why you don't why you don't hear from the narcissist there could be a few reasons why you don't hear from them of course the most obvious reason that anyone is going to say is that it's because they have new supply and i'm not going to say that that isn't true yes they may have found someone else they may have found someone to replace you to take on the role of what you were doing father that is a possibility they may have groomed and advanced a new source to the primary position that's a common thing that happens once they get rid of you they need someone else to take your place that's just how it is but that isn't always the case there could be other reasons why you don't hear from the narcissist which i'm going to get into here another reason why you don't hear from them is because you figured them out you're no longer susceptible to their manipulation they can't manipulate you they can't pull you into their realm you're no longer under their spell and when that happens they notice they feel it they sense it they realize that you have escaped their web their control they pick up on that and when that happens it doesn't make any sense for them to hoover you because they're not going to get what they want you're not going to give it to them so why would you hear from them why would they come back to you it wouldn't serve any purpose they would just be wasting their time and energy which could be spent on something else they could use that time and energy to pull someone else into a relationship someone who is foolish and will give them what they want they would make more sense to them to do that once they know that you figured them out once they know that you're not going to see them the same way and so a lot of times that is what they will do they will go on to find new supply other times they may have experienced a narcissistic collapse where they've just given up they don't know what to do they don't know what to think they've just fallen they've given up they've lost motivation to even try to get you back their ego has deflated and they just don't feel good enough for anything so they don't even want to communicate with you anymore in that moment they may just want to be alone or they will play the victim cards in an attempt to gain new and fly monkeys the narcissist isn't going to come back unless they think you're a fool unless they think that you still believe in the illusion that's the only way that they're going to come back because they're not going to put themselves in a position where they could be humiliated where they could be rejected where they're going to experience a narcissistic injury that's not what the narcissist wants they don't want that to happen but yeah this is a reason why you don't hear from them because they've gone down they've experienced they fall and they don't want you to know about it because they're quite embarrassed about it they don't want you to see them when they're down they don't want you to have the last laugh they just want to ride off into the sunset and make you think that everything's just fine and dandy they never needed you you are the one who needed them and that's is why you may not hear from them because they're using their silence to punish you to make you feel like you've done something wrong to make you feel like you've made a mistake and sometimes that may make you try to come back to them you may send them a message you may call them because you're not hearing from them and you don't know what's going on which is exactly what they want you to do that is yet another reason why you don't hear from the narcissist because they're full of ego and pride they feel like they're so important and they don't want to come down off their high horse and just be real so instead they sit back and expect you to do the work to get in contact with them or for you to try to figure out what's going on that's what it's designed to do a lot of times it's a manipulation tool you can almost guarantee that nine times out of ten their silence is designed to punish you it is designed to hurt you a lot of times that's what it's designed to do pretty much everything a narcissist does is manipulation nine times out of ten that's what it is but don't forfeit i'm giving you this information now so that you don't make that mistake there is a reason why you don't hear from them and it's designed to get you in some way just don't let it get you don't pay too much attention to it just focus on your life find things that are more important to you because honestly if you don't hear from the narcissist that should actually be a blessing to you now you have some peace now you can just sit back and relax and enjoy your life without worrying about these type of things so just enjoy it enjoy the silence this time of not hearing from the narcissist because you never know when they're going to come back and finish the job when they're going to come back and try and bring you down again you never know when that's going to happen but it's no point over thinking about it and stressing stressing yourself out that's not going to help anything just enjoy this time of peace that you have now make the most of it maybe it's here to stay maybe it's not going anywhere but just in case savour it enjoy it because you never know where the narcissist what's around the corner that's all i've got to say for this video i do appreciate you sharing your day with me i appreciate all of you your support means so much to me and the many survivors around the world we couldn't do this without each other out here sharing this message bringing inspiration to millions of people around the world that's what we do and we're going to keep doing it no matter what so yeah if you would like to support this channel please give this video a thumbs up share your thoughts in the comment section i read your comments every day every morning i go through hundreds of comments in here every comment is valued and appreciated and it helps other people on you as well and also click that subscribe button if you didn't subscribe yet then you will receive notifications for my future videos and please share this video with anyone who you believe it may help put this message out there so other people can see it if it's helped you it might help someone else so please take the time to do that it only takes a minute and if you are interested in one-on-one coaching with me i do conduct one-on-one sessions you can book a session with me from my website it's Narc Survivor at the Cody UK or for anything else you can send me an email at coaching at Narc Survivor at the Cody UK okay that's it i'm just out here enjoying the sunshine i hope you all have a great day and as always we will have the two video premieres later today so i hope to see you all there for that other than that you all have a great day and i'll talk to you soon


Hello and welcome to This Week in the City. Highlights brought to you from the Public Relations, Media and Marketing Department. I'm your host, Alicia Utsi. This week in the City, we're calling on citizens to get involved and volunteer to help with the bicycle and pedestrian accounts that are coming up later this month in September. Volunteers are needed to assist with the account process. For more details, visit our website at columbiasc.gov. Also this month, we're encouraging citizens to sign up or participate in various city events. Please register for the Mayor's Walk Against Domestic Violence. The walk will take place on October 5th. However, registration is open now and it is free of charge. Also during the month of September, go ahead and buy your tickets early for the Mayor's Annual Breast Cancer Awareness Breakfast. Funds that are raised will assist victims and survivors of breast cancer. The way that you buy a ticket is visit our website at columbiasc.gov for more details. Again, tickets can be purchased in advance. The breakfast will be held on October 11th. As we roll into the weekend, the City of Columbia will like to invite you to some community events. First up is the Grand Opening Ceremony for the All Stars Baseball Field located near Lester Drive. The ceremony will be held on Saturday, September 7th starting at 11am. For directions to the field, please visit our website at columbiasc.gov. Also on Saturday, the Eau Claire Unity Festival will be held in the Eau Claire community. Again for details, visit our website at columbiasc.gov. Well, it sounds like it's going to be a great week in the city and it's also going to be a busy week. For more details regarding any of these events, visit our website at columbiasc.gov. Follow us on social media on Facebook, Twitter and Instagram. And you can always call the Public Relations, Media and Marketing Department at 545-3020.


Hi and welcome everyone. Good morning to for those of you in the US and good afternoon and evening to the rest of you. Today I'm going to talk about multi-tenant cluster solutions using source service and on this talk I will share our experience at velocity of going through the journey of setting up a multi-tenant cluster. So it was a quite a long learning process and I hope I will manage to provide you with some guidelines and tips on where to start looking. So let's begin. A little bit about myself. My name is Errol Maman. I've been a software engineer for over 10 years. My main areas of passion in recent years were around software architecture, distributed systems, improving reliability and developer experience. These kind of interests were what brought me to velocity where we aim to simplify and make development environments accessible to developers. Also a photo to a three-year-old and a country music addict. I'm located in Israel and feel free to follow me on Twitter. So the first thing that we may think when we hear multi-tenancy at least the first thing that I thought is that it's complex and it may be a bit scary. So what I want to emphasize in this talk is that multi-tenancy can be easier than you may think and that there are tools and methods that can help you achieve that without it being too complex and painful. And before we talk about solutions, let's talk about use cases. Why would you need multi-tenancy? So the first group of use cases is about ways to empower teams in your organization. By talking about teams, we actually are talking about internal multi-tenancy which means that the tenants are located inside of your organization. So one example of such use case is creating affordable environments. So these environments are environments which are created in a temporary way for a specific purpose and limited period of time. So developers in organizations want development and testing environments and they are asking for it in order to help with their development and testing processes. And these developers may want to like test or showcase their features kind of in a rapid way without being blocked by a CI CD pipeline or in other blockers. Another kind of use case is that a developer may want to debug a tough or rare bug in a specific environment. For example, if someone has maybe a resource or a memory leakage bug that can take a while to reproduce. So another kind of use case related to internal multi-tenancy is end-to-end. And in end-to-end, you may have different teams in your company and each team owns a set of services and each set of services communicates with other services of other teams. And together, it forms a complete system. Since each team owns their services, then you prefer the one step on each other's toes and perhaps cause harm to other teams. That's why it may be beneficial to see each team as a tenant scope to their own space. And when dealing with teams, it's beneficial to understand the balance of granting autonomy. So developers obviously want to move fast. Developers, their KPIs and what they're measured upon is mainly focused around delivering features and fixing bugs. And they want to do it fast as possible. And platform teams, on the other hand, will probably advocate and push for reliability and maintainability. So let's really feel free to move fast, but be careful not to break stuff while you're doing it. And platform teams often find themselves as the guardrail for reliability and maintainability. But on the other hand, letting the platform teams exclusively deal with the toil of managing environments while the developers wait for it really beats the purpose of autonomy. So by enabling self-service for developers and other stake orders, while still managing to keep the guardrails in place, it has the potential to keep the balance healthy. So if we jump into another category of multi-tenancy, and that is third-party multi-tenancy. In third-party multi-tenancy, we mainly deal with kind of external multi-tenancy. That can be your customers or maybe other third-parties that you wish to run workloads on your side. So a popular example, if you develop a SaaS product and you want to run workloads per customer. For example, maybe you run the code that the customer provides you. Or another example is that you can run your entire backend app per each customer. In this scenario, since you run the workloads per each tenant, you probably have isolation or regulatory requirements, or maybe even specific performance requirements and SLOs so that no tenant can somehow badly affect other tenants. In this case, usually your application has direct access to the cluster while your customers don't. So another kind of use case is Kubernetes as a service. In this use case, you manage a cluster on behalf of your customers and you want to allow these customer applications to run their own workloads on it by giving them direct access to the cluster. So I believe this one is a kind of a niche use case. However, we at Velocity had this use case and I'm going to tell you more about it later on. But the SaaS use case is probably the most popular use case around. So if we attempt to roughly categorize multi-tenancy into two sections, that will be soft multi-tenancy and hard multi-tenancy. And on soft multi-tenancy, you have a single cluster where tenants are being shared on the same cluster. And on the hard multi-tenancy, you create a cluster per tenant. So even though we have this like soft and hard definitions, these are essentially the edges of the spectrum. And there are hybrid solutions in between that you can implement. So as we've said, choosing a solution is really a matter of a use case. So I'm going to try to compare these kind of two edges, the soft and hard multi-tenancy in a high level based on four categories. So the first category is cost and how much will it cost me? How much the solution will cost me per tenant? And this can be very important once you have a lot of tenants. Second category is speed. How fast is creating or destroying a tenant? Isolation. How is it for a tenant to affect other tenants? And maintenance. How difficult it is to maintain the solution? Each of these categories is just kind of subjective and use case dependent. You may care less about one and more about the other. So let's try and compare these kind of edges. So the scale would be here that the red is kind of the worst kind and green is the best and yellow somewhere in the middle. This is kind of an opinionated comparison. Just bear that in mind. So in terms of costs, in hard multi-tenancy, you have a cluster per tenant. And the cluster per tenant is expensive because you pay for an entire cluster, which means control plane nodes and supporting infrastructure. Also, you can really share resources between clusters like node pools. This may be negligible when you have like a few consistent tenants. But once you grow in scale, this can be a serious problem. In a soft multi-tenancy, you make use of an existing cluster and you basically paid for the added workloads. But you don't pay for additional control plane. And you have the ability to share node pools and supporting infrastructure. So in terms of speed, in hard multi-tenancy, creating and destroying clusters takes a lot of time. It takes time in the scope of minutes, sometimes even more than half an hour.  On soft multi-tenancy, creating and destroying tenants inside a shared cluster can be a matter of seconds. It really depends on how you choose to implement it. But in many cases, creating a tenant would be just creating some Kubernetes objects for that tenant. And that should take a couple of seconds. So if you have like workflows or APIs that depend on the cluster being created or destroyed, this could be a serious consideration for you. In terms of isolation, in hard multi-tenancy, since these clusters have a separate control plane and potentially even reside in a separate cloud provider or a cloud-wide project or account, they have a high level of isolation like out of the box. In soft multi-tenancy, you get a lower level of isolation out of the box. But you can reach high isolation. It requires some efforts from you. If we go to the simplest level of isolation in a shared cluster, which is a namespace per tenant, you get a quite low level of isolation. If you want to increase the isolation, you have to invest effort and maintain various restrictions and controls in the Kubernetes level. Most likely, you would want to do that and we'll talk later about ways to do it. And in terms of maintenance, in hard multi-tenancy, managing many clusters is just hard. It can lead to a situation often described as a cluster sprawl. So you have to take care of the lifecycle of the cluster. You have to upgrade the version of the cluster. You have to monitor each cluster. And as said, like if you're confident it's in the ballpark of up to like a few consistent clusters, this may be inoculable for you. In soft multi-tenancy, in this case, in terms of infrastructure, you have like a single cluster to take care of. But you still have to manage different tenants' configuration and restrictions. Later on, we'll talk about open source projects that can help you make this a lot easier. In terms of maintenance, in both cases, you can actually reduce operational toll on your platform team by taking the advantage of self-service. So after we're exposing use cases, let's talk about how to do it. I will not going to focus today on hard multi-tenancy mainly for two reasons. First reason is it wasn't a good fit for our use case. So we didn't explore the solutions around it very much. And the second reason is I don't, I think there isn't currently in like an easy way to do it. Hard one-tenancy is just hard. If you do consider it, do take a look at the cluster API project. It's an interesting one. So let's talk about how to do soft multi-tenancy in Kubernetes. So the first thing that comes to mind when you want to separate resources in Kubernetes is using namespaces. If we go to the Kubernetes documentation, the definition really says it all. So in Kubernetes, namespaces provides a mechanism for isolating groups of resources within a single cluster. But we can quickly realize that namespace per tenant doesn't really mean isolation. And we can really get to this understanding by asking ourselves the following questions. What prevents a tenant from creating or deleting workloads in other namespaces? So by default, a tenant is not blocked for doing that. So by default, the cluster users allows you to access and modify all namespaces, including other tenants. What prevents a tenant pod from communicating with pods in other namespaces? So Kubernetes does not limit pods to talk with pods in other namespaces. And the DNS part in Kubernetes can actually resolve other namespaces. So your pod can actually communicate with other namespaces. And what prevents a tenant pod from abusing the entire cluster, starving other tenants? So as a tenant, you can bring up a workload which is not resource restricted. And this workload can eat up all of the CPU and memory in the node, starving other tenants workloads and affecting them. So after we get to understanding that namespaces doesn't mean isolation, we have to think about how can we isolate namespaces? So Kubernetes offers us a few ways to do it natively. Allow you to restrict actually on a namespace basis. So let's go each one of them in a nutshell. First, you have RBAC, Resource-Based Access Control. And RBAC allows you to restrict which users or groups can access what resources. For example, you can define that tenant A can only access its own namespace and not other namespaces. So anyone who ever worked with RBAC knows that RBAC can be very complex to manage and to understand actually. Next up is network policies. Network policies allows you to restrict what kind of network traffic can go inside and outside of your pods. So for example, you may want to restrict a tenant namespaces from talking to other tenant namespaces or from directly communicating with the Kubernetes API. Next up is pod security admission. It allows you to prevent workloads from having spatial capabilities security-wise. For example, a pod can ask as part of its manifest to be run as root. And then you can use it, for example, to spawn a reverse shell and then compromise the Kubernetes worker node. So pod security admissions allows you to prevent that by enforcement. And next up is limit ranges. Limit ranges allow you to set up a range or default value for memory on CPU resources for a pod or container. For example, if you want to prevent overcommit of memory or prevent CPU starvation, and these kind of situations can really affect workloads in other namespaces and affect other tenants. Next up is resource quotas. So resource quotas allows you to set a limit for how much CPU on memory resources I can consume on a namespace level. And last one here is node selectors. So if you want to reach a higher level of isolation, you may want, for example, to run workloads of different namespaces in different nodes. So if you look at all of these and there are a bunch of other ways to restrict and isolate namespaces, think about having to apply these for every namespace and having to manage different kinds of configurations. Seems very complex, right? So that was about isolating namespaces, not tenants. Can we really assume a tenant means a namespace? So what if a tenant needs a few namespaces? So at that point, restrictions really becomes complex, because as we saw just before, restrictions are namespace based. So you need to figure out how to apply these for a tenant using a bunch of namespaces. And visibility is kind of hard, because there's no single place when you can see which tenant owns which namespaces. You need to do the digging or build something on your own. Okay, so letting the platform team do all of this setup work and restrictions will essentially bottleneck developers waiting for ITT kits. And also it's kind of a boring kind of toil. We want to aim for developers and other stakeholders to act autonomously within their tenants. So by shifting left to developers, we should aim for self-serve alongside proper automation. We will start by talking about open source project that aim to automate the namespace per tenant approach. The first one is Kiosk. It's an open source project by loft. Seems like it's not entirely currently maintained by looking at their Github, so I'm not going to elaborate on it. Second one is Capso. It's an open source project by Classics. Seems to tackle similar approaches as Kiosk includes more features, and I think it's a bit more flexible. So let's start talking about Capso. And Capso really extends Kubernetes and allows you to, for implementing multi-talented environments, and it actually has a few interesting things that it allows you to enable. First, Capso will introduce you with a kind of a new entity in your Kubernetes ecosystem called a tenant. And you focus your effort on this entity called a tenant and not on namespaces, which is, if you think of it, it's really closer to reality, right? And you can actually, you can configure restrictions on a tenant. So maybe if you want to have like a different kind of tenants, like if you have a free tier and a premium tier, so you want to have a free tenant and a premium tenant, and each of them may have different restrictions, and you can configure that on the tenant object. It also allows you to, for having a tenant, having a few namespaces. So as we follow before, restrictions are namespace-based, so they need to be created by a namespace. And Capso really automates this for you, so you don't have to think about namespaces, just have to think about tenants and self-serve. So you can really extract the platform team to be the main bottleneck of this multi-tenant show. We need to provide autonomy for other individuals to manage their tenant and manage their workloads without the ability to cause harm to other tenants or to the cluster. So to understand how Capso aims to enable these things, let's explore the way it works. So Capso, the main thing about Capso is it defines a new CRD of a kind tenant. You can look at this quite simple example. You can give this tenant a name, and you define who is the owner of this tenant. So the cluster admin persona, which let's say someone who is working at a platform team, it defines a tenant of this kind of CRD. And it also defines who are the users, who are the tenant owners. And from this point, after the cluster admin applied, this tenant starts the self, now begins basically the self-service part. The tenant owner, which is defined on the tenant, can work on this tenant without involving the cluster admin. They can create namespaces, and they can create other workloads on their own. So all of the resources that they create will be actually tied to this tenant. So if you run, for example, kubectl described on this tenant, it will show you all of the namespaces that are tied to this tenant. And tenant owners are really scoped to the tenant limitations. For example, if a tenant is trying to access other tenant resources, or maybe system resources, they will get an access denied error. So as we've seen in Capsule, you can actually define, in top of the existing configuration, you define restrictions on the tenant object. And Capsule will automatically apply that to all of the namespaces of this tenant. So all of the restrictions that we saw before, the resource quotas, the node selectors, the network policies, et cetera, all of these can be defined on the tenant object. And after we've seen that, you can ask yourself, what if namespaces aren't enough for us? Like what if we have a tenant or a bunch of tenants that wants to have cluster level accents? So just for a few examples, if you have a tenant that wants to install a cluster scope controller, maybe a CRD, maybe wants to install an help chart that requires admin roles, or for any other good reasons, a tenant wants complete control plane isolation and the ability to apply things on a cluster level. So just like an example, if the tenants are actually developers, for example, in the company that develops controllers, for example, this kind of use case may be very important for you. So to tackle this kind of use cases, there comes virtual cluster, which is a unique approach. And it really means control plane per tenant. So virtual cluster, it's an interesting approach where the tenants actually share the same cluster, but each tenant is running inside like a sandbox virtual cluster. It's like an embedded Kubernetes inside a Kubernetes. And if you look here at the diagram, then each V cluster actually has a separate control plane. So that includes the Kubernetes API, that includes DNS, Datastore, which is usually at CD, controller manager, and optionally a scheduler. So every tenant that is working with a V cluster actually has a notion that is running inside a separate cluster. So by default, a tenant won't be able to access namespaces and resources of other tenants. So what we get with virtual cluster, at least of the V cluster implementation of virtual cluster by loft, we get kind of out of the box isolation, we get control plane isolation, which means the API, Kubernetes API and the data storage are isolated from each other. So it allows you as a tenant to install a cluster scope resource such as CRD and webbooks. Also, it allows namespaces names to be reused across different tenants. You also get a DNS isolation, which means a workload cannot access workloads in other V clusters or in the host cluster via DNS. However, you don't get a complete isolation out of the box. So if you have a privileged pod inside of a V cluster, you can still escalate to the host node. There are no default resource restrictions. So if you have a pod in a V cluster that use a lot of CPU, a lot of memory, you can cause starvation in the node for other tenants. And there's no complete network isolation. So we've seen that the DNS isolation exists. However, you can still access, for example, by IP addresses, other workloads in other V clusters. So V cluster by loft actually offers you, there's an alpha feature for enabling stronger isolation, or you can actually do it yourself by applying restrictions to the V cluster namespace. So if we try to compare like capsule and V cluster, it's a bit like apples and oranges. But I think like if you put these into these categories that we saw before, it can help you get a better understanding of what's best for you to look at. So in terms of costs, if we look at capsule, a tenant really still resides in a bunch of Kubernetes namespaces alongside additional restrictions. And from that perspective, the costs are pretty much like soft multi-tenancy, which means they are low. In V cluster, there is an additional workload of running the control plane per tenant. And this can translate to a bit more cost. You can choose the control plane actually to be a minified layer of Kubernetes, and that may have lower costs. In terms of speed, in capsule, when you create and destroy a tenant, it is a very quick, like a matter of few seconds. So if you think of it as just an automation to a namespace and some Kubernetes restrictions, which most of them are pretty rapid to create. In V cluster, for each tenant you create, new pods are launched with the control plane. So it's much quicker than hard multi-tenancy because you already have no pools and you have a cluster, but it's a bit slower than capsule. For my test, it usually took more than one minute. In terms of isolation, in capsule, the isolation really depends entirely on how much you invest in configuration. So you can use all of the Kubernetes restrictions in your disposal and achieve a rather high level of isolation. But in the end, different tenants still share the same Kubernetes API and DNS. In V cluster, you get some restriction built out of the box, like we just saw. But if you want to reach a high level of isolation, you still need to invest in restrictions. However, because you get these separate control planes like Kubernetes API and DNS per tenant, in my opinion, this puts V cluster higher in terms of isolation. So there are some use cases that we saw that requires cluster scope resources like installing a controller and CRDs. In this case, I think V cluster clearly wins, as these will be isolated in each V cluster. Internal maintenance here, it kind of depends. So in both cases, you have additional moving parts. And when you had moving parts, you add a bit of complexity. And that other complexity means farther maintenance. In my opinion, the cluster may be harder, a bit harder to maintain than capsule, because adding control planes seems more brittle than automating Kubernetes existing mechanisms. But I also think like both cases are easier to maintain, like then do it yourself hard or soft multi-tenancy. So to sum it up, it's a trade-off as everything. If you need the extra control plane isolation, and you're okay with a bit of an added cost and a bit lower speed, start looking at the cluster. Otherwise, capsule may be a simpler choice for you to explore. So I will now share a story of how we implemented multi-tenancy in Velocity as part of our Freemium solution. So a little bit about Velocity. Velocity allows developers to spin on-demand self-serve environments.  You can develop and debug locally while the rest of your mind is still in the cloud. You can work directly on your IDE or your existing toolbox. You can configure your environments to be destroyed after a period of time. And you can actually use real cloud resources instead of using mocks. So when you think about motivation, so we had a kind of interesting use case of Velocity, we wanted non-paying users to be able to give our product a try. And we kind of aimed for these Freemium users to have like minimal effort, like we wanted the cluster to already exist with our application installed. And we wanted this experience to be very quick, like when they register, they get a working cluster in a matter of seconds. So first we kind of gathered some requirements. So first one is that first by the apps will communicate with our Kubernetes cluster. So as we saw in the beginning of the talk, we are here at the Kubernetes as a service use case. One does support a large number of users. So we wanted to open this to a large audience like in the hundreds. So we wanted to get like a ballpark estimation and limitation of how much will it cost us. So we are a startup and we don't want this to be a major spend for us. Also, since it's a free offering, we can't allocate some of the costs on behalf of the customers. We want to have the ability to have to upgrade these three users to a premium kind of tier because the free tier is actually quite limited in terms of resources and features. And if a customer wants to upgrade and pay to lower restrictions, we should kind of allow that easily without having these customers start off scratch. Observability is important like knowing which tenants owns what resources. And this is mainly for our internal troubleshooting purposes and internal visibility. So the isolation should be pretty high. We don't expect these premium customers to run their production system there. So we don't seek like kind of extreme isolation. But since we do care about security and privacy, we wanted a rather high level of isolation. And lastly, we didn't want this to be a big overt for us because we're a startup and we don't have large teams dedicated to maintaining this platform. We also need to move fast as we explore our market. And we can invest a lot of time in implementing and bringing this up. So when we took our first steps, we quickly understood that how multimodal tendency is not a way to go. It was just not a feasible solution for a free user scenario because of the cost and speed and the need to scale to a lot of users. So we started off by exploring VCluster. It seemed like a very good trade-off between hard and soft multi-tenancy. And we definitely needed a high level of isolation. So we experienced some technical limitations because our app, and this kind of specific to our use case, because our app relied on, for example, dynamically injected Kubernetes resources. So we have a cluster component that copies secrets into namespaces. And we want to do that on behalf of the tenant. So since each tenant is encapsulated with a virtual cluster, we needed these components to connect to these clusters. Not impossible, but it required significant work for us and complexity.  So the VCluster DNS per each tenant won't allow you to access services located on the host cluster. There were a few more general limitations. It seems like a lot of moving parts for us because it's a Kubernetes control plan per each tenant, which seems like kind of like we have to manage these control plans in some extent, which is a bit added complexity for us. So also, it wasn't quick enough for us. It mostly took over one minute. So for many use cases, that is totally fine. But for our use case, we actually wanted to expect this to be a bit more rapid to, in order not to affect user experience. And we had to implement additional isolation. So we didn't get all the isolation built in out of the box. We still had to invest some effort in configuration. So next thing, we started exploring Capsule. And if we actually, first thing, when we read Capsule documentation, it seems mainly aimed at like sharing cluster internally in your organization for internal employees or people in the organization. But our use case was a bit different, like we wanted to share the cluster externally. So we actually surprisingly, we found it as a good fit for sharing the cluster externally. And then it will be used by apps rather than people. So there are a few things that we initially like about Capsule that it's mainly automating existing Kubernetes controls, which makes it a bit more simple in terms of logic. And it makes our troubleshooting a bit more less complex than the control plane approach. So it's really fast, like if we create a tenant, it usually takes us like a few seconds, almost as fast as just creating a namespace. And it meets the requirement of the multi-tenancy benchmark, which is a cool benchmark made by Kubernetes 6. It includes like a list of tests that validates if your cluster is really ready for multi-tenancy. So we wanted the tenant lifecycle to be automated and self-serve. We couldn't rely on a human in the loop creating and destroying these tenants. So we wrote a health chart for setting up the tenants and the access to the cluster. This health chart includes two things. First is a service account, which is a way for our app to access the cluster as the tenant owner. And another thing is a tenant CRD, which defines this tenant. So Helm really gave us the power to define the template. So all of our tenants will share the same definition, but there will be some variance in the restriction because of the templates. Also Helm gave us the ability to use versioning, which may help us in the future with upgrades. So this is like an example from our usage of the tenant CRD. So we define the tenant and the tenant owner is a service account and not a user. And you can see here that we actually defined resource quotas on this tenant. So we limit the CPU and memory per the entire tenant. So if you remember a bit while back, the resource quotas are usually on the namespace level. And this is on a tenant level, which makes a lot of sense when you're dealing with tenants. So this is really cool in my opinion because me as a cluster admin, I don't have to think really in terms of namespaces. I can really deal with tenants. And this is also where the self-service part really comes clear, as from like this point, when you create a tenant, Capsule will automatically impose the Kubernetes in limitations when the tenants create its namespaces. So here's like a general explanation of our application flow. So on the left, we have a flow of creating new tenants. So when the user registers in our system, we have a flow that creates a new tenant, which means we install this tenant help chart, which as we saw, installs a tenant CRD and a service account. And using this service account, the external app, which is now a tenant owner, can access the cluster as a tenant. And on the right, we have the opposite flow where we destroy the tenant and then we uninstall the help chart and revoking from this point the access up for the app. This is kind of another view of this flow. So when the app authenticates towards our back end, it authenticates as a tenant, let's say tenant A, and then the back end kind of creates a Kubernetes token that impersonates the Kubernetes service account that we created in the help chart. And this service account owns tenant A. So after it creates this token, it returns it back to the application. And then when the application has the token, it can access the cluster directly with the token and it create and can create namespaces which will be owned by this tenant by tenant A. So for example, if it created the namespace and then it tried to access a namespace that doesn't belong to its tenant, like tenant B, it will get permission denied error. So after we've done this, we came to the confusion that we really had a very good experience with Capsule and it made us realize that multi-tenancy doesn't have to be that difficult. Just to be clear, we did have some specific issues with Capsule which I think are mainly related to our use case. So as I said, there is no control plane isolation. So you can really create cluster scope resources with Capsule. And we had a few occasions where we modified the tenant CRD and had a syntax error, but we didn't get a clear error for that or any failure. And we only discovered that later, either when the tenant managed to run some commands or when we looked at the controller logs and then we saw the error. So it was a bit too late to get this error. And sometimes when you're trying to delete a tenant, it was stuck because this tenant has some hanging resources. So like if you bring up a resource which is stuck being deleted, that prevents deleting the namespaces and the tenant. But this is generally like a Kubernetes thing, not really related to Capsule. But in vCluster, for example, you could have just delete the entire vCluster and get that with it. It's not always a good practice, but it makes the experience a bit more like a thermal. So to conclude, we did put together a bunch of tips from our experience and from our journey. First, isolation isn't really all about security like you may imagine. It's also about reliability. For example, if you didn't really set proper resource quotas and range limits, then you may cause problems for other tenants because you may starve them. You should really consider your use cases and tradeoffs. Like it's, don't really head straight to the hard solution because sometimes when you think about security, we always want to implement the most secure solution. But if we look at hardwood detalency, like maintaining many clusters is kind of a nightmare. And personally, I would try to avoid it as possible. And especially if you know, like if your cluster will serve internal employees in your company, then you probably don't need the strongest isolation. You should really use the kubectl MTB benchmark tool. It's a very cool tool. It will give you a validation if your cluster is properly configured just in a few seconds. So worth giving you the try. So you should always really assume that someone will be able to abuse your cluster to some extent. And you should really apply some limits. We personally use the client provider budget alerts to make sure that our costs aren't really going way beyond what we expected. If you expose a cluster outside, you should probably consider pentesting. So many of us, you know, don't deal with security on a daily basis. So sometimes an experienced pentester can really benchmark your isolation level. And if you want to reach a really high level of isolation, do have a look at the NSA Kubernetes Hardling Guide. It's really cool dock with good references for higher isolation. So that's it. I really wanted to thank you for your time. And I really hope you gain some insights about multi-tenancy that will help you start off your journey. We do plan to write some blog posts about enforcing security policies on multi-tenant clusters in our blog at Velocity. So do give us a visit. Link is right here on the slides. And I really want to thank you all and have a good one. Thank you so much, Jarell, for your time today. And thank you everyone for joining us. As a reminder, this recording will be on the Linux Foundation YouTube page later today. We hope you join us for future webinars and have a wonderful day.