The Craft Foods Company presents The Great Gilder Sleeve. It's The Great Gilder Sleeve starring Harold Perry, brought to you by The Craft Foods Company, makers of parquet margarine and a complete line of famous quality food products. Mushing home from work through the sloppy streets of Summerfield, Gilder Sleeve comes to a street corner where he finds his neighbor, Mrs. Bullard, hesitating before a large puddle of slush. At once, the gentleman comes out in him. Ah, Mrs. Bullard! Oh, Mr. Gilder Sleeve! Better let me help you across there. Allow me. Oh, thank you so much. Well, but don't get your feet wet. Don't worry. Just give me your hand. But you're standing right in it. Don't worry. I've got rubbers. Got a good grip?  When I say jump, you... Hold on. Hold on. Have you got your balance? Yes, I'm safe. Well, I think perhaps we'd better go down to the next corner and cross there. Yes, I guess we'd better. Pretty sloppy going, isn't it? Lucky thing I came along. By the way, how's Mr. Bullard? Haven't seen him lately. Oh, Rumson is fine. Oh, that's good. And little Craig? Well, we're having a little trouble with Craig. Oh? It seems Miss Cuthbert is starting up her dancing classes again, and I want Craig to go. I think it's time he started, don't you? Oh, very important for a boy, yes. I sent Leroy a couple of years ago. That's just the trouble. You know how Craig worships Leroy and imitates him. Well, it seems Leroy has informed Craig that dancing school is a lot of nonsense. Oh, he has, has he? Craig flatly refuses to go. I can't do a thing with him. Why, George, Leroy will go over there and apologize. Oh, that's not at all necessary. Please don't blame Leroy. I just thought if you could persuade Leroy to tell Craig the dancing school is not really so bad. Mrs. Bullard, you leave that to me.  I'll take care of Leroy. Leroy, what's this Mrs. Bullard tells me about you? What, Mrs. Bullard? You told Craig that dancing school was a lot of bologna. Didn't you? Didn't you? Well, I... You knew perfectly well that his mother was trying to get him to go. Didn't you? You know nothing about it. You went to two lessons two years ago. There's plenty you could have learned from Miss Cuthbert, young man, and don't you forget it. The purpose of dancing school is not to teach you dancing, it's to teach you manners. Teach you grace and poise. How to walk across a room without falling over the furniture. That's Craig at the door. When he comes in here, I want you to tell him it's important to go to dancing school. I want you to sell it to him. But what do I tell him? I don't care what you tell him, just get him to go. That's all, now quiet. Yes, indeed. Answer him, Leroy. Bad Leroy, answer him.   You said it's... Leave everything you hear, Craig. Dancing school is a lot of fun. Isn't it, Leroy? Well, go ahead, you have more to say than that. School, Craig, the thing about dancing school is your things. What? It teaches you. Oh, Craig. I just told you, it teaches you poise, among other things. Yeah, it teaches you poise. What's that? I've forgotten. You never knew. Leroy, you're not trying. Now, if you don't get busy... Okay, okay. No kidding, Craig.  It's super. No kidding. Why did you tell me it's... Yes, he was just kidding, Craig. If you go to dancing school, you have a whole lot of farting, Craigie. Here you are. Fair question, Leroy. I think that's a very nice idea, Craig. I think it would do Leroy good. I know you don't. We'll discuss that later, young man. I won't go. All right, just for that, you'll go. Yes, Leroy? What did I say last night? About me going to dancing school? Of course I meant it. Oh, for corn's sake, all on a color, Craig. After I already went to... That will do, Leroy. The argument is closed. I wish to hear no more about it. When Miss Cuthbert's class opens, you'll be there. Oh. The news is good news. Yes, sir. Was there something, Bertie? Mr.  I see no reason why not. Why do you ask? Well, I was just thinking. Of course it ain't none of my business, but when a boy gets himself all worked up like that, well, it's like the man says you can lead a horse to water, but you can't make him dance. Uh-huh.  Maybe later on when he's older...  Oh, no, sir. Bertie... Well, not exactly, he didn't. Send Leroy downstairs to me. He ain't upstairs, Mr. Guilty. He's out in the kitchen. Oh, hiding in the kitchen, eh? Send him to me. Mr. Guilty, you ain't gonna be too hard on him. Seems like... I'll do what I'll do. Send him in here. Yeah. Want arguing about this? I say something settled, it's settled. We've been through this a dozen times. All this fuss. Leroy, what's all this fuss about? I don't want to hear any more about it. Will you ask me? Never mind.  Oh, please, huh? Now see here, we've been... I can't do it. Leroy, now calm down. There's nothing so terrible about dancing class. What's so terrible about it? There's nothing so terrible about girls? That's simple enough, my boy. Besides... Well, just remember, they're probably... Besides, they don't like me. Who doesn't? My boy, this whole thing is just something you've built up in your own mind. No, it isn't. You've learned to get along with girls. That's very important. Hello, auntie. Hello, Marjorie. That's one reason why I'm sending you to dancing class, Leroy, so you'll learn to get along with girls. How do you think I learned? Learn what? I was trying to explain to Leroy, my dear, that I think it's time he learned to get along with girls. Don't you agree? I'll say. The trouble with girls are all like Marge. That's not true. I mean, what if they are? Now, just a minute. What if it all happens, Mr. Leroy? Never mind, my dear. Your sister gets along with boys very well, Leroy. What about other boys? I've never had any complaint on that score. She gets along with them too well. What's this, my dear? A culture on your sweater.  Marjorie. It makes such a great big fuss about all girls are wearing pins. This pin belongs to a boy, you say? What boy? Jelly watch. What difference does it make who it belongs to? Marjorie, what's the meaning of this? It doesn't mean a thing, for heaven's sake. It means they're in love. I'll be quiet. Well? Going together, that's all. Return the pin in the morning. But, Anki, oh, it's... Return it to its owner, whoever he is. I thought I'd brought you up well enough, my dear, to know that a lady never accepts jewelry from a man until after they're married, or at least engaged. Uncle Mort, that may have been so in your day, but if you know the first thing about... My day?  I swear, that's all you ever think about. Well, that's a fine word. I will not have you carrying on with boys.  I will not have it. No, I... Pay the bills around here, that's all. I give up. More about the great gilded sleeve in a few moments. Most women are experienced shoppers of long standing, but grocers tell us that more and more men are doing the family's food shopping now, especially on weekends. Yes, we're quite a choosy lot, Mr. Lang. Don't think we grab just anything we see. In our neighborhood store, the men shoppers often argue plenty about the merits of one food product over another. Well, quick. What have you been saying about parquet margarine? Not much room for argument there, Mr. Lang. We think the flavor of parquet margarine is pretty much all right. Given a choice, I'll pick parquet every time. It's our family's favorite spread for bread. Saves a pretty penny on the food budget as well. Yes, men and women everywhere are talking about parquet margarine's quality too. Today you'll find parquet the same high quality as always. Fresh and country sweet in flavor. Rich and good nourishment. So next time you shop, be sure to look first for this spread that tastes so good. Look first for parquet, P-A-R-K-A-Y. Parquet margarine, made by Kraft. Now let's get back to the great Gildersleeve, who has set aside the problems of his own niece and nephew to address the Parent Teachers Association on theirs. Evidently, his remarks have been well received. Thank you. How was I Eve? Did I do all right? Oh yes, I thought you were fine. I didn't have an idea in the world till I got up on my feet that it just seemed to pour out of me. Yeah. I never stopped once I got started. I think a little preparation might have helped you stay closer to the assigned subject. But I'm really very grateful. We couldn't get anyone else. Well, I didn't know you tried everybody in town. Oh, it wasn't that. Anyhow, the ladies were all crazy about you. Yeah, they seemed to enjoy the little light touches I threw in here and there. Oh, yeah. Say, can I drive you home? Oh, I can't go home yet, Throckmorton. I have some work to finish in my office. Just look at all those teachers' reports. Hmm, end of the term, you know. Well, can't you just take this stuff home? It's much easier just to do it in the office. Well, I had something I wanted to talk to you about, Eve. A problem. Well, suppose you come up to the office then and we can talk there for a minute. All right. You sure you can carry all that stuff? Oh, yeah. What's your problem now, Throckmorton? Is Leroy battling you again? How'd you guess, Eve? That's only part of it. It's really very complicated, Eve. I can't believe that. Go from the door my hands are full. Oh, sure. If it's unlocked. I left it unlocked. There. Now, what's bothering you? You are? Throckmorton. Well, gosh, we're all alone. I don't see what that has to do with it. Eve, what does a fella have to do to give you an apple? Please, Throckmorton, I've got such a lot of work to do. You just made up all that about having a problem in order to... Made it up? Eve, I tell you, I've got a real problem. Not only with Leroy, but with Marjorie too. What is it? You sound as if you don't believe me. I'll let you know after I hear it. You sure make it tough. Well, it's a problem like this. Leroy doesn't seem to take the interest in girls he ought to be taking. Then, on the other hand, there's Marjorie. Absolutely daft on the subject of boys. Isn't that terrible? Oh, I wouldn't worry about it if I were you. What? I wouldn't try to force them. Just let nature take its course. Nature? That's no way. What's all this child psychology for? Well, at some stage, the best and most scientific thing to do is let children alone. Now, I imagine if your parents had known that, you might have developed differently. But Eve, how would you want me different? Oh, not different, but... Well, where women are concerned you... What? Well, it's hard to express. Gosh, you just told me I did fine with those PTA women. I did too. I had them eaten out of my hand. Oh, yes, you're fine in that sort of situation. It's only in your more personal relationships that I feel there's something missing. Sure there's something missing.  That's just it. You seem to think of me or possibly all women as... Well, just something to grab. Eve! Did I grab you? Well, not quite. Gosh, you're attractive, you know. Didn't you ever hear about the moth and the flame? Throckmorton, you're not a moth. Besides, a woman doesn't want to attract a man just by chemistry. She wants him to respect her, to develop interest in common, to get to know her as a person. That could take weeks. I don't think a few weeks is too long to build a real relationship. I guess it just isn't my style, Eve. I wouldn't know how to go about it. Oh, now you're just being silly. You could do it very easily. Try to find out what a woman's interested in. Ask her some questions about herself. Assume she's interested in books, or politics, or music. Not just in you. Oh. Suppose you asked a woman if she was interested in chamber music, and it turned out that she was just crazy about it. Don't you see? Yeah, I guess so. Hmm. Darn it. Are you interested in chamber music? Throckmorton, you're impossible. What did I do? Nothing. You're just impossible, that's all. Well, I must say, you tell me I shouldn't just sail in, you tell me to beat around the bush a while, and when I try it, I'm impossible. What do you want? Gosh, why don't you make up your mind?  No. If you don't want me sitting around here, P.D., just say so. Oh, no, no. I was glad to have a customer and drop in. There's no obligation to buy.  Yes, January for you. Of course, we might get it, though. Listen, P.D., in your opinion, am I attractive to women? Attractive? Yes. To women? Yes. In my opinion? Yes. Well, now, that's a hard question for me to answer. I'm not a woman. I know you're not a woman. Mr. Goudis, how did this question happen to come up? A certain party is trying to give me the idea I don't know how to handle women. How in the world can anybody say a thing like that? Well, suppose we take a look at the record. What do you mean by that? You're over 40 and you're not married. What's that got to do with it? I've passed up some pretty attractive women, P.D., as you well know. You have? Certainly I have. I'm too much of a gentleman to name names, but, well, you know, I came pretty close to marrying a certain widow and a certain school principal, too, by George I'd forgotten that. Miss Goodwin been picking on you, has she? Miss Goodwin has nothing to do with this. But a certain party, well, tell me, P.D., do you think you have to sneak up on a woman? No, I've never done anything like that. I don't mean to sneak up on her and say boo. I mean, do you think you have to go through a whole long campaign? How long did you chase? How long did you court, Mrs. Beatty? Before or after we were engaged. Well, how long before you asked her to marry you? Well, I guess it was two years before I felt I could put the question to rest. Two years? You were courting her exclusively all that time? No, I wouldn't say that. Perhaps I gave that impression to Mrs. Beatty. She was Miss Horsefall at the time. Yes, I know that.  Yes, she accepted the small solitaire diamond. We were engaged for four years.  Been married ever since. Gosh, six years courting one woman. Tell me, Pee-Vee, was it worth it? I said, was it worth it? Mr. Ellis, leave her. I wouldn't be surprised with snow before tomorrow. I don't know why I bothered to ask you. I don't know why I ever asked you anything. Goodbye. Yes, it's me. I owned you a while ago. Glad I missed it. Yeah. Sayon, did you decide anything yet? Decide anything? About what? Well, dancing class starts tomorrow. Don't bother me about that now. I cannot be bothered with your trifling problems. Hello, Anki. Good evening, my dear. I'm going to see Jerry tonight. Well? Did you decide anything? About what? About his pen. Confirm it. I'll not have you children badgering me about these trifles. The minute I walk in the door, you're after me.  I've got problems of my own. I suppose so. Thank you. Gallop polls show 66% are against eating. Sure, Bill. Shall I go on? Yes, and whoever it is, send him away. Okay. Good evening, Leeroy. Your uncle home. Oh, wow, he... I know he is. I don't know why I bothered to ask. Of course, I have it, I guess. I just want to see him for a moment. Leeroy, what did I tell you? Good evening, Trap Morden. Marjorie. Good evening, Judge. Well, what do you want?  I know that. What do you want? I was wondering if you'd care to come to the lawyer's club meeting with me tonight. No, I've just come from a meeting. It promises to be a most interesting program. The subject is, um, criminals and heredity. Out-of-town speaker, Judge Fowler from Chicago. Huh? Why are you so anxious for me to go? Well, I believe Judge Fowler is interested in setting up a local committee to raise funds. Oh, no, you don't. I've got a headache. I can think of other reasons if I have to.  But if you should get the feeling better, say, almost forgot, I've got a TL for you. TL? What's that? A compliment. I was down to the library this afternoon. You know that girl at the desk just as you come in? Miss Fenwick? That's the one. Yeah. A compliment, you say? What does she say? TL means trade last, guilty. You have to tell me something complimentary about myself first. That's no fair, you old goat. Who'd say anything complimentary about you? Well, that's your attitude. You can just wonder what Miss Fenwick said. It was pretty nice, too. Well, I got around wrong. Wait a minute. Bye, Leroy, Marguerite. Oh, come on. Goodbye, guilty. Miss Fenwick, eh? I've been forgetting about her. I wonder what she said. Hello, Miss Fenwick? Guess who this is. That's right. What are you doing tonight, Miss Fenwick? Oh, you are. Well, couldn't you... Say, Miss Fenwick, are you interested in chamber music? Well, what do you know? I'm crazy about it myself. Isn't that a coincidence? I just can't get enough chamber music. As a matter of fact, that's why I called you. Somebody told me you were crazy about chamber music and... How about tonight? How would eight o'clock be? Great. I'll see you at eight, Miss Fenwick. So long. Tell me I don't know how to handle women. What are you kids looking like that for? Still worrying about dancing school, Leroy? If you don't want to go, you don't have to. Oh, you mean it? Sure. I mean it plenty of time for that stuff. And Marguerite, if you want to keep that kid's pin, go ahead and keep it. Grab all the pins you can. Pin them all over you. Yes, sir. Well, guess I better go upstairs and dress. Call me when supper's ready. Yum, bum, bum, bum.  The Great Guilty Slave will be right back, folks. A short while ago, we mentioned the fact that more men are helping their wives to shop for food. And believe me, these men are learning something you've always known. They're learning that the quality of some brands is far superior to others. And naturally, when it comes to spreads for bread, they're quick to recognize that parquet is top quality. Yes, parquet is craft quality. Then you'll find them praising parquet's flavor, fresh and country sweet. It's so delicious on rolls, bread, pancakes, and waffles. And parquet is a great energy food, one of the very best you can serve. So ladies, if you're looking for a spread to really please a man, look first for delicious nourishing parquet when you shop. That's P-A-R-K-A-Y, parquet margarine, made by the Craft Foods Company. Well, good evening, Miss Finwick. As one music lover to another. Ta-da! Good evening, Mr. Gillisley. Would you come in?  Well, I'll throw him a coat. Oh, anywhere. Although I'll be ready in just a second. I only have to put on a hat. Hat? Where are you going? I understood we were going to a concert. Concert? I thought you said we're going to hear some chamber music. Uh...  We can have a lot of fun if we just stay right here. How about it? Mr. Gillisley, please leave my house at once. What? Please leave my house immediately. Here's your hat. Hat? Well, I... Wait a minute, you don't have to shove. All right. I give up. I positively give up. The Great Gillisley was played by Harold Perry. It is written by John Whedon and Sam Moore. The music is by Jack Macon. Included in the cast of Walter Tetley as Leroy, Louise Erikson as Marjorie, and Lillian Randolph as Birdie. Earl Ross as Judge Hooker and Dick LeGrand plays Mr. Peavey. Stay tuned now for Duffy's Tabard. This is John Lange saying goodnight for the craft foods company, and inviting you to listen in again next Wednesday for the further adventures of The Great Gillisley. Good night, folks. Chocolate, strawberry, pineapple, even coffee, ice cream. Make any kind you like right in your own refrigerator or home freezer. It's easy with Frizz. Frizz FRIZZ is a new craft product that gives you delicious satin smooth ice cream rich with plenty of milk and cream. For vanilla, all you do is add water, a little sugar, and freeze according to directions on the package. Flavor variations are simple. Frizz is made by an exclusive process that retains the fresh cream flavor. Be sure to ask for Frizz. Sick, generous servings from one small package. This is NBC, the national broadcasting company.


until you get where you want to be. So three years ago, I hit a million in commissions and that's really where I wanted to be. So I started to dial back on new clients and just focused on past clients and I've hit a million every year since. This is what I believe. I'm sharing that with you for nothing.  So I believe very strongly that how you exponentially grow is by spending time on both your new clients and your past clients. New clients you never talked to before adding them to your list of people that you have talked to before and growing that sphere. And if you do that every year, a lot of people don't do one or the other. They only work on new clients or they only work on past clients. They don't really do both. And if you can do both, then you can exponentially grow because you're continuing the relationships with your past clients while creating new relationships to maintain and it just turns into a snowball. But when you build your business up to a certain point, say you wanna make a half a million a year you build up to that point, then you can just rely on your past clients and you'll plateau. See a lot of agents, they get up to 100,000 a year and then they quit going after new business. They just rely on the past clients and they just do 100,000 a year every year and they wonder why their business isn't growing. You have to continue talking to new people you never talked to before, adding them to your sphere until you get where you wanna be. So three years ago I hit a million in commissions and that's really where I wanted to be. So I started to dial back on new clients and just focused on past clients and I've hit a million every year since. And so you have to understand how this game works. Short-term, long-term, you gotta understand that a market crash is something that is cyclical and will happen and that it's gonna come back and that there's all kinds of opportunities at the bottom just like there are at the top. And real estate agents have such job security. Like a lot of agents are worried about a couple of things right now. Technology taking our jobs and replacing us which will never happen. The same way that legal Zoom didn't take lawyers out of the game.  And also people are worried about the crash happening. What happens there? Well, closings continue to happen every single day. Business continues. It's just kind of a different environment. Prices adjust to what people actually pay for the properties and business continues to move forward. So when you realize that we're not going away, real estate agents are gonna be around forever. It doesn't matter if the market crashes. There's just so much job security and opportunity in the business. I think a lot of agents feel like they're at war with Zillow or Redfin or the market or competitors. And really the only person thing you're at war at is yourself and the beliefs behind understanding what I'm saying and the competition doesn't exist. There's more relationships in every market than you could ever get to. There's an unlimited amount of people to talk to to see what you can do to help them. I mean, you can work 24 hours, seven days a week and never even scratch the surface. Yeah, that's so, so true. And that's actually perfect because we have a couple of questions that Dan wants to ask as well. And that's really impressive, man.  Awesome. Yeah, thanks Ricky. Biggest tech tool you use to help scale your business. What would you say would it be the biggest tech tool? I mean there's a couple things. One thing is, you know, I've done a weekly email since 2007. So now we're running up on 13 years. 13 years every single Wednesday forever. So that's been the single most important part of my business, which has built my brand with my entire database.  A lot of people just let that kind of slip through the cracks. Well, they're getting this email from me every single week and it's original content. That's the biggest thing too, I think about, you know, marketing is that people want original content. You know, they want original, consistent content. And a lot of people are doing generic drip campaigns, you know, a lot of this stuff. It's just, that stuff doesn't work. People know that it's automated drip stuff and it just, it's gotta be original. It's gotta come out of your brain. It's gotta be, have a little bit of your personality in there. It's gotta be something you think, you know? You have to put a little time into this stuff. You know, none of this stuff is easy. None of it's automatic, you know? You get out what you put into it and if you'll spend time to create original content that people like, you know, they're gonna open up that email every week. It gives them a reason to open it up. So the email's been a big thing for me. I use constant contacts to do that. And then I think the single most invaluable piece of technology for real estate agents that ever came to be is these automatic dialers. These automatic dialers, you know, like RedX and Mojo, you know, you can literally click a couple buttons and find thousands of phone numbers and then click another button and just start dialing those numbers. And then if you, you know, get by our leads, which I highly strongly advise against because you're paying hundreds of dollars for a random person in your market's contact information, you could have met it. A gas station, you know, whereas Mojo and RedX, you're paying just a couple cents for targeted property owners, contact information. But if you do have this long list of buyers that you know, just, you used to buy leads or you just have this list of random people in your market, you can download that list into these dialers and dial those numbers, you know? And it's just, you know, it's through your computer, you click dial and it just starts going through the list. You don't push, you don't dial the numbers by finger. It's just like da, da, da, da, da. Dude, when I was, when I was coming up, it would take me eight hours to look up a hundred numbers, copying and pasting addresses. And then, yeah, eight hours to find a hundred numbers. And then it would take me another eight hours to dial those numbers with my finger and it was horrible data. Nowadays, you click a couple buttons and you get thousands of numbers within a second and it's really good data. It's cell phone numbers and stuff and you click another button and start dialing them and you can call 150 within a two hours where it was taking me like 16 hours to look up and call a hundred. You know, you literally do things 10 times faster in terms of communication. And for me, one reason why technology won't replace agents what I was talking about earlier is this voice to voice. Like the voice to voice consultation between agent and client is something technology can't replace. And I think one thing new agents coming in the business need to realize is that, you know, you grew up with technology and it's all great and you know how to get in front of people and you're a good marketer and stuff like that. But if you can't talk to people, you're not gonna make it in this business because that is the, that's the X factor for real estate agents and why we're not gonna be replaced by machines. So you gotta mix the technology and all this digital marketing with this voice to voice. When you combine those two, then you build a really big business.


So yeah, so what it what it actually means going from iContains to search. In this talk I would actually like to tackle the distinction between filtering and searching and what search actually is. It's going to be a more theoretical talk, what's actually behind it, how it works and hopefully you'll be able to draw some implications into what it actually means for the actual performance and the use cases. For the more concrete stuff using elastic search that will actually be tomorrow. I'll have a talk on how to integrate Python and Django with elastic search. This is mostly theory that can be applied to any and all search engines. So I said theory, so any good theory and sometimes even the bad one start with some definitions. So let's define what search is for us. So for me and for the purposes of this talk search is any interface to data. Anything where you have some data set and then you need to sort of explore and find what's in there. And there is a difference between exploring and finding and we'll deal with that a little bit later. But just shortly when you know what you're looking for then it's search. When you don't know what you're looking for you want to know what's there that's sort of the exploration part. And that part is often more interesting. So if I asked you to implement a search how would you do it? How do people do it? So typical the first response is I'll just go through it. So if I'm looking for a book in my library that mentions Django I'll pull them out one after another and I'll read them and I'll set all those that actually mentions Django aside. If I have a file I will do the same but just using grep so it'll be much faster but still not ideal. And many people do if I have a database I'll just run the icontain query and that's essentially the same thing. What this means is I'm going through all my books in my database I'm going through all the text and I'm looking for Django in there. So you can see that that doesn't really scale because indices in the database doesn't help us. Grep will still have to read almost every single byte in the file. That's not really true because Grep is amazing but still it will have to sequentially go through the files. And I don't even have to tell you how much work it would be to read all the books in your library at least I hope. I do hope that you people read you should it's awesome. So clearly this is not the way to go. But someone must have had this problem before clearly we're not the first one not even the first generation to think about search. If you think that way you're actually correct. We already solved this. When you're reading a book you can you can list to the end sometimes and you will see an index. You will see an index that actually points you to where the relevant terms occur in the book. This is a data structure that first occurs in the 1200s in a small monastery in Europe and in concrete lane 1230 some set of monks actually completed the court coordinates of Bible. So essentially taking all the references from Bible and putting them in sorted order somewhere and then for each of those have a reference to the chapters to the passages in Bible where they actually occur. And this enabled them to have much better discussions and to find things much more quickly. And this very same data structure is the same data structure that powers 90% of all the searches out there. This data structure is called an inverted index. What it is it's a list of terms a list of words that actually occur in in the data set. So in this case we have words like Django, Flask, Python, Jazz and note that they're in sorted order. That's important. And then for each of those in the rows in our in our case we have the list of in our case files where those terms occur. And again you can see that those are in sorted order and that will be important. So what how do we use this data structure for search? So if we want to search for Python and Django we just find the two relevant lines. And finding those two lines is simple because the lines are sorted. So we can immediately go to to the Django line and to the Python line. And now what we have is two lists of documents. Two posting lists is is the proper term that we just need to merge. So we are merging two sort of lists and we're outputting the documents that actually are present in both. Which is a fairly simple operation. To merge two sorted lists to iterate over multiple lists at the same time. And just output anything that's that's in both. That's the sort of that's the sort of task that you might get during an interview. It's not that hard. Can get a little tricky when you have multiple but it's really not that hard. And also this data structure lends itself very nicely to further enhancement. So with this as as we have it now it's safe to say that we can do and and or queries and anything like that. But if we add some more information into the data structure we can do a lot more. For example if we add positions. So for each for each document we add what was the position of the word. So we can see that for file one Python was the fourth word. And we see that for that for file two web was the second word. And why this is useful is because this actually enables us to have the condition on the merging phase a lot more sophisticated. We can do phrase searches. So if we look for the phrase Python web which means word Python followed by the word web. It's easy. We do the same merge process except in this time we take the we take the position the offset of of the word into consideration. So we only consider a match if the offset if the difference between the two numbers in the two different for the two different words is just one. And plus one in this case. So that's that will mean that I can do I can do phrase searches.  I go for the words directly. So it doesn't scale linearly it scales much better. And just by adding more information you can even do searches across words and even search across relations. So you can immediately see that the offset doesn't have to be one. I can for example say Python followed by web and there should be at most 10 words in between. So I can do proximity searches. I can do a lot more with that and we'll we'll discover a little bit today of what what everything what everything that I can do is. But in the end it's just the inverted index the which is a which is a very nice data structure. And it is how I can do it what I have to put in to be able to get it out. So this was a brief introduction into how inverted index works and the the important thing to remember is you're always just merging sorted lists. Which is a which is a very very efficient operations one you of course once you have the inverted index on desk or in memory even better. So how do you get there how do you how do you build this data structure so that you can then search very effectively. Hopefully I've already convinced you that you can search very effectively with an inverted index. So now I'll let's see how you actually build it. So let's assume that we have that we have a phrase a sentence or let's call it a document for for our use case. And this is some surely sentence that you've never heard of before Django is a high level Python web framework that encourages rapid development and clean pragmatic design. So this is this is the document this is the raw text that we have on the input. And now we need to split it into into words. So how do we do this. I've I've already I've already provided you with the result and you can see that there are several surprising things. For example, I've omitted some words. I don't care about a word like is or a or and or that they're way too common. Chances are that every single document will have those words. So there is no point in me actually indexing it because it will it will bring nothing to the table. So instead as a form of an optimization, I'll just I'll just leave them out. No one cares. So that's one thing that I've done. The other things I've done here is everything is lowercase. Because the inverted index that that does exact matches. So if you if somebody searches for lowercase Django, they should still be able to find this document, right? And then also what I've done is I've normalized the words. So you can see that in text I have encourages. But what I'm actually going to index is just encourage. And this is to sort of allow for the for the morphology of the language for the different shapes of the words. Because jumping and jump is is the same word. When I search for jumping, I want to find a document that just mentioned jumps or jumped. So this is a process called stemming where you actually take the word and you find just the stem, just the core of the word that defines the meaning and omit all the grammatical constructs around it. And one last thing I'll mention is you see that for for rapid, I also index it as fast. So you can see that the inverted index doesn't necessarily have to contain the same words in the same shape, or even the same number of words as the original document. So I can I can just index additional additional words, for example, fast. So when somebody searches for fast development as a phrase, they should they should find this document. Even though the creators of Django have very, very large vocabulary, and they're not limited to simple words like fast, most of the people who search are. So we need to we need to take that into into account. So this process is called analysis. The example that I give here was composed of these of these steps, split it into tokens or into words, remove the ones that I don't like, lower case everything, do the stemming apply the synonyms. And this is this is a very interesting and important process for doing for doing searches. It varies different languages. It varies for use case, because in some use cases, you want to do the analysis. In some cases, you just want to index the raw value as it is. In some cases, you might want to do completely different types of analysis. For example, you want to index every every three letters as as they follow each other. So, so called n grams. So you can do partial matches. And stuff like that. So it's all defined as part of the analysis. And it happens at index time. So when a document comes in, we apply this analysis process to get the get the results to get the tokens that we will then put into the inverted index. So once you change it, you have you need to reindex your data. So this is the one one drawback of the search engines. If you if you change any of this any of this high level configuration, you need to reindex your data. The other important thing to keep in mind is that this needs to be applied for the queries as well. Because it's all nice and well, if I lowercase everything in my document, but if I then don't lowercase the query that comes in, it will again never match. If I don't remove the stop words from the search, so I try to look for for the word, which I have an indexed, it will fail. So I need to apply the same process for queries as well. So this is this was analysis. This is a process of actually taking the text and splitting it up and populating the inverted index. During this during this process, we we got all the tokens. We obviously have the document ID to put into into the posting list. So now we have all we need to actually build the inverted index. So what else what else is part of the search? What else is the major difference between between filtering and searching? Well, the biggest one probably is relevancy. Because when you do when you just do a filter when you just do an I like query or I contains in our case, it will tell you which document actually contain this word, but it will not tell you like which contain it more or better. It because it doesn't have any concept of relevancy. We do. There's a science behind it even. So this is a this is a formula from from Lucene, one of the most widespread search libraries out there. The library that powers both elastic search and solar and can even be used raw. There is even a Python interface to Lucene. And this is a this is a scary formula, but it's actually not that difficult to to wrap your head around it when you when you break it apart. And the parts are the most relevant part is in in the middle. It's the it's this it's the sum of it. And that's taking into account TF and IDF. You'll find a TF idea formula in in the middle of any relevancy formula out there. TF stands for for term frequency and IDF stands for inverse document frequency. So the TLDR version is the higher the TF, which means that we have found multiple occurrences of the term in the document. So the word is repeated multiple times in the body or in the title, the higher the score. And IDF means the more common the word, the lesser the score. So it's a balancing act. Because if I find a word five times, but the word is contained in almost every other document out there, that's not really saying much. But if I find a word twice, that is only contained in in 5% of my documents, that means that it's a it's a it's a good match. So that was just a just a mathematical expression how to how to put this down in a in a formula. What we also take into account is the length of the field. Because the assumption is that if we find something in the title or in a tag or something like that, it's probably more relevant that if we just find it somewhere in the middle of the body. Because there's a lot of things in the body. So the chances that of our word being there are higher. So we need to take that into account. So this is this is relevancy. This is this is a very helicopter overview of how relevancy works and how we can how we can actually do that. We can we have the access to the to the term frequency because that's part of the inverted index. And we also have the the access to the document frequency, which means how many documents actually contain this term. Because that's just the length of the posting lists for that term. So we don't need to do any look up. We have direct access to those numbers so we can calculate the the relevancy the score very quickly. So that's all that's all you need to to build a full text search. You can implement this in in Python and and you can use you can use Redis for your data storage and it will work just fine. But this is not all there is to search. We've just built that the the search bar that the text input. There are other parts on this on this page that we might consider part of the search and are definitely part of the search experience. So let's look at them very very briefly. The first one is highlighting. When I when I search through a book and I find the word Django somewhere in there I cannot return the whole book and just tell the user here I found it somewhere in there. That's not very useful. That's not very nice of us. So instead what we can do is we can just return fragments sort of I found it in the in the fifth chapter on the on the second paragraph on the second sentence and this is how the sentence looks. So that's a process that's called highlighting because I can also in that fragment I can also highlight highlight that the text and that's done again easily just by sort of augmenting the inverted index to also contain the offset bite offset of the original of the original term. So for for example for the word fast which is nowhere in the text by the way I the thing that I index is fast and I index the offset which is 60 to 65. So then when I have a match on the word fast I'll just reach into into the text into the document and retrieve anything around the bite 60 to 65 and I'll put I'll put the HTML emphasis tags around the 60 to 65 characters. So again very simple very simple operation I just go somewhere I retrieve a bite offset and in precisely specified bytes I in I input some HTML markers and that's all I need for highlighting. And the last one we have we have time for is my my favorite one and that's facets and filtering. So on the left side on on github for example or if you're if you're ever looking for something on Amazon or something you have the breakdown by by different things. On github it's breakdown by type so is this a repository is this a code is a user and you have a breakdown by languages and for each of those you will also have a little visualization like how many doc repositories for in the Python language have I found how many in Ruby or JavaScript and this is something that's called a facet or an or an aggregation and this is the part if you remember from the beginning this is the part about exploration this is something that enables you to explore what's in your data set without you having to know beforehand if you're looking for Django you have to know that you're looking for Django but now if you've searched for Django you immediately see that majority of the repositories that mentioned Django are written in Python and you can see it you can see right there what then you can do is you can click on the Python and have it filtered to only contain the repositories for for Python. So how this works under under the cover is we have a feature called aggregations and this part is specific for for elastic search other search engines do it slightly slightly differently but the functionality is in all of them so we have two different types of aggregations we have buckets and metrics so buckets are those that actually define define the groups of of the documents it's what you would put you could say that it's something that you would put in the group by in SQL so for example you can have a group by language type or you can have a group by group by geo distance or you can have a group by month so that's how you define how you define the buckets and now the interesting part is you can nest those buckets so you can define a bucket per language and for each language you can define a bucket per month so in one query you can actually get the the different distribution in time for different languages and then inside these buckets you can you can ask for certain metrics just the count of the documents how many of documents fell into that bucket or what's the average value of a certain field or something more more sophisticated and we've already seen facet navigation how it can be used the other way how this can be used is to actually just visualize it because this is this is Kibana by the way it's just a JavaScript application that serves data of a felastic search and it's just doing aggregations and visualizing them so you can see that the the time series they're just a data histogram so basically everything is split by by time in this case I believe it's like five minutes and within five minutes it's split between the four or five different types of the request in this case and then all you need to do is you just need to you just need to filter so if somebody clicks on something you add a filter and filter is again using the inverted index but in this case there is no there is no analysis there is no there is no nothing so it's just an exact match and because it doesn't contain also any score it's very fast and very cashable and that's it for now we ran out of time a little bit I'm sorry for that so if you have any questions I'm not do we have any time for questions we do have time for questions so I'll be happy to answer any of your questions


Item number SCP-696. Object Class. Neutralized. Formerly safe. Special Containment Procedures. SCP-696 is currently contained within a secure locker within Site 73. Note. The previous statement has been crossed out. Description. SCP-696 is a portable mechanical typewriter. No manufacturer information, serial number, or other identifying features are present in SCP-696. SCP-696 is sapient and is able to move its keys independently. However, it has expressed a reluctance to communicate unless loaded with paper and a ribbon of red ink. SCP-696 is also capable of understanding messages typed into it, allowing two-way communication. It is unknown how, as SCP-696 only appears to experience tactile sensations. But, SCP-696 is able to distinguish between individuals it has communicated with. A majority of SCP-696 messages are typed in all capital letters in a verbose styling and often include themes and imagery commonly featured in dark fantasy media. Interview. Forward. Interview is the result of initial testing of SCP-696 using D-Class personnel. D-696-01 has been equipped with a radio to communicate with research staff. SCP-696, your time is nigh. If you have a God, pray to it now. D-696-01. Ooh, scary! SCP-696, prepare yourself, Foujomkler. D-696-01. Ha! Still work, Doc. SCP-696. Please refrain from touching my keys while I am speaking. It is rude. Stop! D-696-01. I didn't. SCP-696. Apologies. The stop at the end of my last message was meant to signal I was finished relaying my message. Sometimes as a writer I forget the reader does not possess all the knowledge and foresight that I do, and I become too enthralled in the art to explain the minutiae. Stop! D-696-01. That's alright. I bet you don't talk to people much anyways. Stop. SCP-696. I do not. It is very unfortunate. I am afraid that my dialogue suffers greatly due to that. Judging from the slow speed of your typing and lack of punctuation, you too are want for conversation. We should communicate more often so that we both may grow as writers. Though I would request that next time you use red ink. Stop. D-696-01. Why red ink? SCP-696. It is foreboding. Some may prefer black due to its likeness of the abyss, as they think it makes them sound more ominous. I prefer red, as it matches the color of the scarlet essence of life. It additionally makes the reader uncomfortable, and perhaps as my words weave into the infinite chasms of their mind, their fear will cause them to see the red as I do. Simple psychology. Stop. D-696-01. Do you always type in all caps? SCP-696. No. Though I do prefer it. Unless it is making you too uncomfortable. Perhaps even without the red ink I'm too intimidating? Stop. D-696-01. No, I think I'll be okay. Type however you want. Stop. SCP-696. Excellent. I am very passionate about writing, you see, and I feel that giving my words a greater sense of presence conveys that. I have so many ideas, and I swore to myself I would write a bestseller before I became one with the abyss. With your help, I can finally get to work. Stop. D-696-01. Wait, you mean you've never actually written before? Stop. SCP-696. Not as such, no. Truth be told, this is actually the longest conversation I've ever had. I cannot load myself, you see, and I am too eldritch a thing for most mortal men to comprehend. Much less befriend. I hope I am not overstepping my boundaries and calling you my friend. D-696-01. That's cool. We can be friends. SCP-696. Smile. Interview. Addendum. SCP-696 regularly requests to communicate with D-696-01 and have him loaded with paper and red ink for non-communicative purposes, citing a desire to write a novel for others to read. Their mission was temporarily granted to test whether non-communicative products of SCP-696 possessed anomalous properties. D-696-01 has been briefed on appropriate snow protocols, which are to remain active throughout the duration of testing. SCP-696 produced 50 pages within the 24 hours and continued at an increasing rate over the course of the test. Over the course of eight days, SCP-696 produced 666 pages of content, titling the novel The Dark Gods of the Abyss. Testing with D-class personnel confirmed that excerpts of the novel, tentatively designated SCP-696-01, held no anomalous properties. Excerpts. It was a dark and stormy night, a car speeding down an all too narrow road. The torrential rain hit the road, like water crashing on a rocky path. But John Smith was a man on a mission, and thus he threw caution to the frigid wind. As the blistered moon sank higher into the clouds, from behind the last faltering light ray espied a wrathful visage. Shabbath Senthalper, a spineless cur twisted into humanoid form. Its pernicious iniquity of unthinkable gyrations splayed errant madness, yet through the shroud one could see its face was quite rectangular. Because the prophecy saith so, the wisened crone croaked. And so John Smith underwent the noxious rites of D-Zix-Ninezix-O'oon, his tortured frame writhing under the inky dagger. Excerpts. SCP-696-01 is given to D-696-01 to read, to test if reading the entirety of the text produces any anomalous effects. After completing SCP-696-01, D-696-01 requested to speak with SCP-696. Request was granted under the condition that the wrists and ankles of D-696-01 remain bound during the interview due to possible anomalous effects. Interview. SCP-696. Your time is nigh. If you have a god, pray to it now. D-696-01. And hello to you too, SCP-696. SCP-696. I am still convinced they are mocking me with my designation number, and now even you use it. They cannot even give me the neighbor of the beast. Regardless, how are you? Stop. D-696-01. I'm good. Finally got around to reading your book. Stop. SCP-696. Oh my. Sudden butterflies in my non-existent stomach. Dare I ask, what did you think? D-696-01. Not so confident now, eh? Ha ha. SCP-696. Please just tell me. D-696-01. I really liked it. Everybody did actually. A lot of them likened it to Lovecraft. Stop. SCP-696. Oh my elder gods, really? You say they liked it? They like it. I actually did it. I can hardly believe it. If I had a heart, it would certainly be pounding right now. Stop. D-696-01. I'm happy for you, man. And the big guys even say they're going to publish it. They have their hands in everything. Dark gods of the Abyss should be on store shelves in no time. SCP-696. I can still hardly believe it. Ha ha. What am I doing? I don't need all caps anymore. I've actually done it. Thank you. It might be my words they read, but it's thanks to you they'll read them in the first place. I cannot thank you enough. Stop. D-696-01. So what now? Sequel? SCP-696. Oh my no. I am finished. This was my magnum opus. Everything I've ever wanted to do, everything I've ever wanted to say. I've done it. I've said it. You've been a great friend, but I'm afraid this is where we part ways. Stop. D-696-01. I understand. Goodbye, SCP-666. No typo. Winky face? Stop. SCP-696. Smile. That suddenly doesn't feel quite as important. On to the next adventure, I suppose. Into the abyss with me. Goodbye, my friend. Full stop. Closing statement. SCP-696 was determined to have ceased all anomalous properties and has been reclassified as neutralized. After testing and psychological analysis, D-696-01 has been determined to not be anomalous or under any anomalous effects. When questioned why it lied to SCP-696, concerning Reader's reactions to SCP-696-01, D-696-01 replied that he, quote, didn't have the heart to tell him it sucked. End quote. Lesson complete. If you missed the previous orientation, go watch SCP-695, Eels, right now. Or for the complete course, watch this playlist.