Welcome back everyone. This is the third video in the series about doing research or conducting independent research Today we're going to be talking about the scientific method or the method of actually Conducting research the general method that applies to finding new knowledge, okay, so In the last video you might have seen We were talking about asking questions or getting in the habit of asking questions about things in our everyday life Things related to our our topic that we want to research But just basically getting in the habit of asking questions about Everyday objects that way we're not taking anything for granted or at least we know Where our knowledge stops for normal objects? Once you start thinking in that mode And I recommend you go watch the video if you haven't yet Once you start thinking in that mode, then you will start to question Or you'll start to realize where your knowledge ends Also in your area of expertise or whatever area you're working on Where your knowledge ends is where you can start asking questions and questions that are hopefully relevant and interesting You're essentially finding by doing this questioning process. You are finding a problem a problem so you're finding a problem and This problem is essentially in the form of a question, right? You you want to know I think my example was about markers. You want to know how does the liquid stay in a marker? If you don't already know that well, that's a problem that you can solve So what you're actually doing is having making some sort of problem statement Right, so some sort of problem statement that we're going to be working on You have a problem in the form. Hopefully of a question. So how does Fluid Stay In a marker, right? So we have some sort of problem statement or a question that we're asking This is something it's not a yes or no question, right? It is something where we're actually understanding more about the way that fluid stays in a marker for example But If you don't know the answer to this already It's not very obvious how you would find that out unless you make your own experiment or somebody else knew how to do it Next once we have our problem statement and we have a question or something that we want to solve then we make a Hypothesis or a guess about What the answer to this question is so actually I should number this one to hypothesis So hypothesis here, you might have I hear people all the time saying I have a theory but actually what they're talking about is some sort of guess a theory The way that most people use it is just saying a guess hypothesis Is actually a guess about what the answer is based on? My limited knowledge or the knowledge that I currently have so I make some sort of hypothesis How does fluid stay in a marker? Okay? Well now I have to make a guess about how does the fluid stay in a marker? So how does the fluid stay in a marker? I think the tip is Made of I don't know Fluid repellent material I don't something like that. I don't even know what a hypothesis here would be so tip repels Fluid now I'm already pretty confident that that's not true because I can write on the board right so tip repels fluid Okay, so the tip of the marker repels fluid so all the mark fluid stays in the marker Okay, it's reasonable Except the fact that we can actually write on a white board which already kind of proves us false But we'll get there so I have some sort of problem that I want to answer and then based on that question that I'm answering I make up some guess. Okay, and that guess Doesn't necessarily have to be right. Hopefully you're right, but you still have to show that you're right even if I say the tip repels fluid well that is It's possible. It's feasible like it's something it's something that could be true right But that's actually not how a marker stare the fluid stays in a marker now the thing about a hypothesis a really good Hypothesis is a true or false statement. So here tip repels fluid I framed this in terms of true or false see the true or it's false does the tip repel fluid true or false Now you could get into some some details about some parts of the tip do repel fluid other parts Do not so it's kind of true and false at the same time But basically what we want is a hypothesis that is either true or false There's nothing really in the middle if there is something in the middle that means your hypothesis is not specific enough Okay, and then you probably have to come up with another hypothesis or a sub hypothesis to prove and then that hypothesis Should also be true or false Okay, so we want our hypotheses all hypotheses whether it's the main or the sub hypothesis To be true to be a true or false question. It should be falsifiable.  Well, I hope that I'm my guess is right But I don't really care if this guess is right or wrong Because once I go through this whole process, I will know the truth Okay, so even if it's not that the tip repels fluid Well, that probably means that I've found the real reason and the real reason might be actually more interesting than what I've come up with here So I'm not I don't really care if my hypothesis was true or false. I want to know more about the issue How does the fluid stay in a marker? Okay, so that's really what I want to answer and then my guess is this if my guess is wrong that I can make another Hypothesis but very often through trying to show that my hypothesis is correct by trying to report or deny it I Often find why the fluid stays in the marker as a as a different reason So I'm not too worried about the hypothesis right now. It's a really important starting point But if I prove it wrong, hey, that's also something That's also something potentially interesting. Okay, so the next or third I should say Once we have our problem statement something we want to find We have a guess about what the answer what the reason is that we're looking for in a true or false statement Then the next thing we should do is go do background research And I will have another video By background about about background research completely separate because it is such an important topic That a lot of people just kind of skip over because they don't They don't really see the need to do background research, but it is very important And if you can get background research right your your total research Will be much more amazing actually much more interesting because you will actually push Where knowledge is rather than being a little bit behind where everyone else is so background research is basically looking up What every? Everyone else has talked about in relation to Fluid staying in the marker now what most people do is say okay I want to know about fluid the ink staying in this marker So I'm going to go to Google and search for how does ink stay in a marker? And then I find every paper with a title ink stays in a marker The problem with that is that maybe you didn't search for the right thing Maybe authors aren't actually talking about ink staying in a marker But maybe they're talking about ink staying in a pin So for example I'm talking about my case is a marker But another author that we found their paper or that we missed their paper Let's say was actually talking about ink staying in a pin right so a pen And a marker let's say the mechanism for the ink staying in them was the same But because I was focused on my marker hypothesis. I completely neglected Pin I say ah pins aren't related to markers. How can they be related? No way Well, actually, maybe they're talking about exactly the same thing just a different case of the same thing so What tends to happen is that people have a couple key words in mind, which is a good place to start Key words in mind to look for background research that's related to what I want to talk about So we have key words related But then we also need to look for different concepts That are related to those keywords, but maybe not in the same area as us Okay, so in this case, I'm focused on marker So my keyword would be marker, but if I look about research if I look at research about pins Maybe I find something relevant to my marker research and very often we do In digital forensics, for example, if you're talking about forensics or you're talking about the way that data Is structured in a computer? well looking at information theory looking at compression algorithms Even though it might not be directly related to Digital forensics the way that those algorithms work or the way that those areas deal with data Is very related sometimes to what we do in forensics even though It's not exactly the same thing looking at those areas might be related Dealing with like sociology dealing with people Very often it's somehow related to digital forensics because people are the ones using the computers So if you're more technical you might focus on actually the data side if you're more investigation focused You might focus on the society side. How do people use computers? But then if you're talking about society then psychology can come into play so you can look at psychology papers and apply it back to digital forensics as well, so My point here is that whenever you're doing your background research. Don't just focus on One particular thing. This is what I'm researching and this is the only thing that's related to it. No Many things are related to your topic and you should know a lot about this particular Your particular area you should know a lot about your particular area But also try to study outside of your area as much as possible and see how that relates in okay That again goes back into asking questions about Basically everything not just your area now. You do have to focus otherwise. You'll never finish a paper, but Try to gain as much knowledge outside as you can and Background research again is one of the most important things you can do. I'll talk a lot more about it later Okay, so once we do background research, let's say that we find out that's people Have been doing some research related to this but they didn't actually find or they're not actually researching exactly what we Want to find or we couldn't find any research on that We found some things that are related to it and maybe they help us to structure some sort of experiment so Once we have our background research then we start If we can we try to use background research as a jump point if we can't then we start to structure our experiment Okay, and this experimentation is basically data gathering Okay, so background research and then we start Constructing our experiment, which is a whole process by itself We'll talk about experimentation a little bit later, but building up your experiment and gathering data from that experiment because we need to prove or we need to show Try to show that our tip repels fluid. We need to show whether that statement is actually true or false Okay, if the statement is false Then that does not answer our question if the statement is true that partially answers our question Right, if the tip repels fluid, maybe that's one reason Why the fluid stays in a pen for example, but that's only part of the answer We'll talk more about Comprehensiveness, I guess later Okay, so starting to structure our experiment or design our experiment and then trying to find data from that experimentation Then afterwards once we have our data, what do we have to do with the data? Always analysis Okay, so we capture our data Okay, and then we do some sort of data analysis. What does this data actually mean in relation to our original question? Sorry in relation to our hypothesis, and what does our hypothesis mean in relation to our original question basically? gather our data do analysis of the data and then try to either Support or deny our hypothesis notice. I didn't say prove or disprove because it's very hard to prove something Sufficiently think of it like a court a court doesn't know that somebody Murdered somebody else, but they have evidence that supports that that person did murder them. They'll probably find them guilty So we're trying to either support or deny our hypothesis And with the data we've gathered through our analysis. Okay, and then the last part of this is Reporting so doing some type of reporting or really what this means is kind of Reflecting on what all of this actually means in terms of our original question. So now our hypothesis What does this analysis mean for our hypothesis? The reporting stage is normally where If this alone was interesting enough where you would be writing papers things like that But normally what happens is after your analysis You get more questions Okay, so after your analysis If you're very lucky and you've already structured everything properly you have a good answer All right, you already have an answer to what you're trying to to show But what tends to happen is that after your analysis you will generate more questions About whatever whatever it is that you're trying to show so your reporting stage then is Reporting about what these questions are and then figuring out how it's going to fit With our problem statement and do we need to generate new hypotheses? Normally at this stage we come up with a new Hypothesis Okay, if we don't come up with a new hypothesis and we probably answered what what research question we had sufficiently enough at least But this whole process just keeps going until you're satisfied with whatever the original problem statement or problem area was So this is very basically the scientific method Okay, so very basically the scientific method come up with a problem statement What problem are we working on make a guess about the answer to the question that's being asked? Do background research to see if anyone else has already answered that question or can anyone else provide? Some support for you to start from maybe they've already planned an experiment or something like that that you could reuse to try to solve your problem and then Do experimentation or design your experiment experimentation method and gather data so do experimentation gather data from that experiment then analyze the data Through this analysis, you'll probably You're trying to figure out. Do you support or deny your hypothesis? Most likely you will have more questions after your analysis so you need to do reporting either produce the results of your experimentation or and Generate new hypotheses to try to talk more about your problem statement So this is just a repetitive process. You just keep going over and over this until you are satisfied With your Answer to your problem statement and by satisfied. I mean you have enough data to support or deny what you think is true I don't mean you found data that says that you are definitely right I mean that you found the truth related to that question, okay? At no stage do I actually Have a particular opinion about what something should be I have a guess about what it could be But if my guess turns out to be completely wrong Hey, I at least I learned something and that might be worth reporting either in a journal paper in a thesis in a blog post Something like that, right? So as you do experimentation try to report or give that knowledge to everyone else because People do use it people do gain a benefit from it even if you find out that what you thought was true is not true So let's say that your hypothesis is this the tip of this pin repels fluid And then you find out that actually I have no evidence to support that the tip of this pin repels fluid well Someone else might come along in six months and say I have an idea the tip of pins repel fluid Right, then they're going to do their background research And if you did not talk about your negative results or That this was found to not be true if you'd never posted that anywhere Then the next person that comes along is also going to do the same research, right? So reporting is extremely important for getting the information out there Even negative results even though journals don't publish negative results very often. It's still very important so I recommend running blogs or just Writing a paper and putting it on archive or just on the internet somewhere and people will find it and it will be helpful to somebody Even if it's only one person you've helped somebody and you need to be documenting all of this anyway for your own research So your documentation is already there. It'll help you to document better It'll give you practice on writing papers and you might actually help somebody in the future Even if it didn't prove what you wanted it to prove. Okay, so negative results extremely important I have no bias about my hypothesis and what it should be Yeah So this is the general process if you stick to this You should you shouldn't go too too wrong the places where you can get mixed up and Yeah, the places where you tend to get mixed up first off Problem statements are normally way too general the ones that students usually make that I see problem statements are very vague very general Not actually answering or asking a specific question Then hypothesis is Also too general it's not specific enough. There's no way to test it. There's no way to measure it It becomes very difficult to Even understand. What are we actually testing? If we make a hypothesis that's too vague So you need to be pretty specific about your hypothesis as specific as you possibly can be I should say Background research the biggest problem here is it's not comprehensive enough Most people do a Google search and then find maybe you know five papers skim them maybe look at the abstracts or something like that and then that's their background That doesn't help you at all. You should be reading papers as much as you possibly can And your background research should be as comprehensive as possible. That will give you the best Benefit for your own work. That way you don't end up doing what somebody else has already done before and There's so many benefits to background research, but anyway, most people don't do a comprehensive background background research then with experimentation and data gathering the biggest challenge here is designing The experiment designing the experiment properly. So it's actually answering or finding data Related to your hypothesis without being biased and hopefully as correct as possible. So experimentation design is a huge area that is People think is easy, but it's more difficult than it seems okay So experimentation a bit of a bit tricky there analysis A lot of people are not very good at statistics actually. So once you gather your data Doing an analysis over that data and doing statistics and Actually finding out how your statistics or how your data might be biased in some way that's not as straightforward as you think depends on the types of studies you're doing but Statistics is usually where most people get kind of caught up on the analysis side So much so that most a lot of people don't really do statistics over their data They'll just say yeah, I saw what I wanted to see there it is but in most cases that's not comprehensive enough and you do more studies to actually show that this is a Normal thing or not. Okay, and then reporting The biggest problem that people have with reporting is not Documenting enough so through this whole process a Researcher should be should have a researcher notebook and should be writing down every single thing that they're doing Every single thing that they're thinking how did they do background research? How did they design their experiment all of those things should be documented now those documentation that documentation is just your notes I'm doing this today today. I had this idea. Maybe I should do my statistics this way right Those notes and actually what you did you should be documenting them all the time The biggest problem that people have is not Documenting everything all the time. So whenever they go back to write their reports. They think How did I design that experiment exactly? Was it a five or was it a six? Right, so then what ends up happening is in their reporting They're not specific about the way that they set up their study Because they don't really remember they wrote a program a long time ago And it worked for them. So yeah, you guys can get the same results as me just Do with the same way I did it, but I'm not gonna tell you how do I did it most of the papers that I read I cannot Verify their findings they just tell me this is the truth But there's no way to verify it because they didn't tell me how they got that truth or the data is not available And you can't be reproduced They didn't tell me how they're did their analysis or exactly how they set up their experimentation all of that should be in your documentation and For your reporting if you're writing in paper you should make sure that everyone else can do the same study as you It's getting a lot better now that we can actually write scripts and then upload them to github Then people can see your scripts and possibly see your data Download it from github and then run the experiments themselves. So Reporting the biggest issue is that people didn't document properly and then their reports or their paper is very Vague on how they did something. Okay, so those are some catches To think about whenever you're you're designing these for your problem statement I hypothesis try to be as specific as possible and as clear as possible So you're not Not too vague For example on this just one example If I say How is this pin? How or how is this marker, right? What exactly am I talking about? How is this marker? Well, I could be talking about the design Right, so the aesthetic design the visual design. I could be talking about the liquid inside I could be talking about the function of the marker But how is this marker is way too vague, right? It could be talking about any aspect of this marker So I need to pick very specific language to describe exactly what I'm looking at About the marker. Okay, so problem statement hypothesis as specific as possible as clear as possible Language is hard to be very clear Really comprehensive background research good experimentation design. We'll talk about that later statistical analysis good analysis, which is normally comes back to statistics so practice doing statistics if you if you're not good at it and then Reporting reporting in a very clear way a very transparent way and a way that can be verified by anyone else who wants to do it Which means comprehensive and complete, okay? And then if you find results that you actually weren't expecting or weren't Hoping for let's say it doesn't it's not consistent with what you thought Still report that somewhere either in a blog post or just a PDF online Or a journal if they'll take it Because negative results are really important for the scientific community Whenever somebody else reads that then they won't make the the same exact study that you made next time because they know that actually this it Wasn't already Was found to already not be supported okay, so this is pretty much it for the scientific method it is kind of a Repetitive flow until you actually get enough data and enough hypotheses supported to To answer your problem statement, okay? It's not just a one one time through finished It's very rarely that way unless you design a very good Experiment or you design a very good I guess piece of research then you might be able to do it one time, but most of time it is a flow You will get more hypotheses more questions as you go along Okay, so I think that's it for today. Thank you very much


So guys, a video which I hope sees you not make the same mistake I did and completely waste your time. How's it going guys? My name's DPJ and if you're doing enjoyed the video leaving a like really helps out and if you like what you see and want to see more be sure to subscribe. Okay so there are tons of quests in this game and I mean absolutely tons. Some short, some utterly long. There's a quest for me within the PK floating fortress called the Deserter. It is quite a long quest, I'm not going to lie. To cut things short though towards the end of the quest it sees you chasing a certain treasure map. Once you get this map you can decide to either give it to her or keep it for yourself. Well, as you would I decided to keep this. This triggers a new quest called Treasure Hunt. The map is basically a cipher in which you have to decode within the library. This alone took me about half an hour to an hour. Then I had to find a location on the world map. So there I was getting all hyped up about this amazing piece of treasure I was about to go and find. A new weapon blueprint for all my troubles. So I found a location guys, I eventually found a safe, I entered the code and boom literally boom I got a sticker see for people.  The location you can see on the screen now and a code for the safe too. I've never felt so robbed of my time in a game ever. This was almost as bad as a book club quest but at least there I got a happy ending. This just left me wanting to go back and play Destiny and that's never a good sign. So yeah guys don't waste your time here it really isn't worth the effort especially when you don't even have to have this quest, you don't have to have the treasure hunt quest, it can still go to this tower, it can still go down to the safe, it can still open it up and you can still get that c4. I've got 200 plus hours in the game now, I've never even used the c4 people. I'm heartbroken I really am. So yeah guys I hope one person who watches this video doesn't make the same mistake I did and on that note the end of the video has arrived, they sure are what sweet one for you guys today. If you enjoyed the video leaving a like really helps out if you're new around here want to see more Dying Light be sure to subscribe and hopefully people I'll see you on that next one.


Thank you, um Alexander. I'm from Mannheim in Germany. I'm a developer of my own company I'm a organizer as well I'm a speaker sometimes a MongoDB trainer for local for our local community and for the Python community I've served as a program work group co-chair Building this conference. So if you have any comments suggestions what we could do better or so I'm around just grab talk to me Very interested in your intro. My talk today is a MongoDB and what's the MongoDB aggregation framework? We're going to cover the pipeline bottle the pipeline stages and map reduce in MongoDB and at first, um, who knows MongoDB just Okay, awesome, and who's actually working with MongoDB? Okay, and who has worked with the MongoDB aggregation framework? Okay, cool. Okay So let's bring everybody up to speed with orientated document orientated databases and set 15 seconds Basically, we work with J the document is adjacent like object. We can store it in our database. We have no shame or enforcement Yeah, um a collection is basically just a collection of documents actually in multiple collections make up our database So pretty easy. It's a pretty simple Concept or the MongoDB aggregation framework was introduced about like three years ago with MongoDB 2.2. It's a framework for data aggregation basically the documents are Processed to a multiple stage Pipeline on the that's giving aggregated giving back aggregated results. It's basically Designed to work straightforward. So no unions like in SQL and and Technically this looks like this. So we have our documents We do a match which is a find we get less documents because we found some the cell phone set and then we do some grouping and we get even less but Actually, I thought it's a little bit too technical. So because basically I think it's more like a relay lace, you know Like relay racers they they go and they they pass the button to each other and basically this is how The MongoDB pipeline works. Um, so we have a match which is fine So we say to please little doggy get the button. Yeah Please pass it on to the smart fox doing something smart which could be like a grouping and then we want to Present our data a little bit more nicely. And so we pass it on to the projection space and Let me tell you a little bit about The data set we're going to work. I'm going to do present the things and what we're doing and I've also prepared some live demos and This is built with MongoDB Yes Um, we're using MongoB 3.0. Um, the new wide-tired Tiger Storage engine with give us a compression Pymongo is obviously the driver we are going to use it's Maintained by MongoDB themselves and it's pretty well maintained a driver. It's always up to date. It's really good And we're working on a data set of 37 gigabytes, which gives us compressible. I take about 9 gigabytes of RAM. Um, basically, um, as you might remember it is my second career. I used to be in the Record industry with a techno house startup business in the 90s So everything I do still in it is very close to working with music on so we have from a project we're doing It's called chart guys. We have some collection of a playlist from the icons music store a playlist is basically All the information about the release you can find and it's music score It's a set of playlists that appeared in some charts somewhere around the world within the last Three years and basically this is what it looks like So pretty cool. So, um, but Don't worry F. Um I've narrowed it down to What we're going to work with today just to give you an impression about our document structure for the for the demos So basically This is a document and info is all the release information So like the album artist the album name when it was released how much is in store and the children That's that's what we call a sub document. It's it's a list with Objects and that's basically the songs each and every song we have in our playlist and I was wondering actually Which artists to use for my demos because it's really hard to choose music artists making everybody happy and I thought I Found something neutral because I chose Taylor Swift. Yeah, and it's not because I like her music actually I actually I don't know any songs of her, but she did this great blog post Making Apple pay for the trial period for the new Apple music servers So artists get paid more money for people using a new service by Apple So she did the good thing and I think that's really worth mentioning her even at Euro Python. So, okay So, um Let's build our first pipeline I've commented in some Notes for the SQL guys to make it easy easier. So basically This is a pipeline a pipeline is basically passed in as a list into Pimonga and Match is just basically a find as you might remember from our document. It's the artist name So we were looking for the artist which is a variable. I've stored already As Taylor Swift, of course, then we're going to do Projected basically it's a select and basically we all all we want to do is print out the Yeah, all the releases by Taylor Swift sorted. So then we switch to This go here. So basically it's just an import we import pie mongo. This is just like a simple database connection and So let's see our database which is live on this MacBook Um, I must say it's only assigned two gigabytes of RAM for this database So it's not usually we work with a lot more RAM in MongoDB So we have a 1.3 million playlists found and it's about like 17 million songs covered in our data set and Usually you could like with you to imagine could also like just like a query and our query says, okay, we found 4093 releases of Taylor Swift So and with the aggregation framework That's the same code. I'm shown you under the slide before we do a match find project Basically, just like we're just project here. It's just like a renaming of the attribute actually and then we sort by release ascending order and basically that's Looking like this Okay, you see we have many releases She's quite busy artist famous karaoke Okay, so and What else we do we can extend our pipeline We can do a grouping so now we want to group everything by Name which is basically the album tab after you probably see we have a lot of duplicates You have done some duplicates in our data set Which is because albums are released by different companies worldwide So at the actual store they get a new ID. They're basically different products, although it's the same contents from the music so the passing in The name as underscore ID underscore ID is in the grouping operator basically what we want to group by You know, it's it's mandatory and it's always called underscore ID and we want to count how many albums are there Account we don't have a count operator in the aggregation pipelines So basically we're just summing one for each and every Document in our group and then we project and sort just to make it a little bit more nice and This is what we get. We still have Some different versions Okay So, okay now we've this nice pipeline we've got a result and I think it's so it's it's so nice Let's print out what we found with again And we get this and it's so nice. I just want to print it again and oops. What happens? I've just I just actually just rest is where we stored Our query to the aggregation framework and I just wanted to print it again and what what what's happening here? Why why doesn't it? Give us any result back. That's like the first trap. Can we I want to show you is MongoDB aggregation framework it returns a cursor So basically the cursor it shows just it points to the data in the database So you get back to from from the MongoDB aggregation. So once we call list the cursor is exhausted So all the data in and then they're printed and then they're gone. So you can't Just use them again unless of course you store them in a new variable So these are all our aggregation stages, um, I've put the like their SQL brothers On the right-hand side. So basically a match is where or having operator sort pretty obvious is Order by limit is also I think No explanation necessary project is a select and we can also use it for renaming as an SQL form our result group is group by unwinds And we're going to go into that very soon is Somehow a little bit of a join not really and Redact we're not going to cover and out is basically just an operator Please send those a result of the aggregation back to a new collection of MongoDB to store it so To make things a little bit easier for you to fall. It's in the next we're going to do something with the artist name and name is the album title and We're making our pipeline a little bit even bigger Like with the group operator what I've already shown you already This is how it looks like prints out a little bit more nicely and next step is how can we work with lists of soft op so As you see we have a list here just with older Songs on that album and we want to do something with it And of course the natural thing would be I query the database and just iterate over it with mongolub With Python. Sorry. Um, but that's quite an extensive task and we can do it in the database and there's this unwind operator and It's basically from from my experience. It's at first sight a really confusing step because it's quite unusual What I've seen so it confuses people and So I think just like Let's just chill you. I think it's probably the best explanation because what unwind does is Basically take all the sub documents in the the list and for each object in our sub document list It creates a new document and this sounds really like an expensive Operation what I can assure you that I did a really good job and it is not expensive at all It's really handy and basically Um, you sure this is like what we're doing now So, um, of course, I'm really sorry. Let's go to that later Here we are, sorry, so let's do this so So now we have all two hundred three three hundred and thirty two songs by Taylor Swift found them we and we can immediately work with them here in our Grouping our stage and as you see the path has not really changed Although this used to be a list before and we don't do read There's no need to do to do anything ago about like iterating over like a list index or anything like that and I've prepared a little bit more It's like here. This is basically what's happening. Um, so it's basically we just get one one one release limit we get one playlist and Then we do unwind and then I'm just renaming it with the project parameters on this is basically what we are getting we're getting All these are single documents new documents just like rated on the fly we can immediately work with so basically it's basically like Just yeah, yeah, it's an unwinding of the data. It's a little bit unusual concept, but it's basically really are simple so but Okay, another one which is Quite obvious one okay that's We have also like a sort which is also like an obvious pipeline stage and I want all the releases Just sorted by count descending and release ascending and basically it looks really simple and It returns us something like this and What's going wrong? Something's wrong because we said I want by count descending over all our data And then I want to have it sorted by release and ascending order, but our result is basically by Release and then by count so something's going wrong here and I can assure you it's not we're not it's not broken it's actually like a trap because We in the pipeline we pass in a Python dictionary and the Python dictionary, of course is in unsorted and of course so We just pass in something which is not ordered and of course our results get a little bit unpredictable but of course That's that's like This is solution and I can encourage you always to use Son from B zone collection or you can also use collection order dick and pass in all sort parameters like as an In an ordered fashion because otherwise you are your sorting order won't won't really work. And so this I'm oh wow it works Okay, okay So this was just like a really quick introduction to Stages, there's a lot which is as mentioned before like it's a skip. It's just like skipping documents out write your results to a new collection There's a neo-gear which just gives you all the documents around Geospatial point Redactors, I don't know. It's it's some people use it to restrict document access on a document level bottom. I've never really seen in production and These are like the stages. This is like our race and now we have some data and basically This is very limited from what we can do. Basically. It's just like mangling around a little bit with the data so We have more of course There's like a minimum a maximum first and last Operator and this is what we're going to work on again. We're searching for an artist we're using release date and a release date epoch and the subtle distance is that a release date epoch is actually a date and The release date is string. It's no date. It's a string on them We're building a new pipeline. We're doing a grouping arm by What we want to find out I want to find out What's the earliest release of Taylor Swift and what's the latest release of Taylor Swift? So we do a grouping by underscore ID As you see underscore ID is empty So new primary key, how can that be empty? Yes, it can be empty because we want a group of our Complete result set so we can just put none in there or leave it empty So there's no need to look for an attribute which is the same on each and every document Just leave it empty and reintroduce new two new attributes Men date max date and basically it's a really simple operation We just walk the path info to really to to today information men max and project it and And yay, you know Taylor Swift is around things 2006 Things she started really early. She's like releasing stuff and she's been around for a while and So what's what's first and last good for I mean we have min max It also would work actually on on on min max would work actually on a ray Just but it's it's it's just like a little bit different and the cave and it can save you Some extra calculations on what's the difference the difference to our previous pipeline is we have a match and then we do a sort by release date and Then we do our grouping and our grouping instruction is first and last and what does first and last do it's really simple Get the first document of the group and last is get the last document of the group So there's no need to iterate over your complete set within the group to find min or max values Basically, you just can say okay. I want this document. I want to look at this document And what's in the middle? I don't really care So this can be really effective and as expected same results so and With dates we can even do more. We have some nice state operators Pipeline we do basically same result by release date We do a grouping and I want to have releases grouped by year. I'm fan boy now I've talked so much about Taylor first. I want to really know everything So I want to have see which year which release so how many releases per year. Sorry so We have actually we extended our ID a little bit and now it's it's it's an object with our dollar year operator and We pass in the date Epoch which is the date and we just pass it in and dollar year will basically just grab the year from our date and This is then our ID. We want to group by and and This Works like this. It's really easy. It makes it really easy if you have some Data with timestamps and so we see okay account.  She's releasing every year a lot of releases. She's hard worker and So but what if I want to dig even deeper. I'm not interested in getting The releases by year. I'm also interested to getting Each the release count for each and every month. She has released something and of course I wouldn't mention it if we couldn't accomplish it in the year We also have a month operator and the next thing what happened is now the ID Which is our primary key can also be a multi key and So we have a multi key year month and basically we do the same arm As before we get the year and month new attribute our ID key has to is this build out of two attributes year and And month and let's just run that and wow we see Haven't checked. That's probably not a month. No hardly any month. She didn't do anything so Well, there's a lot of more date operators as you can guess. There's also like a second minute Many more date operators and it's not a we're not able to cover them all in this small talk I've covered that but it's I'm getting a little bit bored now with Taylor Swift because I It's early in the morning and we want some some more tension So actually I thought about who could and else who could join and so I thought hey I just Google Taylor Swift nemesis and Google says it's an alien space robot called Katy Perry And so let's bring Katie. Let's bring in Katy Perry and it's really easy We can extend our match operator So Katy Perry's is now stored in our nemesis variable and basically we can also do searches with a dollar in Operator and it's basically just the same as in Python. So I think some really necessary To explain to you guys So and of course now we have a big competition. I'm wondering Who deliver delivers more song value? For my 99 cents. Is it Katy Perry? Is it Taylor Swift? So I want to see the average playtime of their songs I'm interested who gives gives me more songs longer songs. I can enjoy for my money It's not a good thing, but it's just like nice example So what we're doing as you see we now have three unwind stages So basically the first thing is we unwind the songs and then we unwind the song offers the song offers And within those song offers assets is basically the prior to the playtime stored So and we want to access this information. So that's why we have a pipeline of one two three unwind It's unwind unwind unwind and then we can group by just going down the path by the song name Which is a childhood name and then we just do an average of the path of our long of the duration we have stored within the assets and Show you this and Something's wrong Okay, sorry and just fixed it. Okay, something's broken. I'm very sorry Won't waste any time to fix this no life So basically, but I can explain you Basically, it's just like the same we did before with the releases here and Counting the releases and the next step of course would be getting The playtime so I hope My notebook didn't break. Yes. Oh, it didn't break Okay, sorry again, of course We have our group or playtime and we just were projected and as a result we can see okay Taylor Swift gives us more more music like about like 10% more music than Katy Perry for 99 cents and That's a really easy operation, okay now something Something a little bit more challenging I'm interested in thank you I'm interested in getting the prices of the releases of the artists and My own it's basically scraped data. So it's not probably As clean as I would wish Basically, we see a formatted price with the currency in front and the price But it's just like in one attribute and I'm interested in getting The prices in US dollars And that's that's easily to solve with a string operation and a compare operation so We have to speed up a little bit Basically, we do a project phase. So just focus on the things in bold They're important ones here and we have is US dollar Basically is a comparison of the lower strings of the first three Characters in our price formatted, which gives us back US dollar or some currencies or numbers or whatever and the comparison is basically is this US dollars It's pretty obvious and then we just do a new match for is dollars zero Okay, so also feels a little bit wrong but Comparer parameter gives us zero back when it's a match and it's gives us Minus one back if the value is higher and one back if the wire the value is lower. So it's pretty Pretty handy one. We could also do is equal There's also an equal operator which would give us like one or two back as we expected as a boolean true false then we sort group and We can even do Something something else we can also go and push every release we find in our group Into a new list with a price and the product That's basically very similar to JavaScript post.  So there's really a lot more operators And just I can't suggest if you if you fend the application framework is probably useful Just go to MongoDB documentation is very well written. It has a lot of examples. It's it's it's quite easy to get into and And one more it's a variable operator It's it's it's a map operator and as you can imagine it's basically the same as a Python map And what what do we do here? We're getting the ratings count which is actually how many users have given some stalls to the product one we scraped the data And we want to adjust it a little bit because Our management is our back and we need to make it a little bit look at a little bit nicer What we don't really want to do but it's just like a good example So basically we can pass in to a dollar map and input the ratings count As value and then we can just reuse the value on our list And we just like add 10 to each and every object in our value We find in our list and then it's applied Then another thing which is not probably obvious We cannot use some operator on an on a list like we can do really it's really handy in Python Yeah, we have to unwind first. Yeah, so basically for each and every value in our list We have mangled with we unwind it to a new document and then we can do a simple grouping as we done before and Yeah, and there we go Which brings us back of course to the next thing You can also do map reduce in MongoDB and how many of you guys work with map reduce or who knows map reduce Yes, and who actually works a lot with not map reduce Okay, so bring everybody up to speed Map reduce basically. It's a really simple concept. We have all these documents We map them map them is basically we just go through and we find key value pairs Which is actually In our word example, we find to find the most popular words in our release titles And we just emit them as tuples as you can see to the reduce phase Which is run by the producer in our example. It will be just like sum up the counts It's really really really pretty easy operation Basically, we just will use our name operator and You might wonder why would we use map reduce in MongoDB? So because we have this great aggregation framework. You've seen substrings we can so do so many things so what's the point actually being in using map reduce and and It's basically for most for most of the time you can work with the aggregation framework in most of the cases It's faster. It's more accessible. Thank you and But however map reduce gives us more power because you can actually pass in JavaScript there And then you can build much more More complex queries and for example for our example with splitting up The release titles in each in words to count them was This could be quite challenging in the aggregation framework. So Let's do it. So Okay, this is map Okay, let me show you a little bit more. Okay, um, okay, this fits. Okay No, sorry, this is our map and So for memproduce we're just using from be some import code, which we can just pass on Text and this is the JavaScript function and what's this is JavaScript function to it basically stores the name of the the the info which is stored in the name of Attribute it just stores it and splits it. It's it's a really simple operation and then it just we just check for some Punctuation and stuff remove it. It's probably it's not the best way to do this It's just like for the simple example and if we actually find the word we emit it so basically if there's something like teenage We emit teenage one and if the album is called teenage dreams We can do also emit frames one we send it to the reducer and the reducer has really simple code here we just take all the keys and Basically just count just do somehow how often did the key would actually appear from our emitter and Here we get a result and This is some and now we're going to do a little bit more because I'm I'm want to remove stop words Which is not really part of the aggregation framework, but just to make it a little bit nicer That's why I've added the natural language kit remove stop words and this is like the most popular words in Katy Perry and Taylor Swift's albums so you see they probably have a younger audience with dream and teenage and one and boys and Fearless and speak Kissed and stuff. So, um, yeah, it's really it's It's really easy, of course would take tombs to wood Unfortunately, we don't have enough time left We could also run this operation across the complete data set and to see what's Basically the most popular words in album releases being sold at the iTunes music store so To finish arm. I want to give you some more best practices and tips you can Use with the aggregation framework. First of all database think about your indexes Especially if you do queries on them, of course, if you have a huge data set and you don't have an index MongoDB has a collection scan and if it's a slow computer It's of course taking time and probably frustrating for you Think about probably getting your data set your database to to your RAM You can just touch commands in MongoDB, which actually do something like yeah similar to Unix touch You touch it and then it fills up your RAM As much as possible as masters will ever get from the system to store data You can work with life and in in memory You have to mind that the result can be only like 60 megabytes because that's the you that's the maximum We can store in a BJ's and document, but I mean like 60 megabytes is still huge Pipeline operation has also a limit of 100 to be but you will hardly ever sounds not much, but you hardly will ever really hit it. Um On your queries, you can improve your queries up front. There's this nice Oh, sorry for the break here. That's um a nice explain operation, which will basically give you information What would MongoDB do when query doing your query? Um and you get some results You see how many documents scanned if indexes were hit if it was all index and then you can really go and say, okay I can really optimize all my work with just like introducing a new index Hardware is of course really important, especially run more is better. That's really simple equation here Mind the disc performance, of course SSDs and cloud computing makes it really easy and yeah, and You can also think about working about and with a dedicated server in case you have something like a replica set and a right heavy Database So you can also say okay, just do another copy and work locally and do your aggregations without Having to worry about if you have a lot of traffic and your database and the last flight is some useful resources, of course as I mentioned MongoDB is Has a very good documentation. It's pretty updated by Mongo as well And I also want to mention Asia Kamski are she works for MongoDB also as a trainer And she has always like awesome tricks and tips Here we go We don't quite have enough time for Q&A So if you want to ask Alexander questions, then try to find them instead. Yeah, I'm around just ask any time. No problem