So yeah, I'm Yael. I'm an investigative tech reporter and I live in Phoenix. I've been writing about technology for about 11 years for a bunch of different sites, including the Intercept, Motherboard, Wired, Wirecutter, Arts, Technica, Consumer Reports. And you can find me on Twitter at Yaelbrides. So about four or five years ago, I started writing about data brokers, which is a bunch of sites that collect information about people. Basically, everything you do online leaves a paper trail, whether you're using a search engine or making a purchase. And even stuff you do offline, like getting married, buying a house, it all kind of leaves a paper trail. And I found out that these sites that I hadn't even heard of were just sharing this information and selling it to other sites and creating data profiles. So that's, so I became interested in that back then and started looking into ways to kind of try to mitigate that a little bit or even understand the risks of what information they have and what they can do with it. So there's basically three different types of data brokers. There's people search sites, which is when you plug in somebody's, maybe their name or where they live, their email address or just any information about them.  Sometimes it will show you like home addresses, other addresses, relatives and that kind of information. Sometimes it'll show you different social media sites that they use under different names just across the internet going back like decades. And so those were the ones that I found personally the most concerning. But there's also marketing sites and a lot of the marketing data brokers will create a profile of people based on information that you search for or purchase. And they do this to be able to sell kind of groups of names of people that are interested in certain products or maybe fit certain profiles. And it can be kind of like a lot of people that seem that think that it seems pretty harmless because they're just you know, giving you better deals based on information that you're interested in. But it can actually be kind of nefarious because sometimes something that you're searching for will make you look like you're higher risk or like you have a medical condition and it can cause you to have higher rates for stuff or not be able to kind of access the types of services you needed, the prices that you would normally get. And then some people just find it kind of creepy. There's been like Facebook and some other sites have allowed you to see which kind of groups you're placed in. And it's something that some people find uncomfortable with are uncomfortable with. And sometimes information isn't accurate. And it's not always easy to kind of get it removed because like people are selling, they're not just selling the raw data, but algorithms that they created based on that data. So they'll put you in these categories, and they'll sell the actual categories. And so it's easier to sell that. And it's difficult for the people who are placed in these groups that they don't even know about to kind of be able to pull that information back. And then the third type of data broker is risk mitigation firms, which is probably the least harmful. They're basically checking for fraud. But it can be harmful sometimes if say you're living in an address that's associated with fraud and you're not able to make purchases that you want because they're incorrectly associating you with that address. So that is the third type of data broker. And then some of the risks that have come up aside from people just thinking, it's creepy to be marketed to based on these categories. And also some of the information is incorrect. Some of the other risks is identity theft. If people can find that information about you, it makes it easier for them to try to steal your identity. And also like stocking, people are really concerned about people being able to find their contact information, everything that they've ever written on any social media site, going back decades, and their home address. And that's probably the emails that I get the most from people who are concerned about this is they're worried about people being able to find them, find out where they live or their phone number and that kind of thing. But actually another risk that I haven't talked about is data breaches. So like even if the material isn't available publicly, sometimes people can just, even if you've opted out of having it displayed publicly, which we'll talk about optouts a little bit later, there's just been these exorbitant data breaches of some of these companies, where they're just able to get tons of information about everybody and share that. So that's something else to worry about because we saw like with Equifax and we've seen with Axiom and other companies that sometimes they don't secure the information to the level you want. So basically they're collecting information about people in many cases without their knowledge, let alone consent, and they're not keeping it insecure in some cases. So there are some restrictions for what people can do with information. And that's mostly based on credit or consumer reporting agencies. So if you have a credit report and people are using that credit report to decide whether to give you a job or to give you credit, you can't you have the ability to access and report and correct any errors. And the companies that are trying to get that information, they can only do it like in some cases, they need you to sign up on it, or they can only access it in really specific circumstances. And there's also another restriction is the Federal Trade Commission Act, the FTC Act has Section 5, and that prohibits unfair or deceptive acts. So basically if a site is making misleading statements or causing injury to you, and you can prove that then the FTC can take some action. But it's really hard to kind of prove harm. And there's been some losses with the companies that have done this. So it's really hard to prove that like this person got my information from your site and then did X. So there are some restrictions, but they're just a little bit more limited than people would like. And one of the more frustrating things about covering this is that there was this amazing report that the FTC wrote in May 2014, 110 pages, and they had all these suggestions for what data brokers should do for improved transparency and accountability. They said you should share with people what data you're going to share and with who. They should be able to opt out. There should be a single portal that they can opt out nationally. And they had some other ideas and they wrote this great report and none of it was implemented. And some people thought it would be implemented after the Equifax data breach, but it wasn't. So the report's still there, it's still relevant, and there hasn't been any national legislation in the US based on it. So then that leaves it to individuals to try to figure out like what am I going to do about all this information about me that's online. And you can try to opt out of some of these sites for the marketing sites. It's a little bit easier. You can freeze your credit. So basically like companies that want to give you credit can access information in your credit report, but not if your credit is frozen. So that's something that you can do.  So it can be a little bit of a hassle. You can file an FTC complaint if you do find that a site is misusing your information in a way that's misleading and harmful. And then just being careful about giving out information. But that's again, that's really challenging because you leave a paper trail everywhere you go. A lot of this information is public. Like voter in many states in the US voter records have your mailing address and those are public or easy to access.  So there's only like so much you can really do to avoid giving out information. And then for people search sites specifically, and that's what a lot of people contact me about because they want to make sure their address isn't online, especially if they're in a public facing position or maybe they do activism or some kind of reporting or something like that where they have people that don't like them online and they don't want them to know where they live. And you can actually remove those.  And it has a bunch of information on how to remove your data from these different sites. The only problem is that it's tedious and time consuming. And sometimes it reprilibrates because they're pulling that information from public sources. And, and they'll keep doing that. So you have to repeat this process. You know, every maybe three to six months, probably. You can also pay somebody to take it down for you. And there's different organizations to do that. I've used Delete Me and Privacy Doc and there's a few others that will do that. But it does cost money and it's not completely comprehensive. Because some sites will like tell you that you can take your information down only if you do it yourself. They won't let anybody else do it on your behalf. And so, and you have to kind of go through and check to make sure it's actually being taken down sometimes. And so like, for example, my list of of takedown sites is more comprehensive than Delete Me. So even though I've used Delete Me, I'll still go through and try to find my information on sites that haven't taken it down. And they're constantly changing like they're they buy each other out or they change how you can opt out or they have different sites. So when I update my list, like I check this every six months and I'm always it's kind of like a like a game to try to figure out like oh here's the new way to opt out because they don't make it easy or friendly for people. And not all sites will even let you take take your information down. There's also like some and you'll talk about this more but there are some states that do have some local laws that allow you to make this a little bit easier for you to do that some people in those states can take advantage of. Oh, yeah. And there's one other thing about this too is that if you're if you do find your address that's posted on on social media or in a search engine, you can get it removed. You can file a complaint on Twitter and they'll usually take it down or on Facebook. And then in search engines, there's actually like a couple of links that you can use to try to get it removed by just telling them this is my personal data I want to take it down. So it might still end up on like a shadowy site that posts these things but it won't show up in a search engine when people Google your name, which can be can be helpful. So I talked a little bit about the biggest data broker opt out list that I've maintained. I try to update it every six months or so. A lot of these sites will ask for more information before you opt out. And so what I do is I make sure my name is already on there before providing more information to them. There's some that charge money for access to removal, which I don't personally I've never paid for that because I'm just not going to pay somebody for it. Like what if they put it back up again? Some of them make you actually write a letter and send it in by mail. Some of them have you use a fax machine. Some of them will require more information, which personally like for me if my goal is to get my information taken off of the web then I'm okay with providing some information. But I'll cross out anything else that they don't need. Like for example, if they ask for a driver's license, I might send them a copy of my driver's license, but I'll cross out the ID number and any other information because they are reasoning as they say they want to make sure to verify that you are the person whose information they're removing is actually asking for it. But yeah, we do have this list and try to make it a little bit easier even though it is still like a time consuming and tedious process. And these are some of the links of articles that I've written in that are online about data brokers. The first piece by the data broker call for transparency and accountability from the FTC, which is that 110 page report that wasn't followed. But I do think it has like a blueprint for what states can do and what you know the federal government could do in the future if they decide to do it. And then the opt-out list, I have it on both Google Docs and GitHub. And then there's this really great article about getting that Maria has sent off wrote about trying to get your name off of people search sites for consumer reports. And they also link to a sort of how-to guide for deleting information that I wrote, which links to the opt-out list but just provides more information about the process. And yeah, I'll be available to answer questions. Hi everyone. My name is Ginny Foss. I'm a social entrepreneur and I work at the intersection of technology and policy. I've worked on a number of different tech policy projects, but recently I've been laser focused on data privacy and consumer rights. And so I'm really interested and excited to be here with you all today talking about tools for making data rights meaningful for everyday people and how kind of tooling and products can fill the gap between what policy provides and what everyday people are realistically able to take action on. So the big question that I'm focused on right now is how might we meaningfully expand data rights to individuals, to everyday people. And in the US there's this new tool that makes this question particularly compelling or possible to explore. And it's a piece of legislation called the CCPA, the California Consumer Privacy Act, which is a really exciting milestone for us here in the US because it's the first comprehensive commercial privacy law in the States. It was signed into law in 2018 and implementation began this year in 2020. And it's making a lot more possible as far as individuals' ability to meaningfully exercise their data rights. So what is the CCPA and why should we be excited about it? So there are a few main privacy rights that the CCPA extends. The first is the right to know the personal information that a company has on file about you. And practically that means that individuals should be able to retrieve a copy of the data that a company has on file. There's the right to delete personal information that a company has on file about you. If you don't trust this company or don't want them to have your data anymore, then you have the right to request that they wipe it from their databases. And then finally, there's what Gael was speaking about earlier as far as the right to opt out of the sale of your personal information by companies. I think most consumers or everyday people might not be aware of the extent to which companies sell data about them. And under the CCPA, any individual can opt out of the sale of their data such that companies aren't able to monetize on the activity and personal information that everyday people are sharing with platforms and products they use. Or platforms and products that get their information some other way. They may not even be a user of those products, as is the case with data brokers. So this seems great, right? Like we have this great new privacy law. We have all these rights. What's the problem? Well, the problem is that these rights practically are actually just very challenging to use. Gael mentioned the list that she has, this big ass data brokers opt out list, which every six months she's going through and for these dozens of companies, she's updating how one might opt out of the sale of their data by these companies and even maintaining that's a challenge. But then as an everyday person who's not a privacy expert or privacy professional, who may have never heard of a data broker or never really understood what it means to opt out of their data being sold, maybe they don't even know that data is sold about them. And so there's this big gap between what policy allows and what everyday people would reasonably be able to do if they cared about their own privacy, or even just how much they would have to know in order to take action towards their own privacy. Consumer Reports published just last month at the beginning of October, a study in which they recruited a few hundred volunteers to start issuing request to companies, request to opt out of the sale of data, request to know the data companies have about them, etc. And it turns out that everyday people have a really hard time using these new rights because for the most part it's still really early days for CCPA. Companies have set up really different processes for handling these requests. They don't always make it easy to submit the request. They don't always know how to process the request. And so for any person off the street, these privacy rights are sometimes a little bit more hypothetical than they are actionable. So what tools might exist to help people exercise their privacy rights? One initiative that launched a few weeks ago that's really exciting is this global privacy control initiative, which works with a browser signal. So the idea is that you could create settings in your browser about your privacy preferences and that any business whose website you visit would have to respect those privacy preferences that you have designated up front. This is a consumer product. It's a technology. It's a tool that lives in the browser. It can and will be very helpful to a number of people. But it doesn't account for the fact that there are all of these industries and companies that are selling personal data. And everyday people don't even know that these companies exist, that they have their data, that they're selling their data. Data brokers obviously are, you know, any company listed on the California data brokers list is a company where probably most people off the street don't realize that that company exists, that company has their data, etc. And so a solution like global privacy control starts to break down when you consider companies that maybe aren't consumer brands or that just aren't popular in consumers awareness. So what else can be done? There's a provision in the CCPA called the authorized agent provision. And the idea is that, you know, there are these rights extended in the CCPA, the right to know, the right to delete, the right to opt out of the sale of your information. But practically, since it's hard for everyday people to take those actions, the CCPA says that an authorized agent can make these requests on behalf of consumers. And an authorized agent is pretty loosely defined in the CCPA. It's any third party. So it could be an outside organization, it could be my mother, it could be, you know, an individual I know who I would trust to make a data request for me. But, but because of this provision in CCPA, essentially, there's an allocation for somebody who's more of an expert to make requests for you. And that's what I'm really interested in exploring right now. So right now, I'm working with a team at the Digital Lab at Consumer Reports, and we're working on trying to create authorized agents that take actions for individuals on their behalf in risk, in regard to their data. So just a few weeks ago, pretty recently, we launched our first experiment of being an authorized agent. And so what this experiment is, is we have recruited some volunteers to help participate. We have 110 consumers who have signed some legal paperwork. And the paperwork says, I designate the Consumer Reports organization to act as an authorized agent on my behalf and to submit data requests on my behalf. We are now currently representing these 110 consumers before 21 companies. So we selected this list of 21 companies for the pilot based on a number of criteria. We wanted a mix of companies that were popular consumer brands, as well as major data brokers, as well as a mix of industries. So we have some retail brands, we have some food and beverage brands, we have supermarkets, we have e-commerce. We really tried to, it's only 21 companies, but we tried to get as much diversity as possible in the sorts of business models and even consumer awareness. And over the past few weeks, we've issued 210 opt-out requests to these companies. So each company in the cohort has received 10 requests from Consumer Reports saying we are opting this consumer out of the sale of their data by your company. So essentially what this means at a high level is that we as Consumer Reports have created a technology by which we are acting as a virtual assistant to individuals who have desires for their data, but maybe don't know enough to act on it, maybe don't have enough time to act on it, maybe just are too busy to act on it. We're doing the work for them and making their data rights actionable by taking on responsibility of representing them to companies over concerns related to their data. And it's been really fun acting as an agent. And I would say in general authorized agent is a pretty nascent part of CCPA. There are other parts of the law that maybe have more in common with GDPR and other major privacy laws around the world. And so companies have a little bit more experience with how they might implement those parts of CCPA, but authorized agent is really singular to CCPA so far. And so when we submit these requests to companies, there is often a little bit of confusion about what this request is, like what part of the law is referring to how to take action on it. And so part of this pilot also has just been developing a better understanding of where industry is and starting to be able to implement the requirements of CCPA. And frankly, even beginning the conversation with companies or DSAR providers who are data subject access request professional groups that sometimes process requests on behalf of companies, we're starting to have a conversation more broadly in the industry of what's currently possible under authorized agent and what should be possible. And kind of we're all learning and exploring together what this often overlooked part of the CCPA privacy law might mean for individuals and for corporations who will need to be prepared to receive these requests. So this is what an agent request looks like in case you're curious. The CCPA really doesn't specify very many requirements about how authorized agents make requests. So our team has created a request format that seems to be working with the companies that we're reaching out to. But in order to be an authorized agent, you have to have the consumers information on files. So all of the consumers who have signed up for this pilot have provided their email and phone number. And then we've also verified their email and phone number so that we have confidence that the consumer is who they say they are. We've asked every participant in this pilot to sign a permission letter. And this is it's kind of a form legal document. And the permission letter says I, Ginny Fos is an individual authorized consumer reports to act as an agent on my behalf. And those letters are signed and dated and provided to companies when we make these requests is evidence that we have, you know, we are not fraudulent and that we really are representing the consumer. And then finally, because we're an organization that's making these opt out requests for everyday people, we need to prove that we are organization in good standing that we pay our taxes, those sorts of things. And so in the agent request, we also provide a certificate of information of consumer reports, which is just paperwork that demonstrates that we're an organization with license to operate in California. So again, we've sent out 210 of these so far to 21 different companies. And seeing the responses back has been really interesting. In general, what I'm really interested in right now and what the team at CR is working on as well, is this question of how might we meaningfully extend data rights to individuals. And we're very early in building tools that do that from the global privacy control solution I mentioned earlier that is the browser signal to this authorized agent idea of becoming an assistant that does work for everyday consumers who may not have as much context or time to deal with their own privacy. So these are just kind of our first two ideas. And so as we enter into Q&A and start to have a broader conversation, I'm really interested to hear from all of you about other ideas you have for tools that will help everyday people meaningfully exercise privacy rights that they are afforded by law, but that maybe are not practical to access. And I provided links to some of the materials I've referenced in case people want to dig deeper and understand some of what we've been working on. So yeah, that's everything from me. You can find me on Twitter via email and look forward to hearing your questions and having a broader conversation with you all about this as well.


Good morning everyone. I am Akanksha your moderator for the day. Next up we have Sanjay Aurore and Uli Draper as data scientist from Red Hat over to Sanjay and Uli. So yeah, I mean I'm Sanjay this is a joint talk with Uli. We are both within the AI center of excellence at Red Hat. Can you guys hear me at the back? This is mostly Sanjay's work so I just delegated to the Indian in the room. We have given a couple of these talks at various conferences on the feedback in general was good but it very quickly got over people's head if they're not into this topic. So Sanjay tried to really really hard to make it tangible so that people can take away from this so just don't assume that after that you can put on your resume data scientist. Okay so let's get started. So this is a talk about unsupervised deep learning as you all know. So what is unsupervised learning? So very simply unsupervised learning is finding patterns in unlabeled data. So the next question is what is unlabeled data? It's data that's not labeled. Label data is data where you have some annotations. So as an example if I give you a bunch of images and say this image is a cat, this image is a dog, the dogs and cats are labels or I can give you a bunch of log files and tell you which ones are anomalous and which ones are not. Those are labels but in real life most of your data is not labeled. Imagine Red Hat, imagine any company, forget companies, imagine the data on your phones. That's not labeled. You have a ton of pictures and you just collect them. So one of the holy grails in machine learning is figuring out how to do unsupervised learning which is learning from data that doesn't have labels in an efficient way because if you can do that then you can actually use the data you collect otherwise you have to manually annotate it then feed it to what's called a supervised learning algorithm and use it. So you can imagine so other companies which you've seen this new economy on some they wouldn't exist except for unsupervised learning because they're all collecting gigabytes, terabytes, exabytes of data. No one could annotate that. So it's all about unsupervised learning. Supervised learning is something, well it's a luxury which hardly anyone has. It's all about unsupervised learning. And so even if you don't do machine learning there's some examples that you might have seen or at least heard of.  So I give you heights of people I randomly sample on the street and you plot them, you make a histogram, how many people for each height and what you'll see is something like a bell shaped curve right so you see a peak and then it falls off. Now technically that's a normal distribution or a Gaussian distribution and you can fit it. You can say what's the standard deviation? What's the mean? Can I compute that from the data? And that's the simplest possible unsupervised learning algorithm. You just measure something in the data set you have. Of course you can make it more complex. You can say you know real data doesn't have to be something that's normally distributed. It can be something more messy, something more complex. And so you can get a mixture of Gaussians where you are taking weighted means. You can make it more complicated. But the fundamental idea being there are no labels there. It's just the raw data. Another example is clustering. Clustering is give me some data and put it in groups, right? Make these disjoint groups that don't overlap and there are no labels there. No one tells me what cluster you should belong to. I just get the data and I tell you these data points are similar. That other group is similar and they get an ID each. Yet another example is that of finding latent or hidden dimensions in your data. So what does that mean? The simplest example I can give is imagine images of dogs and cats again. I'll use that example a lot. And you have 100 by 100 pixel images. So you have 10,000 pieces of data for every image. But as a human being if you come to me and say what's that? I won't say 10,000 values and read off the values. I'll say that's a dog. That's a cat. So the dogginess or the cattiness of the image is a hidden variable, right? One can make it slightly more complicated and say it's a dog where the image was taken during the day or a cat where the image was taken at night. So what animal it was and when the picture was taken are two latent variables. But generally humans are very good at doing that. We are very good at summarizing information and transmitting that to each other. Computers are not. So if you have algorithms that can find these hidden variables and maybe you can interpret them in physical ways, maybe there's some complex variables that you just write down. But that's an example of unsupervised learning. And I'll throw some buzzwords out there. The simplest possible one is principle component analysis where you're looking for these linear subspaces. Don't worry about that. Just know that it exists. And then there are generalizations called manifold learning, very abstractly where you're learning nonlinear generalizations. But again, the techniques don't matter. It's the idea of extracting hidden variables. Yet another thing you could do if you have unlabeled data is you can say, can I use supervised learning that Uli was talking about techniques for dealing with data sets that are labeled. So the simplest example is linear regression, predict the price of a house from variables for the house, location, age, and so on. Can I use those supervised techniques to solve an unsupervised problem? And here things get a bit more hacky, right? You could say something like, okay, give me all the data and I'll predict one of the features of my data set from the other ones. It's not really an annotation. It's not really a target variable. I'm just making it up. But the idea is if I predict maybe the age of the house from the rest of the variables, I can then use the learn model and inspect it to learn something about the data set. And so that's the technique that's often used. Another one that's used is the anomaly detection example. You have log files. We have a lot of log files for some system, open shift for something. And I want to find which ones are anomalous and which ones are not. Now it's unsupervised. No one gave me labels on these log files. I can label all of them as zeros, right? I picked some arbitrary ideas. Indeed, this is class zero. I can pick some log files from a completely different system and label all those log files as one. And then I can say let me train a model to distinguish between log files of my system versus the other system. And once that my model is very confident that zero are okay, the less confident it gets, the less okay. So this is sometimes called semi-supervised learning. But again, the crucial part is you can use supervised learning techniques to do unsupervised learning. So what is unsupervised deep learning? That's the title of the talk. So what's deep learning? Deep learning is, I guess the only thing you could say is it's the art and science of training neural networks or deep neural networks. And the art is important. It's not always well defined in practice. There are many tricks and heuristics. But you can think of a neural network as a machine or a box that can learn any function or any reasonable function very well to arbitrary accuracy. So if you give me images and labels, the function that imagined writing code that says, here are the pixels of an image that has a cat. Let me predict that it's a cat. Imagine writing that code by hand for the pixel values. That's messy. But a neural network says, give me enough examples and I'll tell you what the function is. And it gives you the mathematical function. So if I have a device like that that can learn arbitrary functions very well, how can I attack unsupervised learning? What can I possibly do? So hopefully I'll get a world record entry from this. This should be the shortest neural networks course. But maybe it's not. And I just want to give you an overview because the core ideas are always very simple, right? The details are not but the core ideas are simple. So what's a neural network? Well, what's a model, right? So in this case, we are looking at supervised learning just for these two minutes. That's that blue box in the center can be any statistical model. It has one job in life takes an input, which is x, x can be pixels from an image. It can be tokens from a language, words or characters from a piece of text can be any features. It can be the housing data that we were talking about. And there's an output, which is the Y, which is what I want to predict, right? That's my target. And that depends on the problem you're solving. If you're doing fraud detection, it's is my transaction fraudulent or not. If it's house prices, it's the price of the house fits an image data set. Maybe it's what the image contains. And the neural network takes a specific choice when it comes to building that box. And one way to look at it is a neural network is a very simple iterative procedure. It takes a vector by vector, I mean a list or an array of numbers, which is your input. So for an image, a vector would be all the pixels listed in an array. That guy comes in and a neural network says, I'll just map it to another vector v1, then v2, then v3, I'll iteratively keep mapping this vector. And at the very end, I'll predict a number. And so the question is, what are these sequentially mapped vectors? You make a very simple choice, right? You make one of the simplest possible choices. You say, I don't want to make this messy. When I go from one vector to the next vector, I'll take the previous vector multiplied by a matrix, a algebra object, which has one job in life to map vectors to vectors. That's what it's constructed for. So you map a vector to another vector. If you look at that equation, that's the m, m stands for matrix, v1 gets multiplied by the matrix. You add a constant called B to that matrix. And then you need to do one more thing. This is not enough. You need to feed that to a nonlinear function, right? And there's a technical reason for that. But basically, it's this simple procedure that I repeat. And again, for the rest of the talk, just forget about that equation. What's important is that there's a matrix in that equation. And there's that B vector in that equation, which is not dependent on your data comes from outside your data set. And those things, the matrix and the B's are called weights. So if you ever see a talk on deep learning or AI or anything like that, they'll often say weights and biases. That's what they mean. Your neural network takes the data, but it also has some parameters inside it that decide what this mapping from vectors to vectors would be. And the whole point of training neural networks, right? If you if you wanted to basically summarize all this buzz about, you know, distributed training and GPUs and this and that eventually all we are doing is finding those weights. That's that's the main goal. What are the weights that I should put in there? That's the whole point of training. So of course, I skipped a lot of things. And I'll go through these quickly just to mention them, right, just to plant a seed. So we didn't talk about the exact layers. So these blue bars are called layers. And there are specialized layers, depending on the kind of data you have, you have ones for images called convolutional networks, you have ones for sequential data called, so there are recurrent neural nets, vanilla ones, LSTMs, GRUs, lots of buzzwords. But you can basically have specialized layers for special data. The other thing is how do you actually find those weights, right? We know we need to find those weights. But how do you actually do it? The dominant method is something called back propagation and gradient descent. Again, we won't get into that we don't need to get into that. The third piece, which is important is something called the loss or cost function. And the idea is very simple.  And maybe your prediction is it's a dog, but the actual label is it's a cat. And I need to be able to measure the difference between your prediction and the actual label. That is measured by something called the loss function. So it's very simply if I give you two numbers, right, if I give you four and five and say what's the difference, you can subtract and take the absolute value, can subtract and take the square. That's a loss function. And the goal of deep learning training is to minimize that cost function, to minimize the difference between your prediction and the actual values. And then there's a lot of theory and heuristics that of course we won't get into. One thing is so these kind of things are examples of what could be done. They are just there because people found they are useful. And most 99% of all the people are using this just copy the useful things. It's not the only way. They have merits and millions of different ways how this can be done. They are just a couple of things like back probation, for instance, as the requirement that it's, you have differential functions. Well, this puts a certain requirement on this, but this does not mean it has to be exactly like this. It's just that if you want to use existing software, you follow this. But if you are clever, you can come up with something yourself, it might work just as well. And so at that stage, you can come back to the original question. If I gave you this thing that maps vectors to vectors sequentially and finds the weights so that it can make accurate predictions, how do I solve my real problem unsupervised learning?  It's mostly trying to come up with techniques to deal with unlabeled data. How can I possibly do something useful with it? And so when I started writing this talk, I wanted to be ambitious and I had two different techniques. One that we'll talk about is called autoencoders. The other one, they're called generative adversarial networks. That would be a two hour talk. So we are going to eat lunch. So we'll just talk about autoencoders. And I think you capture a lot of the essential bits of using deep neural networks for unsupervised learning. So autoencoders have the same problem that we have in unsupervised learning in general, which is you are giving me some data, log files, images, something structured. I don't have a target to predict. So I can't do supervised learning. So what should I do? And you make the easiest choice. You say if you don't have anything to predict, predict the input itself. And you say one, like you're giving me four, you want me to predict four? Like, well, why is that useful? Right? So in this picture, we are saying, take that Y, cut it off, predict X back. Or in slightly more mathematical language, learn the identity function. Identity function does exactly what I said. You give me something, I'll give it give it back to you. So let's see how this would work, right? Okay, you say, I'll trust you for a second. Let's learn this identity function. This is useless to me. You're going to predict who wants to build a model that predicts the data I put in. But let's see. So let's say, you know, for, as an example, let's say X, the input is an image. Let's say it's a 100 by 100 image. So you have 10,000 pixels. And what you do is you're doing the sequential mapping to vectors, but I get to choose how big each vector is. So the first one I'll choose to be a thousand smaller than 10,000. Then 500, then 100. So it's getting less, lesser and lesser. That's where the bars are getting narrower. And then we'll reverse the trend. We went from 10,000 to 1000 to 500 to 100, back to 500, back to 1000. And then X is 10,000. And so what you're doing is you're taking this vector, the pixels, and you're squeezing it. And then from that squeeze version, you're coming back to the original one. And the central part, the squeezing part acts as an information bottleneck. Why? Because I had 10,000 pieces of information. If you give me 10,000 memory locations, I can of course give it back to you. That's easy. But if you force me to only store 100 numbers or 10 numbers, then I'm forced to pick out features of those images that are actually useful for reconstructing that image. Because I have to give you back the image eventually. But I have to reconstruct that image from the point where I have only 100 numbers. So those 100 numbers act as a compression or as latent variables. And that makes it hard to learn this identity mapping, right? You're compressing something and re-uncompressing it or decompressing it. There are many other ways of imposing this information bottleneck. I might mention them at the end if you have time. But this is one of the most common simplest ways of doing it. So how do I actually use this thing? Right? You say, okay, fine. You squeeze this, you reconstruct the image. What do I do with this thing? One thing you can do is you can go to that red bar in the center. That's your compressed representation. That has only 100 numbers for that image. So you give me an image. I'll pass it through this thing. I'll go to that red bar and see the 100 numbers. And I say, I don't need the image. I'll keep these 100 numbers. That's my proxy for that image. And hopefully those 100 numbers are something meaningful, right? Maybe if I give it lots of images, it will say, oh, this is a cat. This is a dog. This is a plate and so on. Just a technical point. When you do something like this, technically you have x going in, which is your input. What comes out is x underscore recon, which is reconstructed x. It won't be the same as x in general. But an auto encoder tries and minimizes the difference between the input x and the output x recon. Or in the language of deep learning, you have a loss function which measures the difference between your input and your output. And it tries to minimize that. So that's what those equations say. They say there's a loss that depends on capital R, which is a reconstruction loss. And capital R literally takes the two images, subtracts them, squares each pixel value, and takes a sum of the mean. And you minimize that thing. So in all this is fine. How do we actually use this practically? Right? So an auto encoder in principle is something very simple. So again to summarize it's a neural network that squeezes your data in, then reconstructs it with the idea that at the squeeze compressed part you get something, you capture some salient features of your input. So let's look at a few applications. Let's look at four. The first one is anomaly detection. This is a big thing for any company, especially for us. Log files, telemetry data, I'm sure there's like all kinds of low level signals from the CPU from all kinds of systems. Can I find anomalies? What's an anomaly? Well, what's an outlier? There are many definitions. One could say an outlier is if you give me data, what is statistically unlikely in my data set? I declare that to be an outlier. It's unlikely. A better definition, and I forget who came up with this, is when you have data that's generated by a process. Log files are generated by your system. When data is generated by a different process, that data from the new process is called an outlier. Something changed, and I want to capture that. What's an anomaly? An anomaly is a judgmental data scientist. You say I have an outlier. An anomaly says it's negative, but I can't really tell you if an anomaly is negative or not negative. My job is to give you outliers, you as the domain expert, tell me if it's anomalous or not. How do you actually use an autoencoder for anomaly detection? What you do is you say, give me data that you know is normal. You just make sure you carefully look at it. There are no anomalies there. Everything's okay, and I'll train my autoencoder on that. So again as an example, give me images that were of images of cats and dogs, and I'll train my autoencoder on this, and I get very good at reconstructing these images. If you give me this image, I'll feed it through my autoencoder, and my autoencoder is really, really good because it's trained, and it will give you that image back with minor distortions because of the compression. But if you give me images that are not dogs or cats, something else, then my autoencoder says I never saw this thing before. It's what's that, a plane? I've never seen this, so it can't reconstruct it very well. And so if I look at the difference between the prediction, the output, and the input, for normal data it will be low. We'll say I know how to reconstruct this. For abnormal data it will be high. So I can use that loss, my error in reconstruction, as a metric for anomaly detection. I can say the higher it is, the more likely you are to be anomalous. So I'm just making sure that the definition of anomalies, of course, context sensitive. So, for instance, one other use of anomaly detection is in spam crates. So if I train my system on spam email, no legitimate email I will ever receive will be written in Chinese. So, but if you're actually speaking Chinese, it will have these kind of things. So my reconstructed emails will always only be Latin characters. But for someone else in a different context it will be completely different. And this is, these are two papers, so once from 2007, once from 2018, and the one on the, I mean they're both doing the same thing. They're both taking time series from some systems. And they're taking the values in the last 10 minutes, right? So five minutes, 10 minutes. And they're reconstructing that time series. So you give me the last five minutes worth of measurements of your time series. I feed that to my autoencoder. It compresses it. Then it uncompresses that and tries to predict the input again. And when it cannot reconstruct that input, it says something's wrong. When it can reconstruct the input, it says you're okay. And so if you look at the two plots to the right, the smaller ones, one says normal, one says anomaly, the x-axis is that reconstruction error. This is for a high performance computing system, an HPC system, and they have 166 metrics. So they have fan speed, co-loads, temperatures, all kinds of things. And they're trying to reconstruct that data. And what you see in the central plot where it says normal is that the x-axis which measures the reconstruction error is pretty low. All the values are less than 0.2. The y-axis is the number of counts, number of examples. Then they artificially go and inject anomalies in their system. And if you look at the plot on the right, the histogram on the right, suddenly your reconstruction error increases. Right? So of course there's some overlap between the two histograms, but you get a lot more points higher than 0.2. And that's the fundamental idea. In a real life system, you keep running your autoencoder, and then as soon as I see that my reconstruction error is increasing, that's the signal that something's going wrong. Yeah, again, also something else. If everyone who is in this kind of specific around machines, then know that things are changing over time. So you have drift usually in the data itself, which also means that the models which you're building up, so in this case the neural network which is provided as conserved by the autoencoder, will have to be redone regularly. Without that, sooner or later, everything will look like more like the right figure there. And same thing for the left plot. The left plot is the x-axis is now real time. It's 2001. The y-axis is reconstruction error, so the higher it is, the worse, the more anomalous you are. And I know nothing about routers and networking, but they took time series of BGP router updates and announcements. Apparently these were all internet worms, and routes would die and they would signal to each other that there's a new route open, something like that. And I think the spike corresponds to one of the famous internet worms. I forget its name, so you can look at the paper. But they were trying to see if they can identify these large-scale internet worms which were disrupting network traffic purely from router data. Same technique. Let's reconstruct the data and see if the reconstruction loss is higher. Another example, and this is not something that most of us maybe care about except for people who are doing deep learning or machine learning. But this is what really triggered a change in how deep neural nets are trained after 2006. So if you went back to 2005, no one was training deep neural nets, right? Because everyone who tried it said this is actually worse. So I should first define what I mean by deep neural nets. Deep just means add more layers, right? So the more layers, the more vertical bars you add, the deeper a network is. And we know from first principles from simple toy experiments that the deeper a network is, the more expressive it gets. It gets easier to learn more complex functions or datasets. If you make a deeper network. But in practice when you tried this before 2006, it didn't work, right? It would be much worse than a network with just two or three layers. And so everyone was confused. They said why is this not working? Then people figured out that well when I train these I have all these weights, right? The matrices, the M's and the B's, and they have to be initialized to some values before I start training. If I initialize them differently, sometimes the training works beautifully, right? It's perfect. Sometimes you just initialize it a bit differently and your neural network doesn't learn anything. So of course you have these two hints, right? You know that your deeper networks should do well. You know that the weight initialization matters. The next question is so how do I initialize these weights? Like okay I know it's good, like I should do it somehow. I'm not randomly going to try everything. So let's find better initializations. And so Jeffrey Hinton and his students first did this I think in 2006. They did not use autoencoders. They use something called restricted Boltzmann machines. Doesn't matter. You can use autoencoders for the same thing. And the idea is can I find these matrices, these weights using autoencoders? And one way to do it is to say give me a data set. So you have inputs and outputs. This is supervised learning. Forget the output. I don't care, right? I'll only look at the input X. And let me create an autoencoder, which is what that picture is. It's X getting mapped back to X. There's one more twist where you say the weights on the two every time you go from one layer to another one, you have a matrix. And you basically make sure the matrices are the same, right? So that T stands for transpose. Let's not worry about that. What you would do is I would give you images. You would train this autoencoder and then you would take this second layer and drop it. We don't need need that stuff, right? We just need the first part. And you say, OK, all my images have to some vector V one. Let me build one more auto encoder. And now V one gets mapped back to V one. I have some other matrix you that's OK. Let's train the second autoencoder. Take the last bit, drop it.  The weights at least are slightly meaningful. They are part of an autoencoder at the very end, right? Maybe after 100 such layers, you say, let me now finally use the targets that I'm trying to predict. Let me add that in. Right? So let me add one last layer, which is I'm trying to predict whether it's a cat or a dog or something else. And that last V that weight is trained using your target. So this is called unsupervised pre training, which is greedy. You're doing one layer at a time. There are other versions of this. But this made it possible to train very deep networks because all your weights are now initialized to reasonably good values. They're not randomly selected. And for people who want to do these experiments in today's world, no one does this as far as I know, no one does this. We have other techniques again, buzzword time, batch norm, relu dropout, which make it easier to train deep networks. But I don't know if anyone has ever tried those with this kind of greedy pre training to see if it works even better. But that's it's not an application that most people outside deep learning would care about. But this really triggered this revolution in training deeper networks. This one we would, I think a lot of us would care about. There's something called semantic caching, which is you give me images log files or the stuff. I can create a hash, right? Or I can store them in memory. What I really want to do is I want to store things that are similar to each other, closer to each other in memory. That's where the semantic comes in. I want to take the meaning into account the meaning of the images, the meaning of the log files, or anything else. So can I find a mapping from my documents to locations in memory, where similar documents get mapped to similar locations. And then searching is super easy, right? If that happens, you say, find me an image that's similar to this image. I say, oh, let me just look in the neighborhood around it. And here are more images. And this was also done by Hinton and his grad student, where they I won't get into many of the details. They just took a lot of documents, right? Articles from Reuters and from news group.  And there were no labels, right? They just trained an auto encoder on this. And they took the central part of the auto encoder, the one that's compressed and said, let me throw the documents out, let me take this compressed piece. And you can do tricks so that most of those entries are zeros or ones. So it looks like a bit vector. They say, okay, this is my bit vector for every document. After they had trained, they said, let me plot these, right? Let me see where each document lies. That's what these plots are. So each point is a document. And they're projecting these bit vectors into two dimensions. After the fact, they color each dot. They say each dot comes from this topic. So the topics were never used during training. Training was just auto encoder, take the bit vectors, map them in 2D. Then they said, let's color this. And maybe surprisingly or not surprisingly, they found that documents are the similar from the same topic actually grouped together. The blues are together, the reds are together. It doesn't have to be that way. If I randomly hash it, it will be all mixed. And the fact that it could do this is enormously important for better search, right? Information retrieval. So auto encoders are great for this kind of work too. And as an example, they didn't do it with log files. With documents, all you can do, all you have to do is get every unique word in your document, create a long vector which is called bag of words, a count of every unique word. So you said this document has five instances of the word open shift, three instances of the word linux, create this vector and train an auto encoder to predict that vector back and take the compressed piece and that is your hash function. So what you also see here obviously is that there are what you would classify now as kind of application stay in your mind. Right? So anomaly detection, especially when things get complex, right? Images or log files, that's a useful application. Training deep networks, unless you're doing it, if you are interested in machine learning, you should definitely try it out. But if you're not, semantic hashing is a very useful technique, right? It's very useful to be able to do this on any kind of search space. Another one that is very useful is graph problems. So you have graphs everywhere, right? We have a actually we have a talk tomorrow on graph neural networks to graphs as in nodes and edges craft, not charts or plots. So I have this collection of nodes and edges. And I want to answer all kinds of interesting questions. This can be a social network. It can be a physical network doesn't matter. One of the interesting problems is if you give me two nodes, are they connected? So for some nodes, I know they're connected for sure. For some nodes, I know they're not connected for sure. Have no idea. And I would like to get some indication if they're connected, maybe a probability of connectedness. Can I answer that? And so one way auto encoders for that is you create something called an adjacency matrix from a graph, which is basically if you have n nodes, you have an n by n matrix, not sure you put a null. And the job of an auto encoder is to try and recreate that matrix, but fill in the blanks, right? Say, okay, this is probability 3.3.7. And here's a paper that does exactly that with a lot of experiments. And it seems to work very well. They actually do two tasks. They say, not only will I tell you which links are likely and which are not, I'm also going to some of these nodes have data associated with them, right? The nodes of their physical routers have something related to them, social networks, every one of us has data associated with us. You can actually predict data about the nodes based on the data of the nodes, of the neighboring nodes, right? So they actually train a neural network to both do link prediction and they tell you what kind of node it is. Again, don't worry about that equation to the left, but that's the basic idea. That's very useful too. So these are just four applications, right? And I have a ceiling Tommy to I have like 200 papers and in a spreadsheet with interesting applications. If you're interested, let me know, I'll send you that spreadsheet. But the idea is train a neural network that learns the identity mapping, but make it hard by squeezing the central part. And there are other ways to make it hard. And I'll quickly mention all three. So the first one is what we did, squeeze that red bit and make it hard. The other one is you not only squeeze the red bit, but you try and force most of those entries to be zeros. So that's called sparsity. You can impose that. That helps. There are many other ones. There's one where you put noise in your input and you try to recreate the out the noise. That's called a denoising auto encoder. So there's a whole slew of techniques there. You can try and set derivatives to zero. You can do all kinds of things. But at its core, just learning the identity mapping, but make it hard for it to learn the identity mapping. So it has to throw away irrelevant information. So squeeze it, throw away relevant information. And then what you see in the center is telling you, hopefully, telling you something meaningful about the data you have. And for applications, there are many more. Anomaly detection, semantic hashing, better hashing, graph problems, link prediction, people have done clustering on graphs with autoencoders and, of course, training deep networks. And lastly, so it depends on if you find this interesting or not interesting. If you're interested in unsupervised learning, feel free to contact us. There are many resources. We don't know everything, but we'll work together. But I think what's really important is in your day jobs, you all see some kind of data. There are log files, there's telemetry data, and just keep this at the back of your mind if you think that some of this might be useful. Or even if you suspect it might be useful, get in touch with us or, you know, start working on it yourself. That's even better. But some of these techniques can give very surprising results in a good way when you try them out. So that's it. The last thing is when these files get uploaded, you'll find a whole appendix on another technique. So again, if you find it interesting, skim through it. If you have questions, just email us. So I just want to add one thing to the last slide, basically. So for those who are interested in this, so implementing these kind of, actually, implementation for this is nowadays very simple. So you can just use some of the existing tools, which are Python based on still implementing neural networks that arbitrarily depends on. So that's actually not that hard. So what might help you actually is to get some illustrative example. And one of the most famous ones out there, which has been used for generations of data scientists now, is the MNIST dataset, which is just tiny little pictures of hand-drawn digits. So that's actually also not that large of a dataset. So get that, which you can find everywhere. Just type MNIST into your favorite search engines and train a network on that. The dataset by itself is labeled. So it has, for every picture, a digit to it. So, but ignore that for a time being. Feed in these tiny little pictures into the neural network, make it some architecture, but make the compressed side in the middle, make it 10. This will force basically the autoencoder to actually give you 10 possibilities, which correspond to 10 possible digits, and see what comes out of it. This will give you an impression what you can do with this and how it should work. And at any point in time, you can try to translate back the intuitive back to Wednesday in the reconstruction. You can translate it back into a graphical form and look at it. So this example specifically, I found extremely useful to understand what is actually going on in these kind of things. It's very easy to do. That's it. Any questions? Yeah, really nice talk. I had a few questions regarding the anomaly detection part. So you said that it gives you, it predicts anomalies, but what do you think about the interpretability of those, of the neural net? Basically, why do you think those anomalies are anomalies? If you wanted the neural net to tell you what parts it is looking at, to tell you that it's an anomaly, can it do it? Compared to something like classical machine learning, which kind of tells you which parameters are affecting its score and things like that? Yeah, so I think, and I mean, I would argue even classical machine learning, depending on the technique, you have to do more work to get more information out. So at the surface, the neural net doesn't tell you why it's, why the reconstruction error is lower on this versus something else. But then you can always do more careful experiments where you say, okay, I have these inputs. What happens when I just vary one of the inputs, right? One of the elements? Or what are the activations in the intermediate one? But that is almost always more manual. The interpretability is never straightforward. So it's not going to tell you this is why I think it's anomalous. All you can do is basically explore the input space close to the input you gave it and try and see what exactly is triggering this loss. But that's, I think, a general problem with deep learning. Yeah, but specifically for, I can't express it this way, so specifically for other encoders, you have the advantage that you're controlling the compressed layer. So if you managed to create an autoencoder with a very, a very strict in a layer and so on, you can look at a limited set of your examples and try to see whether you can recognize a pattern in the values of the most constricted area of the autoencoder. Oftentimes, and again, I'm going back to this amnesty example. So hopefully you will see something that, especially if you're forcing the many of the entries of the inner vector to zero, you will force that there's actually a binary relation to something like, and the neural network will not tell you what it means, but humans are extremely good at recognizing patterns. So you're looking at a couple of them, looking at the representation, the compressed representation and assign yourself the appropriate label to it. The label could be, oh, it's this derivation, it's this, it's this. So this is actually, autoencoders are better, I would argue, than many of the other machine learning techniques to actually answer the question which you just tried to answer because if you have a highly dimensional regression system, it doesn't really tell you anything. So you cannot envision a hundred dimensions of regressions. Anything else? In the deep neural networks, at what point do you decide whether the neural network is generalized enough versus it has learned a lot of overfitting specific details of the given data set that you have? Because I think that until and unless you get the test data, that's the only part you can say about it. But is there any way that you could define that or determine that? No, I mean the universal test always is you have a hold out data set and you look at performance there. The thing you can do is control the expressivity of your neural net in some ways, so you add, like I said, drop out or all kinds of regularizations which you do in other machine learning techniques too, you do it in linear regression, regularization. So you can basically impose constraints on your model that make it harder for it to learn in the hopes of generalizing better. But the real golden test is always have a hold out set. Let me look at performance there. So this sounds like something abstract at this point so they make it harder for it to test. But this is really the thing. So if you think about, if you want to learn about this, just read about dropouts. That's the simplest example when it comes to these kind of things for the regularization techniques and so on. Dropouts just means that you're forcing certain connections inside the certain values inside the matrix which are the weight matrix to be zero. So and this and occasionally so this also at random. So this is the whole thing. There's a random process going on and so on. This just means that you're introducing basically noise into the system at some point and you're forcing the system still to learn things. So this will make it even if you're feeding in the same input over and over millions of times it will never actually get the same activation sense on going through the system itself. And therefore you can imagine that it will not lead to something which is overfitting ideally. Hi. So when you're explaining how will you how you're going to train your auto encoder for anomaly detection, you said you will train it first on a good data, right? Like the data which only contains cats and dogs. So isn't it how is it unsupervised? Isn't it like semi-supervised or something like that? I mean it's not totally unsupervised, right? Because... Yeah, but one can... I mean, so I think there's a smooth... So when I say it's not supervised, it's unsupervised. I mean the labels are never used by the model explicitly. Now one can argue unsupervised learning will in general never work because what if I put bad data? What if in your log files I start typing nonsensical things? But the idea being that your model is not looking at the labels during... when it computes the gradients. The model doesn't know it's a cat, cats and dogs. Yeah. But that's the only data we have for training. So it trains accordingly in that manner. Yeah, and I think the second thing is so this is more relevant to GANs, which is the second part, where one of the interesting experiments is if you can understand the process generating a data set, which GANs do very well, right? You can give it MS images or cat images and it generates novel cats. You'll see all these what's it called? Celebs face data set or something. They generate new celebrity faces and new dog faces. The idea being a GAN looks at your data set and tries to learn the joint distribution of the variables. Right. And so a central question there is if I give you a small amount of label data set. So it's supervised. I'm giving you a tiny data set. Can you learn enough to then use something like a GAN to generate more of each class? So it's technically that's supervised too, but there's a smooth spectrum between how much label data you need and how much unlabeled data you need. Are you talking about few-shot learning or something? It's like, yeah, it's basically few-shot learning but using a GAN. And maybe VA can also be... Yes. Yeah. So a VA you mean a variational auto encoder, right? So there are there's a whole stack of variational auto encoder generally, yeah. I mean they they perform very well especially. Thank you. So but you're pointing to right from there's a problem there as well. So Sanjay mentioned that yeah, you're just feeding it the systems of a good system, the locks of a good system and you're learning from this. Well, in reality if you're asking any sister admin, that's hard because at who guarantees out that of the million locks which are collected all of them so you might be actually not learning what is good and what is bad but you're learning well what one system behaves so there's a real problem actually getting good data in the first place and there has to be some supervision of some sort going on there and therefore you could argue even that is not completely unsupervised learning. And as an example the internet worm one these guys literally manually went and said let's make a list of all the data around and let's pick a time period when there are no known worms but you don't know maybe there's something else there so there's also a healthy dose of hope sometimes within the thing. But it's better than nothing that's the argument. It seems like if what you're really doing is just a compression and if your anomalies are rare enough you might not need any known but yeah this one. I think a lot of people find it unsettling when you know you're trusting a neural network to help you predict things or learn something but you have no way of understanding why it's saying what it's saying and I'm curious has there been any work in terms of what if I take the neural network I got my output now I run it in reverse maybe I drop out some of the intermediate nodes essentially the reversed output onto the input does that help indicate where some of the signal came from has anyone learned from that? There's one very famous example and again back to what Sanjay mentioned a couple of times of course there was a training of pictures which are random taken from the internet and has been learned and one of them they simply say well let's reverse the whole thing and it turned out certain pixels correspond to cat pictures you could clearly recognize the cat again at the end so in theory yes this is possible to some extent but not in general so they had to search really really hard but the the topic of interpertibility of models is something which is ongoing research topics and so on so this will hopefully result in a lot better explainability in the future but so far yeah we're trusting and therefore this is what I mentioned at some point before here that you have to be careful when you're actually using these kind of techniques so if it's only about search results or something like this you don't care if it's about driving your currents on it's something else there are these examples there where people are putting stickers on the stop sign and all of a sudden there because the picture is completely changed and that's not going to weigh any time soon I think that's it so thank you all very much and please feel free to even email us questions or talk to us after this thank you


As I've said all along through this process, I believe on camera but definitely in person. This process doesn't have to be hard. It doesn't have to be any harder than putting some stickers and collage on a page and doing some writing. Maybe that's a challenge to you, the writing part. The quotes, as you can see, I always get from the internet, I always Google the word of the page or day as you will and then the word quote and whatever comes up that appeals to me is what gets written on the page and then as I'm writing that I'm putting my own thoughts on a little scrap of paper and doing my to-do list and that's basically my process in a nutshell. If you want to see a video of me talking my way through the process from minute one to the end, there is a video over on Patreon but you have to be a member over on Patreon which I think is only $2 a month or something anyway. I hope this gives you some ideas of what you can do and if you choose to do something for this December and your December daily, that's a different take on a December daily like mine is for positive affirmations rather than focusing on what we can't do, turning it around and focusing on something positive, not ignoring what's going on but trying to work through it. I would love to see what you're doing for that process. Please tag me in your posts and you can follow me on social media. I'd also appreciate the support for the free content here on YouTube and over in the Facebook Art Groups. There is a link tree list of links down in the video description along with a bunch of other stuff so check it out. There's my Etsy shop and of course Patreon and everything else so check it out. If you have a favorite content creator here on YouTube or over in the Facebook Art Groups I know without asking them they would appreciate the support times are tough right now for everybody and if you can't support them that would be great. If you can't figure out a way from their video description then maybe private message them and ask if there's a way that you can do that. I got asked recently about my Happy Mail address. That's down in the video description too which is FYI. Anyway I hope that this gives you some ideas of what you can do with this crazy weird December that we're having and hopefully we don't have another one next year. Cross your fingers and please stay safe, stay healthy, stay creative. Please wear a mask and go out and do something nice for yourself because you deserve it. Big hugs everybody and I'll see you later. Bye guys.


I'm Malan and Derek and Tristan, and I went for breakfast this morning with a beautiful thing at the cafe on the water, and there was these two beautiful white angels at the other table. Who know, and neither. But Malan came up with this beautiful metaphor, and this is why I wanted to share, because I was thinking, well Jesus spaking metaphors all the time is horrible. And maybe if we just went from the other side with a story, or an analogy, like a metaphor that might also just help drill the forgiveness thing. So Malan said well, and then also Susie's got one, so remind me of Susie as well, but just to come on another side. But Malan said well okay, so you're wearing this beautiful white shirt, like the two angels at the other table. And the fly comes and lands on the shirt, this doubt forgiveness thought process metaphor. And do you swap the fly, and leave a nice stain on your shirt, which our peer would laugh at all day, or would you flip the fly off, like Tristan did yesterday with a March fly, and then hurt the tall March flies will be hurt. He was being my hero. He was. I'm just, you know, because that's the reaction if you like, or the resistance to the fly, and acknowledging that's existence, and reacting to it, or doing something better, trying to push it away. Or do you just leave the fly there, and love it, or whatever.  Trust that's correct. But maybe if we've got that problem with word forgiveness, or we're not quite getting it, should we relate it? Sorry, replace the word forgiveness with release. Release. Because if just, I don't know if it's like the sega just stepping in all the time with mental amnesia again, you know, five minutes later. I got it, and then you sort of forget it. You've got to go on live for it again, and try to remind yourself, and stay in line. It's getting down to the core of things, because, yes, I think what Derek was sharing, his revelation about his joining with Elfie, and one more thought added on, would take away the experience that they were in. And then Raj shared with me, if you want to just share a little bit about what you shared with me at the break, because this really relates directly to this forgiveness process. It occurred, what happened was I got up to go for the morning tea on, and to go in to bring up the bits. So followed me in, and they gave her a hug. While I was hugging her, I realized that I was completely one with her. And as we sort of separated, I suddenly realized that in the moment that I would speak, she would have to interpret what I said, like Derek's words, would have to interpret what I said, and in that moment, the joining of oneness split into two, separated.  But the moment I focused my eyes on one person, and identified that person in my mind, suddenly I was no longer in oneness. So if love is constant and only love is real, that has to be the place that we start. But the temptation is to start from the other end, I mean the ego and all the other things that I've got to get to love. Let's just go and start with, I am love, only love is real, only love is here, that's all that exists. Unless or until I choose to separate from it by either saying something which requires a response or by doing something which requires a reaction or something. So I'm making the choice always to separate from a place of love. I love the fly analogy on the shirt. Suddenly reminding me of a really spiritual friend I have called Joshua, who taught me for a walk one day, and we stopped on a building site in its dusk. Beautiful, the lights all coming on on the Gold Coast. We're sitting there together side by side, and I can feel his bare arm, because he's got no shirt on and I've got a t-shirt on, and we're sitting so close that we're just like this, I can feel his thigh, I can feel his shoulder, we're kind of leaning through each other. And the mosquitoes are all over me, and I can't believe it, I mean it's not just one or two, they're like everywhere. And I said, these mosquitoes are driving me nuts, I'm getting bitten to death and I'm sweating and doing all this. And Joshua's sitting there, he said, there aren't any mosquitoes. And I look at him and he's got no clothing, but he's just got a pair of shorts on, and there's not a mosquito near him, and we're attached to each other. I've got the mosquitoes, he doesn't have any. I said, I don't understand. He said, let them go. He said, they're not real, they're just in your mind. And that's with the fly. So, you're still trying to get rid of the fly or deal with the fly, go back one step, there is no fly. Where did the fly, my question to you is, where did the fly come from? Somehow, somehow at some point, I'm just using you Frank with this white shirt, the person wearing the white shirt, for one second split away from being loved, just being the white shirt, and the fly came in. So then let it go. Then just deny though, if it would say they're beautiful, it's like they have white people with Kirsten. If you deny that the people are there though, saying okay, those people don't exist, you're going to say that. Well isn't that the whole basis of the quandary of the course, isn't it? Yeah, I guess I'll refer them back to I asked David in Tilber, do you see them as your brother, or do you see them as not there? And at one stage in Tilber, there was this, just think of, love me, stamped on your forehead, and I've just put two more words with that now. Just trust me as well, love me and trust me, trust me that I'm there to teach you something as well, let's just love me, hand over the fly to the Holy Spirit, and He removes everything in this process I wrote down the other day to clarify, He removes everything that is false, that is an idol of dreams, and gives me back only what represents truth. The white shirt. Well, if there's any truth, what's the way of being the fly? Because we're talking about the fly, the white shirt was the backdrop, because I didn't want to hurt my backdrop. But isn't the white shirt love? Isn't that who you are? And then the fly is the thought, for one minute that I'm separate, there's more than just, there's something other than love here, there's a fly. Now there's a fly on the love, which is separate. Suddenly you split from only love, which is just the white shirt. Two, white shirt and a fly. Now you've got separation. That's related to everything we're talking about, but Suze talked at the very beginning about just that, a decision to all suffering, it's a decision to leave love, then we're getting down to the real line points with this perspective about the fly, using the metaphor of this shirt, white shirt and the fly. And I've done many talks over the years, I've talked about a blue bottle cap. We had a whole, I had whole sessions on the blue bottle cap. Let's try to go at salvation or enlightenment from the blue bottle cap, and really the fly on the white shirt is the same. I want to read something from A Course in Miracles, because I think you'll be able to begin to grasp the answer from these words from Jesus. He says, you live by symbols. You have made up names for everything you see. Each one becomes a separate entity identified by its own name. By this, you carve it out of unity. By this, you designate its special attributes and set it off from other things by emphasizing spaces surrounding it.  All happenings in terms of place and time, all bodies which are greeted by a name. The space you see is setting off all things as the means by which the world's perception is achieved.


Down in the holler. That's a hollow for those of you who don't know. There are traditions beyond the state that people don't talk about. Needless to say, there's old bodies beneath a few sycamores I know. We've heard all the jokes before. Talk of webbed feet and dating your cousin. Hardy hard. But us yee-haw simple folk have a strange connection to the earth. Above the double doors of the cow barn hung the remains of an old creature with twisty bones and a long snout. Everyone supposed it was a buck at some point. But things were never that simple. Sometimes me and my cousins would look up at the thing up there, nailed into the thin wooden boards of the barn and make up wild tales about a ravenous creature. Dad fed the rumors that it was one of those monsters from urban legend. He'd tell us kids around the fire at night that the thing hanging up there was a creature fowler than anything we could imagine. Of course, it was a child eater, because that's the monsters you would make up to make kids more agreeable. Dad would tell the old man that he was scaring the kids and that maybe he should lay off the mason jar whiskey. Normally granddad would retort with a hand wave and continue his wild tales of the thing stalking children through the rocky slopes of the holler. Lit by the flames of the fire, our small faces would be consumed in a red glow while we looked on in wonderment at the creeping wavering shadows just beyond our circle. It felt like those moments when granddad would go on about the creature, we'd lean in real close and crowd to the edges of the fire for warmth. How'd it end up up there? asked cousin Joe while pointing to the dead thing across the yard. Granddad loved it when people would ask him about that, because then he could put up his hands like he was holding an invisible rifle and say, I shot it. His eyes glassy caught the deep embers of the fire. When he'd follow this up with bang, a few of the younger more timid children would jump in response. The way he'd tell it, I swear, you could nearly see the creature come alive. When I was really little, sometimes I'd have nightmares of the thing peering in through my bedroom window while I slept. Perhaps it would reach in with one of its misshapen hands and snatch me from my bed by an uncovered ankle, and I'd claw and scream and kick. But in the end, there would be no fighting the snarling beast as it pried my bleeding fingernails from the window cell. I'd go into that black night, and no one would ever know I was gone till the next morning. Maybe dad was right. Maybe we shouldn't have heard stuff like that when we were growing up. Living in the holler was like nothing else that I'd seen. It's one of the reclusive pockets of Americana that only existed in some previous era that never really was. We had a big family that filled the small valley along the creek in five or six houses. The way it was is that we would all come together to hold the land running the length of the creek and plant what granddad called a garden. It never really felt like a garden though because of its sheer size. There were the rows of corn stalks high over my head and the beans we'd snap and clean and nibble raw as we did. Tending the garden was a job expected of the kids and so it went that we'd come home tired from learning and go immediately to weeding or harvesting the plants till our blue bibs legs were caked in red mud clay from crawling in the dirt lining the creek instead of tending to our chores like we ought to. But the pantry was always stocked and we never went hungry. Time passed and the legend of the creature dangling over the cow barn drifted into absurdity like all childhood fears. Probably it was just the remains of a buck like everyone said and granddad just liked telling stories. It most likely was as simple as that. Still I didn't like looking at the thing as I grew older perhaps 15 or so I asked dad if we could remove the bones of the creature one day while we tended to the cows. He told me no I don't think Paul would like that. It might just be some made up fairy tale that you and me but to him. Well, he really thinks he killed that thing. You don't think he did? He shrugged. That thing's been up there since I was younger than you. We can take it down once the old farts gone. Why do you care anyway? It creeps me out. Welcome to the club buster. It was strange how we accepted its existence so easily when we should have probably been asking why its body looked so human at night or why the eye holes of its skull went on forever into a black abyss or why it felt like it was biting its time that surely must have been my own imagination. Dad always said I read too many crazy books. Then I caught granddad in the barn. That was a strange day. The air was thick and different like the weather knew it was important. He'd taken the creature from its place over the doors and I found him standing with the skull in his hands. Oh, he raised his bushy gray eyebrows. Didn't see you there. It was an alien experience seeing the skull of the thing in his hands. It seems so small and inconsequential like it never mattered at all. I'd never seen it so up close. And as I approached I reached out an index finger to one of the antlers sprouting from its head. Granddad jerked away. Whoa, there. Don't touch it. Why not? I asked. I know you've gotten to the age now. Were you thinking old man's ramblins or just stories? He held the skull up to my face.  But this thing carries with it. A magic. Something from the old world. I blinked so as to hide my eye roll. Still added with the stories. Still trying to frighten me. Why'd you take it down? I think it's time we buried it. He said plainly. Too many years. It's hung over this place. His eyes trace the sun bars coming in through the walls of the barn. I wanted to keep an eye on it all this time. But sometimes I wonder if it wasn't doing the same damn thing. He shook his head. I'm not a smart man. But I know when the end of something's coming. His eyes met mine and stayed that way. You think I'm crazy, don't you? No. I did. But there was no way I was going to tell him that. Thank you for humor in me. He was shaking but smiling too. At his feet was a large burlap sack. As he lifted the mouth of the sack, I could hear the faint rattle of the thing's other bones as he deposited the skull within. I need you to do something for me. You have to bury this thing out at the edge of the woods. The earths rooted there by trees. And I can only hope that will make sure it stays in the ground. Don't tell anyone where you've put it.  For a moment, I wondered why he didn't do the deed himself if it was so important. But looking over his scrawny arms, milky eyes arched back. I knew he was asking not because he wanted to, but because he had to. I nodded without fully comprehending the levity of the situation. One of his thin, leathery hands must my hair. Ah, you're a good kid. He tied the sack closed with twine, checking and double checking the knot before he handed it over. His hands shook and granddad withdrew a bottle of clear liquid. It smelled like kerosene as the lid popped off and he took a brief swig before gritting his teeth. Then his hands were steady. I left with a spade as the rattle of the bones banging against the side of my leg grated my nerves. I stomped towards the tree line over the crest of a big hill. It never looked so heavy when it was hanging up there on the high barn wall. As I came to the shade of the trees and scanned my surroundings, I could hear splashing and laughing from the creek. The house I grew up in, a two story farmhouse with a wraparound porch looked like a graham cracker box from so far away. Just beyond that, the rows of corn stood out among the crops with their deep green stalks. Turning my attention to the bag of bones, I undid the knot granddad made and spilled the bones out of the ground. I crouched down, lifting the dense objects carefully. They didn't even look real. It was like they'd been carved from hardwood with expert hands and even the cracks along the beakish snout looked put there by a person unknown. But those vacant eye holes couldn't have been anything a human would dream up, not in a million years. A chill blew through me and I felt like I was being watched by some interloper among the trees. No, I looked at those hollow ocular canyons that stretched to infinity and back. I held the skull up to the light, hoping that maybe the sun would shed some light on the supposedly hollow cranium. Then I felt pain and dropped it. It thudded to the ground and I grabbed my thumb. One of its cragged teeth had gotten me somehow.  Running along its front row of teeth was a smear of the sticky red.  Then I pulled my thumb from my mouth and examined the wound closely, giving it my full attention. It was a bite mark, clear as day. I glanced to the spot where the bones were and froze as stiff as a pole. They were gone. I spun in all directions, heart pounding in my ears like a drum. I staggered, lightheaded. I could almost make out the slender form of a human body slink into the trees. After a blink, it was gone. I was wrong, of course, that wasn't possible. My eyes went back over the ground, searching for the bones that had only moments before been gathered in a haphazard pile. Lifting the burlap sack with both hands, I moved to the dark shadows of the trees, if before it had felt like someone was watching me. It then felt as though I was being hunted. Stalked. I twisted the sack into a ball and chucked it towards the trees before lifting the spade. Moving back to the house was slow going backwards. Somehow, I felt that if I turned and ran, it would be on me in moments, clawing me away like I'd thought it might so long ago when I was a child. It was getting dark, too dark, too fast. It was only midday and a thick cloud passed in front of the sun, making the chill I felt ever more so. Then, I felt small and scared. I gripped the spade handle across my chest, eyes darting back and forth over the length of forest. Once I'd come down the slope and stepped carefully over the rocky terrain dividing me from relative safety, I went to the house and passed the barn. Standing in the great structure's massive doorway was Granddad. Half the jar in his clenched hand was gone, and he gave me a toothy grin with a raised eyebrow. I nodded as to not let him in on the fact that I had unleashed one of those evils from the old world. The crickets that normally brought the sounds of the echoing holler to life at night removed themselves from a closing performance. The cool air swept through the valley, and I felt something was different. Dad, lounging on the porch while smoking a pipe, called me over, and we sat there together with him in a rocking chair and me on the top step, looking out on the creek bed in rows of corn and damp trees that hung limbs over long-like fingers towards the stream. Seems it's going to rain. Why is it so cold, you think? Ask Dad. I don't know.  Silence hung for a moment. I saw Pa take that old thing down. I nodded and said nothing. You have anything to do with that? I helped. Well, good. I'm glad he finally decided to get rid of that eyesore. My voice cracked. What was that thing, anyway? Dad shrugged. Like everyone knows, it's a buck. The world was so cold, and I felt so alone, and it was the damp weather that beckoned the rain of that creature. I'd so long convinced myself of being a regular animal. But regular animals don't walk upright. Regular animals don't speak with the devil. The evening passed on to full black night. Shadows consumed the holler, and we gathered around the dining room table for supper. Mom and Dad chatted idly about the mundane while Granddad snored in his chair. I could hear him sleeping in the living room by the warmth of the wood stove. All I could think of was the lurking monstrosity. The rain came, rattling the tin roof in a matter that should have been calming. But I could only feel the paranoia creep in. Beyond that damn tin roof, I could make out no sounds. With the fire going, the rain, the full belly of greens, corn, and beef, I should have felt cozy. But a kind of sickness ensnared me. I closed myself in my bedroom on the second floor, attempting to bury my head in a book. But the pounding of the rain became unsettling, to the point that all I could do was watch out the window to the dark shadows in the yard. I was totally blind. Helpless. The thing, matted fur, hot breath visible even in the rain, came from the shadows and stood in the yellow square my window cast upon the ground. The shapeless spaces in its head turned up. It looked up at me, directly at me. My hand clasped over my mouth with cries coming out as moans, lingering there. With its starved skin clinging to its ribs, it arched its head into the air like it was sniffing. For a millisecond, I wondered if it could smell me. Then it dove into the shadows and disappeared. No sound but the rain. I moved from the window and pushed myself against the edge of the bed to sit. Minutes or an hour passed as I watched the unmoving window, waiting for the thing to come in and take me away, waiting for the thing to tear me to ribbons and eat me. Because that's what things like that do. They always eat you. The thuds of boots downstairs snapped me out of my thoughts, and I moved to the door and scampered across the second floor landing towards the stairs. Descending them in slips more than steps, I met the men standing in the doorway. They were wrapped in thick coats, lanterns held high, flashlights aiming around in the darkness of the yard. They were the men from the other houses in the holler, extended family. Granddad jumped from his nap, angling like his legs were stilts. Then my parents entered from the kitchen, mom wiping her hands on a dish towel and dad puffing his pipe. What's the matter? snorted Granddad, blinking his eyes awake and rubbing his cheeks. The man standing just inside the door spoke. Joe's gone. He wasn't in his room. Granddad's eyes shot to me standing on the bottom step, and my whole body went cold. I tried blaming it on the wind coming in, but that wasn't true. Dad stepped forward. What do you mean? The man looked over his shoulder with a flinch, but it was just thunder. Then he continued. Someone broke into his room. His folks are a mess. They keep saying it was some kind of animal.   Granddad bawled his fists. He was staring directly at me. As he crossed the space between us, his eyes went crazed and his bony hands grabbed my shoulders, thumbs driving into my collarbone. What have you done? He said.  I told you to bury it. He said. Dad stepped forward, placing a hand on Granddad's arm. Pa, calm down now. He didn't know. He's still only a boy. Granddad sneered and let go before snatching the lantern from the man in the doorway. I thought I'd be long dead before it came back. The night drew on as mom and me gathered in the kitchen, moving furniture in front of the doors of the house. The men told us to keep quiet, not a peep. They marched in the heavy rain with rifles, shotguns, axes. I sat at the table with my face folded into my hands at the noises that broke the rain. Loud shouts echoed through the holler. As did the cries to human yelps of pain and death. When dad returned home near foggy morning, he came with a rattling burlap sack, but no granddad. Mom looked at him as if she was going to ask a question, but he just shook his head. And then we buried those bones out near the trees.