Hello and welcome. This is active inference guest stream 65.1. It's December 6th, 2023. We'll be hearing from Phoebe Klett and Dan Simpson on Bayesian world models for explainable transparent reasoning. There will be a presentation followed by a discussion. So thank you for joining Phoebe. Very much looking forward to the presentation to you. Awesome. Thanks for having us. Yeah, I'm really excited to chat with you all about how we might start to integrate today's state of the art language models into more probabilistic machinery and what that might bias. All right. Let's get right into it. So some of the things that I'm hoping to discuss today include why we might use a language model for something that isn't long form text generation and how we might do that and then motivate a little bit kind of why we might need a world model, what simple self organizing world models might look like and even in the simplest cases how we might start to use those as effective recommendation engines in the wild today and then a little bit of discussion about kind of where this research is going. All right. So what are language models good at? Today's especially large language models are trained on next token production. So this means given a sequence of tokens, we're going to estimate which token is most likely to come next. Maybe rephrasing that a little bit. We might also say that language models are trained to estimate which sequences of tokens or words are likely. The caveat being likely to appear in the training set, but today our training sets are quite extensive. So given this simple training objective arguably surprising that we've seen language models do as well as they have in really impressive tasks. So they start to demonstrate like really great language understanding meaning like the semantics of language itself and not just the syntax of kind of like which sentences make sense. They've also started to demonstrate some world knowledge. So implicitly learning kind of how the world works just through our own human abstraction and how we articulate that. We start to see language models falter at tasks which look more like symbolic problem solving. So math in particular, we see this in programming although we're getting better at this and in general any kind of like long-term planning tasks which require abstract reasoning. So we see this also in problems that look like this which are word problems but which are really about kind of like understanding abstractly how these kind of abstract objects relate to each other. And again this shouldn't be surprising that language models struggle to do things like this since it's so far off from their original training objective. And in particular even when we arrive at the right answer in some of these cases it's really hard to know after the fact kind of the reasoning for how we ended up at the right answer in this particular case and maybe more fundamentally if we don't know where the abstraction is happening or where the reasoning is happening it's very hard to guide that process. And so this starts to motivate the need for something that looks more like an explicit world model. And now to start to borrow some words from Yasha Benjio this is one of the kind of biggest issues with today's language models is arguably that we're asking the language model to be both the inference machine and the world model implicitly when this doesn't quite make sense. And so some things that we might hope for in a world model are to model causal relationships to be really adept at modeling using uncertainty and to be modular. Yann LeCun puts this in a similar way so we might kind of ask a world model to be able to distinguish between which details are important versus irrelevant and to be able to make predictions that can be performed in sort of this abstract space of presentations. And so hopefully in the next following slides we're going to motivate what it might look like to use language models as inference machines in maybe like the simplest case of self-organizing world models. And even in those most simple cases how we start to get something much closer to the kind of explainable reasoning which I think lots of us are grasping for right now. So in a very simple case we might think of a world model as simply a collection of hypotheses where we model confidence over each of those hypotheses. And in particular we care about predictive world models so given some evidence or data that we have observed in the world we'd like to propose the world models which best explain the evidence that we've observed. And if that language sounded leading as it was indeed we are proposing in this case to use Bayes' Rule which we all know and love which tells us exactly how to update our beliefs given some new evidence. And so when our hypotheses are kind of these Bernoulli random variables this bottom term can simply be expanded into this equation on the slide. And so the real tricky part in this computation is the likelihood piece given that our hypothesis is true what's the likelihood that we would have observed the evidence. And the claim that we're making is that language models are actually this is like a natural task to ask of a language model when our evidence and our hypothesis are both semantic objects as we discuss language models are kind of trained exactly to understand which sequences of text are likely so asking it to prompting it in a clever way such that we can extract this particular likelihood is actually really natural. So how do we do this? So we come up with this kind of clever prompting scheme that allows us to extract again exactly how likely is this should have occurred. So given that our hypothesis is h in this setting and our evidence is curly e, we only need to change our evidence into the conditional form and then phrase the question like this to the language model so the input sequence of text goes like given that potentially if our hypothesis is like Walmart has been severely impacted by COVID-19 pandemic would we have observed the evidence that Walmart laid off 10% of the material stuff. And so either using something like screenshot prompting or guidance or some kind of control generation technique we can ensure that our language model outputs either yes or no and then use the logits from its answer to estimate that probability. And again the claim here is that like this is actually really natural task to ask of a language model that leverages it's like innate reasoning engine better than just kind of allowing it to ramble using text sometimes. And we can even make this updating scheme a bit more sophisticated using things like precision weighting. So this will bias our posterior over hypothesis either towards our prior or our evidence that we've observed based on how confident we are in our prior and in our evidence. So the world models that we've seen so far are fairly simple in particular we're modeling all our hypotheses as independent from each other and this seems like a large simplifying assumption so how might we support more complex world models where we condition our hypotheses on each other. Alright so another thing that hopefully will be familiar to this crowd. Basinets have been around for forever and are often used to model mechanistic failures and large systems and can be understood simply as distribution where each variable depends on some small number of ancestor variables. Perhaps more intuitively we can also think of these as directed graphs where we have edges between variables which are directed in the sense that the parent is conditioned on the child. So let's see an example. Alright so now we have two different kinds of hypotheses the ones which are kind of parents in our child set up are kind of qualitatively more abstract than the children. So our two children nodes are similar to the hypothesis we saw before. Mark a sentiment for Best Buy is Poor or Walmart will grow its physical footprint this year and then the more abstract hypothesis being retailers were negatively impacted by COVID-19. And so then we specify this conditional structure kind of in a classical sense by specifying all of the joint probabilities upfront or alternatively learning it given some healthy amount of training data. And again we're proposing that instead of doing that intensive process we can use language models to extract those probabilities in a natural way. And so one natural complaint at this point might be like well this space is going to start to get very large. If we're trying to update these beliefs in time given some large data stream this is going to become intractable given our current framework. So how might we start to augment that? Luckily there's been a lot of work historically done on this problem and we can start to use things like message passing to update our beliefs in an approximate way. So as long as our Bayes net has a tree dependency structure we can use things like the sum product algorithm to update our beliefs. And I don't want you to worry too much about the equations on this slide. If you've seen this before this kind of recursive structure will look familiar but if not the general kind of intuitive idea that we're going to compute these messages from all of the children or the neighboring nodes and use those to propagate through the graph to update each independent belief. But let's just see an example. All right so suppose we observe some new evidence and we'd like to know how should we update probability over hypothesis A which as you remember is the parent node in our graph. All right so B here on the left hand side is the belief. X A is essentially like just the variable representing the hypothesis A and then the fee and the side terms are specified by our language model or the Bayes net in the classical sense. And so we're summing over all of the variables, all of the values that the variables X B and X C can take which in our case of course is just zero and one. So if you might recognize at this point that this is indeed the exact marginal probability for the hypothesis A and that's because our graph is simply connected in this example. So in general this isn't true but turns out to be the case in our example. And again we can compute these fee and side terms using the language model itself. All right so because we're all lovers of free energy here I'm going to walk through kind of how this is maybe the first example of self-evidencing or minimizing free energy kind of model. So as we noted belief propagation isn't exact for more complicated graphs and so it makes sense and might be useful to ask the question how far apart or when are our beliefs close to the exact marginals. And so we often use things like KL divergence to compare the difference between two probability distributions and that is explicated on the right hand side. And then those of us who are like have a background in physics might recognize Boltzmann's law as well. So this is just the idea that we might represent the probability of a given state using an energy function and we're not going to accept this as truth maybe as some of us have done in the past but we are going to just use it as a definition for this energy function such that when we plug in that term here and expand out we start to see these first two terms look a lot like the kind of energy and entropy functions which we are used to. And indeed we can just classify those two terms as the Gibbs free energy function. That makes me happy to see this all coming together. And in particular it might make sense just to note at this point that using world models which are self-organizing in this sense seems to be very compelling since the kind of world model which we want is one which promotes the evidence that we've seen so far. Why is it useful to formulate this in terms of free energy besides the fact that we all find it compelling here? There's a lot of progress by constructing analytically tractable approximations of Gibbs free energy often. I'm not going to go into the details here but here are two examples where that's been fruitful. Alright so now I'm going to chat briefly about kind of how we might use these systems as recommendation systems and indeed in the world today. Alright so one prime example for a system like this might be useful is a situation where we have lots of data incoming at very high frequencies and we always want to have some set of naturally discreet hypotheses that we're modeling beliefs over which are being kept up to date at a very regular cadence. And so actually a lot of the muscle here is just reformatting documents or however our data comes in as evidences. This is not always obvious or easy to do but once you've kind of figured out that part and in particular we've been using things like RAG, retrieval augmented generation or embedding based systems to kind of figure out when data that's incoming is relevant to a given hypothesis. Once you've kind of built up that machinery the actual updating computations as we've shown already is actually pretty simple. So we do these likelihood computations and we update our beliefs and then at any given time we can query that model for our marginal distribution over any given hypothesis. And it turns out that this kind of setup has many practical applications. It's also noteworthy that even with very simple systems like these these are like out of the box controllable and explainable. So just by storing the magnitude and the direction of the update to the posterior for each piece of evidence that we observe we have a very natural built in explanation for our belief at any given time. And that makes kind of these applications where folks might really like to use a language model but really require a robust causal relationship between the outputs and the explanation which you don't get from a language model on its own. So a system like this can be very appealing in those situations. So now on to further work. Everything that we've discussed today is early work towards integrating language models into more probabilistic frameworks and there's been a lot of exciting work done in this vein right now. Some important questions which are especially interesting to me are which parts of the world model should be learned versus encoded and how do we want intelligence to scale. So I mean that both in the sense of composing systems naturally there should be some very like natural way that we can compose to intelligent systems and also such that we can scale them with compute. And I don't mean to restrict myself either to the kinds of compute that we have today. We're also working at some exciting new computing paradigms at normal which might be more compatible with software of this nature. Also the two folks that we referenced at the beginning of the talk, Jeff Hinton and Yann LeCun have done really exciting work in this area which is very inspiring and so in particular G-flow nets are also probabilistic graphical models which I think folks will find a natural next step in reading if you so desire. And that's it for me. Awesome. Thank you. Very cool. Dan, do you want to give a first reflection on our thought and then meanwhile anyone who's watching live please feel free to write questions and I'll relay them in. Absolutely. So hi, I'm Dan. I work with Phoebe on this project. And yes, the I think the thing that's most exciting about this for me personally is sort of two fold. One of them is that it's a way of avoiding sort of having to trust a language model to understand and reason about text. Because they're not, it's not that bad, the thing is that the extremely strange thing about language models is they're quite good at being almost good enough. But they're never quite what you could use. You could never use a language model to, I don't know, sort of triage like an important sort of situation where a bunch of different things are coming in and you have to make a decision about which is important. The reason you can't do that is you simply cannot understand the encoded biases, you cannot get it to reliably generate reasoning. You can ask it for reasoning, but the thing that it prints out is not the reasoning that it used internally because it doesn't reason. Fundamentally, while these have input and output than a natural language, they are not artificially intelligent. They are just prediction machines. And so we have to be very careful about not anthropomorphizing them. So this is a way of using those incredibly powerful prediction machines in a framework where we can make sure that we essentially keep a record of what we're doing so that a human can look at it. Because there's a lot of talk in this world about sort of post-human AI and those sorts of things. The idea that the machines will become intelligent enough or the machines will rise up in a slightly more alarming type of way. And that's all great and wonderful, but that's not particularly interesting to us at normal. We're much more interested in sort of having mimicking explicit decision processes so that a human can audit them and can make these things work. So that's kind of the area that we're coming from. Awesome. I'll go to a question from the chat. So Josh asks, where does hypothesis relevance enter the calculus? Is it folded into confidence? Not sure if it ought to be. Just saw hypothesis relevance mentioned. Hypothesis relevance. Does that mean which hypotheses are conditioned on each other? Is it possible to ask a clarifying question there? Maybe Dan, you have a better idea. They can follow up, but I also wondered about this. You might know what was relevance. Maybe the temperature and the rainfall were relevance, but then how does this approach help us understand when one of those relevant factors no longer is relevant or when a new relevant factor comes into play? Yeah, these are great questions. So I think in terms of understanding in an automatic sense, when two hypotheses are relevant to each other, we can leverage embedding type language models for this kind of thing also. If we don't have a more structured human intuitive sense for when two hypotheses are related, in terms of how those relationships evolve over time, this is something that's really interesting to me. And I think looking at the theory behind structure learning or when we propose to add new nodes to the network or propose to add a new edge to the network or things like this is a really exciting research direction. Although I don't have a silver bullet answer to how we should do that. Just to add a little bit more to that, it is a really interesting research direction. One of the things that Phoebe mentioned in the talk is that there is a difficult step that we're not talking about which is actually translating this natural language into reasonable hypotheses. So there is a step in there where you take essentially a chunk of text and you have to decide if this is a hypothesis, if this is a hypothesis we've seen before, if this is a sub-hypothesis or a clarification of a hypothesis that we've seen before and so on and so forth. So that in some sense part of the data processing and it is an important one that we are continuing to work on and refine. The other thing, a different interpretation of the question around relevance is around is the hypothesis relevant to the thing that you're looking at? We could have a hypothesis that the sky is blue but if we are deciding whether or not we need to check that part's oil like the truth or not of the color of the sky is very irrelevant. And that then comes into the nice thing about having your world described as a collection of statements with truth values associated with them in that you can directly reason over them. So you can put a classical decision framework over that to take into account both the knowledge you have of the world and also which parts of these worlds are sort of unknown. So in that sort of situation the person using the world model to construct a sort of decision or an output will be responsible in some sense for assigning a weight or a cost to each hypothesis being true. And for some of those hypotheses obviously it will be zero because again we do not care about the color of the sky if all I want to know is if I need to change the oil in my car. So that's the sort of the other end of the answer. So there's a version of the answer at the start of the information flow and there's a version of the answer at the end of the information flow. But it is a tricky point and one that we are sort of continuing to iterate on to try and find sort of good ways on both ends of that. Yeah well a lot there it's very interesting how in that presentation in response I heard both about probability distributions on rules and rules on probability distributions and like which one whether it's the tail wagging the dog or the horse in the cart how to design these synthetic intelligence systems that appropriately bring together aspects that are more symbolic more rule like and then more probabilistic more embedding like. So where does that end with you or how do you see the design of these systems with mixed symbolic and probabilistic components Yeah that's a great point and I think this really gets it like which parts of the world model should be learned or should be represented in some like more discrete space versus like encoded based on our own human intuition for rules and structure and I think like maybe this is would be fairly represented as a cop out answer but I think it depends a lot on the application I think like when we're developing systems like this and and just trying to iterate through as many different hypotheses as you can quickly like choosing an application and benchmarking and testing and seeing like what actually works is a go to strategy for us in terms of like well which parts should be fixed and are actually helpful to increase reliability such that like we can use our human intuition for how this particular you know system is built versus like well this is something that we want uncertainty over that's like a really important part of the learning process for us in terms of yeah that kind of iteration so I think it probably depends on the application Yeah it definitely depends on the application it's also like depends on where the actual challenge points are so we've got like outside of this we've got sort of a few other things that we've released publicly that kind of look at this idea of external rules to the system and whether or not we can add those in so one of them is something called constrained generation where we sort of force the model to only produce something valid and that's sort of quite a useful way of removing one particular aspect of stress from the model which is that it may make sort of syntactically or sort of in somehow incoherent outputs that don't follow the rules and then we can then focus with the rest of our energy we can then take that as given and focus with the rest of our energy on improving the bit that we don't have rules for so those sorts of things and sort of a different version is trying to improve something by saying no you broke a rule go back and make this sort of true so this kind of sort of chain of thought prompting type of idea so yeah the symbolic and the probabilistic I think in our minds live very closely together as two tools that don't completely solve the same problem and I think there's sort of in the world of I'm not sure how familiar anyone in the audience is with language modeling but like in the world of language modeling before this sort of explosion of neural networks and artificial intelligence type method there was a lot of work on symbolics of language and grammars and all of that sort of stuff and that work pushed quite a long way forward and this work is pushing quite a long way forward and I suspect the next thing is going to involve them joining up again because they each have good points they each have bad points and you know who wrongs don't necessarily make a right but they can make the less wrong nice yeah recently we heard from Elliott Murphy and talking about the neuro linguistics and about how the statistics of language are not the rules of language you can always come up with a new expression that's never been uttered that's not going to be in the training distribution okay I'll ask a question in chat from upcycle club they wrote what are some of the key challenges associated with developing such Bayesian world models hmm I think we've touched on a bunch of them the ones that are most top of mind for me right now are the structure learning thing that came up so how do we understand like when to propose new hypotheses and how to integrate those into the models and then yeah just figuring out like yeah I guess this like proposal and evolution process of the nodes themselves since everything else like the framework works pretty automatically in a reasonable way thank you Bayes thank you to the development of language models but kind of moving from this like more discrete case into a continuous case more fully represents the space that we're learning over can be challenging yeah I would also say that like it's a sort of a maxism that max maxism what on earth did I just say there's a common saying let's go in that direction there's a common saying in this world that sort of no model ever survives its first encounter with data and that becomes true here as well so there's lots of like as we've been building these things and using them we found lots of little spiky edge corners with sort of making sure that the language models are actually doing what we want it to do so there are a lot of questions in building these things around how do you actually test that the components of it are actually working the way you want and then on like a broader level how do you compare something that is fundamentally trying to solve a different problem to other methods so we are solving a problem under the constraints that we want a fully auditable system we could also solve all of these problems by a thing called in context learning which is basically putting the context into the prompt of a large language model and asking it the answer and that also works especially when you've got things like GPT-4 which are just wonders and glories it works really well so then we come to the question of how do we actually make the case from a bigger picture perspective can it be more than just a can we find benchmarks that reflect the structural advantages of this approach over something like in context learning that don't come across as false so that's kind of like a stranger answer because it's not really about actually developing the world model it's about convincing other people that it's a good idea and that is a thing that is true of essentially all of the things on this slide as well they are all quite complex and odd little methods that you know there's a degree to which we definitely can solve this an easier way so what is the thing that the application or the benchmark where we can say no if you do it the easier way you will fail at this measurable thing very interesting so you mentioned the self-evidencing advantages of using world models that are self-evidencing rather than reward maximizing for example so how do you see that playing out and I can connect it back to active inference of course but how do you see this self-evidencing centrality play out in the kinds of models described here there are a couple reasons why it's so compelling to me and the first just has to do with explainability it's really convincing to people to say why did we predict this, why do we believe this well this is the actual real world data that we've observed such that this is the impact that that's had and then I think I don't know you hear a lot about designing these really complicated reward functions which are often very clever pitfall because they very easily become disconnected from the complex world that we're trying to model and so you end up in weird local maximums or minimums and you start solving the problem that you've designed versus the problem which actually exists I just have always loved the idea that what we should be doing is self-evidencing from an intuitive sense that feels like what an intelligence system should do yeah I don't have anything interesting to add to that I just agree with Fabi that explainability and the capacity to explicitly reference previous data including like leave one out so techniques from non-parametric statistics about the effect of adding in another piece of data or removing a piece of data it's like to bring it to like a homeostatic setting which is commonly considered an active inference like we're trying to be within a homeostatic temperature range 37 yes we could propose reward functions but as those start to include open-ended exploration structure learning just like you described it Fabi like we're solving the problem as designed rather than the actual question of the homeostatic temperature and the sort of path of least action first principles physics grounded intelligence perspective from active inference is like make it the kind of thing that measures itself at 37 and then as long as it is it is and when it isn't it's dead and that's the kind of mortal computing crossover which is like outside of its zone of surprise it's not just that it's getting a bad grade in the class that is like a deeper failure signal than that and to understand okay when is it a yellow flag when is it a red flag in terms of the new scientific literature coming in those have plain straightforward ways to interpret that developing larger higher order apparatuses will never return to that kind of basal simplicity yeah couldn't agree more yeah I mean absolutely it also like the other thing that it can do quite well is deal with essentially outlier studies so situations where you have a new piece of information that is strongly conflicting with all the previous pieces of information and trying to sort of work through what that really means and there's a like there's a degree to which we can even sort of extend this process to multiple agents that have these belief systems and then look at sort of consensus of experts or weighted consensus of experts so for instance you could have like a weather vane type of situation where somebody really over indexes to every new piece of information and you would do that with you know technically you do that with maybe a power posterior type thing or you can have somebody who's built in strong priors in a particular direction and you can then like take your consensus of artificial sort of decision making all of which has within their universe well reasoned updates to the data and then you can look and try and work out what that swarm of experts can tell you and sort of do very empirical things like try and you know work out which of these experts is doing well at a particular moment in time because you know there could be there could be times when the world's very or the problem you're solving is very chaotic in which case the over indexing expert would probably be a pretty solid bet while there are other times where sort of things are pretty stable and you probably it would be possible that the sort of the more conservative expert is more sort of empirically making good decisions and good recommendations so there's like a lot of ways that we can not just like incorporate these sort of homeostasis ideas but we can also change what that means for different agents and artificially like do that artificially and then combine them together to try and get a almost like a blanking on the word but you know a forecast under sort of a hypothetical set of situations and we can actually sort of bring those ideas of the world forward and see what happens when they sort of meet with actual information. Yeah this angle of mixture of experts as it's sometimes called more in the language model space or ecosystems of shared intelligence or diverse intelligences in the active inference area like that's very interesting obviously has connections back to human teams and teams of beyond humans and so on a lot of this is still text based so maybe you did or didn't mention what representation the base graphs are but they're plain text like there was a lot of discussion about bringing from natural language scientific papers or however it is into a structured form and then the explain method that you showed kind of taking the structured form and just giving a little syntactic fluency so it looks human readable so how do you see that essence coming into play with models and then with action in the world that isn't just developing the next text token but a robotic actuator or modifying some other control element of the world. Yeah that's a really good question I honestly haven't thought much about multimodal stuff in this particular context but I think the framework is general enough at this point such that it's definitely could support lots of different modalities. Yeah I'd be really curious to see how this did with something like audio in particular yeah and then to your point about yeah this maybe like discrete versus continuous relationship I think that's part of what we're learning is how to go from these long natural text documents to a system which is appropriately discretizing our hypotheses such that we have these meaningful explanations like you mentioned. So I think yeah I think like continuing to develop robust ways of surfacing those explanations is a big part of this as well like over time we're going to observe lots and lots of evidence how do we make sure hypotheses don't get stale and how do we use evidence to know when they are and things like this are part of that also I don't know if that directly answered your question but that's some of the stuff that I've been thinking about related to that. So in the I mean in examples like sort of moving towards robotics and sort of tech video generation and image generation other audio other sort of multimodalities to be honest I think of these processes in general as enabling like building a world model to enable a sort of sequential decision process. So if that decision process happens to be should the robot turn left and that's what the decision process is it's multimodal in like a very classical sense that you can put any type of decision framework over the top but it's not sort of generatively multimodal I'm not saying write me a song that sounds like Beyonce and a song that sounds like Beyonce comes out I think this sort of this sort of Bayesian world model layer is blocking towards that sort of thing but that's really not sort of the aim of what we're trying to do it's also like within normal like our almost I don't want to say mantra or manifesto because that sounds culty and no one wants to sound culty but like our basic aim is to always center like humans within our process and so some of this multimodal stuff it's less clear where the human lives so for instance like a video generation type thing where does the human live so keeping it at this abstraction of sequential decision making then it's a decision that a human could do you know human with their thumbs could be moving like a robot around and doing that sort of stuff but yeah it's really all about sort of controllability and auditability for us in sort of a sequential decision process so to the extent that that sort of leads in its multimodal world that's sort of part of what we're doing and like to some versions of multimodality is we're just not swearing in that particular space yeah not a great answer but a long one let them distill it down later in the auditability area it almost falls out to me to be like a syntax of auditability semantics at the syntactic level just tagging or versioning when a file comes in or when a given computation is executed that is basically transfer across all settings and then where I see you honing in on with this work is kind of the semantic auditability which is actually how we compose our accounts but I decided to walk because this happens and so bringing that different kind of trace to systems is going to make it what will it open up in science or education or how do you see this sitting at a console somebody is at now and making this different like over what timeline yeah I mean it's really quite nice for storytelling because as you said you can say things precisely like well you know because we observed this thing or because if we had observed something else you know like maybe you can even make statements which are conditional in that sense I think it does like empower whoever is sitting in front of this data to feel like really sure about again like the reasoning engine that went on which to me is pretty different from what it feels like to sit in front of chat GPT even though it's quite useful often you know you try the code and it works or doesn't work or you like ask your friend is this really true and that feels pretty different to me from being able to look at the evidence themselves and say like oh well actually if this is the reason you think that evidence is not true or you know like you can bring your own human intuition or world model in terms of validating or yeah superimposing what you believe on top of what the system believes and so that makes it really easy to make decisions quickly. I think there's also sort of a converse of this which is that it also makes it clear which evidence was not used to make a decision and that can be quite telling in these situations where you could be worried that a particular type of evidence isn't being weighted correctly or isn't being sort of formatted correctly so again like if this is a sort of a like a system that builds an assistant that sort of does surfaces all this information and sort of makes a recommendation with reasoning for a person that person can then look and be like and they know what the data is they can look at the day and say you know why didn't you consider the make of the car or why didn't you consider this or why didn't you consider that and they can then use their understanding of what's not being prominently used by the model to sort of sense test like it's sort of I mean in some sense that usage of it is a reformulation of what Phoebe just said where you use your internal world model but it's like I think it's important to know when evidence is being used and this is like I think you simply cannot get from like a GPT type thing or any sort of like prompting type method we know for instance that like the order of the order that you submit your evidence in is probably going to matter for a prompting based method okay that's obviously not true for a Bayesian update where we have this sort of this this coherence principle where if you shuffle your data and enter it in a different way you will get the same posterior up to computational artifacts so all of that is like in my mind is just as important to as the ability to write a report that says I made this decision for these reasons yeah well that makes me think about this kind of view from the inside interpretability where the rules help and also knowing what evidence is not used is importance for compliance and knowing what information like in a healthcare setting was or wasn't used what about thermodynamics we heard about free energy Boltzmann came up how do you see the info thermo nexus what have we learned from the last hundred years of thermodynamics and information theory and all of this and on the software or hardware side how is that kind of a free energy nexus being used I'm really excited about how all of this seems to be coming together free energy just keeps showing up in all of these exciting areas to me we have like a book club for a singular learning theory and like they talk all about free energy too and I think some of those ideas are really exciting I mean at normal I think like the thing that I would highlight is like this idea of software hardware code design which is really special I'm really trying to do this hard and fun dance towards each other where we're like thinking about these new kinds of systems and how they might support each other and empower each other and how to build full stack systems which is really challenging and also really exciting and I think like from like the first principles of thermodynamics perspective like we're all just in mathematics and physics people at heart so like going taking like all of what people have learned in language modeling and all of that like very much to heart as well approaching whatever problem that we're facing from the first principles how do physical systems work in the world what are the assumptions we're really comfortable with and building up from there is definitely our natural mode so I think that makes it easier to start working together also it's also probably worth saying that we have a sort of a secondary not secondary a very different stream of interest in thermodynamics as well which is the ways that we can use actual physical thermo dynamical principles to build hardware that is specifically has sort of noise in it as a first class citizen and because of that it is particularly well suited to probabilistic tasks and so we've built if anyone wants to look we have a blog I believe the URL is blog.normalcomputing.ar and amongst other things that are on it there is the very first demonstration of using physical thermo dynamic hardware to actually do computations is the computation the most vital computation that we will ever do converting an 8x8 matrix so no we can do that otherwise but it is sort of building up towards this idea that we can use thermodynamics not just in our modeling and our understanding of the world but also in our sort of low energy compute stack to actually realize these things so I think I think it would be challenging to find a group of people on this earth who have more investment in thermodynamics and don't work in a physics department because we have investment all the way through from sort of active inference type things all the way down to this like this this hot thing goes there which is kind of cool I'm not a physicist so I that's my understanding of thermodynamics this hot thing goes there informative thing goes here hot thing over there call it a day um yep that it's a really cool fusion with the kind of parsimony and elegance and the aesthetic of math and physics and first principles and the different parsimony of pragmatism with the actual material basis like of a synapse the size of the synapse and the kind of stochasticity that that size alone um entails with like membranes and all this those stochastic aspects are leveraged for the compute the synapse is not simply a variance reducing machine and so it's like the platonic slash mathematical ish spirit finds a common home in these real simple physical demonstrations and today it feels like there's a big gap between the mesoscale computational architectures that you describe today that are very much running on the kind of von Neumann architecture turing completeness paradigm and yet very tantalizingly close like to a physical object that has a constrained rule de facto like only one thing can come out of this at a time as long as the funnel is this wide and so bringing the rules and the regularities of what we call physical things to bear with the fundamental and the imposed constraints on informational spaces very cool directions note about just where active inference and action plays a role is and also you mentioned like the hypothesis going on in the pro active stance where we're using expected free energy or something like it to calculate future courses of action over observations that we haven't seen yet moves that haven't been made yet there's an explicit epistemic value and so that can be diagnosed and observed as a measure of where a given computation is on the continuum between purely pragmatic value just constraint satisfying and realizing preferences and expectations and then the pure epistemic value where all outcomes are good and the more information gain the better and then being able to take control of that balance and know amidst changing situations again taking probabilistic or global based approach there to when epistemic and pragmatic like gas and break kind of come into play these are very basal control knobs or features in active inference that it's just not going to show up at the 50th layer of scaling is all you need yep cool well do you have any other like thoughts or things you want to add or questions or where things are heading for your works nothing to add at this moment but certainly excited to keep in touch with this community and yeah collaborating yeah absolutely and we sort of share I mean we write papers and stuff but we mostly share most of what we do be it academic in the sort of machine learning space or be it in the sort of the physical hardware space on our blog which is blog.normalcomputing.ai thank you so much for inviting us it's been very fun awesome thank you hope to speak again so peace bye bye bye bye


Good day fellow investors! I will start a short series about the stocks I recommended in 2017 to see what happened and if I would still recommend them. Today I will discuss 10 miners that I have recommended then I'll make a video about the Chinese stocks that I recommended, about global stocks that I recommended and about developed market stocks that I recommended. So you can expect four videos about stocks that I recommended to see how they have done and what do I expect in the future. So let's start with the mining industry and see what happened. The first stock we discussed on this channel was Nova Gold Resources, a company that has two great projects that still have to be developed. You can see that in the last six months it didn't really do well but this is a very very long-term investment. The two projects are the Donling Gold Investment and the Galore Creek project. The net present value is really related to gold prices. If gold prices double then the net present value of this company goes to stratospheric ranges so really a leveraged play on gold. Nova Gold owns 50% of Donling Gold and 50% of Galore Creek. Galore Creek also a copper project very much related to copper prices. Now the market capitalization is 1.2 billion however the net present value current values is 0.9 billion at 7% discount rate. So we can say that the company now is overvalued. If you're looking for jurisdiction US minor then you will pay a little bit more that's why the company has a premium in relation to the other stocks we will discuss. Nevertheless if in the next 10-20 years gold prices spike this will be a great investment. So you have to look it from that perspective rebalance accordingly but you can buy this sit on it 10-15 years and you will probably make a lot of money. It's really a long-term investment. For those interested look into Nova Gold it's still a good investment for the long term. Another company that I recommended was Harmony Gold Mining Company which was extremely cheap when we discussed it. As you can see it has a margin of safety at the price we discussed it as it didn't go much low but it went higher a little bit. The company has a Gopu project in New Papagini where the net present value is 1.1 billion. It has positive cash flows but it has very high costs from African mines and it recently made an acquisition spending cash with some financing to acquire again high cost African mines hoping that they can turn it around. So if gold prices increase Harmony will immediately be extremely cash flow positive. However if gold prices decline it will be questionable and there will be issues. So Harmony is a play on short-term gold price increases that's how I see it. Undervalued from a book value perspective from the Gopu project so long term if the company survives which it probably will because it can always sell some assets if necessary it's again a good investment very very long term again. Turko's Hill Resources is another long-term investment. I said that you can buy this company for the next 100 years as their project in Mongolia will be mine for the next 100 years and given the higher copper prices the stock reacted very well to higher copper prices. I still find it a little bit overvalued or let's say fairly valued however with inflation with what will happen with copper in the future let's say it was a much better investment at 270 now it's perhaps a little bit overvalued and if copper prices fall the stock price can give better opportunities to invest. Nevertheless if you again want to park your money for 100 years you should look at this. My favorite miner I really like the Timok project I might be biased but I own this company so nevertheless it's interesting. We discussed is the company the stock is still cheap the real unlocking of the value will happen in five years so 2022 so if you want something where you can also put your money for the long term Nefsson is again an interesting play. It has the option to be acquired by somebody especially after the pre-fizability study it pays a small dividend and there is a huge project and huge exploration potential in Timok. Fairly valued let's say fairly valued at three dollars now also good good value. Norilsk Nickel a very interesting nickel play especially as demand for batteries increases it is up since we recommended it was about 15 copper prices went up platinum prices went up therefore I expect really good things for Norilsk in the long term very good ore bodies very good grays long term mines so really little risk because the cost of producing nickel for Norilsk is negative still see it as a good long-term investment in the short term as always nobody knows what will happen. Hut by Minerals also went up since we recommended it now I see it as fairly priced but if copper prices go higher then the stock will also go higher it has the potential to go to 15 especially when their Rosemont project gets online but for now again a good long-term investment in the short term it will depend on copper prices. Eldorado Corporation it was cheap for me at 240 I didn't buy it I did bought a bit at 130 125 now it's already a bit higher the market capitalization is 1.1 billion the value is at least 3 billion if you leave Greece out you are at 2 billion a lot of bad news have really depressed the stock price but this is what gives opportunities to the shrewd investor really check out my Eldorado video to see what's behind it. Guyana Goldfields the stock price increased since I recommended it it has a good ore body it has a new mine that is being developed still a good long term gold investment be careful with all these gold investment miners they are very volatile in relation to commodity prices so if you find something interesting put it on the watch list and be ready to invest when nobody wants to invest now the environment is getting a little bit hot landing mining again a very interesting company they announced that they will produce less in the next two years to produce 20 percent more over the next decade this at first was negatively accepted by investors but then it become a very positive thing and the stock price recovered it's crazy how the market works so from a broad perspective I still see all these companies as good long-term investments what will happen in the short term nobody knows I own some of them nevsoon norilsk for my kids account Eldorado and a little bit of harmony so I have diversified portfolio that I hope will do well in the very long term and as this is a part of my old weather portfolio I rebalance accordingly when the prices go higher or buy more when the prices are lower thank you for watching looking forward to your comments and I'll see you in the next video in addition have a great great new year


SCP-087 is an unlit platform staircase. Stairs descend on a 38 degree angle for 13 steps, before reaching a semi-circular platform of approximately 3 meters in diameter. Lighting sources brighter than 75 watts have shown to be ineffective, as SCP-087 seems to absorb excess light. Subjects report and audio recordings confirm the distressed vocalizations from what is presumed to be a child between the ages of 1 and 1. The source of the distress calls is estimated to be located approximately 200 meters below the initial platform. However, any attempts to descend the staircase have failed to bring subjects closer to the source.


I grew up in a small town in North Alabama called Fife. I graduated high school, went to a junior college basically because I didn't have a plan to come to Alabama. After a year in junior college, I realized I wanted to study at a four-year institution and pretty easy choice to come to Alabama. When I arrived at the University of Alabama, I guess I would say my life experiences were such that they were insufficient that I could imagine the possibilities and the opportunities I've had since then. I think I came here as a fairly naive small town kid who was just limited by my own life experiences at the time, which were not all that robust, if you will, when you think about it in today's very global world that we live in. Along the way, somewhere I grew up, I guess, and I transformed into a young man that had self-confidence, had skill set, and I think most importantly, the courage, and I got over whatever anxiety I had pretty quickly, and how to get into the groove and understand what college life was all about.


And like, isn't that what Jesus says? That we can do anything, anything, if we believe with our whole heart. I mean, think about, you know, the instance when the disciples were on the little boat or the ship or whatever and it's sinking and they're freaking out. And they're like calling for Jesus and they're worried and they think they're gonna drown.  You have no faith. Why are you even scared? Have you not seen enough? And he's frustrated.  Okay? And it's the same for us. So let me ask you, like, why don't you believe that you were this infinite, incredible creature?


US troops and Philippine soldiers launched air defense missiles for a live fire exercise at a naval base north of Manila on April 25, as part of their annual joint exercises. Patriot and Stinger surface-to-air missile systems, commonly used to shoot down enemy aircraft and missiles, were fired during the exercise, which aimed to improve aerial defense at the naval base located around 140 nautical miles from the disputed Spratly Islands in the South China Sea. We are absolutely committed to going at the pace and the desires of partners, allies in the region. But also to make sure that we are prepared in order to provide ready forces as necessary, said US Army commanding General Brian Gibson. Tuesday's drills are part of a nearly three-week annual bilateral exercise between Manila and the US, with this year's addition being the largest in scale yet, with over 17,000 participating soldiers. Last year in other locations around the end of Pacific, we have done similar things in different ways with new partners. So we are absolutely committed to going at the pace and the desires of partners, allies in the region, but also to make sure that we are prepared in order to provide ready forces as necessary.


Titanium dioxide nanoparticles, TiO2N piece, have been shown to be effective in improving plant growth and health. This study demonstrated that TiO2N piece can increase the biomass production of mung bean plants, which could potentially lead to improved crop yields. Additionally, the study showed that TiO2N piece can stimulate the growth of beneficial microorganisms in the soil, which may help to improve soil fertility. This article was authored by Ramesh Raliya, Predin Bisboss, and J. C. Tarftar.


And welcome to this quick recap of section 7.3 on Euler's method. Euler's method is an algorithm for finding approximate solutions to an initial value problem. An algorithm is a repeatable set of steps that can be used to get to an answer. In the previous section, 7.2, we found that we could estimate solutions to a differential equation by following a slope field. If we had an initial condition, we could follow the rough trend of the slopes to get an idea of what a solution would look like. Euler's method is a more precise way of doing this. Euler's method is named after a famous mathematician named Leonard Euler, and his name is pronounced Euler. Right now you should pause the video and say Euler out loud five times until you start to get used to it. Alright, now that we're back, let's take a look at how Euler's method works graphically, and then we'll take a look at the formulas that make this work. If we're given a differential equation with an initial condition, then we can start at the point represented by that initial condition. We want to follow the slope on the slope field for a little distance. To do that, we'll actually choose a fixed horizontal distance called delta t, and we'll follow the slope at our starting point for that distance. We'll also follow the slope up or down by however much it indicates we should. In this picture, the horizontal axis is the t-axis, so delta t represents a short horizontal distance. After we've arrived at that point, there's another slope, and so we'll follow that slope horizontally for a short distance delta t, and we'll follow up or down however much the slope indicates that we should. When we arrive at that point, there's another slope, and so on. At each point, we follow the slope for a short horizontal distance using the slope to determine how far up or down we should go along that time. Remember that even though we haven't drawn a bunch of slopes here, in fact at this point it looks like there's no slope, we've only drawn a selection of these, and we can use the given formula for the differential equation to calculate the slope at any point. That means that anywhere we end up, we can always find a slope to follow for this short horizontal distance. Here's a picture of a more exact solution given by a better delta t value, a much smaller delta t value. The size of delta t can improve the Euler's method approximation, but it comes at the cost of having to do many other calculations. You can see that the one in red that we calculated first looks reasonable and does follow the same general trend. Now let's take a look at how Euler's method works using some formulas that allow us to repeat these calculations precisely. In this method, we're given a differential equation, an initial condition, and a step size called delta t. This is the short horizontal distance that we'll be following. We'll start at our given point, t0, y0. In order to follow the slope at that point for a short distance, we're going to calculate the t and y values that we arrive at after following the slope. To get the t value is relatively easy. We take our old t value and add delta t to it. Take a look at where this is represented on the image to the right. This is a short horizontal distance. To get our new y value is a little more complicated. We take our old y value, where we began, and we want to add the change in y represented by the slope. To do that, we need to actually calculate this using the items that are highlighted on the screen right now. To do that, we'll use the slope given to us by the differential equation. Remember that we have a formula that tells us the slope at any point. So we'll substitute into that formula our original point, t0, y0. That gives us this slope, which we can interpret as rise over run, and we'll multiply it by the amount of run or the amount of horizontal distance. Together, this gives us a change in y distance or delta y. We'll take that and add it on to our original y value to get our new y value, which we call y1. So now we've arrived at a new point, t1, y1. You should take a look at the picture on the right to make sure that you understand how each of these is related to the previous one. Now that we've arrived at this new point, t1, y1, we can repeat the process to get to yet another point, t2, y2. Again, we add delta t on to our current t value, and we add this formula, the slope at our current point, times our delta t or step size. We add that on to our current y value. So at every step we take our existing t and y value, and we add on to them a formula based on delta t and on the slope at our current point. We can keep repeating this at any point. So if we're at a point tk, yk, where k represents how many steps we've taken, we can calculate our next point called tk plus 1, yk plus 1, by using these formulas. Once we do that, we'll arrive at a new point. We could use Euler's method to calculate the next step beyond that and so on. The formulas on screen right now are the fundamental formulas that allow us to use Euler's method to calculate an estimate for the solution to a differential equation. The picture here represents the general form of an equation, and we can think of this final y value as an estimate for the value of that solution at the given t value, tk plus 1. This method is called an iterative method, meaning that we iterate or repeat the same steps every time using the previous point in these formulas to get a new point. We can keep doing this for as long as we would like. If we'd like to simplify this a little bit, we can see that we're given this differential equation where the function f represents the slope. So instead of writing dy dt at this point, we can write f of tk, yk, which represents exactly the same thing. f and dy dt are both ways of writing the slope at a certain point. We can keep doing this until we've reached the desired t value, at which point we should stop, and we can look at that y value as an estimate for the value of our solution. So how good is Euler's method? Well, suppose that we follow Euler's method with a step size of delta t. Then the error in our estimate of any y value is proportional to the step size delta t. In words, that means that the exact value of the solution, if we knew how to calculate it, minus the estimate given by Euler's method is approximately equal to c times delta t, where c is a constant that depends on the differential equation. The way to understand this is to think that if I made delta t half as large, then the error, that is the difference between the exact value and the Euler's method estimate, would also be half as large. So by making delta t small enough, we can get an answer as precise as we would like. Now that we've seen the idea behind Euler's method, let's take a look at how to use it in practice.


I'm Masin Jueh, I run a software accelerator based in Kenya. We train executives as well as communities to understand technology, especially in data science and cybersecurity. That way they're able to mitigate the risks and technologically speaking the youth are able to have the technical skills that they require to be able to help beat the risks that are out there in cybersecurity. The course program has been very beneficial to us. We've been able to meet so many people from different countries and understand what they're currently doing as well as now we will have interesting conversations as a result of this conference in Africa, in Ghana, in Nigeria, in South Africa, in Mozambique where we will try to explore business opportunities for us in those countries as well as being able to network with other women who are doing great things within the program and communicating with them constantly. They will keep us in check as we progress with BotLab. Being at ITU Telcom World brings a lot of memories and good thoughts especially when you think about the policy makers and the conversations that they had and all those people who are interested in knowing what we actually do and seeing most people wanting to take some of the key things that we learn like for example we went through a session on storytelling storytelling as a communication tool which is very informative in how we are supposed to do our communication strategies as well as the imposter syndrome where it was a workshop that will enable us to feel like we are part of the table and the lastly about investment which was very in-depth because we really understood if we wanted to be funded what are those things that we should think about and start preparing for them now and not when we want to get funded.


Hi everyone, Joe for Jasper's casebreaks.com coming at you with our final allure pack Which is the last giveaway break for the cup hockey that's coming up in a separate video giving away 10 spots And this is break number 22 big. Thanks to this group here for getting in on it. Appreciate it All the teams are in it's a quick little pack break Roll it randomized names and teams three to six nine times different dice roll for the for the giveaway part But nine times for names and teams one Two ninth and final time nine times for the teams And ninth and final time after nine There's the first part of the list right here, and then the next part down here. Let's just print this out Rip that pack, and then we'll re-randomize that list of names here Then we'll see who gets what we'll see who ends up with what ends up with the top 10. I got Jack Hughes Top 50 and one of these open ice and Conor McDavid.  I think that's gonna be huge for not only the sport, but probably for the hobby Probably for Conor McDavid. Let's flip back here It's a quick little pack break. Let's grab everybody's names top ten after Top ten after nine Nine again. We had nine the first time top ten after nine. We'll get into that cup hockey break. Good luck one two three four five six seven eight and Ninth and final time after nine So it's top ten once again after nine So from 11 on down right there Eric. Sorry Very close. No cigar, but I appreciate you giving this a shot Thanks everybody Sad times for them, but happy times for the top ten Steve Jonathan Jonathan Steve Steve two different Steve's Sean Jonathan Mike Jonathan and Jack Happy times for the top ten. You are in That cup hockey break. We'll see you for that in a separate video. Jasper's casewrakes.com


So this week we're going to be talking about mobile device forensics, starting to work with mobile devices, all the different types of mobile devices, and how we normally go about acquiring data and analyzing the data from them. So first off, the biggest thing to think about with mobile devices is that there's no single standardized method for accessing all mobile device data. So think about all of the different types of phones or tablets or anything we consider a mobile device now, even smart watches and things like that. They don't all have the same, necessarily the same interface. They don't have the same software. So trying to make a general approach to acquiring data from mobile devices is very difficult. With computer systems we basically have PCs and Macs more or less. With mobile devices or even these kind of smart devices now we have a lot of different architectures that we're dealing with and that makes it very difficult to first off acquire data and also analyze the data structures because there's many different types. Luckily within the last few years we have somewhat standardized at least smartphones and the way that smartphones are accessed, so we essentially have a few different types but they're very, very common. Most people have more or less three or four, one of the big four types of devices that we can access. So we pretty much know what we're going to find whenever we get, for example, a mobile phone. So there are a lot of tools now that can access the more common types of phones. However, there are some, for example, Chinese brands that completely change the way that they do wiring or they completely change software. We run into problems with those sometimes. So just be aware whenever it comes to mobile, I will talk about basically the most general approaches to acquiring data but technically the way that we get that data will be very different depending on what type of device you're looking at. And it also depends on the software and the tools that you have available. I'll be going over some of the free software that we use and I'll be also covering mostly Android devices just because at least in Korea they're the most common and they're in some ways easier to work with than other types. With mobile devices there are some companies that have software that can do quite a bit of automated analysis of mobile device data but in my experience most of the time you're going to have a lot of manual analysis with mobile device data. The process of acquiring data is starting to get a little bit easier but the process of analyzing that data is still pretty manual. However you can write your own tools, we'll talk maybe a little bit about writing your own tools for examination. That being said there's no single tool that will cover all mobile devices and all situations. So all of the tools that are out there, there are some very good tools that are very comprehensive in both acquisition and a little bit of analysis but they won't cover everything. So especially if you're dealing with law enforcement you're going to get a lot of different types of devices in and sometimes your tools just won't work. Some devices will be too old so your tools can't support them. Some devices will be too new so your tools can't support them. Some devices maybe the manufacturer just never saw before so they don't have a way to acquire that data. So when dealing with mobile devices the investigator has to know a lot more than just one simple tool. They have to know how to access this data manually essentially. So for forensic preservation we always start just like with computers. We want to preserve the data on the suspect or victim's device. In this case it just happens to be a mobile device, most likely a phone. In Korea it's most likely a smartphone, not just a phone but probably a smartphone and also could be things like the smartwatches like I said. They also potentially have some information on them that could be relevant. So the easiest way just like with computers, the easiest way to preserve information about the device is just manual examination. So the examiner manually accesses the phone through the user interface. So you have a cell phone for example and you want to know who are in the contacts list so you unlock the phone if you can and then go to the contacts list and scroll down and basically take pictures from there. This is the least forensically sound because you are modifying data on the phone. You are accessing the phone directly but sometimes that's the only way we can get access to data. So this kind of manual operation of the user interface can produce valid results but your documentation has to be very good and you have to justify why are you interacting with the phone directly. So it's potentially the easiest but also the least forensically sound. To ensure that all details are documented in the chain of custody is preserved this process is normally photographed or videotaped. So there are some devices for example like a video camera that essentially hangs over the phone and you can access the phone directly so it records everything that's going on. That seems to be one of the most common ways if we have to do a manual analysis of the phone using the interface. Only data available through the operating system is retrievable. So the big if we're looking only for user information this might be an acceptable way to get access to it because we can potentially access like chats, Kakaotalk, Facebook, Instagram, all of those things using the user interface just like the suspect or just like the victim would however there's a lot of information that we will not be able to recover that is limited essentially to the operating system only. So we're going to be missing a lot of information. If we're just looking for basic account information it's okay if we're looking for contacts or who called the phone, who was this phone used to call, things like that then this method might be a way to go. Next is a logical acquisition via the communication port. So all phones or all mobile devices have usually some sort of communication port. On a lot of devices it's the same as the charging port. You can for example your charging point can also be USB connection to your computer so you can transfer files or whatever. It could also be for example a wireless connection. So Wi-Fi, Bluetooth, all of these connections are on a lot of devices these days. So you might be able to get access to the device from one of these interfaces and basically do a logical acquisition or an acquisition of the partition information on the disk. We normally go through the USB port on newer phones, whatever the connection port is, we connect it via USB and we can normally get access to most data on the system but not all. So logical acquisition methods interact with the mobile devices using protocols such as AT commands and OBEX or object exchange commands. Mostly we're using AT commands to talk to the low level hardware of the phone and try to get that low level hardware to give us raw data. There are tools that automatically do this. Again it depends on the operating system we're using but I'll give you instructions on how to do that with your own phone if you want to try. But AT commands are basically very very low level computer commands that we can give most devices. And this only extracts data that is available again through the operating system. So we're not getting any special access whenever we do this. The operating system itself is still restricting us from getting access to all of the data. We can only get access mostly to user accessible areas. So the next is a physical acquisition via communication ports. So again using USB connection to try to get a physical image of the disk rather than just a logical image. This is actually going around the operating system rather than using the operating system directly. This extracts the memory contents and their entirety through the communications port. It's also much more difficult to do. So out of the three that I've talked about so far this is much harder than the other methods. The other methods can get us a lot of user data but maybe the court doesn't accept it for some reason. They want a full disk image. The full disk image is possible but also much more difficult to get. Interpreting the extracted binary is dependent on understanding how the phone stores data in memory structures. So again we're getting raw data from the phone and we have to understand how to parse out those data structures. So not only is the acquisition more difficult but the actual analysis of the data becomes much more difficult as well. If it's a popular type of phone or a popular operating system there are tools to help you. But if it's a new phone or a new operating system or just a new data structure that nobody knows about you'll have to do it yourself. So it becomes very tricky to analyze mobile data if it's a physical acquisition. Next is physical acquisition using JTAG. This is quite a popular method for really advanced mobile device forensics. I'll give links to JTAG and what JTAG is, how it works on the form. It uses a JTAG interface which is just kind of a hardware interface to extract memory contents of the device. So we connect via JTAG and then we can read out all of memory from essentially a hardware port. Using JTAG usually requires us to take apart the phone and on most phones that completely destroys them. You can't put them back together. So we only use this in extreme circumstances like terrorism cases or something where we can justify the fact that we don't want only a logical acquisition and we probably destroy the phone. So in those cases we might use JTAG if it's absolutely necessary. It allows the extraction of full binaries so the whole point behind this is that we can extract all of the data. We still have to know what those data structures are but at least we can get the data and then analyze it later. Acquiring digital evidence view of the JTAG is less intrusive than relying on the device operating system. We're not going through the operating system in this case but interpreting the extracted binary requires a lot of in-depth knowledge. So mobile device analysts who are expert at it might want to do this method but we're nowhere near there. I'll give you information on what JTAG is but we obviously can't practice it in this class unless you want to tear your phone apart. Then you can try physical acquisition via direct memory chip access. So another method we can use is accessing the memory in the phone directly. We again have to take apart the phone if the phone isn't designed to be taken apart we probably destroy it and can't put it back together. But we can get access to the memory chips directly and then just read data directly off of them. It's the most low level and potentially complex acquisition method. It has the same problems as all of the other methods and the fact that we get direct raw data and we have to do all of the interpretation for that raw data. Nothing is interpreted for us. We can provide access to all device content but requires knowledge of interpreting the raw data structures. And this technique should not be used for cases when the original device must remain operable. Korea for certain types of phones are very good at this technique. They have really good experts that can do this and then put the phones back together. But in most cases the phone is going to be completely destroyed. Which is another problem because if you destroy the device that means that first off you might lose some data. But it also means that nobody else can verify the data that was on the device. The device is gone. No third party can come in and audit you which raises huge concerns in cases. So it's advisable but maybe not practical to acquire data from multiple using multiple methods. So for example collecting data using a logical acquisition is relatively easy. We can get quite a bit of information out but not all information. If once we have that information out we already have it in our in our custody let's say then we might want to go to a lower level acquisition method that may lose data or may destroy the device. Because at least we have a backup copy and we can say okay the user data is here and it's valid. So really the question is how do we connect to these devices? How can we connect and how can we get information from the device? The easiest way is basically connecting via USB. For a lot of different types of phones we can install a piece of software on the phone and that software will get access to quite a bit but not all of the data and then send that data to us. That's a very very easy way to do it. And if we're using free tools that's probably what we're actually going to do this week is that method just because it's the easiest. There are some forensic tools that exist that either automate that process or they're able to get lower level access and I'll give you links to those and we'll talk a little bit about them. However those tools tend to be quite expensive. Wireless access and Bluetooth in some cases we can get access to data over the network. Depending on what our case is we might just be able to leave the phone on and connecting over the network and monitor the network connection to get access to data that way. However in most cases like if we're monitoring an app for example it's probably going to send all data encrypted from the phone so we won't get access to much but it might be possible. We might also be able to attack the phone from the network and get remote access to it or load a program in and copy data out. There's a lot of different approaches we can use. Just think about what are all of the ways that I can connect to the phone and potentially install and extract data and we'll talk about the most common ways this week. Extracting data we're normally using AT commands like I said very very low level commands. We use developer tools so for just as an investigator of mobile devices you have to get used to understanding what developer tools do, how they work, where you can download them because those developer tools have low level access to most data on many different types of devices. We can use those developer tools to copy data from the phone in a reliable way and sometimes even send data to the phone if we want to send a program and run it for example. Physical memory acquisition on mobile devices it's more complete and it includes deleted items. It works with damaged devices so sometimes for example we've had cases where suspects had a phone the police came and they immediately threw the phone down and stomped on the phone. It completely broke the glass and the display didn't work but the phone still works in some cases most cases the phone will still work and we can get access to the data from that device even if we can't necessarily see it. It also makes fewer alterations to data it's not easy and it doesn't provide the logical structure of file system. So again the lower level you go you do not get this structure of the file system you can't just pull files directly out you have to actually parse the data. Again a lot of tools can exist or do exist that you can buy that will help you automate parsing of those structures. Okay and then I have some examples of some tools connecting different types of phones to tools. These tools are again quite expensive but also very useful in most cases. If you join law enforcement, if you join a company doing investigations they will definitely have at least a few kits around for different types of phones. So I give you some examples of those. And then we're just thinking about what exactly is the point of the analysis or of acquiring and analyzing the phone. So we have to go through quite a bit of trouble to get data from a phone. So why do we actually want to go through that process? What are we looking for exactly? If we're just looking for a phone number do we need to analyze the entire phone or can we just analyze the SIM card? If we're looking for proof that somebody downloaded MP3s to their phone maybe they downloaded them to an external micro SD card and we can just analyze the micro SD card instead of the entire phone. What exactly are we trying to prove? And again where is that data located? So just like we talked about in the other lectures whenever we're doing our acquisition we're thinking about what am I trying to prove? Where is the data going to be located? With phones it's even more important because there's lots of different types of data. Some data is very easy to get access to, some data is very hard to get access to. So do you need to go through the difficult process of getting all of the data or can some easier data suffice for your investigation? A general investigation approach is first off survey any available items you have. So for mobile devices, for say smartphones, smartwatches, tablets, whatever you have from the suspect what devices do you have and do you know how to get access to data? Do you know how to get access to all of the data or just parts? And do you know how to verify that that data is the same as the original? What are the main sources of information on the mobile device? So again for mobile you might be mostly interested in like cacao talk chats or something like that, MMS, just communication between people. If that's the case you might only focus in on certain programs and not the entire device. We want to recover any deleted items. So just like computers we can recover deleted items, deleted information from the phone. I mean it still has an operating system and on the disk at least it still has a file system just like a computer, the file system is a little bit different but it still has a file system. Which means that in some cases we can recover for example deleted items. So I think there was a case with, what was the program? I think it was Snapchat which said that it's, whenever you send information and you delete a file then that file is completely gone and we found out we could actually recover images that were sent that way. So a lot of these programs make claims that they are completely secure, they're deleting all the information, they might be deleting the information but just because of the way that file systems work, just because the way that data is stored we might still be able to get access to some or even all of the information that they claim was deleted. Then we harvest metadata from the active recovered items. So what kinds of metadata or what kinds of timelines can we create? So with metadata we're mostly interested in timestamps and creating timelines. We're mostly interested in metadata and creating timelines of events. When were these pictures downloaded? When were these pictures taken? And what kinds of timelines on the phone can we actually reconstruct from that? So metadata is a very important component of mobile analysis. And then finally conducting a search and inspection for evidence, basically looking through all the data, looking for keywords, looking for basically anything that we think might be related to our investigation much like we did with computer systems. So the analysis, the acquisition on mobile devices is going to be a little bit different but the analysis is going to be quite similar to desktop computer systems. So that's it for this week. We will actually go through acquiring and doing a little bit of analysis of a mobile device. So that's the homework for this week. Thank you.


Great job, scholars. I love the way you all came over showing me that you are ready for math this morning. The learning target for today is on the board. Let's read the learning target together. One, two, three, read. I never regret and saw the big digits of finding the quiet and decomposing remainder in the hundreds place. So scholars, I need someone to paraphrase in your own words what the learning target is asking us to do today. Ray Sean, what is the learning target saying that we will do today? This learning target is saying what we want to do today is we want to divide and decompose our remainder by using the place value chart in the hundreds place. By having our remainder in the hundreds place when we're trying to divide and then we can decompose it and then we get our remainder. So you're saying that we will be dividing and then we will decompose a remainder in the hundreds place? Try again. This learning target is trying to tell us that we're going to have a three-digit dividend requiring we have to decompose it and then we're going to have a remainder in the hundreds place so we're going to have to find out where to put the remainder at. In a different place value. So we'll have to decompose the remainder and you're saying we'll do what with it? We'll move it to a different place value. Did you want to add to that David? I would say that in the hundreds place you have a remainder in the hundreds place but you can't have a remainder in the hundreds place because it's not the smallest unit so you would have to decompose that number and bring it to the tens place and then if you have a remainder in the tens place you get it to the ones place because the ones is the smallest unit. Okay well David and Reshawn thank you for your responses and I like how you're thinking about the learning target.


Alina Rabakina vs. Igor Sfiantec is the second semi-final for the ladies at Indian Wells for 2023. Both players have had different roads to get to the semi-finals. Sfiantec started the tournament off as the number one seed and will get a buy in the first round. In the second round she'll take on Lu, the American. But this was no problem for Sfiantec, getting through in straights. Six loves, Six won. In the third round she'll take on the 32nd seed on Drescu, who she had played last year on the clay. And after a close second set Sfiantec would have to fight to get through 6-3-7-6 to get to the fourth round. In the fourth round she'll take on Emirado Canu, who had been very impressive this week beating Hedaj Maya in the third round. But this didn't faze Sfiantec getting through in straights sets easily. 6-3-6-1 to get to another quarter-final. Sfiantec entered the quarter-finals and would take on Kasteya, the unseeded player in that quarter-final. But again, this was an easy road for Sfiantec getting through in straights. 6-2-6-3 to advance to another Indian Wells semi-final. Rabakina started the tournament off as the number 10 seed and will get a buy in the first round. In the second round she'll take on Sofia, Kenan, a fellow Grand Same champion. And this would be a very tough match for Rabakina getting through in straights 7-6-7-6. In the third round she'll take on 2021 champion 21st seed Bedosa. And after a second set that was close, Rabakina would eventually get through 6-3-7-5. In the fourth round she'll take on the qualifier Gracheva, who had just beaten Kazakina along the way. And this was by far the best match for Rabakina getting through in straights sets 6-3-6 love to advance to the quarter-finals. In the quarter-finals Rabakina would take on the unseeded Mukava, who had been very impressive this week. And after a close first set, this match would go down to three sets, with Rabakina getting through 7-6-2-6-6-4. Into her first semi-final since the Australian Open. These two have played two times before with the record standing at 1-1 with their previous meeting being the Australian Open. Rabakina taking that in straights sets. Rabakina is going to win this match. She needs to serve big and push Fiontek around and use that power of hers to her advantage. In the Mukava match she was running around a lot and that's really not her strength. So she needs to make sure she dominates those baseline rallies pushing Fiontek around. If Fiontek's going to win this one, it's simple. She needs to push Rabakina around. Mukava did it very, very well but just fell apart towards the end of the match. But we know Fiontek doesn't fall apart in matches like this. So she's going to push Rabakina around and maybe come to the net and finish points off in that service box area. This is going to be a very highly anticipated matchup because of course, Australian Open was only a couple of months ago and it was an easy win on score to Rabakina. I think Fiontek's going to get revenge though in this one and win in three sets. I think Rabakina's pace isn't going to work as well on the slower hard course like it did in Australia. And Fiontek's going to take advantage, get to the final. Let me know down in the comments below who you think is going to win this one.


We had another handshake gate in this pit Georgia Tech game pits leading by 14 points with 30 seconds left Georgia Tech has a brand new quarterback and I think that's like his second pass ever because their first quarterback just got hurt goes for the fake spike to the end zone.