Okay. My name is Dean Wampler from Object Mentor, that's Uncle Bob's company, and I've had too much caffeine this morning, so if I talk too fast, slow me down. What I'm going to talk about is functional programming and why that's interesting for Ruby programmers. Jim Weirich told us the other night that we need to start our presentations with a story, so here we go. Once upon a time in a kingdom far away, there was a law that ruled the land, and it was called Moore's Law, and everything was hunky-dory for a number of years. The processor speed, performance, et cetera of the processors just increased, basically doubled every 18 months, which is an exponential increase. But then there was a tremor on the force, if I can mix my metaphors, and a few years ago we started to see that turn over to go flat. And so in order to increase our continual gains in performance, people, basically the overlords that tell us what to do said, well, let's just add a lot more CPUs and add a lot more cores in our CPUs, and we'll just keep scaling that way. Unfortunately, the scribes in the kingdom said, so how the heck are we going to write robust concurrent software for all these CPUs? And it turns out the ivory tower scribes in the kingdom said, we've known how to do this all along. It's called functional programming, and if you guys had just listened to us all this time, you wouldn't have this problem. So the scribes ask what is functional programming, and that brings us to today, and hopefully the outcome of the story will be a good one. But what we're going to first do is talk about functional programming a little bit. So isn't it true that we already write functions in our Ruby code?  Well, when we talk about functional programming, we're really talking about what functions mean in the mathematical sense. And so, for example, if I pick the value of x in sine of x, then I'm going to fix the value of y. Therefore, this is not working, variables are immutable in functional programming. We can assign them once, but we can't keep assigning the same variables. We're used to thinking of variables as variable, but once you've picked all but one in your equation, then the rest are fixed. The other thing that's very interesting about mathematical functions is that they don't change any state. There's no state variables or object variables that they're changing. They do whatever they do, and then they return everything that was done in the return value, and that's it. And the beauty part is, we say these are side effect free functions. And what's nice about that is they're very easy to reason about. I don't have to think about the global system to understand what this function does. I can call it as many times as I want, and as long as I use the same value, passed in, I'll get the same results. I can run as many of these at the same time as I want, and I don't have to worry about threading issues. So they're really handy for all the problems that we face in software today. And then getting back then to concurrency, if we don't have any mutable state because their variables are immutable and we don't have side effects, it turns out there's nothing to synchronize anymore. And almost all of the problems in concurrent programming deal with synchronizing access to mutable state. It's important to know that it's mutable state we were worried about, because if it's immutable, we don't care how many people are reading it simultaneously. Turns out Jim Wyrick, right after lunch, is giving a great talk on threading. And it would be really good to attend that talk if you want to learn more about the issues of concurrent access to mutable state. Another property of mathematical functions is they are composable. I can build new mathematical functions by lopping together a bunch of other ones, like this definition of cosine x in terms of square roots and sine x. And we say that these are first class citizens. So I'm passing these functions on the right-hand side as if they are values, like data. And one of the ideas behind the list is that code is basically data. What we like are having first class functions. We like to be able to assign our functions to variables, pass them around, manipulate them, and so forth. So let's talk a little bit about how to do this stuff in Ruby. Let's talk first about immutability. So here's a classic definition of a person object. And of course, one way to make it immutable is just to have attribute readers but no writers. Now, I'm sort of elitding the definition of the initialized method. You know what it's gonna do, so it's not really worth showing. So if I actually instantiate one of these persons, I can read the first name, and that'll return Dean. But if I try to set it to some new value like Bubba, that's not gonna work. I'll get an undefined method error. So I can't change the state of this object. Pretty trivial stuff. Shameless plug. Turns out this is that same definition in Scala. It's one line. Well, it's two lines because I had to wrap it. But you basically, it does all the magic for you that you have to do in Ruby. So we think of Ruby as very succinct and elegant, but there are some languages that are maybe a little bit more succinct in some ways. I'm saying this because I'm actually writing a book on Scala. This is a shameless plug. There's another way we could actually handle immutability. I suppose that we don't want all person objects to be immutable. We want some immutable, some not immutable. Well, we could always have our read-write accessor. But then we could just freeze the individual objects we care about. So before I called freeze, I could rename any field in the class. But afterwards it's frozen, I can't modify it. However, it's important to remember that I can assign new objects to the same variable, the Dean object. That's not disallowed. Even though I've frozen it and all kinds of stuff, it's still letting me do that. What I froze was the object, not the thing that's pointing to it. So this is fine. So it's important to keep those kinds of things in mind. Now there is a drawback, nothing comes for free in this world. And if you're creating lots of little objects because say I'm merging two hashes together and just creating a new one rather than doing an in place merge, there's the risk I'm going to create a lot of objects. There's a lot of overhead, garbage collection, all the usual stuff. So there's always performance issues to be wary of. Okay, let's talk about side effect free functions in Ruby. What does that mean for us? Which ones of these are side effect free? Well, certainly the attribute readers are side effect free. We just talked about that. Obviously the accessors, well, this is generating readers and writers. And of course the writers are not side effect free because they're changing things. Obviously the initialized method can't be side effect free because it's got to set up the state for the person object. But that's the sort of thing we're willing to live with. And hopefully we don't get objects referenced before the initializer finishes. I'm not actually sure what the Ruby VMs do about that. But like in the Java VM, for example, you cannot access an object until the constructor finishes. Now this method is, at least on the outside, we don't know what the inside is doing. That is inside the select method. But this is also side effect free because I'm not modifying the array. I'm just building a new one by selecting on the elements. However, it's important to think about what might happen if this array that was passed into me is being modified by another thread while I'm iterating through it. What's going to happen then? So even though this method in isolation is side effect free, maybe in the context of a concurrent system, I have to worry about what's happening to the array. And which ones of these things are side effect free and what's actually mutable? Here's a problem right here. I have to change the value of this loop index every time I go through it. So that's an issue for us. Now down here, it might look like this is side effect free. But in fact, when I'm printing to standard output, I'm changing global state and effect. So in fact, these methods, like prints, are not actually side effect free. Now maybe we don't care about that, that might be okay. But it's still something to think about that there's something going on, some status being changed in my global system. So then what do we have in terms of first class functions in Ruby? So a little example here, let's define a predicate. And all this is going to do is return true if the first character in a string is the A character. And then I'll have like a filter method. Actually, we saw this before. Well, I'll pass in an array and I'll yield to whatever block was passed in, sort of not shown on each element in the array. And then I'll just set up an array of arbitrary strings and I'll call filter. And notice what I had to do here, I had to wrap the predicate in a block. And then that will actually call that, is this an start with an A kind of string. And then it'll return these three strings out of this list of strings. So pretty straightforward Ruby. You could also do this trick where you call the method and get the predicate thing as a method object. And then I could use that by basically turning it into a proc. Personally, I think this is cleaner, clearer. You might like this better because it's cooler. I don't know. But what I cannot do though, I can't just pass that predicate method, even at predicate, directly as a block to this function. I can't really do that. I have to do at least something like this or maybe some other trick that's like that. So we can really think of blocks as being first class citizens in sort of the functional programming way, but not methods. The methods really are not first class functions because we have to do those little tricks, but it's not so bad. We can live with this, I think, most of the time. Another thing you'll see a lot in functional languages is recursion. Because again, I can't change this loop counter. So somehow I have to do looping when I can't use each for some reason or whatever. So this would not be allowed in pure functional languages. And the way they do that is with recursion. So I might have this print array, or I just want to output the contents of the array. It's basically the same thing I showed on the previous slide. I'll pass in the array and the index into it that I care about. I'll return if I happen to be at the end of the array and there's nothing left to print. Otherwise, I'll print that ith element of the array. And then I'll call the function again. This time, passing the array unmodified, but passing a i plus 1 is the value. So notice I'm not actually modifying any state in this method. These are all going to be new values on the stack. But the question you might ask yourself, is this really better than a loop? I mean, it might be better in terms of thread safety, but it's maybe not as clear what's going on as that previous loop thing. So it's kind of nice in a language like Ruby where we can use imperative techniques, sometimes when that's best, and functional techniques when they're best. So let's actually look at a slightly more interesting example. Does anybody know what this is? This spiral. What was that? Yeah, it's actually part of the golden ratio. This is actually the Fibonacci spiral.  This one has side 2, side 2. This is 3 by 3, 5 by 5, 8 by 8, et cetera. This is actually called the Fibonacci spiral. And it's basically based on the Fibonacci numbers. Now, by the way, did anyone, if you went to the Aristotle talk this morning, you saw a very similar spiral, just one of the pictures he showed in the talk. So this is actually something you see in nature a lot. This is like the shell of a nautilus, for example, follows this spiral. And it is actually, I think it's in the asymptotic limit. It turns out the ratio between the Fibonacci numbers is equal to the golden ratio 1.6, whatever. Really, a cool example of how simple mathematics has profound implications in nature. But anyway, here's how you would define it in a math book, for example. If n is 0, just return 0. If it's 1, return 1. Otherwise, return Fibonacci n minus 1 plus Fibonacci n minus 2. Nice elegant little definition of this sequence. And if we write it in Ruby, it has sort of a similar elegant quality to it. I'll just use a case statement, so I think you can pretty well see what's going on. I think there's a bug in this, actually. It should be case n. Sorry about that. It's funny how you notice things right when you're doing the talk, but not when you've been reading the slides over and over and over again. What I think is interesting, though, about this definition is that I'm using case matching. And we'll get back to this as something else that's very profound in functional programming in a minute. But what I love about this definition, except for the bugs in it, is that it's very clean and concise. If n is 0 or 1, then we'll just return n. Otherwise, make the recursive call to F of n minus 1 or F of n minus 2. Now, there is an issue you have to be aware of, though. And that is stack overflow. If I ask for the Fibonacci number of 1,000 or something, I'll probably blow out the stack. And it turns out that while this is not a good example, there are tail recursion optimizations that can be done and are done in most functional languages. And also is coming, I noticed yesterday, in Ruby 192. But you do have to be aware of this. And you could certainly rewrite this as a loop to avoid stack overflow. So again, nothing that comes for free. There's always drawbacks to any decision we make. There's a set of three classic operations that you see a lot in functional languages. And the list data type is like the classic thing that everybody uses in functional languages like Lisp, which isn't pure functional, but we don't need to get into that. First of all, there's mapping a list from one set of elements to another where I do some transformation on each element. I might want to filter the list like what I showed earlier where I just want to pull out a subset. And then there's this thing called folder reduce. Say I have an array of integers and I want to add up the total of those elements, then that would be a folding operation where I'm basically reducing things. And of course, we have these in Ruby. So for any of the standard collections that we're used to, we already have a map function by name. The filter is like find all or grep. And there's obviously synonyms for a lot of these. And then we're used to using inject as our fold and reduce method. Now here's something that's important to remember. And that is no matter how complex your object graph is, it always reduces at some level, in some sort of tree structure, down to primitives like numbers and strings and stuff like that. And collections of other objects and primitives. And if you keep that in mind, then sometimes it's actually better to have an external function that's not like a method in the class and just apply that function to your collection in the class. And just to provide an example, let's go back to our person class. And let's say we're going to have a list of addresses. And that's just part of the stuff that's passed into the initializer. And notice that I want to freeze this array so that nobody can modify it. I want to keep my immutable object here. Come on, here we go. And then I'll just create a new person, pass in a bunch of addresses like the Apple address at the last line there. And it's fine for me to call each on those addresses. That's OK. But if I do in place sort, then that's going to blow up and fail because the object is frozen, the list of addresses. But the interesting thing, though, is if you think about what behaviors might I implement outside the type? Well, I might define a validate address method that'll make sure that any address in my list is actually in the United States Postal Service database. And you can actually buy these databases, for example. And then I'll just iterate through my addresses. I might even do this inside the initializer. And I'll make sure that this is a valid address you've given me, otherwise I'll raise some exception. But that's not necessarily something you want in the person class, right? Even in an address class, maybe you don't want to have this kind of context-specific logic embedded in it. So pull it out somewhere else, but use the fact that you have a collection that you can work with to do this validation, to get access to the data when you need it. Another quality you see in functional code a lot is it's very declarative. Remember the initial definition of the Fibonacci sequence that was this nice little curly brace? And then if n is 0, it's blah, blah, blah. And pretty much this buggy code kind of looks the same way, where I'm using the case statement to define the different clauses. Well, it turns out domain-specific languages often have this quality of being more declarative than imperative. And the classic example, of course, is from Active Record, where I describe just the relationship between person and other types in the system. But I don't tell you how to do it. I just say what I want and then let the system figure out how to do it. So it's nice and succinct. It's obviously high productivity for the developer to just sort of declaratively say what I want and let Rails figure out how to do it. And it also gives Rails the freedom to do it in a bunch of different ways if needed to optimize and so forth. So when you can avoid being imperative and be more declarative, that usually has a lot of benefits. OK, so we talked a little bit about concurrency being interesting for us. And I hope that you also saw that there might be other qualities, other benefits of functional programming beyond concurrency. But I want to talk about a particular approach that you see in some functional languages. Erlang, for example, and Scala use the actor model of concurrency. And the idea here is that you have these autonomous agents maybe running in a separate thread or a fiber or some mechanism like that. And the only way they communicate is through messaging rather than trying to manipulate the same shared state. And I picked an example from one of the three actor libraries.  There's going to be a talk tomorrow, I think it's tomorrow, on another one called Dramatis. And there's also one called Revactor that's saying primarily at Ruby 1.9. But here's the idea. So I include all the gem stuff. I'm going to use two objects that will basically be the messages I send back and forth. The first one will be request a greeting. And I'm going to pass in the reply to will be like the mail address of the actor that's asking to be greeted. And this is sort of a common model, too, is that all these actors have a mailbox and you're just sending messages to a known address to talk to them. And then the actual greeting will be the object that comes back, and that will be hello Bob or whatever. And I apologize for this slide. It's the most complex in the talk, but I couldn't really trim it down any farther. So let's just go through it. So I'm going to spawn a new actor that could be in a separate thread or a fiber or something like that. And here's my default greeting, which will just be hello. And then I'm going to loop forever and wait for messages. And when they come in, I'll receive the message. And then I'll do a comparison. And these are like Ruby case statements, actually. And in fact, there's a gem that's part of this package called the case gem that adds more behaviors for doing matching on types. So if you happen to send me a message that's a greeting object, then I'm going to save that as the default greeting. And notice I'm going to duplicate it rather than risk having access to shared states. So I'll just duplicate the value. So now I've reset the greeting to something else. But if you request a greeting, and I apologize, I had to abbreviate the word request, again, for space reasons, then I'm going to actually reply to you. And this is how you send a message to an actor. You use the less than, less than operator, if you want to call it that. Remember that this request greeting had a field that was the reply to mail address. And so now I'm going to send you a new object that's basically the greeting. Again, these are structs. So that's why we have the array syntax here. I'll send you that greeting that was maybe set previously, or it was the default, whatever the case was. And then, obviously, I have to close all my blocks, if you will. OK, so I hope that's clear. But really, the key point, though, remember I mentioned pattern matching earlier when I was talking about the definition of the Fibonacci number? This is an extended version of essentially what a case statement might look like, where I'm now matching on type fields. And then here's how I might actually use it, where I'll have some oracle that does the greeting. And I'll send him a message to change the greeting to howdy. There here's a way of getting the current actor in my current thread or whatever. And then I'll send another message to the oracle to request the greeting and pass my email address, if you will, so that I can get the reply back. And then I'll actually loop myself as an actor and wait for a greeting to come back. And then I'll just print that to the screen. And to actually show this, I think I didn't. Anyway, all right, I just said that already. But it'll come back in this case and say howdy, Dean. It turns out that in functional programming, they use pattern matching as sort of this fundamental concept of modularity in kind of the same way we use polymorphism in object-oriented programming. Now, it turns out that pattern matching like case and switch statements is kind of a bad wrap in object-oriented programming for reasons like the open-close principle. If I add a new type, oh my god, I have to open all my case statements again and handle this new type. But it turns out that that's not necessarily the case. And I wish I had more time to go into this. But I think there's a great synergy between using case statements or pattern matching at boundaries of things, like boundaries of messaging coming into an actor and then using polymorphism as sort of the internal behavior. And it's something that's worth investigating. Maybe that's homework, if you will. It turns out Scala has some really elegant ways of combining these two things. There I said it again. I used that word Scala. How about that? All right, let's recap some of the messages learned from this. First, prefer immutability in your code. A nice thing besides concurrency is that if I have an immutable object, I don't care who I give it to and have to worry about them messing it up in some way. So if you ask me, give me your list of addresses. If that list is frozen, I don't have to worry about you doing something malicious like adding another address or deleting addresses. So this is really a general principle that whenever you can use immutable data in your code, it's usually going to be better code, more robust, easier to reason about, and all that good stuff. Also, prefer side effect free functions, methods, whatever term, or even blocks. The advantages are many. It's very easy to reason about the behavior and isolation of that method. It's very easy to just call it as many times as you want and not worry about what's going on. It's, hence, like a concurrent environment where you have multiple threads calling the same method. You just don't care. Now, it's important to mention, too, that the sort of distinction between what's visible on the outside and what may be going on on the inside. My Fibonacci method might be side effect free on the outside.  But internally, I might actually do some imperative stuff, like maybe cache previously requested value so I don't compute them over and over again. That means that if I choose an implementation decision like that, I'm going to have to be careful that I don't break this quality of side effect free-ness, at least from the outside, in particular that it remains thread safe if somebody actually uses it concurrently. So the nice thing about that, though, is that I give somebody something that behaves in a thread safe way and they can sleep well at night knowing it's not going to cause them problems. And I basically isolated the problem of making sure that it's thread safe in sort of a well-known context. Whereas if we don't think about thread safety at all, if we just have side effect functions everywhere and mutable data everywhere, then we've made the problem much bigger of making sure it's thread safe. So it is important to think about sort of what's the boundary of where I want to use imperative, non-functional idioms versus when I want to stay in sort of a functional mode. The other thing to ask yourself then is if I have these collections and I can apply functions to them, maybe I don't need all the behavior possible inside the type, maybe I can have some behaviors like validating addresses in an external place and just keep sort of the essence of the type pristine. Sort of the platonic vision if I can use the philosophy metaphor since we've had a great talk about that already. What is the essence of a person versus what's all the external stuff that a person might need to do or I might need to do to a person and so forth. So sort of defining that boundary is useful as well and thinking about functions and applications of functions and composability is a good way to kind of frame that discussion. And be careful not to get too carried away with these functional idioms because you're going to run into issues like what's the overhead of making all these little copies, what's the overhead and the risk of blowing the stack up if I do a lot of heavy duty recursion, performance as well as what's the boundaries of the allocatable memory for the stack and so forth. And this is your homework assignment. Think about how to combine pattern matching more effectively with polymorphism, especially at the boundaries of like subsystems, actors, whatever. And I do encourage you to go to Jim's talk after lunch and the dramatic talk tomorrow to learn about actors in more detail. And I actually finished a little earlier than I thought. Any questions? So when you were looking at functions and their side effects with collections, you were only concerned with the collection itself and not the elements in the collection. What's the reason for that? If you're changing elements within the collection, couldn't that cause problems? Right, so if I'm only thinking about, to repeat the question, thinking about manipulating the collection but kind of ignoring maybe what might be happening to the items in the collection, then that could be a problem, right? Did I say that reasonably well? OK. Yeah, and that's one reason why I called freeze on that list of addresses. I wanted to make sure that you didn't modify the elements in that. Actually, does that work?  See, you get into these issues when you start composing data structures of what am I actually freezing and what's actually not frozen. So for example, if I didn't freeze the addresses in the person, maybe I can't modify the address list directly, but I would have been able to manipulate the stuff inside it. So yeah, you do have to think about that. And I did mention that even though I'm iterating through an array, say to print it out, it may look like it's, well, it's not entirely side effect free, but the array looks like I'm not touching it. In fact, somebody else might be touching it. And functional languages tend to be a little better about giving you these primitives to lock down things. And in Ruby, we kind of have to apply what we have available to us. And also partly, be thinking in our minds, is there a way that I can preserve immutability throughout my code to make sure that I don't have to worry about somebody damaging the type or rather the object somewhere? In the back. I think we would like to pause and process the specification there, that there are arguments. If that variable is already in scope. So it's worth to be saying, X is side one. From say, array 1, 2, 3, we need to do X, what's X? When you come out of that, you're gonna have X as an X, and now we need to do 3. I put the scheme back on this, basically an X. For me, it makes Ruby pretty close to unusual as a functional programming language. Because every time I pass a lot of work out, I know I made these side effects. If I have an array, and this didn't even happen if I assigned a block to a numerical array. So I say, F is a property, whatever. So I may not even be visible from people in source. How do you know what's visible? Right, so if I can paraphrase the question. I think it would make correct me if I misunderstood it. But for example, a closure is basically freezing some states so that you can, it's available to the closure later. For example, if I reference a variable outside the closure that was in context, then that variable might be changing, even though I think I have a side effect free block. Did I say that right? So that's the case where Ruby doesn't let us preserve correctness in this sense. We just have to kind of be sure that we're doing the sensible thing, of not referencing stuff outside the block that's out of our control. If we can avoid it. Yes? Yeah, you know, right? If you grab a problem from somewhere, if they have a main variable, two bar, and you have a very two bar, you'll decide. Right, so an issue of what if I'm using someone else's proc that they give me? What if I've been very careful in my library to write side effect free code, but I have this proc that comes in that fails all the tests. And maybe, and you cited the example of maybe they actually use a variable name that collides with a variable name I have locally or something, then yeah, you could break things down. I think it would be, you know, the great thing about Ruby is because it's dynamic and has fantastic metaprogramming. I think there's maybe a little mini market here, if you will, for tooling that would actually kind of try to verify correctness in these ways, like look for cases where you're modifying data that maybe you're not aware of. I know, for example, the D programming language, which is intended as like a successor to C++, I believe they have a new keyword and I think it's called clean. You can apply that to a function and the compiler will check that you're not doing anything in a side effect failing way. Let's put it that way. So that would be kind of nice to have tooling that kind of verifies as best it can that your program is side effect free or whatever. Yes, sir? I think it would be one of the things that I'm going to do. So Ruby 1.9 block variables may be local now and I'm not sure myself, so that may be true. Anything else? Yes, sir? How do you deal with crazy backtraces and running recursive functions in Ruby? You said that again, I didn't quite get it.  Yeah, yeah, if you want me to. Recursive function when you get in a backtrace can be heinous, yeah. So workarounds that would make that a little bit easier. I mean, yeah, so it's another case in point of there's no free lunch. You're kind of paying a price if you're using recursion. Maybe clarity isn't so good, like the for loop I showed, and debugging can be hard. Besides the fact you might blow the stack. Anybody else? Okay, thanks a lot.


Back everybody our next speakers are here from code think giving us a talk on open QA They've updated their talk in the last few months. So if you saw the one in Berlin, it's gonna be there's something some new content Lawrence and I can't pronounce your last name And James Thomas are your are your presenters, so please welcome them Thank you very much for the introduction. Yeah, so the title of this org is open QA testing on hardware automated full system tests actually open QA is one of the main tools that we've used in this kind of journey that we've been on over the last few years when we're Thinking about functional testing on on hardware and automating that But actually in addition to that there's quite a few other tools that we've created that we're also going to be talking about today so Yeah, my wave introductions. Well, just did a great job of that already really but My name is Lawrence the project manager at code think James a senior engineer at code think and We've been working on automotive projects for many years now And we're also both involved in free and open source software as well And I'll talk a little bit more about what code think is and what code think does in a couple of slides But yeah today's talk is going to be really Talking through Like I say this this journey that we've been on thinking about automated testing Functional testing on hardware so I'll start with a bit of context around why we think testing is so important and Trying to set the scene and then I'll do a little recap on what we've talked about before and Then we're going to hand over to James and we'll talk about some some new advancements that we've made some new developments in testing and automation So why testing matters in automotive? So actually even before we start talking about testing I think it's useful to set the scene and talk about software complexity in the vehicle and Probably I don't really need to labor this point because I think everyone in the audience will be aware of this kind of overarching challenge that we're facing at the moment as an industry I Think I read I read in an article a few weeks ago Lines of code in a vehicle 10 or 15 years ago was about 10,000 lines of code on average And now today if you look at a GL IVI plus the cluster It's just shy of 200 million lines of code and I can't find that article anywhere So if anyone does find if anyone sees that please link it to me I can't I can't find it but I definitely did read that and You know as an industry I think that's just indicative of the challenge that's being faced And this article by McKinsey. I won't read it through all of this, but it summarizes it quite nicely the level of complexity of the features that consumers are demanding the difficulties posed by integration of those features and Especially software updates over the air it points out specifically Essentially the automotive industry is kind of struggling to keep pace the productivity is not quite much in the complexity that we're seeing And I've put this comment on the bottom here, which is about Most automakers are not really set up to support software for the life cycle of the vehicle I just bought a car in December and I'm pretty sure I won't get any software updates Although it's going to be on the road for 10 15 maybe 20 years It's a pretty basic car. Maybe a better car might get more software updates and I know that You know this situation is changing people are working on this and trying to do You know full system updates more regularly and easier, but Generally speaking I think most most products out there not just vehicles, but most products are not really Designed to be maintained for the life cycle of the whole product Okay, so code thinking our view so could think is a software services company and what that means is that We work on lots of different automotive projects So we have quite an interesting vantage point I think we work with lots of different tier ones and lots of different OEMs as well and invariably We we see the same kind of things happening on projects, which has led us into thinking about testing We're an open-source consultancy with a strong expertise in embedded Linux and Excuse me. Also, we Spent a lot. We have spent a lot of time thinking about The integration story and the construction of software. So, you know, the software delivery life cycle is called build test deploy integration That's kind of you know an area where we've been really focused on So the theory of constraints I wanted to add a slide about this because I Think it's a very useful frame of reference to think about problems On on projects and generally, you know in software delivery as on an ongoing basis so I first learned about this from a book called the Phoenix project which is it's kind of about dev ops and an IT services and how this the theory of constraints it originally comes from production line manufacturing process, but how that can be applied to Dev Ops and IT and it's it's a novel actually it's it's it's a really really good book I recommend it to anyone who if you haven't read it the Phoenix project. It's really good but the theory of constraints is is useful because It's kind of founded on this principle of bottlenecks in the system. So in a rough summary the Maximum throughput of your system is constrained by the maximum throughput of your your bottleneck So if you can alleviate the bottleneck the Overall capacity of your system will be alleviated in line with that so an improvement of the bottleneck is an improvement for the whole system if you think about this example of a Highway with six lanes for all the different vehicles, and then there's a crash in the middle Or there's something that's happened. You can see that you know that the maximum throughput is only As wide as the the bottleneck is so Yeah, improvements at the bottleneck are improvements to the whole system and then conversely any effort that you put in either side of the bottleneck is Not really having any impact. It's it's good. It's an illusion. It's a complete waste of energy so when code think comes onto projects and If things are delayed or something like that and We often try to look for the actual bottlenecks and see where they are So yes what that usually means I mentioned earlier. We see the same patterns repeating almost always we start to think about testing and Usually that's trying to make testing more reliable more repeatable More robust almost always we're pushing for automated testing in CI pipelines But also we try to bring testing as early in the development cycle as we possibly can what people call shift left Because the earlier that you find an issue the less expensive it is to fix And I don't just mean in terms of dollar signs, but in you know people people's time and and all the other resources that go into projects So why is it more? Why is it less expensive to fix an issue the earlier you find it? Well testing is generally an expensive part of a project overall We we still believe that testing is generally overlooked on a lot of projects a Lot of projects rely solely on manual testing And on large automotive projects you can be looking at hundreds of people Sometimes in the biggest projects, you know, it can be into a thousand people who are validating the vehicle at any one time and Hardware is quite a prized resource especially in the early days of a project, you know people can Kind of not literally fight over it Certainly, you know, it you know, it's difficult to get your allocated time with the hardware in the early days of the projects in our experience, so if you have For example, if you have a really critical issue on the software that it doesn't boot or something like that You really want to find that in emulation Before you put it onto the hardware because if you know that you have a really critical issue You don't need to waste that valuable resource. You don't need to waste people's time putting it onto the hardware and Then if we extend that one step further The vehicle environment is the most expensive testing environment on a project, you know, it takes a long time to get the Though the vehicles wired up into a prototype state You have to have a special license to operate them and all this kind of thing so Any issue that you find in the vehicle environment that you could have found earlier is again It's a it's wasting people's time basically and in an extreme example on the previous project that James and I worked on If you've bricked a unit in the vehicle, you had to phone Bob from some other department. He had to come down Take out the seats change all the wiring and replace the unit and everyone's just stood around for a few hours You know days your day of testing is down the down the drain and yeah, just a waste of time And it's it's these kind of hidden costs. I think that makes testing so important because It's very hard to kind of quantify that kind of thing, but but it's always there So and conversely if you improve your testing and you start finding more issues earlier That's very hard to sort of to measure because you're measuring something that never happened in a way You know, you know that you're finding issues earlier or you're not letting them in the software But it's it's kind of hard to justify. But so it's it's these hidden costs. We found So, yes, we want to catch them as early as possible Also, we want to automate repetitive and manual tasks wherever possible. I'm certainly not saying that Manual testing can completely be removed. It absolutely can't it's it's essential, but obviously in any we want repeatable tests and manual testing by its definition Can't really achieve proper repeatability. So The current project that I'm working on They have a full Full system validation. It's a three week test and I think they do it every every two months or so at the moment the tier one that I'm working with and I asked the testing manager How many times do you plug and unplug the USB media in order to test it? Does anyone want to guess in a three week full system validation test? How many times? Thousand's not bad. It was 800 Which you know, I thought was fair play. It's quite a robust test. That's good. But yeah a huge amount of tests so If you were able to automate the USB media test not only would you be able to Not have somebody doing it during the day and you could make better use of the the hardware resource because you could do it Overnight but that person would actually be free to do more important things stress test the system Try and find edge cases use it as a real user would something much more interesting So it's not about removing manual testing or removing people who are manual testers It's just about freeing them to do things that are more valuable and more interesting And also maybe saving the sanity of this poor soul who had to do it I think if we worked it out, I think if you think it's 30 seconds per test Which is probably quite optimistic because you have to check everything loads properly It's obviously 400 minutes, which is about a day. So a full day of someone's time doing that Yeah, so a USB media was Was something we were focused on but it's just a good specific example Okay, I'll do a quick recap of some of the previous things that we've spoken about So we spoke at ELC 21 and FOSDEM 2022 and that was talk was lava and open QA was automated continuous system testing, so I'll talk a little bit about How we started how we sort of identified open QA as a tool that we wanted to use and How we combine that with lava to get some testing on on hardware and As Walt said, we I gave this talk at the AGL AML AMM in Berlin in March But we have some Advancements in Sten as well So the talk that we gave at FOSDEM and ELC We sort of use this to mind the gap To sort of set the context here. So it wasn't specifically Focused on automotive at the time, but I think the same principle does apply so Modern testing modern software platforms is extremely time-consuming and there's generally a lot of manual testing And there's a lot of products like I said at the beginning where you might You might use the kernel version that supported for two years But you know that your device is going to be in the field for ten years or plus So if you speak to upstreams, they will you know, upstream projects will always say Use the latest version and that's absolutely right. You know, that's that's why they work on the latest versions But in reality, we've found that there is this gap because the the sheer amount of Regression testing that you need to do before you have the confidence to upgrade In many cases, you know, it wasn't happening and certainly This wasn't a novel observation. There's a lot of people have had this observation before as well I think there was a talk by a Pengu Tronix from FOSDEM. I think a few years back Which is very good on the same topic So obviously, you know upstream first that kind of thing is is all going in the same direction So we thought that basically more testing Making testing more available is Is going to help maybe reduce that gap So we looked at OpenQA OpenQA is a very well-established testing framework. It's For the OpenSUSE project and maintained by OpenSUSE. So it's designed for Linux desktop environments and Linux distributions But we wanted to see if we could use this in an embedded context it works on a principle of screenshot comparisons using these things called needles and I'll talk about that in the next slide Well, basically, it's a click and a search and a time our value and a screenshot so you can compare what you What you expected to get in the UI versus what you did get and what we really liked about it was that It's actually replicating how a user is using the software because it's a touch input. So it's not You know, it's not sending loads of can messages from Canalizer as a simulation or something like that It's really is how the user would use the software And it's designed to work with QMU So what's a needle and test in this context? So a needle is simply it's the screenshot of what you expect to see and then it's also some JSON coordinates so you can do quite clever things like Outline an area on the screenshot where you need to click and then you can have an area that you want to mask so for example if the clock is changing in the corner you don't want that to fail your test so you can mask that area and you can Change the tolerance that you expect in different areas if you expect the colors in the background to be different you know shades of brightness and so forth and Then a test is separate to that and the test operates on the needle So in the test you can say where you want to click and how long you want to wait for So with this system you can you can run multiple different tests on the same needle to navigate through to different areas and things like that so you can relatively quickly you can you can get the workflow of the All the different screens in the UI So brief overview of the architecture The test themselves are in git and that's quite useful because You can tag them with every release of the software so obviously as your project matures You're going to be adding more tests so you can always keep track of where they were up to And this we this is git lab because that's our weapon of choice But it works with any other CI orchestration as well So that acts as a launcher and it spins up an open QA worker The open QA worker can be Anything we use git lab runners, which is a VM of course where you can be a laptop or a Raspberry Pi And then separately you need an open QA server and this does two things It stores the needles because they can get quite large so you you store them separately And then also it has this kind of dashboard of results Which I've expanded on another slide as well so Yeah, and then every time you want to do a test the open QA worker will spin up an instance of QMU And that's done ad hoc so it's per test and then it reports the results back to the open QA server And a view would look like that so you can see here just on the bottom We've got a mainly green test run with one fail at the end you could drill into that and see exactly You know what what you expected to see versus what you got So that's open QA with QMU as it's designed to work We push this to the GNOME project So GNOME OS is now using open QA in its CI which is very nice There's a GNOME open QA server But like I say we wanted to get these exact same tests running on hardware So we look to lava and I think probably most people will have heard of lava here It's was originally a linearal project. It's used a lot by the kernel CI projects as well essentially you can it's orchestration of hardware testing so you can set up what's called a Board farm and have lots of different boards with different architectures on and you flash your flash your image to it And then run your tests on it So using VNC we got that working with open QA so that we could run the same tests there instead of QMU in emulation and Yeah, we have a lava instance at code think now and we have an open QA instance as well and We are testing the Linux kernel upstream project every night with this so we've got a devian image that tests and That's had some good feedback from Linus, which is great. So that's actually being used So yeah, we managed to get the open QA tests working on a dev board The next step after that was getting it on to representative hardware on For example an embedded project an automotive project and also getting something in place that allowed us to put it onto any generic hardware and this led us to developing QAD and A host of other things so I'm going to hand over to James now and he'll take it from there So, okay. What is QAD? QAD is a very very lightweight Linux Daemon That you can put onto pretty much any Linux system that uses DRM for rendering and EV dev for input So for example, this actually works on Android you can run it on natively compile it for Android automotive and Pretty much any type of system that uses DRM for rendering So this runs on real hardware you can slot it in the dependencies are incredibly minimal So it's very easy to compile. You don't have to change anything about the software you're trying to test And that was a design goal. We did want to make this In a way that didn't require a special test version of the software We want to consume the version of the software from the CI that will end up in production and run the tests on that So you build this statically linked binary and you put it on there And it's essentially creating this Automatable test engineer. So what can it do? QD is very simple. So we have a very very simple HTTP API So you run this Daemon on your system And it exposes to HTTPM points. So you can get HTTP get the screenshots of the display Or multiple displays and you can also post input events to it. So this is JSON formatted input data So it's very very simple. You specify the X the Y duration the type of event velocities and things like that So byproducts of this it was originally designed for testing But because we know how access where you can get a screenshot and send an input event you can actually use this as a method to remote the access the hardware that you want to test so say you're working from home or You want to test something in a vehicle This could be installed in the vehicle and then you can remotely connect to it and then control the UI So if you want to test a new piece of software you want to see if your changes have gone into effect you can use this and then do that Depending on your network configuration of course What isn't QAD It's it's very dumb. So it will not do anything by itself. You have to call The screenshot so you have to get that and you have to post an input event It doesn't provide any type of remote shell access. So keep on using SSH It doesn't do any type of Test orchestration so it will not do any automation for the tests at all. You need to use something else for that And it's not a silver bullet. So just because you have this Doesn't mean your software is going to magically get better So how do we actually automate QAD then? As I said, we have a very very simple get and post API so we get the screenshot and we send input events so this ties in very nicely with the The architecture of open QA and how you write open QA tests Because open QA is based on this concept of asserting a screen and then clicking a point on there and making sure that worked So for this we've added a open QA console and back end into the QAD repository So you can actually integrate this with your open QA tests So if you go back to this slide where we're using lava for the hardware orchestration This talks VNC to the Ubuntu instance. That's running on the Target hardware there. So lava has orchestrated this It's provisioned the board. It's given you access to that boots. Ubuntu is running VNC and then you use open QA And VNC is kind of the default protocol that open QA works Works with But that's not what we want.  It could be on a developer's desk. It could be a vehicle You know, it's got multiple ECUs speakers and all that type of thing And we need some method of communicating with it That isn't going to introduce software that is going to change the parameters of the actual system So I've never come across an automated project that had a VNC server on for example. So we can't use VNC And whenever people use Western That does have a remote desktop protocol, but that usually is installed afterwards So you're kind of making a test build in order to to do that So this is the architecture that we use for the client that we currently working with So we have a git lab instance And then a developer will make a merge request into the yachto recipes for the system. So they want to update the system That merge request is then built. So we build the image and Then we send this to a git lab runner. Now the git lab runner itself is a small It's a small Intel machine, but it doesn't have to be that powerful. It could be a Raspberry Pi or something It's something even more minimal, but this Git lab runner is essentially doing the orchestration for the test hardware So this is physically attached to the automotive rack. So it will have a can USB interface It will have serial interface to the rig. It will have ethernet to the rig and that Particular device is running open QA worker and that's the thing that spawns the tests So we download the image we just built from the CI pipeline on to that runner We then flash it onto the rig So this is going to be rigged dependence, of course, but we send it over and then run the Flashing procedure on that new image. We reboot and then we start QAD So once we started QAD we can now run open QA tests Using the QAD console that we've developed for open QA One of the real benefits of actually using open QA for this type of thing is if you're a developer as part in the CI Pipeline during the testing stage You can get instant feedback. So you get a link to the test reports that one showed earlier But this will also give you a live view of the testing. So as the tests are running you can see The tests running on hardware in real time So you do get this instantaneous feedback there And then of course once it's finished you can pass or fail the pipeline depending on the policies there You know some tests are not as important as others But perhaps if Cal Play didn't start you would want to fail that pipeline And here's a video so this is a demo of this so the rigging question here. This is actually the AGL demo platform It's running on a Raspberry Pi and there's a multi-touch touchscreen attached. So I Think we'll go to the terminal in a minute So on the right hand side there, this is the open QA worker being Run on a piece of hardware on the right hand side now They're launching QAD on the device on the left hand side is EV test just to show you that the input events are coming through So now they start the tests and this is the type of thing that would be done by the CI So the CI would start the test and you can see there. We've got tests that goes on the home screen to the HVAC And you can see the input events arriving now. We have another test that goes back to the home screen and You can see the input events there Now that test is finished. That's the only test we did for that. So it's a very very simple demo So it's finished you can now go to the open QA Server and see the results of that Now this is actually one of my favorite things about open QA the UI for this is exceptional for reviewing tests So you can there's a slider to compare screenshots if something fails There's a needle editor So if you need to update the needles if you need to update the screenshot because something's changed in the UI You can use that So it makes it very very easy to kind of update these tests Now as I said, there is a needle editor in the web UI So if you need to update the test and you took the Jason in the test then Use that is very very simple and it will push it to get and then you can start using that test straight away But what we found when trying to retrofit this approach onto an existing platform that had no testing Was creating those needles in the first instance was a bit of a chore so we've got Dozens of tests here and we have to create the needle and the Jason file for them in the first instance and It wasn't that easy Which is why we invented This QAD web UI So this actually is useful without writing tests because you can actually use this to remotely access your Hardware here, so you can connect to QAD running on a piece of hardware and use this to then Navigate through the UI and what have you But its main purpose is actually to create those Jason files and take the screenshot for a particular needle so you can specify your needle directory you can give the project directory you can edit existing needles from there you can easily as a whizzy wig editor so you can define the Areas that you want to mask and the areas that you're interested in because at the threshold and then you can export all of that to the Png Jason pairing there Now you don't have to use this. This is I don't think it's actually currently open source, but we're about to press the button on that But I think it's very useful and it's very useful as this remote access technology So whenever there's a failure on a piece of hardware that is running in a different country we use this to see what's happening So the future work for QAD It's it's a new project. So it's not the most mature thing in the world. It does work We need to open source the web UI And we probably need to do more screen capture back ends So the default one that we use now is a DRM back end. So it literally just grabs them Rendered frame buff from the DRM subsystem. We do have a Western back end, but that's a Western back end specifically for the Geneva layer manager So this is the ILM protocol and that that that is useful because you can then speak ILM to anything that's Using layer manager on a system so you can ask for the screen and Configure it that way More input devices so in terms of the input device We either piggyback on an existing piece of hardware that's on there So let's say the multi-touch monitor or we do have a new input back end so you can create these virtual Touchscreen keyboard and mouse devices If for some bizarre reason your Linux system is not using EV Dev for the input you would have to create that yourself But it's relatively trivial. You just have to implement the primitives in the input It's MIT licensed and if you scan that QR code, that's the repository for it So this gets us to a very good place. So now we have a lot of tests where we can exercise the UI We can see what's going wrong. We can verify that The system looks good, but of course an automotive Project is is a lot more than just touching the screen and seeing what the reaction is Now we need to automate things like the can interactions So if I went to the climate application and I increased the temperature That will send a can message that which we then need to respond to so if we did want to have a test that did that to Check that the temperature was increased We also need to provide that can simulation and we want to do this without Relying on a Windows laptop running canalizer You know, we want to put this on the control device So this is an example that just came up with before the talk So we have a very very simple JSON parsing can Damon that runs on the control unit and it's connected out using a USB can device and we can just simply define The types of signals that we want to send the period whether or not we want to send them at the start and we can map them to Signals that we receive so for the example of the HVAC unit there. We will get a fan Event received in a particular format. We then need to send that back with the Temperature or the fan speed and we can do that quite easily with a Simple JSON signal map there It that definitely needs more work And That's not open at the moment, but this is something we are thinking about Now another piece of hardware that we developed for this project is This USB switcher so the example Lawrence gave of inserting a USB stick taking the USB stick out 800 times Can now be automated using this piece of hardware This this actually has been Brilliant for the testing because now we can actually test for every merge request USB media Does it does it come up is the salt order correct is Does it play? If we remove it does it go away if we put it back in does it come back? If we change the language is the salt order in Chinese correct is the salt order in Korean correct and all those types of things are Open using this piece of hardware Again, this is attached to the control unit that's attached to the rig and Then you can put two USB devices in and switch them using a very very simple serial API there And another amazing thing is that we can now test Cal Play as well so We can have a test where we plug an iPhone in Does it come up? Does it come up in two seconds? Does the functionality within the Cal Play application actually work? So here's a bit more about the switch It has so you can connect two devices to one host That's to a USB stick and an iPhone for example to one rig if you did want to connect connect more so if you wanted an Android device an iPhone and a USB stick you can daisy chain these and Use it that way so you can switch them on and off So you can have as many devices plugged in as you you want That's completely open. So it's open source hardware the firmware everything about it is open source and It's it's Generically useful. So it's it's not just for automotive testing if you need to automate any type of USB procedure This type of thing is it is an absolute godsend So the QR code there will link you to the Git lab repository for all of that Now all of that Brings us to something that code think has been working on for the last couple of months and that is testing in a box So everything that was just described there in the process kind of looks like that You can see on the left there's a control unit, which is the git lab runner in the open QA worker There's an iPhone attached a USB switch Ethernet dongles there's a can device there peak can USB device And and it's it's a lot And it's not that easy to put into a rack or To stack up anywhere and it's certainly not the type of thing You would want to take into a vehicle for testing because it's just a bit of a mess and all of that requires setup So you need to set up the control unit. You need to install the software. You need to Find provision all of these can devices. You need to buy the hardware. You need to plug it in And that's not ideal and that takes time so the solution is to literally put all of that into one box and it does exist is is a Is the very first one? I believe and so what this does is it allows you to Connect to a piece of hardware that you want to test and it has a number of interfaces, so it has a serial IO interface has can bus emulation The USB switch that was previously mentioned is built into that. It has a USB hub It has a Bluetooth and Wi-Fi Chip, I think it's a risk 5 chip in there And that actually is quite exciting because now we can do tests Where we mock a phone for example, so in our current test rig, we actually have a Google pixel That's close to the test hardware so we can then test whether or not contacts download or whether or not it's paired correctly Or whether or not the music's available But with this we could then mock that we could send contacts We could change the name we could un-pair we could repair we can do all that in an automatable way Which isn't that easy because you know you'd have to do that on the Android side Has I swear see SPI HID emulation so you can use this as a keyboard and mouse for a device so if you Wanted to automate sending keyboard presses or mouse movements You can use it for that and it also has a host PC and I Can't remember what the actual host PC in there, but it's some lightweight Armboard and that allows you to also then put the software directly into the box so You can install the GitLab runner You can install actually run an instance of GitLab CI in there And it has all of the software QAD open QA ready to go and ready to deploy You know this this is this is really going to be very useful because you can take this on to any project Plug it in and then start writing these tests almost immediately And the provisioning for it for this it's It's all in Ansible so you can deploy to this using Ansible It contains all of the software We're planning to add CI templates so you can more easily integrate this type of thing into your Existing CI so you know we prefer GitLab, but you could be it you could use this on anything And we're working on a version 2 of the board so there's going to be more features Now this is also going to include software again that we have open sourcing to monitor performance of tests So we can gather data about a test run and Then compare it with a previous one. So let's say you have a merge request That comes in you can do the test get that to performance data and compare it against the one That's in the current master branch so you can easily spot things like the CPU usage has gone through the roof or memory consumptions high And things like that and that's all going to be part of this testing in a box So the qi code there Links to that group So it's quite exciting and if anyone wants to take a look at it afterwards then feel free I think it works not actually plugged it in But yes, that's Yes, that's it for me. Thanks James. So yeah just to recap We started off talking about testing is expensive and it has hidden costs automating testing Functional testing in this way presented us with quite a lot of challenges. We found it quite hard but with the right tooling it can be easier and Obviously all the tooling is open source, so I Think at the end as well that the the great thing with testing in a box It's just worried about why we really wanted to do it With open qa and all of the other tooling obviously I've said before we were on this journey to sort of Figuring out as we were going We everything was ready to go so that would have saved us months really setting up open So that would have saved us a lot of time Okay, and we are out of time. So that's the end of the talk. There any Questions is no time for questions. No. Oh, sorry. I thought we had 50 minutes Okay, well, we're here so you can always ask us please to feel free to ask us any questions


Hi everybody it's Sam at Mixed Up Craft, thank you for watching today. So this week I've got a few things to share with you. First is the latest issue of Paper Craft Essentials magazine with this one here this is issue 179 and you get this lovely luxury shaker kit. Now I've not even opened this yet so I'm just going to see how do we get in this one oh on the side. So this one's been designed by Simply Made Crafts which is Helen Griffiths, I always get Christina Griffiths and Helen Griffithn. I get the the Griffithn and the Griffiths mixed up so I'm just going to take it this is how you do it. It seems a shame to rip into it but that seems to be the only way in. Oh that was a fight anyway.  So in here you get everything you need to make a shaker card.   Now you get loads in there there's an example of one of the cards you can make and you've got some other nice Christmas ones there as well so 78 pieces inside. So here is some acetate, you've got these bags here of sequins I love the iridescent clear ones because they just work with everything and then you have all of your card blanks these are your windows they're so cute you can use these in so many other projects as well not just for the shaker cards they're really really nice do like them and they're white so obviously they're easy to add your inks to just stress them you can yeah do whatever you want there then you get a little pad which is full of all the strips so that's one long strip which is great and then these are your papers so these are going to be a six size card I believe looks like it four by yeah it's about four by six isn't it so then we've got all these background papers you've got this one I like those ones Santa Claus Joy Laugh Snow Merry Christmas Starry Night Noel we've got these with the snow theme and you've got your antlers brick that's nice see these would be great for new home cards as well it's not just Christmas you've got your like wood effect there the decking there's another festive one and then you've got these smaller scenes here I'll have to have a look on how they oh these are I see so this is to look out the window so your window here will sit perfectly over these how sweet is that that's a really really lovely just completely transforms those images and then you sit that on your chosen so there's to say your house has got a wood effect or you live in a brick house I think they're going to look wonderful so that's those you get two of those and then that's nice so this is the the other way now this is like you're looking in the house in this Father Christmas where you've got the Christmas tree really like it nice and then I spots on foiling lovely I think it's needed as well because these don't have any foiling in the papers they're quite matte it's nice to have that pop so you get double there as well you get the little cat so you could have the cat sitting on the windowsill because he's just that yeah look perfect and then you've got your Christmas tree so yeah they're really really good and I think with these kits I need to almost like schedule in some time to actually work through them because I love them and I they're quick cards so they're handy to have and like I said some of these I donate charities once I've made them so I'm going to pop all that back in its case you actually get two bits of acetate there's another one there as well so it isn't just that one piece that would be enough for four cards because I imagine actually no probably no you probably get two four so yeah that's enough for eight of the cards because of the size of the windows so that's the lovely kit and then the magazine does as always have lots of inspirations you've got inspiration there I feature in this one as well I have done oh look yeah they look really really good because you can change the window as well that's a different window effect so that one's been made love it there with all the sequins inside always lots of inspiration trying to see where my one is because I haven't actually looked yet oh oh my god I get really touched you're seeing this as a first reaction guys I've just seen two projects and both of them have been inspired by my tutorials that's so lovely so Jane Kennedy in Bristol thank you so much Anna Elizabeth Fienham from via Facebook that's so lovely she was inspired by my wreath because I used the the the leaves the autumnal leaves that were the free gift I was inspired by sound Clayton from mixed up craft plus the fabulous kit from issue one seven seven oh that's so lovely and then this one here I do love that box and I love that she's made both sizes as well because that mini one's really it always chokes me up I'm such a silly one right anyway this one says I love sound Clayton's projects in issue one seven seven and I couldn't wait to make her step a box they are so cute this is my version and I also had a go at the large box too oh thank you Jane that's so lovely and they are amazing this is I love that huge it's like a Hessian bow that looks absolutely amazing and then there's more amazing these are great this this was such a great kit with all the flowers so many of you made some wonderful things with that and you can get these back copies as well so again I'll share all the links below because they are so handy and if you're someone who maybe I always say is just starting out or you know you don't like to spend a lot of money on cross supplies these kit things are great I mean that's going to make you how many cards did it say eight luxury shaker cards that's brilliant and this cost of this one is really competitive this is one of the cheaper ones 6.99 so it's always I think really good but where is my one there we go I just show you how to make your own wrapping paper so I created these little um these boxes here with the wrapping paper on them and I used the Altenew that was the bells and bows it's lovely that one I got it last year but it's still available it's just such a lovely stamp set and I'll be pulling that out again this year but that was my feature all wrapped up so it was really nice and you're like next month's next month's is next month's no it's every fortnight this one it's I think it's going into that one it's amazing loved making that one so that is the latest papercraft essentials magazine okay and then I picked up the creative stamping magazine this is issue 76 so let's just open this one and the reason I wanted this one although I nearly buy I'm kind of starting to buy every single one of the magazines again I did stop for a while because I was just taking up too much room but this one is Pinkfresh Studios and it is lovely big images of your well it says Christmas florals but you've got your Ponsettias here and I don't have nice big images of this and I'm seeing so much card inspiration with Ponsettia flowers I thought I want some and then I saw that this was coming out so I thought yes I'm having it I also love the little houses anything with houses on is brilliant I mean that one does have a wreath on it certainly this one here would work great for a new home card you even got here from our house to yours lovely little images here and just all these sprigs you've got the berries it's just a really really nice set let me take the packaging off okay just so you can see it a bit closely look at all the detail so much detail but easy to color you know still nice blank areas so you're really going to see your um yeah all your coloring with them love this one as well just think they're very nice and modern and I think that's something you know it's pink fresh studios they are very modern with their stamps and contemporary feel to them and I just think it looks lovely love all the the lines through this one here so that's that one it's got a little bit of inspiration on the back there as well and then you have your magazine and again lots of inspiration see this is the kind of thing I like I love that look that's been stamped using embossing powder so it's golden embossing powder and then you color within and it's just such a lovely effect so I'm definitely going to be giving that a go and I'll show you some more inspiration here love this one it's the diamond fold card see this everywhere some more there that's on like a craft card so that looks lovely and then here yeah see they've used it there from our house to yours that's really lovely and there's that one there and that's so quick to do that doesn't take long even with just basic coloring you don't have to do all your blending you can keep the coloring quite blocked so yeah so I'm really pleased with that one so that's the creative stamping issue 76 and then I also picked up the crafts beautiful this is the october 2019 issue so this one you have 25 festive stamps plus a festive edition craft snowflake die and trim craft papers on the back which are here turnover for 32 festive papers free 164 page coloring book worth 9.99 you've just got to pay postage so that's something else you can get and then there's a competition there and an interview with one of the designers so let's just get into this one first of all there is the stamp set now the reason I was drawn to this one if I just take the belly band off that snowflake dies actually really sweet because it's smaller than the ones I've got so I'd be nice to mix in with them you can just see the nice little snowflake but look at these stamps you've got the Christmas tree there and you've got the elf with his legs sticking out the top so he's got stuck inside the Christmas tree you've got lovely images of Father Christmas but it's these that I like I like the elves eyes only until the 25th of december and I love this one welcome to the north pole and you're going to see that on a really nice 3d project which I keep hinting at but I also like these because they're nice stamped as like a silhouette so I can use them with some of my other ideas you've got santron is sleigh and just these boxes and things like that they're just handy to just mix with other projects so that's the stamp set so let's have a little look at the papers this is the enchanted Christmas it's 32 designer prints so it's the trim craft ones and I'm on their design team so it'd be quite nice to see what papers we've got here so start from the back so you've got that lovely like traditional color there this is great for masculine style cards again great for background papers these and not just for Christmas I've said it before this is this was like a golf print reminds me of love that one with the holly it's really nice and then you've got your Nordic style recognize this one I think some of these are maybe from paper addicts or they've evolved from that some of them I do recognize love that one that's great for again all year round you've got the wood effect there with the snow you've got your stag head again more of a Nordic or Scandinavian feel and stars again great for any kind of card it's nice with the craft it's like wrapping paper Christmas paper that reminds me of and it's actually you could easily use that to wrap small gifts as well that's cute love the houses and there's your mistletoe you've got like your candy cane stripe and the candy canes they're great love those ones and then that one there so you can even use your top sheet as well and then there's just a little welcome letter there so that's the papers they're always handy and then you have a very colorful magazine here full of inspiration it's actually really thick this one 100 Christmas craft essentials gadgets papers pens and more okay so that's the crafts beautiful and like I said that's issue that's the october 2019 issue and then last week on my facebook page I said that I joined the Kleina Palmer designs design team so Kleina is or has been designing dyes for other companies for many many years and she's been crafting for a long time and she's now done her own collection and it was really nice that she um she approached me and asked if I would like to you know have some of her product and join the team so I jumped at the chance now I'll show you why in a moment because I think they're a really cool idea but what she sent me is these magnetic sheets I've got a four and I've got a five so they work on your smaller machine and your larger and these are your they're called a linem so they're your die cutting alignment tool okay now you get two so I'll show you here all right that's the smaller ones and then you've got your large ones here and the real thing I like about these as well is that she's done all of the in different colors so it's really easy to see like my five by seven is that turquoise color so I will you know I'll get used to seeing that you've got five by five a six six by six seven by seven a five eight by eight so you and I've got these lovely stencils so I've got the heart stencil that's the embroidery on glaze um probably see that a bit better there actually really detailed and then that one there and that's your doily I like that it covers the whole thing actually it's going to look lovely with my embossing paste and glitters and stuff on them and then I've got these two dies which are your nested heart and the nested doilies they coordinate with the stencils but the nice thing about these is they're so big because you've got your outer cutting line and then all the detail cuts into the card stock this will cut it out so I'm going to make a card that size with this I think that's going to look really pretty and again with the heart there you've got your outer cutting line and then these all cut within so I think they're going to be so detailed and lovely then I've got these I couldn't believe how big they are they are so nice and these fit the alphabet letters in perfectly but just look at those now you can cut these in half and layer them up which is what she shared on some of the cards that she'd made but I think they're lovely I like that they're just plain and then you can add everything to them lovely cutting decorative papers they're going to be really nice and then these are the linkums so basically this is a modular system so these are 44 this is your bold lowercase alphabet and you'll see there she's got a little dovetail and they link in to each other so that you can get them perfectly aligned so there's no kind of wonky sentiments and things so that's the 33 bold numbers bold lowercase and then you got bold uppercase and then you've got your shapers so these here will allow you to have like a curve on your sentiment so that's nice and these here allow you to again get them perfectly lined up but also a nice kind of gap between them so you've got all of them but you also get like with these here you get loads of spacers so it's just yeah I'm going to really look forward to playing around with this and yeah just watch your space because I will be showing tutorials and if you watch Hachanda where this is all available and I'll link all of that below you may see some of my cards as well so thank you Kleiner and in due course you will see some lovely projects that I'm going to make using this product and then I think it was the beginning of last week that I saw Paper Mill Direct done a call out for design team and I thought oh yes please so I applied and it was really nice and this week I found out that I joined the team so thank you Paper Mill Direct I am chuffed to pieces because now I get sent really nice playing card stocks so that's like your bread and butter that's like the main thing you need they do so much over there they do your envelopes they do all your different card stocks papers there's loads and loads of different things so I'll link them below and go and check them out but they so she sent me some glue there which is always handy but then this is the paper I've got so I've got the buff I like the name 240 gsm so she's wrote on these what they are this here is the white smooth this is going to be I think amazing with my coloring so using with my alcohol markers then I've got a mixed non-shed glitter pack let's have a little look at these colors so I've got silver red or real lovely Christmas red that one's beautiful sparkle and I've got that rose gold Christmas green lovely like baby pink and then a gold or like a bronze actually it's more of a bronzy color so some real lovely glitter card stock there I'm going to pop them all back in because I don't want these to get mixed up then I have so she sent me a lot for Halloween because I'm working on Halloween projects so she sent me some that we're going to work well with out now this is charcoal sparkle I don't have anything like this and just look at that oh my god look at all the colors you got everything it's like rainbow sparkle on a black background so she's sent me those and they are just gorgeous and it's relatively smooth you can feel it but it's not it's non-shed again that's definitely not coming off but isn't that gorgeous gift boxes oh can't wait so yeah that's charcoal sparkle and then this one here is dark violet that is a really good Halloween color love that one then this one here's black and this is this is a dark black and it's not until you put it against another black so this is my other black and that is considerably different that is now looking gray it's a real lighter black so this is the darkest black I have so if you are looking for that black black again I think this is going to be a good one it's really really nice and this is 240 gsm so it's a perfect weight for your card blanks and for making you know your 3d projects then we've got mandarin orange I'm not going to take that one out you can see there how that looks and then this is navy blue and I've said this before it's a color that I never have a lot of but once I get it I use it and I know I just don't seem to replenish it so this is going to look lovely with the moonlight song the paper boutique pad that I've got and also the nautical one that I've got in that collection as well I think that's going to really complement them well and then I've got this is fleck craft I've only ever used two other craft cards before in the past so it's going to be nice to try a different one this is 250 gsm and it's a fleck because you do have that fleck detail throughout and then you've got that slightly different tone on the back so yeah really really nice thank you Charlotte I'm going to get cracking with these and then lastly I picked up a couple of bits from craft stash so they've asked if I would like to try this product here so this is sussie crafting by Emma Jewell so she is one of the newer of the branded product that they have so yeah I just thought I'm going to give it a go because I don't like I said before I don't buy halloween paper pads I never really have I've always looked for downloads online or just used stamps and kind of made do or just used black card stock but I thought you know I'm going to give these a go so I've got here that's a bundle there so these will die cut these here so you've got your skull but I do like her playing with the words so she's got love at first bite and you've got the lips stay spooky my kinder and then you've got poison or you've got watch your poison and you've got happy halloween and I just think you know what they're a little bit of fun you don't have to use the traditional colors you can do like hot pinks and yellows and stuff like that to make it look quite bold you've got the little spider and stuff so I've got some yeah I've got some fun ideas and I'm going to be doing a table kind of layout this year so things like this are going to get used a lot for that and then you've got a nice sized spider here but I thought about doing the spider in like glitter I think that's going to look really cool and I'm thinking I might do a ring as well and then you've got the bat and you've got the skull and crossbones there so they're fun then she's also done these here are your this is sticker pack in this one here yeah just loads of different stickers if you've got it haunted stay spooky um you've got some strips there got that one there with all the days of the week and then you've got freaky friday so I'll use this one on my planner for the the week the run up to halloween and the week after and then you've got again the poison keep it creep it real I put a spell on you I'll be watching hocus pocus this year happy halloween actually when I was making some of my crafting products projects the other day I was watching adams family I forgot how much I like that movie and then here she's got these die cuts it's 22 pieces 11 designs if you don't want to do the dies and coloring with this set here then you can use these because she's done it for you so I don't think they are stickers no they're just on a nice thick card so it's these images here and you've got the coffin but blown up so I'll probably do a scrapbook layout and I will probably do I'm thinking I might do a mini album actually with the papers I think it's going to look really good because I've got so many over the years of like halloween fancy dress parties that I've been to and I went to Alton towers over halloween one year and I think if I pull out all those photos and put them in a halloween album I think that's going to look really quite cool and this is a bit more of a modern halloween which I like so that's where I think I'm going to use a lot of this and then I'll show you the papers which I'll be using so here you get 12 sheets and this is trick or treat they're double-sided again they're going to work really well for a mini album I love that it's like the cover and then you've got this one here I believe these are all Emma's sketches they're her like images that she's done I like the pink with the orange and then you've got there the the lips with the fang teeth and that's the second one she gets two of each page love that one I really really like that that would be actually good for the front cover I think yeah I think I might use that one instead and then you've got the coffin with the rose and then you've got these cutouts if you've got it haunt it creep it real stay spooky love it first bite trick or treat they're going to look really cool as little journaling cards because you have got room there you could with a white gel pen you can journal above here and you can journal here so they will work nicely there and then that's the same and that's the same so yeah that's going to be just enough papers I think for a nice mini album halloween mini album so yeah so that was the sassy crafting halloween collection okay and then it wouldn't be a good what did you get video without something from a charity shop and we scored big so first of all picked up these so these are lindy sprays if you're into I guess more mixed media then you may know of these I believe they are American in fact they are American and I've seen these from my scrapbooking earlier kind of days this is you know a lot of people that I followed or follow still on youtube use a lot of lindy sprays I've had one before but a lot of mine are cosmic shimmer and these ones here are your flat fabios I think it was so all you do these have never been used they're brand new you can still get them online and they are they're about four to five pound per bottle I got these for two pound and all you do to activate them is you add boiling water and then you leave them to sit for 10 minutes and then you will have a highly pigmented spray color so these are just really really bright colors there's no shimmer in these but you can also use them for paints you can just spray some onto your acrylic block and then use your paintbrush and use them with your stamping and stuff you can see the colors there they're on the fronts so I just thought that was a bargain so two pound for all of those sprays so I will activate all of those later on and I will share any links to anything that I can find that I brought in the charity shops because like I said I think these are still available but may not be those exact colors and then these were like the gift that kept giving because they were in a basket and honestly I thought when is it going to stop because I kept pulling one out after the other and these are your new boat embellishment mouses I've had one of these before and they are really really nice for using with your stencils and they add texture and dimension but with a metallic finish so I'm going to be using these on cards these are all still available online in stores and they were a pound each and they are £6.49 between five and as much as seven pound each online I couldn't believe it and they are all brand new not one of them has been opened they've all got the foil on top so yeah I was like squealing inside and then we spoke to the lady at the the desk and said look who who is the lady don't ain't in all this this craft product because you know we've been coming in for the last few weeks and not every time has been successful but it's it's been that same kind of stuff all really good quality product and never been used so she was either a crafter that hoarded and never used anything or she might have been a demo some rep or something I just don't know but it's all in perfect condition so I have for here I've got like a black the colors are there but the one pound price tags on a lot of it you've got this beautiful blue if I just put them like this and you can see there the colors I love this this is like a peony color it said I remember reading that one this is a bronze or copper color in fact that's more of a copper there so that's more of a bronzy color blue black lilac or lavender this is like a golden yellow you've got that green at the end there like a pistachio color love that lovely minty color they're just I've got such a nice range I can't believe it so yeah that was um a huge score at the local charity shop and then my mum done really well because this was all in the same thing but I was grabbing all of the nuvo and the sprays and she was grabbing all the dyes so it's all that sheena stuff again and I got some of the embossing folders I think either last week or the week before so again none of it has been used it's all brand new in the packaging so it's this one is home life and you've got the teapot cup and saucer this one's beautiful sweet peas sweet peas are just so lovely look at them really nice sized dyes then this one here was really good she's got so excited about this because it's really nice and big so I've got the small ones from the secret garden collection these are slightly bigger ones so they're gonna look lovely with lots of flowers you could actually have like sweet peas coming out of this and you've got a nice size little you know what's it called planter marker or whatever it is and then these are 3d embossing folders but we knew we all know how you know these are can be quite pricey you know you're looking at like six pound for a 3d embossing folder and they're five by seven size again my mum's a five by seven card person like myself so this one's gorgeous with this it's got a bit of a steampunk background there with the cogs but then on the front here you've got these beautiful large flowers so it's called clockwork oh daisies they are sorry clockwork daisies that was a pound all of this they were a pound each that one was 50p and this one here is autumn leaves so yeah was that the one yeah and that's the top part of it that's the bottom part so it's like snow and you've got this one here with the butterflies this is a lucid butterfly this one here is up up and away and you've got your hot air balloons so it's the sky and then this one here is cherry cherry I think that's really sweet it's going to be lovely if you just highlight these in pink for your cherry blossom really really nice again never ever been used and then with that was this here which was the spots and stripes this is do cross paper mania it's their capsule collection I've had some of the capsule collection ranges before in the past they are really nice but with this one so she got the 32 pack of 12 by 12 papers again brand new all in the packaging for two pound then got the matching buttons I think that was 50p that one didn't have a price so I think the lady scanned it through 50p and then these are beautiful they're a pound they're huge flowers look at these there we go my mum's been crafting a long time she's not the kind of person that has to get the latest craze she is you know happy using anything she just appreciates all kinds of you know patterns and colors and things like that plus I hand a lot of my stuff down to her so yeah these are just look at all these absolutely wonderful so you just get a couple there and like offset them and put a nice embellishment in the middle put some leaves behind them but look at these it's going to look lovely on gift bags I said to her you've got to make some lovely big gift bags with these great big so there's two of each you can offset each one like so or add you know clash them I mean that's a lovely lovely look there I think they're brilliant really really good quality lovely and thick and there's tons in there how many do you get 32 pieces they're 160 gsm they're so strong so yeah so that was all my mum's charity shop finds so that's what I've got this week so I've got plenty of card stock which is going to be awesome to now kind of finish the rest of my Halloween projects I'm going to plan a Facebook live and I'm going to be doing these along with the Christmas ones from the last issue and then yeah look out for tutorials featuring these because though yeah thank you for watching there's always I'll be back tomorrow with another tutorial see you later bye