The Databases for Machine Learning and Machine Learning for Databases Seminar Series at Carnegie Mellon University is recorded in front of a live studio audience. Funding for this program is made possible by Google and from contributions from viewers like you. Thank you. Hi guys, let's get started. We're excited today to have Montana Lowe, who's the co-founder and CEO of CTO, I forget. CEO. CEO.  It does. I'm in the same boat, yeah. He's the CEO and co-founder of PostSML. It's a hosted Postgres instance that also I am deeply integrated with ML frameworks and he's here to talk about what the stuff they've been doing. And again, as always, if you have questions for Montana as he's giving us talk, please unmute yourself, say who you are and find your question, Adam. And feel free to do this anytime and that way he's not talking himself on Zoom for an hour by, you know, which can get lonely. Montana, we appreciate you being here. The floor is yours. Thank you so much. Thanks for having me, Andy. I really appreciate it. Yeah, as Andy said, feel free to interrupt me. It'll probably improve the coherence of my words coming out of my mouth. But we'll dive right in. I like to start with a recap of the whole talk of summary so that you know, like what we're actually going to talk about. Zoolander is one of my favorite movies, but the reason the reason you should care about Postgres ML and the way that we do things and the way that we use databases. I think Andy was mentioning this a little bit before the talk is we're not, we don't deeply change the internals of Postgres by, you know, using columnar storage or doing distributed compute and things like that. Most of our usage of Postgres is much more similar to application usage of Postgres, but we move a lot of the important machine learning things into the database. And the reason we do this is because it gives us more efficiency or reliability and scalability in the database. And when you look at, compare it to a lot of the point solutions that you'll did Postgres ML is competitive with in the ML landscape. It's a lot more capable because it's built on top of the entire Postgres ecosystem and foundation. Obviously Postgres is one of the most compatible cps of software, every language in the world has bindings for it and can call into it. So you don't have to worry about like, is there a client or whatever. And when we, one of the big components of Postgres ML is that it's part of the open source ecosystem and not just the Postgres is open source database. Our extension is open source, but all of the models that we serve, and there's lots of publicly accessible models and algorithm implementations. It's all open source and to end. So that means, you know, as a as an end user, you have a lot more control over how the system is going to work. It's also a very opinionated system. There's there's a lot of what like any application there's 101 ways to skin a cat. But Postgres ML we've we've made decisions that work well together to give you a complete platform, rather than having individual touch points like yes, sure. Pinecone is an excellent vector database, but I hope that you have bindings in your language for it. Otherwise it might not work very well. It's like if you're trying to ETL data from your data warehouse, for example.  It's, you know, we're, we're fairly small project at this point, and we have fairly new technology so that we don't have all the technical debt of some of the machine learning systems that I've worked on in the past.  So I'll dive right into some of the motivations for why we actually started thinking that it was a good idea to move some of these machine learning workloads deep into the database. Because I mean, I'm sure Andy has had too many times in his life where somebody's added something to his one of the databases he's responsible for. It's an obscene workload that should never, that just takes the whole thing down. And this is a very common refrain. And so a lot of DBAs. And for all of you who are taking a databases course, you know, you'll need to protect the database from unwarranted abuse. And this happens all the time. But one very classical use case for a database in the world is the common web application architecture. You know, you've got your software application up top, and you've got your database. And then of course it's connected to the internet. And there's, there's a couple important things here is that apps are stateless. And the databases are responsible for maintaining all of the state in the system and persisting that long term. But, and because these things are connected to the internet, there's latency inherent in the system. And so any of the latency introduced by separating the state and the statelessness is, you know, very small compared to the latency inherent in crossing the continent of the world over a network connection. And so, when you when you look at like, actually, this is this is great for your first, whatever prototype MVP. Everybody should start with something very simple like as a very simple architecture, I would discourage anybody from getting overly complicated with their database or their application technology choices. It's much more important to get something working quickly. But when you talk about the scaling of the system like this, you often think like, okay, as soon as we get more users, what we really will do is we'll scale the out because scaling stateful processes is hard. But what inevitably happens if you're successful is that your app keeps scaling and every time you add more application workload that does increase your database workload. And eventually you reach a point where your database is starting to get a little bit hot. And so you start looking at ways like how can we remove database workload. Quick, quick, easy ways to do that are let's catch more stuff in the app. And so your app starts to become stateful over time as you pull database workloads out of the database. And it gets more and more complicated and hopefully your application is or your business is very successful and you have this problem. It's a great problem to have. But eventually, the amount of state that you'll be managing in your app will continue to grow. But at the same time, you're still going to be adding more and more load to your database. Until the point where you finally cave and you actually start googling on like, how do I actually scale a database and you'll realize that, oh, it's actually pretty easy with Postgres, you just stand up a replica. And there are so many workloads in the world that can be handled by read only non transactional queries that are executed, you know, outside of the normal scope of things and so it's perfectly fine for those queries to go to a replica. And they can be dealing with data that's one to maybe seconds, maybe days, they'll, and so you a lot of systems can be scaled pretty far this way. Just to give you some examples, I think Instacart was doing hundreds of millions of dollars in revenue before we got to the point of needing a replica. I think Figma and Spotify are similar stores where they scaled these massive, massive businesses on a single database, basically, and then they're like, Oh, what do we do next? Oh, okay, replica, that's easy. But eventually, you know, Instacart went through the pandemic and they became a household name, and that meant more and more app servers were required. And, you know, the dreaded day came when we had to shard databases. And when you have to shard databases, it gets pretty messy. You have like the application has to decide what goes where. If any of these databases fails, which happens more and more frequently, you have to have failover logic. You're now managing a whole fleet of databases. And it's like, how do you, how do you operationalize database management? It's not quite as simple as like, Oh, we'll just spin up Kubernetes and we'll dockerize this stuff. There are more considerations that you have to make when you're dealing with stateful systems. And so, in the end, you end up with a lot more database if you're successful. And so this, this architecture, you know, there's, there was a huge backlash in the industry in the maybe ops in the web 2.0 days. And everyone was like, Hey, let's move to microservices. And you know, one, one way of thinking about or defining microservices is that you're trying to design a service that will never require that second database. You know, replication is old, but it's not that old. In terms of functionality that's always been easily available to people. And so there's this idea that like, Oh, we'll have, we'll just have services and we'll just keep breaking up the application into smaller and smaller services. And anytime our database gets too big, we'll just break that database apart. But this, this actually gets pretty slow because you get more and more network latency. And eventually what happens is in a, in a clean service oriented architecture, an entire web request can be serviced by a single service. So like one easy example here is like maybe you have a metric service and all of your user metrics that you collect in from the client side data, all it is a single post request and just record some JSON about some event that happened somewhere. And that can be, that could go into one database table that's your events table. That can be a standalone database. It's very small, very self contained. But eventually you get cross cutting concerns. One example of these cross cutting concerns is like the search system at Instacarp.  But the logic that occurs there involves multiple machine learning models. It involves half a dozen microservices. You know, I've used closer to a dozen actually, at the worst point, many of these have circular dependencies. And they have their own statefulness and they become like these big app level things actually that start to have all the same problems that we had with our monolithic architecture. And then you eventually have to figure out how you're going to shard or scale that that final database. One of one of the great things about microservice architectures that's also a terrible thing is that you know this frees up every team to choose their own database to be suited exactly to the purpose that they need. In Instacarp, you know, we were running Postgres, but also Redis, Memcache, Cassandra, Druid, Redshift, Snowflake. I'm forgetting several of them, but I think if there was a major database that was popular in the last 10 years, we were probably running it behind some microservice. Most of these many of these were for machine learning models as feature stores or model stores.  It's very important of SQLite, of course, SQLite in S3 because why not? That's the outer circle of hell, right?  And you're just like, okay, I've got to get out of this circle. Where do I go next? The only way, I guess, is down. At least you can say the big O word, right? Oh, yeah, thank God. But no, we had people who were like, why don't we use my SQL because, you know, Postgres has vacuuming and I don't like that and I'd rather have anyway. You can, once you open the can of worms that is microservices, it's pretty hard to put it back in the box. But eventually we did to a large degree at Instacarp. The way I think about microservices is that like, you thought it was hard to manage a sharded database system, but it was a single kind of database. It is much, much more difficult to reason about what happens when, for example, your memcache cluster goes down and now your service that was depending on that is now backfilling memcache and how is it doing that? Of course, it's hitting your primary application database, which then takes your primary application database down, which then brings down the whole site, but you thought you had microservices with database isolation and it's really, really difficult to actually achieve that level of isolation when things get complicated. To give you an idea of how complicated things can get, this is a chart created originally by Andreessen Horowitz, who's one of the big investors here in the Valley. For example, they invested $100 million in Pinecone recently, which is one of the hot new vector databases. They are very in touch with what companies are doing, and I've included a link in this graph to their original write up and blog post where they presented this. This is only a small expanded box in their much larger data infrastructure diagram, but it gives you a peek into what you need to actually build a machine learning model and a machine learning service. And when a request for search comes in, and it's using a dozen models created by a dozen different data scientists, you'll notice that in each one of these boxes, there's always a handful of different competitive technologies that can be used for that function. Inevitably, your data scientists will all make different choices just like they did for their databases. And if you don't have a really strong machine learning platform, it's sort of solved all of these problems. Every single request will go through a different microservice that's virtually an entirely new and different stack. Each one of those requests will take anywhere from 50 to 500 milliseconds. You have a search system that first has to be named into the recognition, and then it has to do synonym detection, and then it has to do query expansion, and then it has to do its initial query, and then it has to query for potential replacements for low stock items. And all of these involve multiple models. At one point at Instacart, when we had all of these microservices that were Python based, our P90 query times for search were up around eight seconds to get through. And sure, that's P90. It only happens one in 10 times, right? Except people do more than 10 searches on every visit. So basically, every customer was hitting at least one of these. And when you make somebody wait, you know, eight, nine, 10 seconds during their shopping cart, some significant portion will churn out, or they'll just give up on whatever it was that they were looking for that time in search or something else. So if you don't just lose, if you don't completely lose the customer, you'll at least lose some fraction of sales. I realized this talks not about Instacart. Talk about your stuff. Like, was that eight second P90? Was that like a, you flipped a switch on something new and it was just that slow? Oh, no, no. No, machine learning, they always want to add like a little bit more data or a little bit more sophistication or one more microservice. And so it's constantly like each microservice they add to add some new functionality. It's like, oh, this one's only 50 milliseconds, right? Yep. But then when you've got dozens of them involved at the end, you know, after years of iterative development on a search and recommendation system, then all of a sudden the CEO of the company is like, Hey, guys, I was trying to search on my mobile and it was just timing out what's going on with your team. Yep. Yeah, I would say the other thing we noticed is some of this is auto and stuff from when the, when the CFO or the CEO knows the name of the service or the name of the database. That's a problem that like you got that's when people, you know, people got, you know, actually, or have a motivation actually fix it. Yeah, no, it's definitely true. And so I got to be part of a task force that was tasked with like, you're going to bring down search performance speeds and you're going to make this system better. And that's actually where, you know, this talk is about Postgres ML, but a lot of the thinking for Postgres ML came from those explorations and those learnings. I mean, a lot of it was my fault too. Like, you know, when I got to Instacart, I helped build the machine learning platform and I helped set up a lot of the data engineering principles and, you know, a lot of service oriented architecture makes sense. I don't want to bag on like microservices are terrible idea. If you can like fully isolate something like I mentioned, then it's actually great and you should, you should try to do that. And it's a great way to break load out of your primary database without complicating your life too much. But in the end, if your business is as successful as something like Instacart is, you will have database problems. And so in the end, we moved just to an architecture like the one I've got on the screen now, where we did end up sharding Postgres. And we ended up sharding multiple Postgres databases, but we're now Instacart is fronting all of their Postgres clusters with PGCAT so that it makes it much cleaner at the application layer so that they can go through a proxy pooler like this that is shard aware and it can handle your recovery and everything else. And if, so if, if you know this is where you're going to end up, this is actually the desirable end state for your app where you're not having to cash so much at the application layer, but you can you can actually to remove database load, but you can actually handle scaling your database horizontally, both with replicas and sharding and that it's actually not that horrendous and terrible of a process. And especially if it will save you from all of the other pain points that can be involved in microservices and having many different kinds of databases out there, then I think this architecture is a pretty sweet spot to live in. But the really nice thing about this architecture is that you don't have to start with it. You can actually start with the original web app architecture of like you've got one app you've got one database that you pick your app server, whether that's node or Python or Ruby or Java, pick your database postgres is a really good choice, because it's so general purpose can handle so many workloads. But just know that it's rather than having to like make your app really complicated. You can just throw a pooler in the middle of your databases, and you can keep that and you can't really get out. And so I think companies that go this way in the future will have a much better time. There's something I've seen at a lot of startups is that that I've worked at is when we have all the engineering buttoned up really well and we have our architecture being clean and crisp, and everything's properly refactored all the time. Usually the business isn't doing very well. And I think that's why you have all the time to get all of your engineering right. I think that if your business is really growing really quickly, that there's so many urgent priorities for engineers to work on that there's very little time to go back and clean everything up. And so if you don't have that pressure to move quickly, then maybe your engineering will be okay. But maybe your business won't. And after sort of coming to that conclusion, I decided, well, I'll just found an engineering specific company. So engineering will be the whole business and we can make the engineering really good so that I can have my cake and eat it too. Because, you know, as a software engineer, code cleanliness and best practices, strong principles are important to me sort of spedically. But, but in the end, you have to be willing to do what needs to be done for the business, not necessarily for that, or whatever idealistic engineering principles I happen to have. So don't. Sorry, do you accept questions along the way? I tried late sorry. Oh, yes. Yes. Okay, so just a question on the sharding. I was wondering, not sure if you're aware of like systems like our coach DB or you go by it. Would you consider those instead of sharding or would you prefer sharding over those systems that have the partitioning sort of speak in their schema building. No, I think that those those can be great. One anecdote I'll share as well is that, you know, we moved to elastic search. That was one of our first projects of we had a postgres catalog database holding all of our product data. And so we that database was overloaded and falling over because we were treating it like a data warehouse, not like a application database. And one of the first projects was to move all of that data into elastic search and front it with a sharded elastic search cluster, which was great for about five years. But we didn't have good enough control over the elastic search sharding schema and algorithm and everything else to get where we needed to be and eventually we had to redo our sharding on postgres where we could have more control over the end to end solution. We escalated all the way up to the CTO of elastic. And because of some of our needs that were like, Oh, well, yeah, it says right here in our documentation, you can't you can't do cross shark joins. Those are always going to be slow. And so if you if you and we don't have any indexing types that will help get you out of this jam that your business requires you to be in. So I think long story short. Yes, there are lots of databases out there that will do the sharding for you all you have to pick a key right if you can use those at some point. You may find that your needs get more complicated and need and you have these cross cutting concerns. And I think particularly with machine learning again, you find yourself where you end up in these cross cutting concerns, where there's no single sharding key that will do it all for you. And you and you have to have to reconsider but yeah. One of the things I really like about Postgres is just how how much control ultimately you can have over everything. And if it's not built into Postgres itself, it's it's one of the most extensible databases out there. Always write your own extension to do the thing that you want. And that that's actually a lot of where the motivation for Postgres ML came from. It was, it was a realization of if we have this horizontally scalable database. And all of the, all of our machine learning date, like the scalability of these machine learning microservices the hardest part was always scaling the feature store the hardest part of engineering and the most complicated engineering was about getting from wherever it lived into the feature store so that it would be there in time to make the real time online prediction. And, and so trying to figure out, you know, how we simplified those systems that in Scott we ended up replacing our elastic search cluster with a system very similar to what I've shown you. And that is big Postgres charted thing and then we started moving all of our feature store data from various databases into this big charted cluster and we did this in the middle of COVID. I don't think that you know had the business not been exploding exploding in a very good way. If we were doubling every other week or something, then we wouldn't have had the license to start making these huge engineering moves and having sort of all hands on deck of pulling everybody's favorite database out of their hands and saying like this is the one scalable system, your system is like quickly going down under this load. So, so the only option at that point was to build something like like we've shown. At the end of the day, it was shockingly successful. We went from a acidic, for example, if you wanted to add some new data to the Instacart catalog, and a product manager was like oh we want this feature it would take literally the last one we did on elastic search took three quarters of iteration, nine months. Okay, the product manager says they want this feature they go to the catalog team the catalog teams that we're going to put that in snowflake, and then we're going to figure out a way to like etl that through Druid to do some future computation and then we're going to go from Druid and just that no elastic search and then, oh by the way we didn't get it in the right format that the search team needs it so we'll just start this whole like cycle over again and cross like three, three VPs of engineering and half of a half a dozen engineering teams. The coordination overhead there was terrible. It's like oh wait we don't support your your data type for time stamps that's use strings for time stamps throughout the whole system, or or whatever. And so, when we said everybody was going to put everything inside of Postgres, we're going to let anybody get out of any table they want. Then when we had issues it's like oh we're just going to change the column type in Postgres, the two engineers going to agree in a meeting, and it takes an hour now, instead of multiple weeks of like oh we've got to reconfigure our whatever service. But anyway, I feel like I've wandered a little bit away from the slide. If you if you don't buy all of the are all reasoning for simpler database architectures and how those will make your life better. They're in especially in terms of machine learning complexity. There there's this notion of data gravity that like the more data you get into a system. The more data it will also attract the more applications will get built around it. And again you'll have this snowballing problem of like unconstrained growth in the data layer. But in machine learning you have a different option. You can be running your, your model as if it were a stateless service, and every time your model needs to make a prediction you can go fetch data and pull the data up to the model. Or you can do what Postgres ML does which is push the model down into the database into the data storage layer. And then you know you're not pulling data out of the database to the application layer, you're you're just passing a pointer from Postgres shared buffers through the model and so there's no more data movement. That does mean you have to redeploy models and so you're moving models, instead of moving data. And in my mind, this is, this is fundamentally better and it's provably better, because any model is always smaller if it's a good model. It's always smaller than the data set that it's trained on, and it's always going to be smaller than the data set that it will be used for prediction. And it will always change less frequently than the data set that it's being used to model, otherwise it's just not a good model. If you have to constantly update your model then it hasn't generalized and you've really failed at the machine learning aspect. If you build your model well, then there will be fewer electrons involved in a Postgres ML kind of process than would be in a microservice architecture process. And there's a question of like, how much it does that actually matter? Aren't computers fast? Aren't networks fast? Isn't ML inherently slow and expensive anyway? So are you optimizing the right thing? And that's a really good question. You should always benchmark and optimize the right thing. There's another question of like, a lot of Postgres ML thinking applies to classical machine learning. The systems that we were building this for, we did have some deep learning models involved in our search services at Instacart years ago running like TensorFlow 0.4 or whatever it was in production. But the new world that everybody's really excited about is like vector databases and GPT-4 and open AI. And like, can't you just make a call to open AI? Why do I need to consider anything else? But if you're even in the new world, your open AI chat GPT model in the open source world where you're hosting this thing yourself, it's still a massive thing. It's still 70 gigabytes. It will break all of your traditional software application continuous integration deployment pipelines, because most people aren't deploying 70 gigabyte Kubernetes containers. So you're going to have to rethink your deployment system as it is and like how you actually manage these systems. But at the same time, these models are still incredibly data hungry at inference time because you need to go pull back not just one vector, but potentially hundreds of vectors. Vector databases will do cosine similarity, but cosine similarity is actually a really bad predictor of relevance compared to having a trained model that is trained to predict the relevance. And one of the things that is catching on now is like, oh, I'll fetch 10 documents from my vector database by nearest neighbor. And then I'll feed that to a pruning model that will select the top two or three most relevant documents before I actually pass that on to my text generation model. So even in this new world of LLMs and vector databases, being able to have the data, whether it's vector data or traditional and user data, tabular data, in the same process as the LLM, even though LLMs are slow to run, you know, they're anywhere from 10 milliseconds to many seconds of runtime. That data movement is still a considerable case or a considerable expense. I've got some benchmarks we can show later in the talk, but this matters as much as ever. You really can load up your LLM in your database one time, and that's a one time data movement cost that will then save you from moving. And keep in mind, vectors are, you know, a thousand four byte floats long, that's four kilobytes. It doesn't take, somebody wants to pull 100 vectors out of your Postgres database. Then they're talking about half a megabyte of data movement. And then you're going to pull that into a Python process, which is going to blow it up to like 50 megabytes of Python data memory.  And it's just, I mean, actually, you'll spend as much time in pandas in the data frames in the Python world, as you will actually using your model and actually using your data. And one of the really cool things is like, you can see that this is a sequence of events that starts with the app and goes through the embedding model. And then it goes prompt creation, then it goes tech generation and then response comes back to the app. I should have had an entry arrow on this diagram starting with the app. So you can see the actual whole loop on what a request lifecycle looks like. But you can actually write a single Postgres query with multiple comment table extensions. They can do the first, you know, it can do a union between an embedding query and a normal SQL query as a CTE.  That can then be calling a pruning model as a third CTE. And so you can just chain these comment table expressions together until what you really have is like a full program of multiple steps. Except that program instead of executing across a web of Python microservices, serializing and de-ferializing the data at every single step, it all happens inside a single Postgres process. And so you cut out so much network latency. And when we actually look at the load on the database in these cases from serializing all of this data in and out, Postgres load and query times actually drop because instead of having to send back, you know, half mag of vectors on every single query, they're sending back like a 10 kilobyte text string or something. And so just like the data movement in and out of the entire system can drop significantly. And then you've also like dropped off this huge web of microservices that are now non-existent instead of massive GPU bills. So I hope I've convinced you that this is at least an interesting idea. I was going to give you a little bit of an idea of like what Postgres ML actually is. For classical machine learning, it's just these three functions. These are basically UDFs that the extension provides for Postgres. Machine learning is actually a very well-defined process. We have supervised and unsupervised learning. We have classification and regression. These are all tasks that can be done with machine learning if you're not familiar. These can all be provided as just parameters to a training function. You can just say, I want a classification model. Literally, that's the task. That's what it's going to do. The problem formulation with machine learning is still, I think, hard. And I think that's where most people get stumped. But once you can formulate your business problem as a machine learning, either classification or regression or now as a text generation problem for chatGPT, then you're off to the races. Postgres ML gives you the ability to train models. It gives you the ability to then strategically deploy those models like you would expect. And finally, it gives you this predict call that you can now leverage that model given some new data that's been written to the database. Or you can just pass Postgres accepts parameters in queries. You can just pass the, you don't even have to actually, this was surprising to us that people were using Postgres ML as just a model inference server. It's basically stateless service, but they liked Postgres better than having like GRPC or some other H2P REST endpoint because they trust Postgres to be able to serve responses reliably. And they know how to manage Postgres as an existing piece of infrastructure. There's, I think this is more like the new school of vector databases and transformers that Postgres ML also provides. It's worth, I think, noting that, and I'll talk about the technology that we use and how we build this stuff in a little while, but the transformers stuff, the Hugging Face Transformer stuff is still in Python. Everything else is written in Rust. And so we have like good zero copy abstractions and a lot of places we can move data without having to actually copy it. But in the Python case, we still do have to go through Python to access some of the latest LLMs. And those things are changing and coming out like every week. So it's pretty hard to nail them down and standardize them, but there's progress being made. And we'll get back to a lower level implementation on this front in the future. But given those six functions, you have a very comprehensive machine learning toolkit that you can solve a lot of problems with everything in Postgres. So this slide talks a little bit about how we actually bear memory inside of the Postgres process. If you know Postgres very well, then you know shared buffers where Postgres pages data in and out from disk to RAM. And that's configurable. But we store our models in Postgres tables and we store our feature data in Postgres tables. And so that is naturally stored in the shared buffers. And so Postgres manages that global cache resource for us. But each connection that's opened for Postgres, when you actually call one of these functions like predict or embed or transform that leverages a model, that the actual, you know, pulling the model out of all of the weights out of shared buffers and instantiating it with whatever model inference library it needs to use, whether that's XGBoost or scikit-learn or PyTorch. We support all of those under the covers. Your model gets cached in the connection. And because so many of these models run in Python and were originally conceived of in Python, none of them are developed to be concurrent. The support concurrent access, they all have some kind of a lock around their usage. And so this actually works really well in the Postgres connection process memory model because every connection is an independent process. We can load as many copies into as many different connections. And then it's Postgres connections that give you concurrent access to the model as many times as you need. Now this PGCAT is also really important in this picture because PGCAT allows us to keep that connection open even when a client goes away, which preserves our model cache of what models were actually being used. We can also use Postgres roles. And so like if you want to isolate certain connections or throttle certain models and enforce queuing, you can do all of that with PGCAT. And you can say that, you know, whatever user is using model XYZ, we're going to limit them to one or two or 10 back end Postgres connections, which limits their total usage and throughput in the system and their queue.  How big is usually a model? It varies wildly, but a linear regression model is like eight bytes. It's like two floats basically. And you can actually do a lot like that. But like an XGBoost model can be anywhere from like 10K to a couple of megabytes. XGBoost really is state-of-the-art for tabular data inference. But like, yeah, it's on a connection for like a state-of-the-art model for doing, you know, most of the search tasks, it's totally fine. It's even on like a tiny database. But when you get into LLMs, then you can be talking about like a, you know, Lama7DB is like 280 gigabytes if you're using the full precision. Yeah, that's what I was getting at. So like, so every connection, every Postgres worker has a copy of a model sitting in memory. And it's not, is it in shared buffers? So, I mean, because the model is persisted in a table. So if you want Lama7DB in Postgres ML, you can get persisted in a table. It's going to be a 280 gigabyte. Well, you know, there's a row data size limit in Postgres. So we transparently split it up into multiple rows.  That becomes 280 gigabytes in shared buffers. And then that actually, you know, we'll get actually copied into your connection-specific model cache. So then your connection will need 280 gigabytes of workman. Yeah, that's what I was getting at. Yeah, so it's almost to the point for, I understand not everyone's running the full Lama thing, but like, at some point you need to dedupe that memory. Yeah, absolutely. And I think that that will be part of the move away from Python transformers to libraries like Rustformers. This is all very new stuff for us that we've just been, you know, extending in the last three to six months. And we've got a lot of work to do on this. But I think you're absolutely right. And we want to get to a point where we can like share beyond what people are doing in Python, where we can actually share the read-only weights of these models. And then we'll only need to allocate memory for the intermediate computation steps. There's still a lot of these buffers are still intermediate computation that have to be isolated and can't be concurrently accessed by multiple processes at the same time. Because they write to them for intermediate use. But we can definitely do better there. Awesome, thanks. And the GPU cache is also interesting in that regard. Because you're much more memory limited on GPUs. And so I think this is where a lot of the motivation for us is coming from de-duping memory. And we have a serverless cloud offering right now where that being able to offer people, you know, a time slice of a GPU and a shared memory model across multiple connections is really important. Because, you know, a lot of people experimenting with LLMs and transformers right now are hobbyists or enthusiasts or prosumers working on hobby projects. They're not necessarily large corporations that have the budget to spend $5,000 a month on a GPU in the cloud. And so being able to reuse that multiple connections from PGCAD is something that we're pretty excited about. Because then we can charge people 60 or 70 bucks for their little embedding model. They've got a vector database with an embedding model that they can query many times an hour for their chat bot. But they can't, and they don't need to fully utilize the GPU. Either it's RAM or it's compute. So yeah, I think expect to see more there. And so we love benchmarks at Postgres ML. In some ways I feel like we're just cheating because we don't have all of the network overhead and everybody else has network overhead. We can say things like we're 10 times faster than OpenAI for embedding generation. But that's because you have to call OpenAI over the internet. You can't run OpenAI in your data center. The best you can do is like try to guess where Microsoft is hosting their things and then to put your app or whatever in the same data center. But you're still subject to all kinds of queuing and whatever. A lot of people think that OpenAI is a clear leader. And so it's worth waiting for higher quality. I want the best when it comes to these things. But it's not true. OpenAI has lost pretty much every domain except for text generation. They used to be the leading image producer with Dolly too.  Trying to get back to relevance against stable diffusion or against mid-journey, which really took it away from them. They used to be really relevant when it came to embeddings but they now ranked like 12th or 13th and they keep getting pushed down the leaderboard across the metrics that we see. And I mean this isn't a whole other interesting question of like is OpenSource going to win or is CloseSource going to win? Will OpenAI keep their lead and will you always need an OpenAI integration or will we actually be able to run OpenSource models in our database or in our local Lama setup? My money is on OpenSource. I think that we've had multiple reports of like the whole Google we have no moat thing going on and like the fact that GPT-4 appears to just be a mixture of experts of GPT-3.5. And now GPT-3.5 is losing to Falcon 180B and a lot of context so I think like we might even see them lose text generation in the next three to six months unless of course they release GPT-5. We don't owe, they don't tell us. But I think it's going to be very competitive and very interesting and there's still a lot of other reasons to choose OpenSource other components that you can mix and match with text generation. Yeah, it's a similar story. Like if you use Huggingface for your text generation to notice these models and then you use Pinecone as your vector database. I mean, PGVector added hierarchical, navigable, small worlds, H&SW as an index type last month. Andrew Cain's been crushing it on that front. But even with IVF flat, the let it faster to index things. So it's still a relevant indexing type for vectors. But even using IVF flat for query time, which is slower than H&SW and doesn't scale to as large of collections, we're significantly faster because you've eliminated two internet round trips, which are a lot slower than the 10 milliseconds it takes to do an embedding generation and the sub millisecond it takes to do a vector index lookup even with IVF flat. So you can go down the list and you can dig deeper here. In my mind, I think that we have presented a pretty strong case for, you know, this terrifying concept of moving more workload into the database that it's not just effective. It's also safe and it's also scalable. And if you follow those architectural principles, it'll be a lot better in the long run. Oh, yeah, it is important to note that, like, a lot of, I think SQL is awesome. I imagine everybody in this databases course also thinks SQL is awesome, but a lot of people only know Python or JavaScript or whatever language. And so we actually have another Rust project where we generate Python bindings and we generate JavaScript bindings that encapsulate a lot of these common machine learning application paradigms and give you, like, three easy JavaScript functions you can call if you want to index documents and recall them from a vector index from your JavaScript app without having to actually write any SQL or know about an IVF flat index or anything like that. And again, this means that, like, people who don't know anything about retrieval or anything, I guess, can get the benefits of a much, much faster architectural. So, like, getting, I think we're coming up on time. We've got 10 minutes left. Andy, is that about right? Yes. Yeah. So I'll go through this pretty quickly. You know, we use PGRX. It is a Rust extension management framework. It is awesome. Like, I love developing. I never thought that I would love developing a Postgres extension, but it's a pretty nice life. It's all pretty well managed. It's like writing any other Rust app at this point. I love the strongly typed Rust application with strongly typed database schema. Having lived in a world of Ruby and Python and Elastic Search and Cassandra that are all, you know, all of these things are schema and type-less. They're runtime typed, whatever. It's amazing to see, like, I just add a new enum somewhere, and then all of my Rust match statements are broken, and then I go fix that. And then the Rust compiler tells me, I haven't taken care of a bunch of other things, and I'm like, oh, I've got to add this. And, like, I don't really have to think through it anymore to add features. There's enough of a framework in the application. But if I just break the first thing by adding something to an enum, it pretty much tells me everything. I have to fill out. So I'm pretty excited about the full extension. These are some of the libraries that we use under the covers. There's a lot going on to move more machine learning into Rust in the Rust community. So I think this feels good to me. But in the meantime, we do call back into Python. And there's a couple of reasons. We want good, strong reference implementations. People have been using Sykit for 20 years. They come in to Postgres ML, and they're like, I want to see the exact same convergence and statistics that I was getting. And so we need to be able to at least give them that pass out equivalence test before they can move onto a different implementation or different platform that they're used to. So I'll stop there. Open up to any questions that people might have. Awesome. Thanks so much. I will applaud and have everyone here. If you have any questions, Kristoff, you want to go first? Yeah. I have a question on PG Vector. If I look across different vector databases, then they implement different similarity metrics and different indices. And not a single one has all of them. So I was wondering, do you have a sense of if PG Vector wants to become in quotes a superset of all these combinations so that I maybe avoid having to run two or three databases depending on the vector similarity search or index support? So I think if you're running less than, say, 10 million vectors in your corpus, it doesn't matter. It's going to be whatever there is going to be fast enough. And also, and that's for indexing, whether you're using IVF flat or H&SW, or there's at least six that I think are out there right now and fairly popular. PG Vector, IVF flat is the one that's really fast to build the index, but it has slightly worse query performance. H&SW really slow to build the index, but has much faster. So you can pick which of those two extremes you want to live on. It doesn't have all the ones in the middle. But I think from my perspective, that's good enough for, like, 98% of people. And honestly, anybody who has less than 10,000 vectors doesn't need an index at all. You can just run the query and brute force it, and it'll come back in 10 milliseconds. It's fine. I mean, I guess if you need sub 10 millisecond queries, then sure, put an index on it, and then you'll be bound by your inside data center query time, which is going to be a millisecond to get between boxes and connect to your Postgres instance. In terms of the operations, like cosine similarity versus Manhattan distance, those don't really impact query or indexing speed that much as much as, like, the index type or network latency would. I wouldn't worry too much about that. What you should worry about is cosine, all of those are very simple arithmetic distance functions. But they all treat every single element of the vector as equally important to whatever is being measured, and that's rarely the case. And so what you actually want is you want to train a machine learning model using user feedback data to actually tell you how similar two vectors are and not rely on cosine distance or cosine similarity or the dot product or the Manhattan distance. If you really want to improve that, and the only way you can do that is if you can take 1,000 of these vectors and then run them through an XGBoost model in the same memory space. Otherwise, it's prohibitively expensive to pull 1,000 vectors out of your database, feed them to an XGBoost model. But this is what we do at any modern search and recommendation system.  You can't possibly pull 1,000 vectors out. And so there's some negotiation and haggling that takes place of what if we just pull 100 out or what if we just pull 50 out? Can I get some latency budget back for my next machine learning project? But then when you're only looking at the top 50, instead of the top 1,000, there's usually something in the long tail that might have been the product that the user was going to buy that XGBoost would have been able to promote all the way to the top, but cosine distance won't find it. So you just lose some percentage. Thanks. I'll just say we invited the PG vector guy to come give a talk and declined. So we have the neon guys are building PG embedding. And they're giving a talk later in the semester. Other questions? Go ahead, sir. I was going to say it's exciting to see multiple implementations take off here. Yes. Any other questions from the audience? So I'll finish up by saying a question. I mentioned the memory deep issue. And you guys also did a major sort of refactoring, sounds like from Python to PostgreSQL-based extensions. What's another sort of major system task on your horizon for the next one to two years that you guys want to undertake? And if you want to go five years out, by all means do it. But what's a major challenge you think is sort of unsolved that the space you're working in? Well, I mean, off of the top of my head, columnar storage is actually really important for time series calculations. They've got good, there are good algorithms out there and in timescale has already implemented it from Postgres. But unfortunately their license wouldn't allow us to offer something like that. So I think coming up with an actual open source implementation for columnar storage that we can integrate with some of these time series predictions would be good. We do have a long road to go when it comes to adopting rust implementations for the latest LLMs and making sure that we can get back to a fully deduced memory storage. I think that that is what we'll consider 3.0 for Postgres ML. That'll be a big milestone achievement for us. We've got probably 50 or 100 GitHub issues open right now. So I think an enormous breadth of coverage that when you look at what's possible in machine learning, there are so many different algorithms or bells or whistles that people want that we are effectively competing with the entire Python ecosystem. Like as soon as Yandex releases CatBoost, people are like, oh, it's the latest ingredient boosted trees. Can you do CatBoost too? And it's like, oh, OK, yeah, we'll add that. But it's a never-ending fire hydrant. So I think having help on that front would be awesome too.


So in this video, we will review reading a stem plot or stem and leaf plot. The stem and leaf plot shows house sale prices over the last week in a city. What was the most expensive house sold? Give your answer in dollars. So in my stem and leaf plot, the stem is actually representing my bigger place value, which in this case is the 100,000th place, and my leaf is representing the smaller place value, in this case the 10,000th place. So it's important to note that our numbers will be in the 100,000th, that means there will be six digits. So for instance, if my stem is one and my leaf is nine, this would mean a one in the 100,000th place and a nine in the 10,000th place, and everything else is going to be a zero. So that represents the price of 190,000. So they want to know what was the most expensive house sold. Well that would be in the 400,000th, where the leaf is seven. So that is number six digits. So my stem is four, representing 400,000th of my leaf, which is the 10,000th place will be seven, everything else will be zeros. So at the end of the day, what is the price, the highest price, $470,000. That's all I have for now. Thanks for watching.


Welcome to this week's preview show here at Vitality Stadium. BBC Radio Solence Chris Temple is back alongside me as we look ahead to this weekend in the Premier League. Here's what's coming up. We'll be discussing that narrow 1-0 defeat against Arsenal at the Emirates Stadium. We'll also be discussing the international break and who played where around the world. And finally, we'll be looking ahead to tomorrow's game against Norwich City here at Vitality Stadium. But first, we're going to start back two weeks ago and that game against Arsenal at the Emirates Stadium. Chris, it would certainly be fair to say it was a game of two halves, wasn't it? It was. We've seen that quite a few times in Bournemouth where they have taken 45 minutes to get themselves either settled in the game or not be too defensive, which can sometimes happen particularly away at the big teams as well. The first half, you were thinking this could end up being a couple of the four or five that we've seen at the big clubs in the past. The second half, you're looking at some great chances or a couple of notable opportunities anyway, much more like the fluid football that Bournemouth can play, particularly away from home on the counter. And yeah, it did seem like the shackles came off in the second half and obviously they had to get themselves back in the game so they needed to score and therefore they needed to approach the game with a bit more offensiveness if you like. But yeah, in the end you're looking and thinking possibly they could have come away with something from there. And you're talking of those chances. There was none better than that Callum or someone, was there? Where he rounded the keeper and he squared it. And again, when you start to get a bit of form coming together and you're thinking, OK, well, here's a great chance now and the way things have been going. Recently the goals have been going in, you know, Callum's been in form. You know, it was unselfish for him to square it. There was nobody quite there. You think Callum earlier in the season when he wasn't scoring, he probably would have gone on himself there in his desperation to find a goal. So sometimes the decision making, you know, is borderline in the heat at the moment. But away from home at Arsenal, that's the kind of chances that you need to make something of. You don't get too many opportunities as good as that. And not many teams will go to Arsenal and give them the problems that Bournemouth gave him in the second half or certainly give them as much to think about. So yeah, all in all, you know, no points, no goals, but certainly a performance that I don't think has left anybody feeling particularly negative. Yeah, the first half wasn't that good and then he was frustrated that it took them 45 minutes to get out of the blocks. But yeah, I think coming back off international break, that's a performance that keeps the momentum that's been building in the last few weeks, albeit in defeat. And as you say, yes, the first half wasn't perfect. But actually, for all of Arsenal's possession, they didn't actually test Aaron Ramsdale too much. The defence limited Arsenal to their chance. Yeah, and the goal was a bit soft. You know, David Lewis' header was a little bit soft, wasn't it, as well? You know, in terms of set pieces, it'd be disappointed to concede a goal like that as well. It wasn't the most forceful header in the world. It was beyond Aaron Ramsdale. You know, nothing he could do about it. But you know, you think of conceding from set pieces, that's a little bit disappointing. But yeah, for all of the offensive quality that Arsenal have got, you know, and the relatively new look and inexperience back for, we say inexperience, because Rico and Stacey at fullbacks haven't played that many games, you know, in a row. But Rico looks like he's growing with this little run of starts. He's having Jack Stacey continues to grow. And you know, he's up against some very good players, the like of which he hasn't faced on a weekly basis. You can play them in the cut one off here and there. But to be facing them week in week out and to have that mental test every single week. So yeah, I think the new back four looks to have been, you know, for if you put together all the teams, the defenders of how many appearances they put together to go and only concede one Arsenal. I think again, that's shows positive blocks for the future. And just a word on Diego Rico there. He's had a run in the team when he's come away with September's player of the month, which is something last year, to be honest, fans couldn't really see happening. I'm not sure anybody could see that happening with the greatest respect to Diego.  I've been critical on here in the past and we haven't seen enough from him. He would admit that himself, I'm sure. But again, you speak to the players and they say, you know, on the training ground, he's, you know, he shows, shows promise. He shows some of the ability that may Eddie Howe pay 10 million pounds for him. You know, some of his set pieces and his assist is where he's going to be proving valuable. You know, without the likes of David Brooks and Ryan Fraser, not in the team at the moment. So set pieces and things. He's a with that left foot. He's a, you know, a key figure in the team, but defensively he's where there's been a few question marks. And he is an attacking fallback really. He's one who wants to get forward. He's an Eddie Howe type of fallback. He doesn't really have fallbacks that just sit in the defensive line. They're always, you know, almost auxiliary wingers at times, which sometimes can be the undoing of the team as well. But no, really pleased for Diego that he's been voted as the player of the month. And he, you know, hopefully will continue to benefit from this run of starts. He's getting, you know, albeit slightly fortunate because of Charlie's injury and Adam Smith's injury, but as one door closes, et cetera, he's taken his chance. Absolutely. And just on that Arsenal game, when you look at previous performances against Arsenal and, you know, our results there in recent years, I think it was 5-1 last year, it was a much improved performance, you know, going there and sitting tight and creating our own chances, which is something, you know, in the last few years we haven't done. Yes, least we say scored last year, but again it was on the back of a defensive mistake.  You can't come out in the first 15 minutes and just go blazing because you'll get picked off and we've seen that happen before. But at the same time, if you sit back, that's inviting trouble as well. So the balance wasn't right in the first half between that. It was a little bit too defensive and a little bit allowing Arsenal to dictate. But in the second half, you know, whether it was the, you know, the spur of having gone behind and having to find a reaction and to get a goal to get something from the game, but it shouldn't need a team to fall behind to provoke that reaction. So, yeah, all in all, as I say, the ball continuing to roll in the right direction even though it was a defeat, it wasn't a defeat that sort of sapped the momentum out of that recent run. And just finally, when you look back on the last block of games, the four games between September's international break and October's international break, two wins, a draw, and then obviously that defeated against Arsenal, so it's nothing to be ashamed of. No, it's a good block. That's a good block of four. You know, West Ham Games, one that got away, of course, here, but again, a performance that had, you know, plenty of positives about it. Lots of VAR, you know, goals chalked off and chalked on in that little block of games as well. But yeah, striker scoring, you know, Callum has found his form. Joshua King contributes in from open play as well in recent matches. So, and defensively looking, you know, reasonably solid, one or two sort of sloppy mistakes to give away goals, but actually, you know, Aaron Ramsdale continues to sort of exert himself in goal and have authority over the back line as well, which is great to see, and you know, obviously having a great international break as well with England under 21, so he'll continue to grow in stature. So, all in all, that was a good block. Look at the next block coming up, which we'll get onto in a bit, but starting with Norwich, Watford, Man United, Newcastle, you know, there's three games in there that you're looking thinking there's some points to be had. Absolutely. Well, earlier this week, our international players returned to Vitality Stadium and are back in training. Let's take a look at how they've gone. Well, great to have those lads back in training. Twelve players called up overall, and we'll start with Callum Wilson. He had that glorious chance for England, didn't he? Should have scored. He knows it. You know, he should have scored. What a great chance. And he's been crying out for minutes really. He hasn't really had that many opportunities with England. Recently, obviously, he was ill for the first game, but the second game, you know, and he looked actually bright. He came on at left midfield, you know, left wing, which Eddie House said he'd given him some food for thought. The one person I'm pretty sure would be pleased to see Callum Wilson excelling at left midfield would have been Joshua King, because that might give him a chance to play down the middle like he's been crying out for. But no, at all seriousness, I mean, Callum will know he should have scored. A couple of great little link-ups in the few minutes he did get. He looked sharp, you know. He looked like a guy who's found his feet this season. You know, he's right up there in the scoring charts in the Premier League, so quite rightly getting his opportunity. And for him, you know, unfortunately, all the while Harry Kane's fit, England minutes are going to be hard to come by. So, at least in those few minutes he got in Bulgaria, he took his opportunity. So, yeah, I'm really pleased for him. And in terms of the under-21s, the young Lions, at one point we had three of our players on the pitch at the same time, which is quite something, you know, for Eddie and his staff and the Academy guys for Sam Sturridge. Yeah, let's start with the guy who's not here at the moment, Sam Sturridge, because I think that was a complete surprise to him, let alone anybody else, that he got a very late call-up for that squad, but managed to get on in both games and also scored, obviously, in his first game. So that's brilliant for him. And, you know, we think of the impression he made when he got a couple of appearances last season in the first team when Needs must, and they were sort of down to the bare bones of players. So, yeah, great for him. He's having a good season at Swansea. And hopefully we'll continue to flourish and come back here and be a first team sort of contender, if you like. Yeah, and Aaron Ramsdale, you know, he's unequivocal number one for England under-21s. And Lloyd Kelly playing the full game as well. You know, still waiting for his Premier League debut. We've seen him briefly in the League Cup. But, you know, he's still waiting to get on that pitch in the Premier League. And at the moment, for him, unfortunately, the way the defenders have been playing, Diego Rico's performance is probably, you know, Lloyd Kelly is the one who's knocking on the door at left back if Diego Rico, you know, his standard slip, I guess. So he's having to be patient and wait his chance. But again, he, you know, I think he contributed and assisted in that game as well. So, yeah, really good for him to be getting minutes, albeit if you can't get them here. And you mentioned Sam Surridge scoring. Another person who scored was Joshua King, a last minute penalty for Norway against Spain. Yeah, they rescued late draws in both their games, didn't they, Norway? Pressure penalty, that one as well, you know, against, against Spain last minute. Unbeaten home record to protect as well. So, yeah, that showed a good corner. See, doesn't miss too often from the spot these days, Joshua King. So, yeah, that was a notable moment for him as well. You know, and some, you know, good minutes elsewhere for other players. Well, Nathan Hacke, unfortunately, didn't get on. You know, Gavin Kilkenny got a bit of time for the Irish under 21s. Harry Wilson obviously came on for both, in both games for Wales as well. So, yeah, just picking out a few of the names who, and Jefferson Lermer got a full 90 for Columbia and thankfully only had to go to France to get it as well because they played Algeria in France. So, good for him to get minutes without having to travel halfway across the world. Absolutely, and I think in this last international break across, you know, under 21s, under 19s, seniors, we've had 12 players called up, and that really does show how far the clubs come, hasn't it? You know, to have 12 players called up and get international recognition. Yeah, and I think also, you know, notably getting called up at international under 21 level as well shows a real emergence of some of the clubs' younger players. You know, they might not be featuring so much in the first team, some of the guys who mentioned they killed Kennes of this world and, you know, Mark Travers in the Irish full squad who's under 21 but in the senior squad. So, you know, there's some real good prospects for the future there and it shows what we've said before that the club is starting to produce players that eventually, you know, maybe a year or two, going to be knocking on the first team door regularly here and you have got to produce some of your own, you know, these days clubs are, with the amount of money they've got, can go out and buy anybody, but you do still need to produce your own. Not just because it keeps that connection with the fans as well, but because they've watched on the playing for the under 19s at Canford Park Arena, seeing them in the Premier League. You know, for the fan, that completes a bit of a journey as well. So, I think it's really encouraging, particularly the number of under 21s and under 19s who are getting international minutes. Absolutely. Well, next up here at Vitality Stadium is the Visit of Norwich City and Eddie Howesman speaking in this morning's press conference. Incredibly disappointing to see. I watched it live and, you know, it was a tough watch. Although you couldn't hear anything, you could sense what was happening at the stadium. I thought Callum and the rest of the England players acted with real class, dealing with the situation and then getting on and playing the game and played ever so well as well. So, you know, full credit to everyone involved with how they reacted, I think. Yeah, we did some good work. We were left with a small group, but we did some good work in terms of, I think we had a behind closed doors game, a little bit of team building in there. Players went and visited the academy during the week, which was really nice. So, we filled the time hopefully in a productive way for the club. Team news is we're pretty much as we were. Adam Smith's getting closer. David Brooks is still somewhere away, so is Dan Gosling, so is Junior Stanislaus. Well, they've approached the Premier League in a Norwich City fashion under Daniel, really. I think they've been really impressive, and I admire any team that has a clear philosophy and sticks with it, and they're a very attacking team. We know we're going to have to be very good in this game in all aspects of our play, because they're a dangerous team. Well, that was Eddie Howell speaking just this morning. Chris, if we rewind the clock to almost this time last year, we played Norwich here in the Carabao Cup, and they played very well. They didn't win, but there were certainly a lot of encouraging signs there weren't they? They were going well in the championship at the time. They weren't top, they were fourth. They were still making their way through the championship, but they were pretty unlucky not to get something from that game here. As you say, that was pretty much exactly a year ago. Next week, I think it was. So, they've started life in the Premier League as so many teams do. The championship to the Premier League is such a big jump. They're conceding goals for fun at the moment. They're the leakiest defence in the division. Obviously, you've got Pump 5-1 at home by Villa. In a game which actually, if you look at the stats, there were 43 shots in the game. Norwich had 21, Villa had 22 shots. Norwich's problem was half of them, I think, ended up in the stand rather than troubling the goalkeeper. So, yeah, they're finding their balance at the moment. We should say they've got quite a few injury problems. Key players like Alex Tetti has been out. Tim Crawley, the keeper's missed the last couple of games. Obviously, he's a very experienced Premier League campaigner. Onel Hernandez, who scored the goal here for them in the League Cup as well. Tom Tribal in midfield has been out. So, they have had quite a few injury issues, which, when you're making the transition, you want all your best players available, go to that saying. So, I think Daniel Ffark has been dealt a difficult hand. They haven't had the easiest fixtures either, saying all of that. They've beaten Man City. So, how do you judge Norwich and those in the first eight games? I think they'll have learnt a few lessons, that's for sure. They like to keep the ball. They like to play possession football. They like to counterattack. So, all the right ideas are there. But certainly, it's been a case of learning some harsh lessons to find their feet at the moment. And, as you say, they did beat Man City. They came very close to beating Chelsea, just losing 3-2. They got a win against Newcastle. So, they can get a result when they need to. It's something to be wary of, isn't it? I've got to say, their fans are absolutely fantastic. Even when Norwich slipped to League One, they still used to sell out Carrow Road. So, the home form for them is going to be massive this season. I think that's a cliché. Everyone would have said that at the start of the season. Away from home, struggling a bit. They've already scored one goal all season, saying that at Bournemouth, you know, lost at Burton. So, Premier League teams do struggle in the League Cup away from home sometimes. And that goal was at Liverpool on the opening day of the season. So, they haven't scored since the opening day of the season on the road, which is a worry for them. Obviously, Team Uppukki is right up there, like Colin Wilson in the scoring charts at the moment. Scored a couple for Finland. They're on the brink of making their first ever major tournament. Finland, if they can beat Lichtenstein next month. So, that will be a huge part of that, Team Uppukki. And he's settled into the Premier League life very easily. I think only De Bruynau and Aguero have contributed to more goals this season than him. So, he's one certainly to be aware of, plays up front on his own. They sort of play a 4-2-3-1 kind of formation. But it's interesting to see that there's talk that Tim Krull could be back tomorrow. He's been training, I think, this week. I think Alex Teti has been training as well, who's a sort of dynamic figure in the centre of their midfield, quite key. So, you know, they may well have one or two back, which should be big for them. But they're a waveform, certainly, a 5-5. I wouldn't be putting on Norwich tomorrow, given what they've shown away from home so far.  They've had problems with their goalkeeper situation. I think they're number one and number two, both being injured. They had the likes of Todd Countwell, who's been hugely influential for them, pulling out of international duty. So, for them, even seeing a couple of players coming back into the squad, it will be a huge confidence boost for them. Yeah, actually, when you look at the team that played here in the League Cup last season, quite a lot of them have been involved this season in the Premier League. So, you know, a lot of the players who got them up last season, there's obviously a big foreign contingent in there. Daniel Farkas bought quite a few players over from Germany with him, and quite a few of them who've been on the fringes have had to be playing recently. You know, they're missing Zimmerman at the bat, one of their centre halves as well, another key miss. So, yeah, they've got some promising young players. You think of Max Arans at the right back, who's been in England under 21, set up, he contributed an assist this week, played in the same team as Aaron Ramsdale and Lloyd Kelly as well. So, they'll know him well, Ben Godfrey at centre half. They've got some good young English talent as well, Paul Lewis left back, who's Northern Irish International as well. So, it's a nice mix of sort of homegrown young players, and then the foreign imports, if you like. But as soon as you take two or three quality players out of a team trying to get used to life in the Premier League and everything they've worked on over the summer, they've had some strange injuries. I think on Llanders fell down the stairs or something at home. Somebody else pulled a calf muscle doing something in pre-season, so they haven't had, you know, lady luck has not been shining on them. But we heard Chris Sutton, you know, on the BBC say it's Norwich Stryker, if Norwich carry on playing with their philosophy, they're going to go down. I think it's a bit early to say that, but every team, you know, full on with shipping goals for final last season, Bournemouth couldn't keep the goals out when they first came into the Premier League. It happens to everybody who has an attacking mentality. So, it is, you know, it's, we're only eight games in. It's very early to be judging them. What we would say is any team that can beat Man City and nearly beat Chelsea, as you say, beat Newcastle at home, you know, is worth taking seriously, absolutely. And then sticking with injuries. In terms of our injury news, Eddie said, you know, very much the same, but Adam Smith's been training and Chris Meppin's been training, despite pulling out of international duty last week. Yeah, and Adam Smith's been a key player, you know, versatility, you know, maybe hasn't always helped him personally because he hasn't really, he's been right back left back, right midfield, left midfield. He's played, you know, pretty much everywhere on the flanks as Adam Smith. So it'd be interesting to see when he's fully fit. I imagine he'll want to try and get him back in. But Jack Stacey's been playing well. Eddie's usually pretty loyal. He'll stick with players who've been doing well. So if Adam Smith is fit enough for this weekend or whether we have to wait for Watford next week, we'll wait and see. But, you know, he's got a battle on to get back in, that's for sure. You know, look at Lewis Cook. The fanfare about Lewis's, Lewis Cook's return played what was it, one game, two games, and then I can't find himself back in with that Lermer and Billing partnership difficult to dislodge. Jefferson Lermer was obviously on four yellow cards. So that, you know, could be a route back in, a predictable route back into the team for Lewis Cook. That's fair to say. Something's don't change, do you think? Well, Billing's on three. So, you know, no surprise to see that they're sort of suspensions pending possibly. But yeah, and Simon Francis, you know, has come back and can't get back in at the moment. And so you look a bit further down the line at the likes of David Brooks, who Eddie's not so keen to put a timescale on it, but getting closer is, you know, he's been saying that for a few weeks. Everyone's getting closer, I suppose, every day. You and I are getting closer, put it that way. So David Brooks to have him back again. Ryan Fraser can't get in the team, you know. So there's all of a sudden the bench is starting to look strong and when there are changes that need to be made, it looks like Eddie's going to start to have a few options pretty soon. Yeah, the bench certainly is starting to look strong and just before we go, it is that time, I'm going to ask you for your score prediction for this weekend. I'm pretty sure last time I was here, I obviously got it bang on. I can't even remember the last game I was here for, but I'm pretty sure I said it was going to be the right score. I fancy Bournemouth quite strongly this weekend. Home form actually has been a bit indifferent for the cherries, isn't it? I mean, great win over Everton, you know, let it slip against West Ham. Promoted teams don't have a good record here, full on the only promoted team to win here in the Premier League. Let's not remind us of that. Let's not remind us of that one. I'm going to go, I don't think no actual score. I'm going to go 3-0 Bournemouth. Wow, 3-0 Bournemouth. You heard it here first. If you are over 18 and you want to have a go at predicting the score for this weekend's game, please head over to Cherries Champions in partnership with Manchin Bets online. This week's winner will win two hospitality tickets to our next home game with Manchester United. That's all we've got time for today. If you are coming here tomorrow, we look forward to seeing you then. And if not, make sure you keep an eye across all of our social media and listen to BBC, Radio Solon and Chris on commentary for all the latest updates. Bye for now.


OK, Prime Minister Tudor, welcome to Kingston. I also want to say that I am a proud Canadian and a veteran of 36 years, although my question has absolutely nothing to do with being a veteran. I'm also a transgender advocate, and I may am extremely happy that C-16 looks like it's going to pass and through the Senate very shortly. But what I want to talk about is one of the after effects of C-16, and that is with Corrections Canada. Corrections Canada has in the past been doing what I consider torture of transgender inmates by placing, especially trans women, in a prison for men. Now, you would not, of course, consider the fact that a woman of any other type would be put in a men's prison, but trans women have been. This is especially of a concern when you read that one article out of Australia where a trans woman claims she has been raped over 2,000 times. Now, I'm not saying that happens in Canada. I strongly suspect a lot of trans women are put into segregation, which is not much better than solitary confinement. So my question to you is, will you do your best to ensure that trans women are put in prison or prison more appropriate to their gender identity? The answer is yes. I will ensure that. I mean, this is a great example of the value of having these community meetings like this in these town halls, because I'll admit I consider myself to be a fairly strong advocate for LGBTQ2 issues and fairly aware of all the different pressures. And this wasn't one that I had ever thought of. So thank you for bringing it forward. And I will make sure that we look at it and we address it. And we do right in recognizing that trans rights are human rights, that we need to make sure that we are defending everyone's dignity and rights in every way we can. And this is something that, if we're not already addressing it, and I hope we're starting to address it even without my leaning in on it, now we will take a closer look at it. Thank you.


Hey guys, I'm Ryan and this is Nick and we just saw Halloween. Yeah the 2018 reboot Would you call it? Well, I'd say it is the Halloween we deserve in 2018 it's the Batman begins of Halloween films This is the horror sequel you have to see this year But stay tuned you're about to find out what we thought of Halloween and why we are excited for this trick and treat So before we get into things make sure you hit that subscribe button and hit the notification bell And if you fill up to it get the like button, too. Let's get into it Halloween. What's it all about? Well, Halloween is a re-re-reboot of the franchise. So look Jason Blum, John Carpenter, Green, they've got together and they've given us a whole new chapter. Laurie Strode It's been 40 years since the original Halloween. It's actually the 40th anniversary this year and Strode is dealing with the trauma of the past. So Myers has been dormant for years It's been insane asylum, but he's about to get triggered and we're about to go on a murder spree unlike any other Ryan, how did you feel every second? You did plot depth like the best way possible? I'm excited even though I've seen it I'm excited about that. Look, I didn't know what to think about this movie going in You've got people like Danny McBride producing it. David Gordon Green is known for comedies If I feel like and then we've got this Halloween horror film you've got Jason Blum behind it producing which is always a good sign because I mean out of all the 30 films or so he's produced. I reckon it's a good 20 or so have actually been pretty good So there's got good things got good bad things going into it, but The second the movie starts from the get-go you are in just straight from the John Carpenter theme Launching you into the movie. Yeah, what's chilling in the beginning is you see the pumpkin rebuilding itself It's almost like the franchise saying we're back. We're rebuilding what we've done and every moment oozes tension Shout out to John Carpenter and his son Cody Carpenter for this awesome soundtrack Because one of the things that stands out for this movie along with a great performance by Jamie Curtis and Greer and others is the Incredible music every moment Your ears pick up just as your eyes do and that story becomes intertwined and just when you jump that music kicks in You know something's about to go. Yeah. Yeah, they used it so well I think too there were moments where you thought Something was gonna happen and it didn't happen They switched it at times and used it on different characters But I thought it was quite interesting and what you were saying before with the way they were showing the pumpkin Rebooting itself They had a lot of moments like that in the movie where they said almost a fuck you to the sequels that came after it They made comments about Laurie being Michael Myers's sister The sequels that came after it the new The doctor whatever his original name was Loomis that's right new Loomis and yeah stuff like that Which I thought was kind of fun and an interesting way to reboot a film Yeah, and what's interesting is even though it is definitely return of the original slag to genre to its prominence It still has humor and still has drama. It walks that fine line in there And the thing about is it's a story of trauma. It's a story of the final girls It's a celebration of horror and some return to quality for a franchise that has needed this for so long Yeah, it's a slasher genre. I think has needed resurgence for quite a while I remember that Nymer and Elm Street the remake tried to do it And then the Friday the 13th remake and try to do it and they all kind of failed This does it very successfully. I feel like if at first you don't succeed try Try again. Yeah. God damn. Is it a happy Halloween because I've got to say From even when it does end you want more of it hundred cent. Yeah, it's Blum has signaled this isn't the last we'll see of Halloween Hopefully and I truly hope it isn't because I feel that we have just began to scratch the surface The surface of the potential to Halloween franchise again something that I feel like reboots do Quite well is the first film when they reboot it But then it's the sequel that kind of throws it back into that realm of shitty sequels again So look, I'm hoping that this next film if it does happen We've gotten hints from Jason Blum if it does happen I really hope that they stick to this formula They've built with this film and don't just throw Michael Myers into another you know college I got a stabby stabby some babysitters again. We've built a storyline a backstory for these characters. Let's continue that in an era of huge budgets massive blockbusters Halloween's the film you've got Gantus in the most take some of us and seeing the original in fact if you haven't seen the original all the better because To be introduced this way might very well make you a fan of the franchise hundred percent But look if you have you have the chance I recommend watching the first one before you see this film There is a lot of callbacks and you will get a nice little nudge in the neck when things happen in the movie And you will get it one of those people in the audience don't mention the next and the stuff done tonight was just this The one of my favorite things about this movie is that it is Absolutely gut wrenchingly brutal. Oh, there are so many great deaths that this is gonna line up People are gonna make this the line where people need to cross and do better after this film comes out So look if you want B our oodle horror if you want some fun If you want an entertaining film that walks the line between multiple genres catch Halloween under 25th of October I'm Ryan. I'm Nick and this has been moving on So make sure you hit that subscribe button below And if you've seen Halloween or seen a horror movie that we should check out comment below Let us know and until next time. We'll catch you nerds at the moonies