So, my name is Willem. I'm going to speak to you today about one of the use cases we had at Gojek. Basically, it's a story about clearing tech debt that we had. So, I lead the data science platform team at Gojek, which is why this is one of the things that we had to focus on. So, before we get started, I wanted to give an introduction on Gojek because I don't think everybody here is acquainted with the company. So, what is Gojek? Gojek is an Indonesian technology startup. We have a variety of products and services, but we're most famous for ride-hailing on motorcycles. That's one of the reasons why ride-hailing is so popular in Indonesia. It's actually called an Ojek or it's a motorcycle taxi. It's because of the traffic congestion there. So, we launched our first product, which was a ride-hailing product in Indonesia. And since then, we've gone on to launch many other products in Indonesia. So, our aim is to have basically a solution for every workday problem that you have, whether that's taking you to your office in the morning, buying lunch, buying groceries for you. We aim to have in, I think about 15 verticals we have at least one specific product. So, it's lifestyle, payments, ride-hailing, logistics. We have about 18 products in total at the moment and we are classified as a unicorn. So, just a bit about our kind of scale and reach. At the moment, we are in three and a half countries. I'm not sure about Singapore. It's kind of in beta. We should be launching fully soon. If you just look at Indonesian statistics, we have, on an average day, hundreds of drivers online at one time. We do more than 100 million bookings a month. We have, that's one product. And one other product is food delivery. We have 250,000 merchants, making us one of, if not the biggest, food delivery service in Southeast Asia. And yeah, so, other products we have that are also pretty big are payments. So, GoPay is one of the leading e-wallets in Southeast Asia as well. So, the scale is quite big. There are not many other companies that are both digital and offline because we pride ourselves on being also an offline company that are operating at the scale. So, what does my team do? The data science platform team at Gojik focuses on getting ML into production, reducing the time to market for ML. Okay. So, reducing time to market for ML, productionizing ML models with data scientists and then building a generalized platform and tooling for the data scientists so that they can focus more on ML and DS instead of the nitty-gritty engineering details. And if you're just starting, you're starting from scratch, then that's often the problem that you have is you need to solve engineering problems. So, this slide actually captures that quite well. It's a common industry problem where you can easily just clone a repository you find on GitHub and get some model up and running locally, but getting that into production can take months. And this is kind of a dirty secret, but it's the reality because you're integrating with, in our case, product teams. So, just a quick outline of what we're going to discuss. We're going to discuss one of our first models that we made at Gojik, which is our driver allocation model and some of the initial tech debt we built up building that and then how we ended up clearing it and how ML flow fits into that clearing of the tech debt or that solution. And then I'll quickly just demo ML flow for you and some of the functionality. So, the driver allocation model is actually quite simple. I think if anybody is right-handing like grab over Gojik, you've probably triggered an algorithm or a decision like this, which is which driver do we send to a customer that wants to go from his location to his destination. And this is not always an easy decision because it depends on what you're optimizing for. You can optimize for business objectives or customer experience or driver experience. There are many different objectives that you can optimize for. So, this is a very important model as well because of the amount of money that's at stake here. It's at Gojik scale, it's a lot of money. So, small improvements in the model have a very big impact on bottom line of the company. So, when we started the team, this is one of the first models that we had to build. And of course, if you're starting with no infrastructure, nothing really existing and you have to get something into production, you need to deploy and provision everything. So, we had a lot of decisions to make, which, how are we going to go from raw data into production serving these models? So, we could do this manually, of course, but for a production system, we actually want to do this in an automated way. So, we looked at the machine learning lifecycle basically and identified the stages that we need to solve. The first one we looked at was data processing. So, for that at the time, Airflow was basically the primary solution, the most commonly used solution. And I think even today, it's still probably in first place in the open source world. It has a lot of gotchas, which is out of scope for this presentation, but we started with that for data processing and I think we're still happy with that for data processing. So, in Airflow, basically, what you have on the screen here is a DAG and that's at the top, you define a basically a graph and then you define specific steps within Python code and that that graph gets executed and it executes a series of steps in which you can transform data and typically Airflow runs based on some upstream event or a timer. So, either a partition is created in another database or you have like a daily job that runs and then the task is all complete and the graph completes. So, we built that, we implemented that and we needed to do that to get raw data refined. And then secondly, we were looking at how are we going to do our deployments but and then we saw, well, the company's using GitLab so we can just use GitLab CI which allows us to get into production to deploy our models and we can just define a YAML of all the steps that we want to execute to build our model serving application and deploy it into production, which in our case is Kubernetes. Then we were left with the question of how do we train our models and at the time we didn't really think that it was worth implementing a custom ML solution to train our models. So, we looked at our existing stack and we said, well, why don't we just use Airflow as one continuous process and we process our data there, we train our models there and we connect that up to our deployment. So, basically how this works is you have raw data and at some schedule you have your Airflow DAG which is a process, a long-running process being triggered by that scheduler. When it's done processing data, which is your features, done producing the model and your evaluation, metadata and all of that statistics, it sends an API call to GitLab and then GitLab builds the model serving application, it knows, based on that trigger, what model was created and it deploys all of that with the binary and the docker image and the configuration for that model into prod. So, this is basically a long-running push-based system. So, each step requires the previous step to notify it what's coming from upstream, which is important for how things can change later. But what we realized was that we'd actually built a very monolithic, tightly coupled process because you'd have a single instance of that timer trigger happening at the start and then all the way through to production, you have not models producing all of your metadata and all of these things, building, serving, deploying, serving and all the testing, everything is one, essentially long-running process. So, at first this was great because our driver allocation system could get into production, but soon the cracks began to show once you wanted to experiment and change this. So, the first problem we had was it's quite inefficient. So, we have a long-running process like this and there's a lot of complexity in this process. When you need to make changes, you need to wait a lot of time for each pipeline to run. Secondly, the steps are dependent on each other. So, your deployment con, your model con, so you're serving con deploy without the pipeline triggering it from Airflow and Airflow con can get into production without GitLab being triggered. So, those two are very tightly coupled. The second problem we had was that it was extremely hard to experiment because we had still long-running pipeline. So, either two options you have is you can either fork the pipelines if you want different variants. They say you want to add a feature, remove a feature, try a new model type, but forking it would then require multiple pipelines to trigger a single serving application. But if they're running at the same time, which one gets precedence? So, basically you need to hard-code into your deployment which model should be the one being served. So, you have all these complexities that arrive because of this tight coupling. Alternative is to fan in and fan out within a single pipeline, but then you need to have feature flags and you're serving as well. So, we knew that experimentation was a problem. And the second problem, the third problem we had was versioning. So, the problem with versioning is if your Airflow ETL is producing everything that's going into production, you're versioning based on time, not on instances. But it's actually very unnatural to version a model on time. And what ends up happening is everything is based on the timestamp of the original Airflow. So, what happens is that you're eventually versioning your model serving application and even your user clicks with that timestamp of that model was created. But actually that is something that is out of scope or irrelevant to serving. It should be something that you can find out if you had to dig in deeper later. The fourth problem was reproducibility because our pipelines had side inputs. So, they make API calls or pull data from GCS or do all kinds of strange things that even we didn't know about because we allow the data scientists and we think the data scientists should be the ones offering these pipelines. So, we wanted all the platform on which it runs. But if they're allowed to pull in data from the outside then we can't have deterministic runs and there's no reproducibility in the run. There's also no standardized way of tracking which artifacts are produced. So, each project has its own way of tracking these artifacts that are produced from these pipelines. But again, in the driver allocation case, this was also a problem. The visibility was also one of the problems. So, when you're using GitLab, this is a specific example of what we use GitLab. But this is often the case in CI tools that are not meant for observability and evaluation of models. So, in GitLab you can see that you have some idea of, for example, the Git tag, the SHA and who commit, who commit. But there's no indication of model matrix or parameters or how do you compare specific runs or deployments with each other. And then the scalability was also a problem. So, one of the issues was that GitLab, because it's push-based, depends on you defining all the destinations where you want to deploy all of your models. So, if you're scaling out to many regions, let's say we want to scale out to Singapore, Thailand, Vietnam, just from Indonesia, and you have all of these destinations like your tests, production, staging, all of these variants or model types, depending on your objectives, your deployment pipeline can get extremely complicated. And the final problem we had was there's no separation of roles. So, data scientists have to work, all the data engineers, software engineers and data scientists have to work up and down the stack. So, you'd have data scientists debugging issues in production where they make a change with the start of the pipeline and you had software engineers fixing a code that's producing data or artifacts. So, we knew that this was unnatural and we didn't know how we could limit this effect. So, we sat down and we figured out what is our desired state. The first thing we wanted was to simplify experimentation and make it a bit easier. So, that can be getting something into production that's new, but it's also something as simple as just changing some features or trying a new library for producing a model. We wanted to be able to easily reproduce results. So, we wanted to have deterministic, a deterministic way to version pipelines or code that runs specific process, whether it's deployments, training, ETLs, and based on the inputs and the code that runs, have a reproducible way to run that again and get the same results. And then, we wanted to make deployments easy. So, deployments should be something that's an afterthought. And, evaluation was the other thing we really wanted, some way to have an easy overview of both model training parameters and metrics as well as information about how the models and features are performing in production on real traffic. And then, especially for an engineer, the most important for me was how to be scaled as to hundreds or thousands of models because when you're stuck in using tools that force you to take a certain design pattern, then it can restrict you. And in our case, it was restricting us. So, we knew we needed to think about what we wanted to implement to scale this out properly. So, we evaluated some tools and one of the ones that we're looking at was MLflow. So, on the screen is just a simple ML workflow and it basically starts with raw data and this can be your data lake, it can be an event stream.  So, one of the things is that you typically want to experiment with a lot of technologies and this is unique in even though software engineering and ML are very similar, this is one area where they're kind of different in that you want to experiment with all kinds of different ways to process data, ways to prepare data like feature engineering, especially on training, you want to try different models and in deployments often you want to try different serving depending on the actual application. So, you might have a specific model being deployed another one going into TensorFlow serving. So, the next one that, so this is one of the basically the goals of what MLflow is setting out to solve. So, the first was how can we allow all kinds of different tools throughout this MLflow lifecycle. The second thing that it sets out to do is to allow you to tune. So, at steps like data prep training and well potentially even deployments, you want to change parameters and experiment and see what the outcome of that is. Another thing you want to do is scale. So, each stage there's actually scaling required, scaling your data lake, scaling data processing, handling very high load if it's a Kubernetes cluster or some cloud service that's serving your model. And then model exchange, basically how do you transfer or interface between the stages when you have a model. That's not always easy because it could be different teams between different stages and if there's not a common format for model exchange then that's something that needs to be defined or it's going to lead to a lot of inefficiencies. And then governance. So, how do you know that your process is following the regulations that you've set out and how do you have way to evaluate and observe what's going on in your system which is often neglected when you're just starting from scratch and you're just trying to get into production. So, MLflow sets out to solve these aspects and it tries to do it in a way that you're basically unable to use any kinds of tools, basically bring your own data processing, training, deployment, tools, and libraries. And on this platform you can run everything from training to serving. So, it actually consists out of three components. And just to add to this important point is that the components allow you to do that both in local development and in production. So, there's just a thin wrapper or execution difference when you actually take this into production and run your own production code, but otherwise it's identical. So, the three components are tracking projects and models. So, tracking will get to that a little bit later. So, with tracking you can record and query your experimental runs. So, everything from the code, the data, the config, the parameters, the metrics, artifacts that are produced during your training runs. You can actually, because it's a very generic service, you can track any kinds of instances of a product. It doesn't have to be ML, but in this case it's kind of built for that use case. And the second one is projects which allows you to wrap any code basically in a repository with a specific format for you to reproducibly run those or that code. And models is a generalized way for you to deploy models that are produced using MLflow and or actually just using a standard interface using multiple deployment tools. So, the one we were most interested in was tracking because of its ability to track what we are training, the models themselves and metadata about that. And it will allow us to decouple a lot of our processes. This is Dave. Long press. Okay. So, the key concepts in tracking are parameters, metrics, artifacts, and your source. So, the parameters, typically what you want to track are the two most important things for reproducibility are the parameters and the source because with those you can reproduce based on the source code of your training run and the inputs, what resulted in, what are the outputs of that specific process. So, in this case, the outputs would be the artifacts that are produced. So, that can be your models or data or any metadata or even images if you're looking at feature importance. And then importantly the metrics because that's what the data science is ultimately used to measure how well their models are performing. So, this is an example of the MLflow UI. So, you can see that on the left hand side you have some experiments that you've defined and for each experiment you will produce experimental runs and they're all timestamped and you'll see that there's also even a version and this version is the commit where you executed the code on and then you'll have parameters which were inputs run and then metrics between the outputs. So, this is the base of what you'd see in MLflow but I'll have a quick demo about this a little bit later. But for our use case this seems to be quite suited. So, we went back to the drawing board and we looked at what we built and we knew that we needed to make some changes. The main problem with this what we had built was there was a monolithic long running and there was no real separation of concerns. So, we broke the process up into three stages. Processing of data, training of models and then deployments. So, what we also did is we removed GetLap from this whole workflow of ours because it requires push basically API calls it's basically a push-based system and it's good at CI but it's not really good at continuous delivery and what we wanted for scaling something to allow us to continuously do the the training of models we actually implemented systems, a software and we basically in this case it's not really important what those models end up in. So, for example, data scientists can use Jupiter to train a model as long as he follows the right approach and the structure and libraries and his models end up in the right place which would be a model store stream. We also have other ways to schedule those jobs and there are some subtle differences in why we implemented this instead of using airflow which I'll touch on later. And the second thing we did was implement a CD tool. So, basically the change here is I think a pull-based system that has an idea of what is upstream to it instead of one that requires that information to be passed downstream because if it needs to be passed downstream then your ML pipeline needs to push that information downstream. So, for example, if you want to deploy to a specific region then that model needs to know that it has to go to a region but that's not really the focus of the model it shouldn't actually care about that. So, now that we have these ways of executing the stages of processing data which is creating your features, training your models and employing, we needed ways in which you can asynchronously change those processes without affecting downstream processes. So, we needed a way to store artifacts between the stages. The first way we did that was with a feature store which is actually open source. It's a way for us to store data that's processed both batch and streaming in a single place. So, it's polyglot storage. It's used for training and serving actually but in this case we're just focusing on the training aspect. So, now somebody can actually focus on processing data without affecting the training downstream. And then finally where MLflow fits into the picture importantly is between training and deployment. So, with MLflow here you can train models and as a data scientist you only need to focus on getting your model into MLflow. And with the right parameters that decouples you from the whole training, the whole deployment aspect of this workflow. So, this is ultimately what we're ending up with. So, some of the advantages to taking this approach. So, one of the big advantages is that ETLs in airflow are fundamentally different to model training in whatever training system you're using because model training is instance-based where ETLs are generally time-based. They're waiting for a partition to exist or some day to roll over before they process the data. But for model training you just want to be able to, in some cases it's on a schedule but often you don't want to wait for something to happen.  So, if you're using MLflow then to log your matrix from that training and you want to basically train your model, store the matrix in MLflow and have that available to deployment. This is basically the change you make. You're just wrapping the existing training code in some, in the MLflow Start Run method and then you're logging the parameters in that context and the matrix in that context and then finally your model out. And that gets stored in the MLflow tracking server which is what we use. And that's all you do. You don't have to have your code deploy the model or trigger anything down the stream. And then finally because our deployment system is then artifact-based it's looking at MLflow, it's looking at configuration, it's looking at Docker registries, it's looking at Helm charts, it's looking at all the components it needs in a pool-based way for their availability. The second advantage is reproducibility. So, when you're with feature store MLflow, storing the versions of the code and the inputs that went into them, then you can easily trace back from deployment through a backup stream to what produced either the models or the features that went into production. And typically this is how it would work. You'd make a change, get your model into MLflow, get into production and then see some result. And then from there you would go backwards and see what change you made in training, what change you made in processing the data or upstream format. So, we now have instead of just prod serving and raw data, we have two basically checkpoints which decouples the processes from each other and allows us to, because of the versioning, track and reproduce results. So, this is one of my favorite slides and it's quite important because it really shows why MLflow is useful in our specific example. So, on the left is just the previous slide that I showed you, but what the typical experience would be like or the flow would be like is that the feature store would be used to train the model, but all the data, the config that's used to train the model will be logged in MLflow. And then you deploy from your CD system into production. And so in production you just have the idea of the model. Then when you get metrics from real traffic, you can actually log those metrics back into MLflow. So, you don't just have the training metrics, you also have the, for example, the objective that you are targeting which could be conversion rates, click through rates or something, you're also logging that back to the model. But you can even take that further because the feature store and upstream processes that produce inputs that go into that model might also benefit from that. So, you can even log the feature performance back into the feature store. So, now if there's a data engineer producing features, he can see, well, the features of this specific section are being used in many models and have a high importance in reaching our objectives. And then role separation. So, role separation was one of the other things that was really quite powerful in making this distinction. Once you have a decoupling which is essentially an interface between these different processes, now you can have that engineers focusing on producing the features that go into the feature store. You can have data scientists only focus on the model training and getting models into MLflow. And the software engineers can basically, especially for me, protect the production environment from everybody upstream. And, yeah, so my goal is to just basically squeeze the data scientists there until they can basically just sit in a Jupiter notebook and they're very safe there. So, the last one was about scalability. And for our driver education system going multi-region was one of the most important things. And if you look at what we wanted to achieve, just with a specific system, it's multiple environments, multiple markets, many different model types because it's multi-objective, and you want to have many experiments. So, you can look at hundreds of simultaneous deployments. And we knew it couldn't be push-based. But if you have a CD system that's pool-based and the CD system is the one that knows which inputs or materials or artifacts should be in the pipeline. And it knows about all the destinations that it needs to deploy. Reversing that process from push-to-pull unlocks scalability for us. Because now we can define all the markets in our configuration. And as soon as there's a new model, the C-part won't be alerted of the change in MLflow. And it'll roll that change out automatically. So, I'm going to show you just a quick demo of MLflow. You guys can see this. It's a simple example. So, what we're doing here, this is one of the basic MLflow examples. I want to illustrate to you. Let me just show you MLflow. So, we could see demos here. And we're going to train a basic model. And then I want to show you the difference in what you can see in terms of parameters and metrics. So, in this example, you're basically going to read in some wine quality data, just a CSV, and then train in the scikit-learn model, and then produce that as an artifact to your tracking server. So, you can just run this code. And basically, the only difference between normally doing this and with MLflow is that you're wrapping, you can see at the top there, your code with just MLflow's.run, and then you have your logging at the back. The rest of the code is identical. So, we can produce a couple of these runs with different parameters. And they should show up here. Okay. So, now as a data scientist, you can have a look at this and compare the results of these different runs. So, let me give you just a quick glimpse of what these columns mean. So, one of the cool things that MLflow does is it actually looks at the git version of the code that ran. So, you can see that that version column actually tells you the git shot.  So, that's A01D, which is the same one that's over here.  And as a data scientist, you can then have a look at them and evaluate the ones that you want to. So, I'm just going to quickly show you how the SIGI process would look like. So, if you're looking at Go CD or some, it's actually relevant which CD system you use, as long as it allows you to have a pull-based declaration of your inputs. So, you can use Finnaker, you can use Go CD or concourse. There are many tools that like to do this. In this case, we're looking at two things. Basically, the MLflow tracking server as well as Git. So, Git will be our configuration store. Now, we did some new runs here, some three new runs. And if you go into MLflow and one of these runs, you'll see that it has a promoted flag here or a tag. This indicates to the CD system that you want to deploy this into production. Of course, this promotion can be a manual step. It's typically not done from the actual training. Also, when you're in this detailed page of experimental run, you also get these statistics of a specific run. So, you can look at the parameters, the metrics, so the inputs and outputs, as well as the artifacts that are produced. So, you can drill into the model that is logged. So, that model is the one that's being produced over here by the log model method. And you can have a look at the inputs that require the packages and dependencies. And basically, any arbitrary metric can be stored here, sorry, any arbitrary artifact. And it also allows you to backup this or to retain the state in an object store like S3 or Google Cloud Storage or anything equivalent to that. But now, what's important about this is that from an engineering perspective, because I care more about this than I do about the metrics, that's up to the data scientist, is how can I get that deployed? So, if everything went well, we should have a new model deployed here. So, basically, if you have a look here, there's a seventh one that's being triggered. And a trigger is based on a new version being available upstream in MLflow. And MLflow allows you to do this because it has an API where you can just hook up to, where you can just hook up to and just scan for new versions with that promote attack. And then you can even click on this link and go back to MLflow where it's working and see what was actually deployed and draw into your specific new model. So, this information can be passed into serving as well. So, you can put the specific run ID or experiment as the ID that's deployed in production and then you can tag users with that or your user segments and then evaluate the model in a loop. So, to just give you a little bit more detail on what the UI can do, I'm going to show you some data that I've partially cleaned of anything important, but that is actual data. You can actually, you can log any number of columns or parameters. In this case, we have a lot of parameters that are being logged for our driver allocation model. And the data scientists are looking at a lot of metrics here as well. So, in this case, this is not really that much time that's passed, that has elapsed, but you already have 276 runs. So, if you didn't have something like MLflow to evaluate this with, it would be very, very difficult. You need to go manually and or with some SQL database or something or some BI tool and go and evaluate this. So, it's very convenient to have MLflow here, AUC or whatever metric you're interested in, and then you can draw into that specific run. Let's say you want to see what this model, what's important about this model. And one of the artifacts that we typically log is the feature importance. So, in this case, I've blurred it out because I don't want to show you guys what features we're looking at, but clearly there's one feature that's really important. Sorry, great people. So, this is very useful because the training pipeline has this information and both the data scientists are very interested in this. But the point is that you can log in any kind of arbitrary file and the UI allows you to expose that. And another interesting function is that you can actually select multiple runs and compare them. You can do this video comparison between runs. So, this is another way to compare specific instances of a run which you have pre-filtered out. You get some horizontal scaling here and you can look at that. But one of the cool things is that you actually get a graph at the bottom here that allows you to look at, in a visual way, the differences between them, between the specific runs. So, being a human, it's easier to see these hot liars and then you can draw into them and see what make these specific runs unique. And you're able to do this for all kinds of metrics and parameters. You can make changes as you see fit and you can zoom in and look at specific areas. So, the functionality here, it may seem not very different from what you can get with the BI tool, but it's about the decisions they made in the API that they've built. Something that's useful for ML, but generalizable. It doesn't restrict the way that you operate. So, it allows you to use the system both for offline development when you're just testing things out and for production systems. Yeah, so I think that's it from my side. Yeah. Thank you. So, questions, please. I will come up to you and I'll pass the mic if you have questions. Okay, so I'll go. Every time I'll say more questions recent and whoever raises thirsty drink up. I really am excited you are protecting production from our stream. I'm still a bit unclear though on why you're preferring a full-based model for the push-based model. I think you briefly stated that it's probably because you guys have feature clients that you want to serve to specific countries and something we're not going to serve them. But wouldn't that be probably how you buy configuration or something? Right. But let's say you've got configuration or other inputs to your deployment. Then you would need some way, some other store where you basically change your push-based system into a full-based one by having coded the start of your push-based system that goes and asks those stores what's the latest version of what you need, what I need to deploy. Do you know what I mean? There's a slight distinction there as opposed to it polling and knowing that what are the latest versions available. So are you also in effect like using a helifluor as one type of photography service which is actually provided? Yes, yes, yes. So the parameter storage is actually one of the things that makes it valuable as well. I mean as an engineer that's the part that's valuable to me. It's the interface between the two layers. More so than the matrix and artifact storage which we can solve ourselves. But in this case it solves it well for us. But yeah, I hope that answers your question. Storage is definitely not a problem. It's the least of our problems. Will you once more ask because we captured the audio for the recording through your this one. Yes, so then we keep that question from the word this case. Thanks William. My question is that for purposes of deployment can you logically define a setup model? Often you don't deal with one model, right? You probably deal with a group of models for a particular use case. Now, can you logically define a group of models and version them as a unit and then that you are doing at a model level. Instead of doing a model level, you do it as a group of models that are just abstracted out. So basically the mapping is a point to a deployment unit. Right, so the question is can you version groups of let's say artifacts but in this case models together just as you would at the model level or the atomic level. So yes, you can do that. So you can just well in our case there are ways around that. I guess there are better ways but we basically just sum up all the versions from all of the components all the inputs and we produce a unique version from that's basically a aggregate version and then we use that to track the group together. But you should look at the specific attribute that is unchanging or that's relevant. For example, if you have multiple runs producing a model with different timestamps but it's an identical run then that should be an identical downstream version. So I guess that's dependent on what you're actually producing. In our case it hasn't been a requirement actually because the model is the atomic thing that we care about for the future. If you want to ask more questions then feel free to raise up your hands and I'll wrap up my question here please. Right, so your question is which other tools did we identify for model tracking? Right, so the other tool we were looking at was model DB. I'm not sure if anybody has seen that. So I think that the biggest difference for us there was that it was focused a lot more on the evaluation side of it because the engineers are making the decision in this case. So we felt like MLflow had better and more generalizable decisions that they had made to support many technologies and also the API that MLflow has is very lightweight compared to what model DB has. So we opted for MLflow because of its API even though it is lighter weight on the UI side and the ability to compare between experimental runs. That's the only one in that case where I was talking about different technologies. I think I was also referring to non-tracking. So for training and for CD, there's a lot of tools out there. Okay, keep on raising hands. I'll come up with the mic and I have one more question here in the back. Right, so we're not using projects and models because, wait, is that your question? Okay, have you looked at model chip? Model chip? I have not looked at model chip. I mean I was just asking, are there any more features other than traction that you're using because I understand you're serving there is also very possible rather than making MLflow in point to whatever. In this case, it was primarily the tracking that we were using as well as the ability to easily log out the artifacts because it also abstracts the way for us and the API is very sane and lightweight. Okay, one question is, I think you were just coming to us. My question is about role separation. You mentioned that this is one of the advantages. So I'm just curious about like, is it the responsibility of data scientists to introduce ML logic into their roles or software engineering? No, so that should be the data scientists. So the data scientists need to be semi-software engineers in our case as well. That's what somewhat is opposite from what you mentioned that it clearly separates data scientists and software engineers. So now it means that they have to also introduce the software engineering system. Well, it depends on what you define as software engineering because they're running code, right, even if it's just some fine code. So it depends on what level of abstraction you're working at. I think the difference is that they would be working at a higher level of abstraction and the software would be at lower levels. And I wouldn't want the data scientists to be deploying Kubernetes clusters and things. So I'll not anymore. Okay, keep on raising your hands. Okay, you catch your next one. Okay, this gentleman. I have a couple of questions on the serving side, right? First is, does the department need any money on services today? So you have so many developments happening. Does it require the services to go on or does it require downloading on services side? So Kubernetes does our deployment, our current deployments require downtime during deployment. It depends on the service in some cases. So most of the time we're using loading updates with Kubernetes is not an issue if it's state for service. If it's a stateful service like a database that's loading a lot of data into memory. So we have, for example, food recommendations that takes a lot of data into greatest memory that can sometimes have a blip where it's down for a few seconds. Yeah, is that the question? Or does it answer your question? That's part of the question. The question would be, you know, something complicated or close. So for example, before you train a model, you have a complex feature engine that you've done, right? And you have to repeat the same feature engine in the serving side as well, right? Because so how do you handle the sort of complex feature engine on the sort of serving side?  So we have some details on how your structure, your sort of lean infrastructure can be able to do these questions. So your question is how do you account for feature engineering on the serving side to not impact your latencies and your performance? Because that's a requirement. I think that's one of the biggest problems that we all face. And we still haven't solved it. But one of the ways that we're doing that is by moving as much of the data processing upstream. So both data batch transformations as well as how you can have this. Let me just show you the slide. So if you can do all of the data processing upstream using Airflow or some other system, you can also use Flink or Bean or something for your real time and then store it in real time stores and then pre-compute as much as you can because then you can you only need to do minimal feature transformations in your application itself. And so that's what we were trying to get at is not to have any feature transformations except some basic data structure manipulation in serving. Sure. In some other cases, I think scientists prefer to look by governments in the past, you know, historic data, pre-offensive features and so on and so on, right? I think those are the cases where it's difficult for us to be able to do business on their own, which is responsible enough and also in the form of the JVM, so they don't have a hate on the performance question. Right. So yeah, so I don't know if that was a question, but just to present another alternative is a tool called ML Leap, which allows you to take pipelines, I believe it's scikit-learn and run that on the JVM. So there are some other tools that are trying to solve this problem. Okay, keep on raising hands. I know who is the next one. Okay. So, sir. So my question is how frequently do you deploy as a product in all those models? And when there is a new data generated through the source site, what is the feedback cycle? What is the latency in your feedback in your model? There is some point of the new cookies or the new traffic metrics generated from the source site. How frequently do these feedback will go to the products? So the first question is how frequently deployed to production and the second question is how frequently do changes in our product reflect in our models? Is that your question? Yes. I'm not sure how much detail I can give on this. We deploy many, many models every day. So at the very least daily, some models will deploy once every three months if it depends on the actual model and what it's trying to do. I don't think we deploy hourly for some projects. I don't think we ever get to that scale because normally the analysis takes a long time and data scientists don't want to have too many models to slow it down there. In terms of how quickly changes get applied, changes get applied a lot through the feature store, not so much through the models itself. That process is slower. So I think on a daily basis, at the fastest, but in aggregate, many models do. Okay, keep on raising hands. I know there's a bit of sugar still in you, so we'll talk about it. I can do it. So sir, here's the next one. We'll get to the next one. Thank you for a great content, William. A few questions. One is, is it easy to interface annual flow with FLO? For example, FLO runs daily and the daily batch is done, then you want annual flow to kick-start the training process. Second, does annual flow allow for control? For example, if I'm tracking on several metrics, if the metrics don't fall, it will set the threshold, deploy automatically, and maybe all the metrics fall, it will set the threshold, if there's a problem with the artifact, send me a notification, can you take care of things like that? No, I don't. So your first question was, should we repeat that first one? How would you interface? How would you interface FLO? To annual flow? How do you do that? We haven't actually tried to integrate an ETL process that's time-based with MLflow, but I don't think there'll be multi-different than interfacing an ML system with it. And secondly, in terms of how can we have alerting or basically hooks based on conditions in MLflow, I'm not aware of any functionality like that, but I also don't think that they should have that functionality there. I think that should be something that's left outside of MLflow. API that they have built is easy to interface with. So I would rather have MLflow as the central point to communicate with an external service, maybe. Thank you, colleagues. Keep those raised in your hands if you're passing the mics. Yeah, so thanks for sharing. So actually, I'm quite interested in how another can help in the deployment part, because tracking the 20 models is very useful, but at the same time, when we deploy the models, what kind of model version and how is it important for an issue for us? So I think I believe you're a resource of some problem, but I didn't see the size of the issue of someone because of us. Well, we can just have a question as hard as MLflow help with a versioning and solving that. Well, this is a closer look. Like once we're going, you will need to collect it at comp and then trigger the training. So it's like a close look for updating the model. So I don't even want to do this. Right. So connecting that up to training is a different matter. So your question is how do we collect feedback from specific deploys and basically evaluate that and like monitor the performance of the current model. So I didn't see how you do it with MLflow. Right. So in this case, it was just a toy example. But the versioning question, how do we do the versioning or how does it help? If you have a CD system and you have basically combined inputs, let me see if I can find an example of you. You have an instance, right, that's produced, a specific pipeline, specific instance, and the instance is a combination of specific inputs, materials. And that's can be a round charge, your Docker image, your MLflow model. You can have 20 models here actually, right? Because if you're building this pipeline, you're saying, this is my driver allocation system. It requires these six models for six objectives. And in the combination of them, the unique combination produces an instance and that instance goes into production. And then when you serve something to a user, you also tag that specific order, for example, or the user's session with that instance. Then you have a stream that captures all of these users. That is custom integration because it's, you're hooking up into a product team with specific backing, right? So there's nothing that I can tell you there to make that easier for you, unfortunately. But you can log that back and you can store that somewhere. You can even store it in the feature store if you want to use that somehow. But typically we don't do that. We just store it in BigQuery or in some place to analyze that. We don't generally use that in a closed loop, but we're thinking of just hooking that straight up into MLflow as a metric. And then you have everything in line with that model, right? So you can just compare the matrix of training and compare the matrix of production. How are you feeling? I'm fine. Okay, cool. So keep on raising questions. You have online and offline learning. So when you're doing online learning, can you have a change data capture such that you can capture the changes automatically and redeploy the model? Is it possible in MLflow? So, sorry, just repeat that question online. So there's a few types of learning. One is online. Right. So suppose I want to capture the change in the data automatically and redeploy the model here. So is it possible in MLflow? No, MLflow does not allow you to detect a change in data and redeploy the model. So the key thing here is that you don't want to use MLflow to deploy the model at all. MLflow doesn't need to retain the information. It's just the store, the artifact store, essentially. The downstream services use that as a repository to find the information and then take action because that's their role, right? It's not MLflow's intention to do that. So you need to have a CDC in between to capture the changes and... Right. Right. That's right. Also, just to read, Gorgia uses the tracking segment of MLflow extensively. There are two more, which is the projects and the models. So Proplex Micro help. We're not really covering it today. So I think please raise your hand. Who is the person who wanted to ask a question in this corner? Okay. Any more questions? Anyone? Yeah. Any more questions? Any four more questions from the same attendees? Please feel free. William is hopefully a very open guest to our meetup. Maybe next time when? Maybe, I don't know. This year, hopefully... It depends on the stickers and the pens. Yes. The stickers and the pens are over there. Okay. One more question. Yes. A question that MLflow works with certain subset of models where the parameters can be calculated in a concise manner. So for example, if you're using a deep learning model where the parameter is the architecture of the neural network. So how do you use it in MLflow? So your question is how do you use MLflow with models that have metrics that are not easily capturable by MLflow? Or what type of... Maybe to articulate what type of information you want to be able to evaluate on that model? And then I can tell you if MLflow can do that or not. I'm referring to the T side. Sorry, I mean the model itself. So for example, it's a deep learning model. Right. So you can basically log anything. So the only thing that MLflow does not allow you to do is I'm not aware of time series data, but you can log strings or text or even artifacts. Right? So in your case, you want to log maybe an artifact which can be an image or a graph, or it can be some text or, I don't know, it can be any arbitrary binary or data. Right? So it's really up to you. As long as there's a way to do it. Right?  Thank you.  And you can see how the artifacts, the pickle models themselves are getting outmoded. There are several codes. And also MLflow comes with the modules inside that are for different most popular algorithms that are already decomposed. So you can take a look at the GitHub. MLflow examples are the errors here. It's quite easy. Okay, one person's here. And does MLflow change how you do monitoring visibility on different types of things? So do you need more visibility to specific models that are being applied? The question is, do we have any more visibility due to MLflow being part of the stack as opposed to how we previously did it? I don't think that it does not give us any more visibility, but it allows you to debug problems because if a new model comes downstream, you'll see it first here, right? And see new models being deployed. And you know something happened upstream. It allows you to investigate. There's a paper trail. But other than that, it doesn't give you visibility. But it shouldn't, in my opinion. Oh, no, we have not done that. Any more precious? We actually have a Slack channel, so I believe you that feel that the forum should have seen the link to get onto the Slack. We just established it two days ago. So we live there as well. And you may go check out all the details that were not asked in the live session.


is joey vato oh my god oh my god unbelievable joey vato oh my god and just this guy for the angels no big deal just the mic trout autorellic with a piece of letter in there that's all just no big deal there holy crap Washington diamond guarantee oh come on man unbelievable diamond gem logo man look at dodge itch boom yeah Lakers dude somebody has to get rich on the stream logo man freaking Davis kaboom are you freaking kidding me nebula one of one dodge itch is a Hall of Fame flawless case man Tom Brady one of ten Patriots auto boom flawless finishes Super Bowl 38 that's insanity 52 tops throwback Super Fracturado show hey Otani and what the heck is this oh my god oh my god are you kidding me wow one of one multi-gem Tom Brady boom Patriots what eight diamonds and five six seven eight nine ten eleven rubies look at that how we doing how we doing happy quad release day bunch of stuff going on today flawless tribute lead extradition obsidian all ripping tonight rock and roll here everybody starting off with lead extradition got to ten boxes so two half cases essentially and then we'll be in a tribute and it'll be in a flawless and we'll be into obsidian Andrew I bought yeah yeah big time yeah I gotta get out of the bot lobby honestly out there yeah that's like people walking into walls out there man rich city pulled some quad auto the Beatles I don't believe them no I don't believe me that sounds programmed people yeah whenever start getting loaded cases dude yeah right man can't believe they send that to a breaker you know anybody opening person also transcendent please do you please if there is somebody opening a person of transcendent please please god please let me know please I'm sure there's some out might maybe probably happen probably definitely this weekend maybe I'm saying interesting on Sunday baby low key Wallace Wallace six to go in that first serial number yeah it's getting close I mean what 2019 spots have come in random teams filling faster than cereals I know crazy it's cheap guys we have it at price like 10k a box a case is going for like 23 a steel city right now I think so yeah insane and it's short and we don't know where we're getting it back and it won't be that same price no no no this is the time if you are interested in buying flawless today is the day to do it yeah today is the day to pull the tree next like the next like 24 like the next 48 hours right today and tomorrow just being honest no no we don't first first case is gonna be a man cut which it literally was a man cut and I was like second case and then the other day I was like Jake you're you're about to pull our first MLB debut patch you know I'm not I was like you're about to pull our first MLB yeah we were talking about the over-under on amount of comments that's a SMH a breaker got this kind of thing and Devin said it double already yeah I told you they put the over-under on like 13 something I was like oh yeah people are arguing with them being like this is a $50,000 product yeah fine yeah sorry sorry Dino from North Carolina you can't get it at Walmart I gotta do this random good luck everybody what's going on everyone Sam here with late in sports car just doing the random for 23 Panini elite extradition baseball hobby 10 box break number one random team style I got 30 participants and 30 teams all teams are in no combos roll a dice gonna go 11 times we've got one two three four five six seven eight nine ten and 11 Andrew M down to Benjamin H and 11 times on your teams one two three four five six seven eight nine ten eleven reds down to Rocks how was your day my day is good Andrew reds James S. Marlin Stephen K. Padres Kevin a with the Giants Cody G. Cardinals Justin M. White Sox John W. Tigers Matthew B. Rais Jonathan L. Virginia with the Mariners tie W Pirates Brian G. Orioles Andrew P. Astros James S. Red Sox Matthew B. Metz John C. Rangers Jacob H. Dodgers Raymond S. Diamondbacks Kevin a athletics Jonathan L. in Florida with the twins Ron our Phillies Brian G. Brewers Kevin angels Ron our Royals Jonathan S. Yanks Gabriel S. Cubs Brady T. Guardians Benjamin H. Blue Jays KLD Nationals Roy Iran are with the Braves and Benjamin H. Rockies get these coming over guys get started in a couple of minutes thank you all that's why it was at 12 not 11 I figured it out Chris D. picking up a flawless spot Philip T. picking up the last four of the tribute anybody go Phil solace out on tribute flawless left side number five that's not Wallace number five is down to five five to go on five everybody five to go on flawless left side five very nice very nice very nice continue this on today everybody results for you there you go here comes your results link with you guys were Sean for one second you mind Sean thank you be right back everybody no it's the it's the Red Sox City Connect reverse I think there's something I'm gonna give me a pirate's hat though I just got some black and yellow dunks so I'm gonna rock a pirate's hat with it it's gotta find a cool one I know now that's that that colorway one they have that looks like UCLA but this is like the the reversed one I haven't checked face believe you on there was 14 and a half I'm at 33 and have a check come on all right so that's done we had a little comeback last night right it's a late innings but we fell short true just in handy name the first six innings other one yeah um thank you Audrey picking up a spot and flawless picking the Lions in Obsidian a random spot in Zenith and a spot in archives thank you buddy bunch of spots coming in there that means let me go update because I don't remember where those were exactly six to go on that obsidian six twenty seven eighteen to two to go so Sam yes how did both of us debut pitching go against Dylan Cruz not good no very bad danger tell me what happened please we he claimed it's my time to get my revenge on him from Little League right and I guess to summarize it paraphrase they did not get his revenge no Dylan Cruz is now up to on him and he hit one that might not have come down yet dude I think it's still in the air it might still be in the air to be honest so we'll get that I have that clip we will get that clip one of these days and make sure you get to see that yeah not great man thank you Scott with a spot in left side number five Jason see with a couple of tribute spots what's up kale four to go on flawless left shot it's up kale how we doing buddy I think tomorrow tomorrow yeah tomorrow yeah actually interesting story I don't know if any of you have known both are played with Dylan Cruz in Little League yeah both his dad coached right tell the story come here real quick before we start yeah so you how old were you when this happens you tell at that time like this is this guy like it was like about that I mean seven years old little Timmy can hit him on yeah yeah it was pretty serious so did you had yet an opportunity last night to go up against your former arch rival yeah yeah and it didn't go well no it didn't go well no has the ball come down first time on the mound oh fair fair fair got the swings in which was good yeah that was nice take that got a hit didn't go so well with now not super hot with Dylan Cruz out the bat that was more than seven years old was like last week for Bofo kind of story let's go Riggs what's up Joshua so funny bring your kid to work that's not as well be rocking the floor to fire frogs today floor to fire Josh Riggs he went up all right let me grab our timestamp we'll get going on this everybody yeah so wasn't good now not the bet not the greatest of showing wasn't good hello Spencer he's still ducking us man yeah I did we've been telling people you're ducking us Spencer you don't want to play us in the show good luck everybody going on everyone Sam here with late in sports cards from 23 Panini elite extradition baseball hobby 10 box break number one random team style look again at your end results as always find them on the website under the results tab as well as the Lincoln chat so good luck let's do it I did not actually Joshua SC SC is getting ready for some some tribute divisions I know that I am amped tribute in general there's a lot of all right starting off with a ret louder pump ret louder to at 199 got a Noah Shenuel the angels to 349 a Josh Rivera nice one there on the elite extradition to 399 on the on Christopher familiar DPL the DPL's top Michael Michael so man familiar is a New York Yankee nice like this guy then I'd be going to the Yankees I would just any of the DPL's I would just tag yeah and then a Jack Hurley base right you got a higher learning match of the Chicago Cubs rhyme numbers J Bashir's to 60 or the Cubbies Michael Caracol on the auto that is Noah Shenuel for the angels year 23 pick 11 on the Jersey card in Florida Atlantic and then a Dylan Cruz here you go Nets that's national got Dylan Cruz again spellbound number status Ralphie Velasquez for the Guardians to 499 got impact auto of Landon Sims 400 points ran to everybody at the end and then Enrique Bradfield on the base to the oh Kyle Teal are and Alberto and Rios to 399 PJ Nichols that is gonna be to 60 going to the race on the on we've got a Nolan DeVos for the Astros Houston Astros the hidden gems auto and a base of tenor hole we're going to be going on like seven days very cutters of to wrap right louder might be very well might be Dylan Cruz die cut to you 97 got Leo DeVries Leo at DeVries international Padres he's actually the number one prospect number one international new prospect forgot about him to 399 so that'll go Padres nice hit there for San Diego got Nathan Martirella also for the Padres nice one there on the hidden gems auto and Josh Knoth at John birdie I'm not I don't mind that honestly he's a base deal it's still like 41 bases three years ago so I'm in I'm in on John birdie being treated to the Yankees I am all skeins living on boyfriend living on boyfriend living on boyfriend I don't care I walked into that yes Tray Morgan trade the red 499 all right kid collar go play the show and grapes because I'm tired very bad at it Zybel Zyass quite the auto Zyass didn't even see it he said no what's up John what's going on brother thank you oh this guy's got nothing on him yeah this is gonna be a random that's gonna be a random thank you George bring up a couple of spots later brother bye I'm just gonna do all the DPL spots to one break or one so the points will be a separate random and then all of the non like the DPL non-team DPL auto as well go to one spot as well no a shanuel die cut of Homer Bush for the Draze a Sammy Stafora nice one there for the reds he's gonna be a chase I'd imagine in 24 or 25 Bowman 24 Bowman Olivia Pichardo what oh yeah Olivia she's in what's it called what she and one of the Chardo she's in the yeah she's American baseball player at Brown right now so that'll be a random that will be a random she's in Dylan Campbell on the base like one of the leaf one of those probably not leave but like upper-deck skybox wasn't that I think it will break here I believe at some point Phil that is Matt Corbath the 49 year old Cordero again that'll be another DPL spot or not DPL but that will go somewhere Walter Ford nice one there for the Mariners we've got a DPL of on hell at Senial I know he's somewhere on hell Senial I don't think he's anywhere right now I'll double check again on hell Senial and then Jason Woodward let me look really quick I'm not seeing him anywhere all right so that'll be a random this is Paul Skeen's primary colors let me dumb boyfriend let me dumb boyfriend that is Sean Sullivan to a 10 hero Wyatt for the Royals a future threads Kyle Teal for the Red Sox to at 199 and Braylon Morrell hello RPC you pumped RPC is here no Chase Davis higher learning Christian Campbell to 349 Travis Honeyman for the Cardinals on the on how does he keep getting here Hunter Barco for the Pirates instant impressions auto and then Charlie Soto I keep letting them back in then got Aaron W picking up an obsidian full case spot Graham M with the Jags in number five Evan O with the Angels and Tribute and Gerald a with a flawless left side thank you all I'm pretty excited about opening dude yeah me too man it's gonna be a good time just have some baseball back three to go on flawless number five flawless because a Jacob Gonzalez star status Alex Clemmie on the green auto Peyton Pallet the White Sox that is a Braylon Fuentes Nicholas do you go any games up there he's in PA to any minor league games you don't have to go to any of the big league games this guy like straight doesn't exist man that's gonna be part of that's random the DPL random spot and Eric what's home to here I got sure yeah make one for the DPL so yeah one for the DPL you could just put that on like the on the sales sheet which to be like unmarked DPL go to whatever spot that will be this will be a separate and then the points will be separate so those three randoms at the end those all go together yeah there's like a DPL spot yeah Dylan Cruz primary colors blue of Typhaloid the fever Eumann Lin for the Diamondbacks that is a DPL Christopher Familia I think there was 249 and then a Homer Bush Christopher Familia yeah he was he was the Yankee Familia was the Yankee Shipe with the Falcons and six thank you buddy Charles with the Falcons and Cowboys both in five of obsidian Gino Gruver Drew Hathenberg 85 Eric Pena DPL we got to make a DPL spot in this man good good I'm chilling went to my first movie game okay I'm gonna go try to catch aha he's a royal let's go this guy exists he's a royal Dylan Cruz nice one there for the Nats and then it Teddy McGraw he's gonna try to catch something bacon Gonzalez Tampa's easy enough to get to yeah and dry for Vizcayano Typhaloid nice one there to you at 399 Leonardo Garcia boy man it's just such a lame stadium like nothing Larry what up man Larry what's going on brother hope you're having a good day I was your day my day is good Yankees on Leonardo Garcia Yankee yep and Brandon Sprote Matt Shaw Dylan case that Victor her tar her taught her Tato Victor her Tato he is a national in the DSL he is a national trade let's go hidden gems and George Lombard make colors Marieke Bradfield blue up for the dime back rock Wilkin going to the brew crew you start that all right all it takes around here boys you're upcoming comedian call me okay Jason Savakul I got a great audience 9 499 honestly on Helsenio I'm gonna double-check him again really quick miss more yeah I don't but yeah that'll be part of the deep DPL random Cole Emerson going to the Mariners and then and Andy Montos Jacob Gonzalez Tray Morgan do 349 Dylan Campbell the Dodgers a Jose Jorge with the name that looks like it's gonna be part of the DPL random and then TJ Nichols Aiden Miller Felman Celestin a Dylan head to the Padres auto to 399 sunny day she are one of the angels and Francisco Valoria by Loria pump rent louder to 199 right no gold Luis Garcia 10 I'm assuming right that's 24 got a Sal Stewart nice one there the impact impressions for the reds a Ronnie Garcia thinking that's gonna be another part of the random that's gonna be another random and then Jacob Gonzalez that'll just go with yes yes we do Gary on channel to late channel to we upload breaks throughout the day yes yes sir yes sir you can also go back on today's like after or morning and afternoon stream and just click on the timestamp in the description to 199 chase Davis TJ Walton for the Phillies Justin Campbell for the Guardians on the instant impact or impact impressions Yario for Darrow baseball and Keith still another guy who's gonna be a random to or that DPL spot and then chase Davis Enrique Bradfield primary colors Enrique Bradfield nice orange to 275 Phillip Abner for the Diamondbacks a chase Davis for the Cardinals on the Jersey and Brady Smith Jacob Wilson spellbound this guy you know one of eight base signatures Brody Hopkins Mariners Brody Hopkins that it's a redemption congrats Mariners a Kyle Teal for the Red Sox nice one there congrats Boston and then a hey suit or Jose Luis Pena spellbound of ballskins a Victor Hurtado to 499 Eden Smith for the Mariners stop James Nolan DeVos the Astros I believe Houston and Jason Woodward group got Ralphie Ralphie Velazquez Dylan Cruz for the Nets there you go nice stuff there 349 I'll give on Jake Wadsala's White Sox running plate is gonna be a Nathan Detmer one of one for Oakland nice one there and a Mitch Jeb yeah we're Hell yeah oh there's a lot of schemes in this one just get ready get that copied and ready to be pasted there's like one a box man spellbound of Dylan Cruz Phillip Abner a Kumar rocker impact impressions for the Rangers that is a Brenner Cox for the Nationals hidden gems and Miles Naylor goes oh perfect music my ears and Sean's Beat it Max Anderson for the Tigers to 24 Jason Savakool gardenals a DPL of Brian Arousa nope because LinkedIn starts coming up so that'll be part of the DPL spot and then on hell De Los Santos chase Davis primary colors Mac Horvath for the Orioles Conor O'Halloran going to the Blue Jays hey George Lombard for the Yankees nice one there first rounder in this upcoming moment Connor Burns hopefully higher learning Drew Hagenberg I cut of Charlie Soto for the twins Yandell Ricardo the Royals Andrew for this guy you know baseball yeah ask me another guy still in the part of the DPL spot auto 260 and Wilson's Jacob Wilson spellbound got Andy Matos to 499 a Brock Wilkin nice one there for the brew crew on the yard


Okay, we're live. We're back. I'm Jay Fidel. We're here at Think Tech, and we're talking about Think Tech Asia today with, guess what, Ralph Kosa, Pacific Forum, the president, CEO, and also the Honda Chair of Pacific Forum. And he's been spending most of his adult life, my right to say this, thinking about Asia, thinking about American relations with Asia, writing, thinking, visiting, talking, schmoozing, rubbing elbows with all of Asia for many, many decades. And boy, he's the guy I want to talk to about what really happened in Singapore a couple days ago. Welcome to the show, Ralph. Thank you, Jay. Thanks for coming down. My pleasure. Yeah. Can't wait. So we entitled the show The Real Meaning and Effect of the TK Meeting. Let's call it the TK Meeting in Singapore. Singapore is almost an irrelevant here. It doesn't matter where it was. Okay, and, you know, after you deal with all of the pyro techniques that led up to it, and the name-calling and all, so now they sit down for 20 minutes, 20 minutes, I did the math on that, 20 minutes with actual face-to-face, and they'll speak the same, maybe they do speak the same language, but I doubt it. Oh, I don't think so. And you wonder what, how that happened, what came out of it. So there was a thing on 60 Minutes this past Sunday where they had some senior nuclear scientists and military talking about what they thought, you know, real level of nuclear threat was from North Korea. It's almost as if they wanted to educate Trump because they didn't feel that he was properly educated before the meeting. And it was pretty high level of threat, somewhere between 30 and 60 nuclear weapons, and possibly the ability to do reentry without burning up on the mainland, on the U.S. mainland. So we, you know, we can have a serious threat, and if the government does know about that, then certainly that would factor into this meeting. If it doesn't know, gee whiz, he didn't do his homework, did he? Do you think he did his homework? Yeah, I think he did.  That's nonsense. He's playing to the crowd and we bite every time.  He should have done that.   So I don't, you know, I don't worry about that. I worry about the fact that his values are different. His views of the importance of U.S. as the security guarantor in Asia are different. His view of the value of U.S. military presence in Asia is different. And his belief that somehow or other he can talk anyone into anything is at least concerning. You know, I had joked previously that one thing that Kim and Trump had in common was that each one thought he could outsmart the other. And I'm not sure who won the first round, but I would give Kim a lot of points for how things have turned out so far. But, you know, this is a process.  This is the first, you know, it's one small step for mankind. Now we've got to, you know, put the meat behind it and it could either turn out to have a really good outcome or it could be more of the same or it could be a really bad outcome. And right now all we know is that there's movement and nobody is threatening to kill one another. So that's probably good news, although I didn't lose a lot of sleep about the North Koreans attacking when everyone was worried about it. But now there's perhaps even less concerned about that. You know, I just, I wonder on this sort of thing whether Kim is as well prepared as Trump. Whether Kim, you know, understands. I mean, he doesn't come from the same level of experience. And he's avoided diplomatic experience. How well prepared was he? And how much of the work was done? You know, we have 20 minute meeting that the agreement was drafted in advance by probably by other people. And it's a very, what do you call it, threadbare kind of agreement. It's an outline, not really terms. So how did that happen? This sounds like what you always hear about diplomatic meetings in this nature, that all the work is done by the bureaucrats in advance. On the other hand, Trump is unique. And he doesn't operate like a conventional diplomat at all. And he doesn't let his State Department operate like conventional diplomats. And so, you know, a guy like you would be within the framework of conventional diplomacy. But we don't have that here. So it's hard to understand exactly what an unconventional guy is doing. Right. And you know, the reality is they're both unconventional guys. And I think the reality, I was one of the few, I think, who thought the summit was a good idea. And the reason why is because we've tried the standard diplomatic practice. We've had the American diplomats sit down with the North Korean diplomats and come to a great agreement. This was called the Leap Day Agreement under Obama. And then 16 days later, the North Koreans launched a satellite, undermined everything because the diplomats were doing their job, but they weren't in sync with the military who had different marching orders from the big boss. So you've got to get the two big bosses lined up. Because otherwise, you've got Trump tweeting little rocket man. And then that insults it and it undermines everything. So you had to get the two guys at the top, both say, in principle, we are going to agree to, quote, complete denuclearization. In the real world, in the old world, we would have come up with the definition of complete denuclearization, how are we going to get there from here? And then the two bosses would have come in and pat it themselves in the back. But this is not business as usual in either country. So this is what had to be done. That's fine. We just need to understand that. But you don't come home and say, the world is now safe. Sleep tight tonight, America. There's no longer a North Korean threat. How is there no longer a threat? They had all of these missiles before. They still have them. They haven't given up anything. So they said, you know what? This week, we're not going to attack you. Thank you very much.  I mean, if you went back and started doing little rocket man insults again, I suspect just as a matter of humanity and psychology, it would be worse than before, because now he would be returning from a point of relative peacefulness and re-insulting the man. And the man's going to get all the more angry. And then you have even a greater possibility of a confrontation. Yeah, but this is all a reality show, Jay. I mean, this is what frustrates me, and I think frustrates most foreign policy professionals is this sort of game of chicken that's been going on and, you know, the tweets which may or may not be policy and may or may not reflect any kind of sanity. And we all react to that and we all, you know, stop what we're doing and sit there and scratch our heads and say, how could he have just said that? And meanwhile, it captures the news cycle for the next 12, 12, which is 24 hours, which is exactly what he wants. And fortunately, you know, I was not a big fan of my Pompeo when he was in the Congress, because I thought he was sort of heading in the wrong direction. But he is a hard-nosed realist. And he's the guy that's actually doing the leading the negotiations with the North Koreans. I don't believe he believes for one second that you can trust the North Koreans or that the problem has now been solved. So, you know, the boss has said the problem has now been solved. Now you go solve it. And so this is, you know, now the hard work is is going to be gone. And we've got some real hard-nosed realists doing that hard work. So, you know, we could still have a happy ending. But I'm not sure. We're sure ain't there yet. No. But but if you be a realist and look at the terms as revealed to us, however vague they are, this is an outline. It's an agreement to agree, is all it is. But on the one hand, we apparently, at least on a moral level, have agreed not to do any more joint exercises with South Korea. And he, Kim, has agreed or he's agreed to explore the possibility of not doing nuclear anymore, pulling all his sights, which are numerous and all around the country and hard to find. And somebody on 60 Minutes said, well, that the best of it would take 10 years to actually do the nuclear rise in South Korea. So how do we come out? Is that a good deal? Is it a good deal for them? Is it a good deal for us? To whom is it a better deal? You know, it doesn't matter if one side has a slight advantage or the other side has a slight advantage. What you're looking for is something that's win-win. And in win-win, the win-win implies lose-lose. Because each side has to give up something. So we shouldn't, the problem in the Middle East is everyone wants a win-lose solution, so no one will ever agree here. We're trying to find a win-win. And that's good. But there's the agreement, the joint agreement as stated, which is a two-page piece of paper that literally says very little other than general principles. And then there's all these news conferences and everything else. So the president in the news conference says, we're going to give up military exercises. And the military says, oh, say what? They didn't know. And the vice president says, well, no, we're not going to end exercises. We're just going to end this. And then someone else says, no, what the president meant to say. So we still really don't know what... It sounds like so many things about the past. But none of that's in the agreement. The agreement just says denuclearize and work toward X, Y, and Z. So do I believe that there will not be a U.S. military exercise in South Korea? And I sure hope that that's not the case, because you've got to continue to train and exercise.  No, but that will probably stop doing. And, you know, when the president was in his stream of consciousness talking about war games and provocative, those are the types of things when I meet with the North Koreans that they say they're concerned about. They remember bombs dropping on them and they don't like the idea that we're practicing with nuclear bombers and that it's denuclearization of the Korean Peninsula. That means they get rid of their nukes, but we stopped the nuke threats on our side. Fair deal. But so far they haven't given up anything. All they've done is blow up a test site after they stop testing. And that's better than keeping the test site, but it's not doesn't make me sleep any better at night or any worse. There's there's got to be here's a list of all of our nuclear capabilities.  Why there's a big difference between 30 nuclear weapons and 60 nuclear weapons? This is this is the experts guess. So if the North Korean Samar hand us a piece of paper and say, we have 24 nuclear weapons and we're going to turn them all over to you. You can believe that. Yeah, if you do, I've got a bridge in Manhattan. I want to sell you, you know, so so that's that's part of the challenge.  We only have to we only have two nuclear weapons.  Here's three jars of plutonium. Have a nice day. And then what do you do? Well, you know, you can say, peace in our time. There's no more threat. Or you can say, Well, you know, I think we need a little more of this verification. And that's where everything is falling apart in the past. It hasn't been on making the kind of agreement that we made here on Monday. It's on the follow through on the verification part. And that's the that's the tough part. It's tough for the people because the people again have to get their information from the White House. And so we've had a problem getting information from the White House. We've had the war on the press. We've had information withheld or distorted. And I think it's going to be really interesting to see this process you described going forward, where we get only piecemeal information, we don't really find out what's going on. Yeah. And you know, the important thing is that the negotiators know what's going on, and the negotiators know what the president has promised or not promised. And there's even some question about that. And I think that, you know, the the rest of us all would like to know, but that's not really critical. But if the negotiators sitting there and saying, we want X, Y and Z, and the guy on the other side says, Oh, President Trump said that didn't matter. That's sort of undercut your bargaining position. So they, you know, we better be real clear on what it is that the president has promised. That's where the experienced diplomats come in. That's right. And we better find some really quick. And that's where at some point you get the what the president meant to say was, you know, did he say exercises? No, what he meant was he's going to stop doing pushups and jumping jacks. So and then you're going to sort of go from there. It's Ralph Kosa Pacific Forum. We take a one minute break, we'll be back. And then we're going to find out the implications of this on other countries in the region and the world. Wow. We'll be right back. Hi, I'm Ethan Allen, your host on think tech's likeable science show. Every Friday at 2pm, we delve in the magical, magical, fascinating world of science. How science applies to your life. Why you should care about science. What impact science has on you and on those around you. Why you need to know some science. It's a fun, interesting, painless way to learn some good science that you can use. See you there. Hello, I'm Cynthia Lee Sinclair. I have a show called Finding Respect in the Chaos. It's all about women's rights and gender equality.  And a place for advocates to come on and share important resources so that people can get past the abuse and into the hope and healing that's on the other side. I hope you'll join me every other Friday at 3 o'clock for Finding Respect in the Chaos. I'm Cynthia Lee Sinclair on thinktechaboy.com. We are honored to have Ralph Cosa here because now this is like one day after June 12th when all this was supposed to happen in Singapore. And you know, it's like Puff, something happened. A 20 minute discussion and a two page document. Now what? It was like one of those agreements in a business sense where you have this very short agreement and everybody says that's not an agreement. That's an agreement to agree. And now as you said, that fun begins. But you know, it does have an immediate effect on people. For example, I'm just throwing an example. South Korea was not notified of the provision that said no more joint exercises.  Is it? There are people in South Korea right now who are beating their heads against the wall or digging tunnels deep. There are others who think this is delightful. You know, the current government, the progressive government, if we had wanted to continue military exercises, they are so eager to have North South for Approach Month that they may have said we won't agree. We can't hold, the U.S. can't hold an exercise with the South Koreans in South Korea unless South Korea agrees. So we may have been making a virtue out of a necessity. We don't know because we don't know what Trump and Moon have been saying. There might have been discussions. Yeah. And you know, Moon may have made it very clear. Look, we don't, we don't, you know, we want to hug one another and you're standing in the way of that. So that's, there's a, you know, that's, he's the interested third party. You know, I mean, it's the Korean Peninsula, the South Korea, North Korea. And at the end of the day, he's the one that's 30 miles from the border. Yeah. I mean, one of the things that I thought was a shortcoming of, of the agreement was that there was no reference in the agreement to the need for the DPRK and the Republic of Korea to work on peace issues on the peninsula. Oh, interesting. And that to me is, you know, the final solution. First of all, North Korea has to recognize South Korea as a sovereign equal and go forward. And that's where we need to be insisting on things. And obviously in the, in the agreement, it wasn't there. And we still keep talking about this as if it's all about us. And it's not all about us. We're a key player. We worry about the fact that they may have one or two missiles that might be able to put a warhead on it and might be able to reenter the atmosphere and might be able to hit us. But they've got artillery that's within range of soul and 25 million South Korean inhabitants. Mobile artillery that can't be found so easily. Yeah. So, you know, let's remember whose peninsula it is. And we've got to keep the South Koreans engaged. And, you know, the Japanese are concerned because that in mind about the South Koreans a week closer or further away from reunification. There were, you know, there were dreams here in the last few weeks that maybe this would somehow wrap around reunification. Is it less or more likely to define reunification? I think that what first of all, it's we're not going to wake up one day and the wall is going to come down and Kim Jong-un is going to hop on a plane and disappear. So the best I think we can hope for is peaceful coexistence in some type of a confederated Korean states where each keeps its own identity and you start opening up in the Helsinki process, if you will, begins so that maybe 20, 30 years from now there's a peaceful reunification. To have reunification otherwise, either the North has to surrender or the South has to surrender. Neither one are prepared to live under the system of the other one. Maybe a lot of North Koreans they'd be willing to have given the vote, but they're not going to have that chance. And certainly Kim Jong-un is sort of happy where where he is and is not looking for a chalet somewhere in Switzerland to go back to. So when we talk about reunification nowadays, really what they're talking about is peaceful coexistence, some type of confederation where each still exists as a sovereign state, but they recognize the other one's legitimacy. And that's a big step because both sides have in their constitutions that they are the legitimate government of both of the entire peninsula. This is like China and Taiwan, both of whom pretend that they control all of the Chinese people, although there's 47 million people in Taiwan that don't quite agree with that. So well, you there's also suggests that we have an armistice now. We don't have a peace treaty and it's going to stay that way until some other aspect is renegotiated. That's right. And you know, who's going to sign the peace treaty? This was one of the reasons why the peace talks fell apart back in 1999 under Bill Clinton. They were trying to work on the four party agreement. And the North said we'll only sign a peace treaty with the United States. And the United States said, South Koreans have to be part of this. And the Chinese said, hey, hey, how about us? You know, we're a player here. And now in the North South Declaration, after the Moon Kim meeting, they talked about the three sides, U.S., South Korea, North Korea, sitting down and talking about a peace agreement or maybe four, including the Chinese. So the North and South Koreans, for their own reasons, don't trust or like the Chinese and are, you know, they don't want to completely alienate them, but they're not real eager to have the Chinese part of this anyway. So there's all sorts of dynamics. It's complicated before. It's still complicated now. It's still it's more complicated now. Yeah, right. Because nobody knows what that thing means. The other thing, it comes up in the notion of, you know, the 30-year process of opono-opono, if you will, okay, is economic interaction. Economic development, joint economic projects. And there was a lot of press about that. There's nothing in the agreement about it. That's right. And again, if you listen to some of the things that the president has said, it's like there's a pot of gold at the end of this rainbow as soon as you denuclearize. But we've also said, until you denuclearize, you're not getting a penny. And we've also said the U.S. isn't going to pay for all of this. South Koreans and Japanese and Chinese will pay. Well, you know, we're very generous with someone else's checkbook. Maybe the Mexicans will pay. Yeah, maybe maybe the Mexicans will pay. I wish I had a thought of that. So at any rate, it's going to be expensive. And the North Koreans aren't going to give away anything for free because that's not who they are. And they're already talking in their propaganda about how the president is going to lift sanctions and help provide economic assistance and etc, etc. That's also not in the agreement.  And until in our system, until the Congress says, yes, I'm going to write that check, you know, the president can can promise everything in the world, but the Congress has to pay for it. And, you know, the Congress isn't particularly eager to start writing checks for the North Koreans. They're also not particularly eager to bring US forces home or to stop readiness and training.  How long ago was that? And how many are still there? So, you know, when the president says jump, the Congress does not say how high. But you raise, you know, the issue of the effect of all of this on politics and political processes and public opinion in the United States. What kind of effect do you think, Ralph, this is going to have on on the midterm elections coming soon, four months? God only knows, you know, I mean, I one of the things I'm very happy about is that I'm a foreign policy analyst, so I try to figure out what easy things like what the North Koreans are doing or what the Chinese are doing. I don't have to I don't have to predict American politics, because I would I would be unemployed if I was predicting American politics. So far, at least in the last couple of years, my track record has not been all that good. At the end of the day, I don't know how many Americans are going to vote based on what's happening on the Korean Peninsula, to be honest with you, they're going to vote based on what's happening in their neighborhood. And, you know, whether they've got a better job or whether the tax cut has really helped them or hasn't helped them. And whether the other side is so demonized or the other side's candidate is so off the fringe that no one will vote for him. But, you know, the criteria as to what's electable has changed considerably. Yeah, right. Well, I mean, you think it's possible. Is there any real chance between now and November there's a Nobel Peace Prize in this for Trump? Is it possible? I would find that hard to believe for a number of reasons. One, because the Europeans don't like Trump, so they'd have to really be holding their nose to give that. Secondly, I think most people understand we're not there yet. There's, you know, a mouse ago before we sleep, the kind of phrase. So you've got that involved. And then third, you know, the Trump brags about the fact that he got the North Koreans to the table by threatening to bomb them and kill them. That's normally not one of the criteria is for the Peace Prize. It's, you know, it's more an aspirational and and everything else. Who knows? Who knows? You know, I mean, I would be less surprised if the Nobel Committee turned around and gave it to Moon Jae-in. There you go. As the sort of the architect of all of this. There you go.  But again, you know, this is, you know, just ask me what's happening in Myanmar or something. I'm more comfortable. I want to ask you what's happening in Europe, because Europe, relations with the U.S. have changed. For that matter, Canadian relations have changed. I mean, all of our allies, the G7 was the G6 plus one. And he made enemies. I mean, I mean, he he insulted those leaders to the point where the people in those countries were insulted. How does this now what happened in Singapore or didn't affect that? I think it's a great question, Jay, and the answer is it beats the hell out of me. I'm on my way to London on Tuesday for an Asia conference with a lot of our European friends that I'll have a better feel for that. You know, I think we haven't yet refocused Pacific Forum to concentrate on the Canadian threat rather than the North Korean threat. We're still we're still going to sort of look at North Korea and China as the primary challenges. But who knows, we may you know, we may need to change that. And you know, those French, you can never trust them in the first place. Anyway, so, you know, maybe all of that will change. But for for right now, I you know, I think they're all disappointed. It's it's not the way I would prefer to see diplomacy conducted. I've always, you know, been brought up to believe that you treat your friends better than you treat your enemies. Reasonably, yeah. But you know, that doesn't seem to be the case. On the other hand, you know, we've got enduring relationships and mutual interests that may cause us to smile and grin and bear it and hope for their best or hope for change and just sort of muddle through. And that's probably mostly what's going to happen. Yeah. Well, yes, I think so. I agree. One last question. This is a really interesting and provocative question. I ran a little piece about this. So, you know, one of the things that Trump has done in his campaign and now in his office is to criticize ongoing criticism of the Iran deal, the Iran nuclear deal that Obama negotiated over a long time. Great detail, as opposed to two pages. So. Which one is a better agreement, Ralph? Well, it asked me in two years whether or not this agreement has turned into something. I think it was a great mistake to walk away from the Iran deal. We hear these criticisms that the deal allowed Iran to do X, Y and Z. Deals don't allow countries to do anything. That would imply that without the deal, Iran wouldn't be doing it. Well, we're going to find out what Iran is prepared to do without the deal very quickly. And we're going to say, oh, wow. You know, so my view has always been you don't get rid of a deal until you've got a better one laid out. You don't do it in hopes of having a better one. You know, I'll give you all my money in hopes that I'm going to hit the jackpot tomorrow. Hit the jackpot tomorrow, then you start talking about what you do with your old money. So to me, it was it was a bad move. There are a lot of people that are worried, probably more worried about what the future lies as far as U.S. Iranian relations are concerned because here you've got a hostile power that is not particularly intimidated by us and has a lot of friends who can help them a lot more than North Koreans friends. So but, you know, at one time in my prior life in the military, I worked on the Middle East. I was so happy to stop working on the Middle East because it's an area where no one will accept the win-win solution if you can find it. Yeah, talking about complexity. Yeah. It's unfortunate, but let's hope for the best here, but I think we need to be prepared for the worst. And that's why places like Pacific Forum exist to sort of prepare for the best and prepare for the worst. That's why I think TechLog, Pacific Forum, so much. Thank you, Ralph. Good. They're coming down. Thanks, Jay. Any time.