I'd like to thank everyone who's joining us. Oh, your slide's popped up. There you go. All right. I'd like to thank everyone who's joining us today. Welcome to today's CNCF webinar, Scalable ML Workflows with Advanced Data Management on Kubeflow. I'm George Castro, Community Manager at VMware and a Cloud Native Ambassador. I'll be moderating today's webinar. We would like to welcome our presenter today, Vangelis Cukis, CTO and founder at Erecto. A few housekeeping items before we get started. During the webinar, you'll not be able to talk as an attendee. There's a Q&A box at the bottom of your screen there in the Zoom. It says Q&A. You can click on that and submit your questions. Feel free to drop your questions in there throughout the webinar and we'll get to as many as we can at the end. This is an official webinar of the CNCF and it's such a subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that will be in violation of that Code of Conduct. Basically, please be respectful to all your fellow participants and presenters and I hope we're all ready. So with that, we'll hand it over to Vangelis to kick off today's presentation. Take it away. Thank you, George. Thank you. Today's presentation is about scalable ML workflows with Advanced Data Management on Kubeflow. What does this mean? What's the problem we're going to be talking about? Growing up on an ML stack, a full ML pipeline is very hard and doing it in production is even harder. And then if you want to do multi-cloud, then this is when the difficulty skyrockets. And by multi-cloud, we also mean working on your laptop, working on a local deployment. If someone is working on-prem, in a non-prem Kubernetes cluster and on the cloud, then it's a multi-cloud deployment. So it seems a communist conception is that to build an ML product, you need to focus on ML code. You need to write your code, write your model, train it, and then you're done.  But the reality is that it's a DevOps problem, configuration, data collection, verification, managing resources, managing processes, serving, introduction, monitoring. These are much more important and take up a much bigger percentage of time compared to actually writing ML. So we need a platform for ML. And this is where Kubeflow comes in. So why Kubeflow? Kubeflow is an open source project started by Google. It's a meant-to-end solution for building ML platforms, ML workflows on Kubernetes. It aims to containerize workloads and allow users to experiment and explore with state of the art AI. It focuses on easy onboarding. We have contributed a code and a packaging for Kubeflow called mini-Kubeflow mini-KF2 is the process of onboarding new users. I'm going to be talking more about this in this presentation. And Kubeflow itself has outstanding community and it is a support. It's a vibrant community, lots of commits, lots of new features. The code is evolving very fast. So what is mini-Kubeflow? What is mini-KF? It's a packaging contributed by us, a Rictor, to the Kubeflow community that packages mini-Kube, Kubeflow and rock our data management platform in a single VM image so you can spin it up very quickly on a single node, presumably your laptop or a very big desktop machine you may have. It does require quite a lot of memory because Kubeflow has lots of components. It features the latest Kubeflow version, 0.6.x, currently 0.6.2. It's the, we think, fastest and easiest way to spin up a full Kubeflow deployment and start playing within minutes. So this presentation is going to be about deploying mini-Kubeflow and then running a well-known example on top of it with advanced data management that comes from rock. So how do you install mini-Kubeflow? There's a previous webinar we did, so you can see the recording, there's an installation video, there's the documentation, but the too long didn't read version is you use Vagrant to initialize a new virtual machine, you start the virtual machine and this is it. And this is what I'll be doing now. So I'll be switching desktops and I'll be spinning up the virtual machine. The machine is up, I go to this URL and I'm shown the mini-Kubeflow landing page. So I start navigating this very nice script, except you... And what happens now is there's a short provisioning phase that essentially manages the virtual machine, provisions mini-Kube as your Kubernetes substrate, all Kubeflow components, sets them up, rock so that you can have storage and manage the storage alongside Kubeflow. And eventually when the whole platform is up, it gives you easy to use links so you can access the Kubeflow dashboard and rock. This will take a few minutes, it takes in the order of five to ten minutes depending on the speed of your machine and your CPU utilization. So I'll switch back to the presentation and come back and see what happens. So this is essentially the process of starting mini-Kubeflow. You initialize a virtual machine, then you start, this is it. Let's focus on what Kubeflow does. This is a page from a well-known Google paper on TFX. So TFX focuses on building an ML workflow with these steps. Data analysis, transformation, validation, training the model, tuning the training of the model with hyperparameters, evaluating the model and eventually serving it. So Kubeflow comes with these libraries in containerized components. So you can use them as they are. And it also comes with a hyperparameter tuner containerized that can run multiple trials. Okay. Then you need an integrated front-end for job management, monitoring, debugging data scientists like notebooks. Kubeflow comes with a dashboard and a notebook-based UI, Jupyter notebooks. The notebook manager is actually part of what we have contributed to Kubeflow. Then you need a shared configuration and job orchestration framework. Kubeflow has taken a hard dependency on Kubernetes for this. So Kubernetes is your shared job orchestration network. What this means is that all notebook servers and all training components and the hyperparameter tuner and whatever you do runs as pods on Kubernetes. And finally, you need a way to actually store your data, control access to your data, and use it in a reproducible way, which means when you have actually run a training pipeline, have a record of having run this pipeline and be able to go back in time, rerun the pipeline. When you have a model that's the output of the pipeline, go back in time and see exactly what you used as input to train the model. To do this, that's where we come in. So we integrate ROC, our data management component, with Kubernetes and Kubeflow to give you an end-to-end platform for reproducible machine learning. This is what we'll be talking about today. If we go back to deployment, we're deploying the database, waiting for it to come up, it's progressing. So moving on, what exactly does ROC do? When you run an ML workflow, different steps of the workflow usually happen at different places. This is the ideal scenario. Data scientists like to experiment on their laptops, on-prem, locally. But for training, you usually need to move to a bigger deployment with GPUs. For example, the Google Cloud, where you can spin up GPU-enabled instances. And production, actually serving the model, happens elsewhere. Sometimes it happens at multiple locations. It happens in self-driving cars. It happens at IoT sensors. It happens at many different places. So how do you move the process from experimentation to training to production and keep it reproducible? First part, we contributed code to Kubeflow to make it data-aware in general. This means we extend Kubeflow components. So they use persistent volume claims provided by Kubernetes. So they store the data in volumes, vendor agnostic volumes provided by Kubernetes. Our software integrates via a mechanism called the Container Storage Interface, CSI with Kubernetes, to provide implementations of PVCs, of persistent volumes. So when Kubeflow stores its data, this data is managed by ROC. And then ROC forms a peer-to-peer network among all locations and allows you essentially to keep your, to start from your laptop, keep your data locally. But then when the time mark comes for training, take a snapshot of this data, move it transparently to the training location, run training. Training will produce a model, take the model and synchronize it with all the serving locations. And because this is snapshot-based, everything is immutable. You'll find similarities with Git for it. There's a live demo later on. So when you have a model, you essentially know the exact input dataset that led to this model being trained in a specific way. If something goes wrong with the model, you can go back in time, see the input to each individual step of the pipeline and experiment with it and explore it. So what's new in the latest mini-Kubeflow based on the latest Kubeflow version? It uses Kubeflow 0.6.2. It enables multi-user authentication. So you can have multiple users spinning up their notebooks in an isolated way. So there's authorization for notebooks as well. If it's integrated with ROG, then you can do near instantaneous restore of snapshots. So when you have a notebook server, you can snapshot it, shut it down, then come back one or two days or one or two weeks or one or two years later and spin it up almost instantly. We have significantly improved the time to perform this snapshot and we also have the ability to snapshot every step of a running pipeline exactly for being able to go back in time and see the input to each pipeline step. This is the landing page you saw earlier. That's how you control mini-Kubeflow. Why do that? And let's see how the deployment progresses. So provisioning, I switched back to the mini-Kubeflow desktop. Provisioning of Kubeflow is almost there. We are now waiting for Kubeflow resources to come up. ROG is already up. When Kubeflow is up and running, this button will also turn green and then I'll be able to connect. So what I did is I started the virtual machine, initialized provisioning, and I'm waiting for the 39 pads of Kubeflow to come up. Kubeflow does have a lot of components. Okay, so let's give it some time. Starting up. Why did we build mini-Kubeflow? Because people need an easy way for onboarding and people like to experiment locally. So by packaging it for a single node deployment inside the virtual machine, we give a very easy way to spin up Kubeflow and start experimenting with it. But because it's still Kubernetes and it's still Kubeflow, you have the exact same user experience. So you can use the same APIs. You can submit the same Kubernetes resources. You can use notebooks exactly as you should on a cloud provider. And when you're ready, because it's Kubernetes plus Kubeflow, you can move everything to a bigger deployment where, for example, you'll train without having to rewrite things. So our goal was a unified user experience.  And this gives you the ability to use the same APIs and find your data where you need them. The adoption has been going on quite well since we first started this around 7th March. It's reached around 4,500 downloads. Please try it out. Let us know how it goes. Give us feedback on the Kubeflow Slack. Initialize the machine. Start the machine. This is it. The requirements are kind of hefty. You'd need at least 12 gigabytes of RAM to get the virtual machine up and running. Kubeflow is quite resource hungry. You need a bit more if you want to run ML training, depending on the size of your data sets. And because it's virtual machine-based, it will run on Linux, Mac OS, or Windows in the same way. It uses Vagrant for managing the virtual machine and VirtualBox for actually running the virtual machine. So let's move. I like how this synchronized. So provisioning is complete. Kubeflow is up. ROC is up. I can connect to it. I'm logging into Kubeflow using a standard IP address accessible only by the local host. So I can log into it. This is the authentication part I mentioned. And this is the Kubeflow dashboard. So I'm logged in in my own namespace, Kubeflow-user. I'm the owner of this namespace. Mini Kubeflow has a single user. Kubeflow in general, in enterprise deployments, supports more than one users. I can access my pipelines, my notebook servers, Jupyter notebooks, hyperparameter tuning, metadata, and ROC as the snapshot store. So what are we going to use Mini Kubeflow for? We're going to use it to run a live demo of the Chicago taxi cab example. This is a TFX example on-prem on our laptop with Mini Kubeflow. What exactly is the Chicago taxi cab example? The sheet of Chicago produced a data set of more than 100 million trips. You can download the original data set here. This data set recorded quite a few details about each individual trip, including how much money the trip cost, when it started, where it started, where it ended, how long it was, what means of payment the rider used, and whether there was any tip. And the result of this example is a trained model that can predict whether a random trip will result in a tip that's more or less 20% of the fare. So we get this data set. We feed it to a model and build a classifier that can decide whether a random trip produces a tip that's more or less 20%. This is going to be the workflow that will be demoed. OK, this is a summary of the input features. Essentially, start and end location, means of payment, and the amount paid. And what is our demo going to be? We'll start from a notebook. We'll use Kubeflow to create this notebook. We'll create a new volume as storage to hold our data. We'll bring in our data set from an external data source. We could download the full data set from the sheet of Chicago site, or we'll download a subset of this data set so we can run the example of completion from GitHub. Once we've done that, we'll take this is ingestion of data. We'll take a snapshot of our notebook volume. Why do that? So we have a data commit, so we know exactly where we started from, so we can rerun the pipeline with exactly the same input data. So we know that whatever the result of the pipeline is, this is where it started from. At this point, we can continue working with this volume. We can continue exploring things in the notebook, and the pipeline is not going to be affected because the pipeline will run from this snapshot. It will run from a clone of this snapshot. This is going to be the starting point for the pipeline. So then, we will run the pipeline. We'll run a series of preprocessing, training, analysis, and eventually serving steps. And we will also snapshot a step in the pipeline, the final step, so we can keep it and explore it by cloning it sometime in the future into a new notebook. So this shows that we can take any individual step of the pipeline, snapshot it, and explore it again using notebooks. And this process can start all over again. I explore the data, make some fixes, ingest new data, adjust my algorithm, adjust parameters, snapshot again, build a new pipeline, start over. And when I have something that works or when I have something that doesn't work, I can go back in time and see exactly where I start from. Essentially, we're using Rock as Git for our data. So this is what we're going to do step by step. We're going to be creating a new notebook and adding a new data volume to it. So we move to the Kupflot dashboard, notebook servers. We can also, I forgot to show you, log into Rock as well. OK. I'm going to be creating a new notebook server. It's going to be named Webinar 1. It's going to have a small volume where I'll be storing my libraries and my code. I'll also be adding a new volume. It's going to be the data volume. I'll make it a 10-digabyte volume. I'll also use a custom image we have just created. And for this, I'll consult my cheat sheet. This is the custom image I'm going to use and launch the notebook. So at this point, I'm launching a notebook, a notebook server using Jupyter Notebooks from a custom image. It might take some time. Oh, OK. This was fast. To actually download the custom image, the notebook is up. It's running as a pod inside Kubernetes. If I used Kupkatl, I would be able to see the pod running. And I can connect to it. It's got a few secret volumes, my data, and my workspace. I'm connecting to it. I'll refresh to give it some time to actually come up. It's complaining. It shouldn't be happening. So I'll give it some time for Istio, the service mesh, to realize that the notebook is up. Or I'll debug it for a while. Let's debug it. I'm going to my namespace, looking at my pods. OK, invalid image name. If the image is invalid, then it shouldn't be showing as running. So this may actually be a bug. Image is shown OK. OK. Yep. I think I messed up. I messed up for including this. And I think it's going to work if I omit the HTTP part. But it shouldn't be showing as running. So this is definitely a bug. It should be running. I'll remove this. Goodbye, Webinar server. And start again. So Webinar 2 it is. I'm going to be using this custom image. I'm making sure to actually omit the HTTP part. OK. Again, I'll have a workspace, a data volume. And we do have a question if you want to address that while this is firing up. Of course. Sure. Swapna asks, why do we need to take a snapshot in the pipeline? It's not mandatory to take a snapshot in the pipeline. But it's best to take a snapshot so that you don't have the notebook operation messing up what the pipeline sees. So if it was just a shared directory, and you had the execution, that's a very good question, by the way, and you had the execution of your pipeline actually messing up the same volume, the same storage that the notebook or notebooks were seeing, then you'd have your work or your colleague's work messing up the result of the pipeline. So you can't just use an NFS share because it's like working the same Git directory if you have more than one person, right? What happens is you can't be sure about the result of the execution. The result will essentially be random as people are changing the input data. While working with Git, you can say, I started from this Git commit, so I can reproduce the bug. I can reproduce the result no matter what because the commute is immutable. It never changes. It's the same thing with a snapshot. If you've started from a snapshot, no matter what your colleagues or you do in a notebook, it won't affect the execution of a pipeline. So this is why we focus on working with snapshots for having reproducible ML. I hope this answers the question. So the pod is shown as running, but let's not believe it. Let's actually go see the pods. The pod is running. Okay, and we have an upstream bad to fix. So the notebook is up. I can connect to it. It's a nice Jupyter lab. And I can close these extra tabs. So this is my data folder. This is where I'll be storing the input data. I have a terminal. I can increase its size. And for the purposes of this demo, what I'm gonna do is use my cheat sheet again to actually download, so for this, a notebook that we'll use. So I won't have to type too many commands. So this is a notebook. And what it does is it has the commands to run individual steps embedded in cells. So what I've done so far is I've created a notebook and added a data volume. I'll now ingest data from an external source. So ingesting the data will be first downloading the actual source of the pipeline. This is the pipeline we will be running later on for training. I'll show you the code. Then I'm the stored here. Then we'll bring in the actual subset of the data set to use for training. So I'm cloning the git repository and then copying the subset of the data I need into my data directory. This is essentially simulating ingesting data from an outside data lake. HDFS, Snowflake, S3, the city of Chicago site, wherever I find my data, assembling it all into my storage. And then I'll have a reproducible snapshot of this storage so I can start training. So I copy the data into my data volume here. I can actually see the data, that's the data. And what we'll do now is move back to the presentation. I have ingested the data. I will compile the Kubeflow pipeline. I'll show you the code and compile it. This is essentially a description of the steps that are going to take place. And then I'm going to snapshot the notebook. So, compiling the pipeline creates a parable. We're using a newer version of Kubeflow pipelines which produces a few warnings. And let's take a look at the pipeline itself. So this is the pipeline specification, the training pipeline that we're going to use in a domain-specific language specified by Kubeflow pipelines. Let me increase the font size a bit. Okay, so this defines individual steps of the pipeline as components, containerized components, mentioning specific container images, taking arguments and producing outputs that live in volumes. So by snapshotting individual volumes, we can snapshot the state of the pipeline at each step. And why is this important? Because the most difficult thing when running a pipeline is essentially having insight at its execution. What happens if the pipeline fails? What happens if a step fails? If I'm in a notebook, I can experiment quickly. If a pipeline step fails, how can I go see what happened? The easiest way is take a snapshot of its input, clone it in a new notebook and go see exactly what happened. I can actually run the exact command that would be used to produce the data and see what fails. So at this point, I have my input data here. I have compiled my pipeline so I can now snapshot my notebook. For this, I'll move to rock. There's a bucket here. Buckets is where you store your files, your snapshots. I can create a new one. Let's call it webinar. This is gonna be where I'll store my snapshot. I'll take a snapshot of my Jupyter lab. It's gonna be a full snapshot of the whole lab. I could take a snapshot of a single dataset, a single volume. I'll snapshot this lab, this notebook server. I'll name my commit initial commit. I'll describe it as, I don't know, just ingested data, compiled pipeline. This will be the input to the pipeline run. And then snapshot it. So what happens is a new asynchronous task starts. This is a group of tasks actually. One task is to snapshot the data. The other task is to snapshot the workspace. Why do we snapshot both volumes, both disks? So we can be reproducible. You have seen multiple times that the result of execution is not just a function of input data. It's a function of the libraries you use as well. There's bugs that only appear with certain libraries. There's behavior, desired or undesired, that only appears with certain libraries. So to be reproducible, to be able to allow a colleague to see exactly what you did, you need to snapshot both your home folder, your workspace where you store your libraries, what you pick install, and your data. And that's what we did here in a few seconds. So I now have a snapshot that contains snapshots of both disks and combine them in a group. So if I go here, this is my snapshot less than a minute ago. It has a snapshot of the data and my workspace. And it has, let me go back here. I can see more information about it. It has all the information I gave when actually created the commit. So I have snapshotted my notebook and created a new snapshot. At this point, I'm ready to create a new pipeline. So what do I do? I go back to the node, I go back to good pipelines actually. This is central dashboard, pipelines. I have a few sample pipelines. I need to upload the compiled pipeline created here. To upload this, I first downloaded to my box and then uploaded to pipelines. This is kind of a cumbersome step and we can also automate it from within the notebook itself and upload it via a command line interface. So I'm now moving to pipelines, upload pipeline, choose a file, downloads, days file, and I call it the webinar pipeline, upload. This is it. So now we'll be creating a run of this pipeline. Run the pipeline. These are all the pipeline steps. First step is to create a volume. Why do we need to create a volume? So we can have a clone of the snapshot we start with and every other step depends on this volume because every other step manages this volume and creates files in this volume. Then we'll validate the data, preprocess, training, deploy, predict, analyze, and we have a final snapshot volume step so we can actually show you how the cycle ends. And we can start over with a new notebook. So I'm at this point, I'm about to create a run. Create a run, run name is going to be, I don't know, webinar run one, description, no description. I'm gonna use the default experiment. It's gonna be a one off run and now we need to specify parameters to this pipeline. One of the parameters is a rock URL which is what is the input data? Where do I start from? Let's choose. This pipeline will start from this dataset. This pipeline will start from my input data. The data I just snapshot it four minutes ago. Okay. And where is the output data going to go? Where is the snapshot going to go? Let's have it here in a new file named, I don't know, webinar snapshot. Okay. I now start the pipeline. So what happens is a new run is created. This is now the run time graph. We'll be seeing the steps exactly as they happen. And this will auto refresh. First step is to create a volume. So we now have a volume that is independent from the volume seen from the notebook. So I can use my terminal, go to data, mess up data, for example, you know, create a new file. This new file will not be part of the pipeline. The fact that I'm now changing the data does not have an impact to the execution on the execution of the pipeline because the pipeline runs from a clone of this data. And this answers the previous under these question. For example, let's touch this file. This will not appear at the end. Okay. Where were we? The pipeline is running. The validation step is complete. The processing runs. So let's take a look at the steps. Eventually the pipeline will snapshot a step into a new snapshot and we'll clone it if we have the time. So the steps are validating the data using TFDV. This is a TFX library. This step has already completed. It runs from a clone of the notebook data. Then we transform the data in a preprocessing step. We transform both the training part of the data set and the evaluation part of the data set, the input that we have kept so we can then evaluate the model. Then comes training, which will lead to a train model, which will also be stored inside the volume. Let's see, is this progressing? The processing is done. We now run training. Then comes the TFMA library, which is going to analyze the model and produce artifacts. We'll be seeing these artifacts in the snapshot of the pipeline that's going to be produced eventually. Prediction and serving. So training is happening and when training is done, we'll have prediction, serving, and we'll end up with a snapshot that we use in subsequent steps. So why are we doing this? We extend Kubeflow. So each component can store data in a vendor agnostic way in persistent volumes, in file systems that are mounted. We have extended, first we created a Jupyter Hub-based spawner with persistent volumes for 0.4. We replaced it with the native notebook manager that you saw in 0.5. And we have also extended the pipeline's domain-specific language so it can access persistent volumes and snapshots. If you look at the pipeline, I can actually show this. This is the pipeline, right? We specify that we want the output to be this volume. The volume is, let me find the volume. Every step is using the volume. This is the volume. So we created extensions to the pipeline's domain-specific language so we can create volumes as clones of existing snapshots and then we can use these volumes for each individual step. So each individual step uses this volume. And eventually, we'll also produce a snapshot. Let me find this as well. So we have extended the DSL so you can also produce snapshots of this volume with dependencies. So when the final steps of the pipeline are done, a snapshot of this volume will be produced as snapshot volume in the location-specific. So let me move back to the pipeline. Training is still happening. This is the most expensive part of the process. Let's give it some more time. Why use ROCK for your pipeline? Step one starts from a notebook. You snapshot it.  Step two starts from a clone of step one, make some changes, you snapshot it again. Step three, and so on. If this breaks, you can take this snapshot, examine it in a notebook and start over. Hybrid pipelines. You have a pipeline that you want to run in two locations. You want to run a few steps in location one. Then you want to run serving or training in location two. Step one, two and three run in location one. You snapshot step three. You move the snapshot using ROCK to another location. And at this point, you can run the second half of the pipeline on location two. So this is a hybrid pipeline. It's essentially a meta-pipeline of one step running in location one, another step running in location two, and ROCK bridging the gap by moving data from one location to another. If you want to reproduce a pipeline, you start from the same input data, rerun the pipeline in another location, and you end up with the same results. ROCK is in between, syncing data and state among all locations. Back to the pipeline, it's almost done. No, it is done. The snapshot step is done. Everything is completed successfully. I move to ROCK. This is my snapshot, created less than a minute ago. So the very last step, and then let's move to questions, is I start from this snapshot. So I go back to my notebook servers, create a new server. Let's call it webinar three. And this time I'm going to add a volume that's going to be existing, and we'll be using the snapshot I just created. I'm going to call it data again. Don't say notebook. So this notebook would now start from the snapshot that was created in the pipeline, and I'll be able to explore the output data of the pipeline in a notebook. Let's give it some more time. Behind the scenes, is essentially cloning the output data. Yes, George. Sorry, just reminding you you have four questions in chat there. Okay, let me connect to the notebook. I'll speed it up a bit. Go to data. See there's actually extra files being created. So this is not the input data. This is the data after the execution of the pipeline. Move to analysis, output display. This is the result, let me see. Should appear, it doesn't. Something may be broken here, or let me refresh. Anyway, this is the result of the analysis. I cannot show it right now, but I can see that this is the data that was produced by the pipeline. So this is more or less it. Let's move to questions, and then I'll continue. Sure, so, Jordan asks, how difficult is it to share snapshots in between users? Okay, that's a good question. We didn't cover it in this presentation. We've made it as easy as possible using Rock. You essentially publish a link to a published bucket to another service, we call the registry, and then any other user can subscribe to this link and follow your changes as they happen. What we mean by this is you go to your buckets, you publish them, this essentially gives you a link you can share with others, and others can then create a subscribed bucket and follow your changes. So whenever your pipeline runs and it creates a new snapshot or wherever, or whenever you create a new snapshot of your pipeline or of your notebook, any other follower sees it. So this is how you share snapshots across locations. Okay, next question. Can Hyper-V be used on Windows instead of VirtualBox? MiniCube does support Hyper-V. That's a good question. Right now, MiniCubeflow does not support Hyper-V. We are thinking about using it. We are focusing our efforts to moving MiniCubeflow so it can run on public clouds as well. So yes, Hyper-V is an option, but we're focusing on efforts to giving MiniCubeflow as a package product on a public cloud so you can run without having to worry about your RAM or CPU limitations. But this is feasible, yes. Okay, Joan asks, how enterprise-ready is Cubeflow slash rock stack in regards to identity management and being able to do audits? That's a good question. Cubeflow has authentication. Authorization is not there yet. It is there for notebooks. It's not there yet for pipelines, for hyperparameter tuning, for metadata handling. We are contributing actively to multi-user pipelines right now and we hope you have it as part of the next Cubeflow version. Please come in touch in Cubeflow Slack with us and we can talk more. Maybe you can also test an initial implementation that we presented today at the community meeting. That's what we're doing right now. To be enterprise-ready with full support for authorization, there's quite a few things to be done but we're moving towards this. Is the volume getting created is of type Cinder or some shared storage so that all stages can access it? That's a good question. Rock manages whatever primary storage you provided with. This demo was using local disks. So there is no Cinder or another API underneath. We just gave local disks on each individual node to be managed by rock. Rock carves individual volumes on set disks and you can be as fast as possible and have super low latency because of this. You just work over local disks. You can work over local NVMe. That's what we are targeting. And you can still withstand node failure because you can be snapshot in your notebooks once every five minutes, for example. The snapshot operation, as you saw, is very light and we snapshot thinly. This means we have an internal mechanism that tracks whatever blocks you have touched and only copies these to produce a new snapshot. I hope this answers the question. Please pull up if it doesn't. Yep, some nice, instead of copying a snapshot into a notebook volume, would it work if notebook itself is mounted to an object storage with shared folders, et cetera? I'm not sure I understand this question. I think you mean what would happen if I mounted my object storage as a volume in the notebook? That's what I understand that maybe I could use a user space mounting mechanism to access S3, for example, and S3 bucket as a volume in my notebook. This has two main issues. One is you lose reproducibility because it's like using a shared NFS share. While you're making changes in the bucket, you don't know what pipeline executions are actually using the bucket right now to read data from it. So you cannot, if you want to perform an exploration and run an experimental transform on the data, you can't do it because you don't know who else is using the data at this point in time. You actually have to copy all of the data to make sure you're working with a private copy, but then this becomes too expensive. So this is one thing. And the other thing is if you compare performance and latencies when working with a local volume compared to working with S3 mounted over a notebook, performance difference is abysmal. The latency is skyrocket. We have measured in the order of 800 megabytes a second bandwidth and microsecond latencies because we work with local MVMEs with an internal data path. This is orders of magnitude faster than going to an object store every time. Okay, Vishnu wants to know, what is the storage class used? How about on-prem? Okay, that's a good question. The storage class used is hours. Rock exposes itself as a storage class and we provision volumes using this storage class and rock underneath carves up volumes as you need them. For on-prem, rock is deployed as a demon set. It has its own operator. You spin up a custom CR, a customer resource. I think that's what we've done now as well. Rock cluster, yep, how about we change? Yep, this is it. This is a deployment of the rock cluster CRD in the rock namespace. And our operator, which is this, will take care of spinning up whatever pods are needed. We spin up a demon set on each individual node. So for on-prem deployments, we spin up a pod running rock on each individual node and this could manage your storage. Okay, we're running out of time. Since you're in there, do you support encrypted data at rest since you just happened to be in there? We can knock this question out. Yes, yes, yes. Okay, good. Definitely. The internal data path, we can set it up so it also encrypts data trust, yes. Okay, two more questions. How are large data sets trained as a scheduler work like a Spark scheduler? I'm not sure I understand this question. We've tested this with a few hundred gigabytes of data. We just carve up local disk and then we've made lots of optimizations to make sure we only touch the change blocks. So when you have a very big data set and only change a few parts of it, you can still snapshot it very efficiently. Okay, can this solution work with Spark, Hive and or Presto, I assume? If you ask, could you please elaborate? And that is our last question. I'm not going to type four. I assume it can as long as you can mount a persistent volume as long as you can access a file system from said jobs, then can be used with these solutions. But please come in touch with us in the Kubeflow Slack and then we can talk more about it because you're interested in various cases. Please join the Kubeflow Slack. There's a link to join the Kubeflow Slack on kubeflow.org. Please follow the link. I have actually have a slide about this. So this is how you run KFP on-prem with MiniKubeflow and ROG. This is a nice image. This is what we have for the future. GPU support and support for running public clouds. Please try it out. Join us on the Kubeflow Slack and let us know how it goes. Awesome. Thanks, thank you, Ellis, for a great presentation. Thank you. We are done with the questions and that's all the questions that we have time for today. Sorry, and thanks for joining us. The webinar recording and slides will be online later on today and we are looking forward to seeing you at a future CNCF webinar. And with that, thank you everybody and have a nice day. Thank you. Goodbye.


Good afternoon and welcome to today's energy seminar. Our speaker today is Alex Grant who's out of state and he's a principal at Jade Code. Jade Code partners if you read his bio you'll see this is I think his fourth clean tech startup and despite that he was able to be named an honoree and energy 30 under 30 at Ford's magazine last year which is pretty impressive. I should at this point acknowledge the person who suggested the last two speakers and this one Dennis Morjani so I know Alex knows him. He was a kind of early mid-career person in the class last year and has helped us a lot with you know refining topics and picking good speakers and so far we've been extremely successful. So besides getting into into Forbes in this talk Alex is going to describe another dimension of the kind of battery ecosystem that you've heard about last week a little bit and if you were here last spring from another speaker on supply chain logistics. For but I understand Alex is a innovator in the extraction which I think means mining and processing of lithium and concerned about doing that in an environmentally sustainable way but also as we've heard that could have implications for international supply chain and international security issues regarding large-scale deployment particularly of lithium ion batteries. So I'm quite pleased to announce Alex today as our speaker kind of fits in nicely. We will transition from our four clean tech entrepreneur talks at the beginning of quarter more to policy and systems for the next few seminars. So with that said Alex take it away. Well thanks so much for the intro. Somehow tell me if you can't hear me correctly. But yeah so just to start off the reason why I am not there in person is extremely characteristic which is that I am working on a lithium brine exploration project in central Nevada about eight hours drive from the Bay Area and a bunch of schedules got mixed up so I was going to come back to San Francisco yesterday but had to stay an extra two days unfortunately. So that's for the audience members who think you're in the casino in Las Vegas. Good to hear your heart at work. Oh no yeah definitely not at the casino. I'm in tone upon Nevada. I don't know if anyone's ever been there. It is a very small town but it's the biggest town between Vegas and Reno so it's got a supercharger which is kind of interesting. But yeah so I have a very good and characteristic excuse. And yeah so I regret not being able to kind of spend time with you guys in meat space but if anyone wants to chat later on I'm very easily found on the Internet so feel free to reach out. Always happy to make new friends and learn about what other people are working on. So yeah so that's the frame. I'm going to share a couple slides with you today that kind of summarizes a little bit about how I think about lithium ion battery supply chain impact mitigation and how to really make sure that the energy transition as worth it really as much as possible to make sure that the CO2 emissions in that new paradigm that we're building right now are just as low as humanly possible and we're able to achieve the deepest decarbonization that the laws of physics allow without compromising our high quality of life since in the next decade. So just really there you go. So just really just on my background real quick so I'm from Canada. I did my undergrad at McGill at Montreal studied chemical engineering and philosophy in undergrad mostly focused on wastewater treatment and water chemistry research at universities and at startups in Montreal. Went for PhD in chemical engineering at Northwestern in Chicago. I was kind of tired of cleaning up other people's messes doing wastewater treatment research and wanted to help build a new energy system. So I went to study heterogeneous catalysis for CO2 conversion to methanol as part of a solar fuels concept and when I was there I built a high pressure high temperature reactor designed and built a gas chromatography machine like really super hands-on hard core kind of mini pilot development and hated it. So I was really not well suited for academia. Never going to finish my PhD I don't think and you know wasn't particularly happy in the Midwest. So I ended up leaving my PhD with a master's and moved to the Bay Area to co-found a technology startup called Latinx Solutions. So I started working on this company in early 2016 so about six years ago now kind of well to think and that was kind of you know another nail in the coffin of my PhD was the opportunity to go help build this company. So Lilac is a lithium extraction technology company that uses kind of novel ceramic ion exchange materials to take lithium out of unconventional natural resources like low-grade brines like geothermal brines which you may have heard about in the Salton Sea in Southern California for example and oil fuel brines and other types of resources to make lithium chemicals faster with lower land footprint. And I was full-time at Lattoc for about a year and a half kind of covered in salt building mini pilot rigs similar to what I was doing in grad school fully automated kind of robotic ion exchange systems to test our technology on different types of brine resources left just under three years ago in 2019 and kind of accidentally stayed in lithium one of our customers just kind of wouldn't let me leave. So I ended up starting my own consulting company which is Jacob Partners which is still kind of my you know official title is principle of my consulting company and I've been consulting on lithium natural resource projects all around the world for the last three years and trying to understand really really just ultimately trying to understand how can we make lithium chemicals faster from natural resources and how do we make them better with lower CO2 emissions lower freshwater use lower lower land footprint etc that's really been my my mission the last couple years and a big part of what I've been doing the last couple years which we're going to talk a bit about today is working with a company called Minvaro in London doing life cycle assessment so two years ago we actually published the first ever life cycle assessment of lithium hydroxide manufacturing from natural resources no one had ever published numbers on it before and that became kind of a benchmark for the industry and we've since done probably between 20 to 30 different natural resource project LCA's for making lithium chemicals so we've built up sorry my mom can sort of call me we've built up the the best database that anyone has ever had anywhere on the CO2 emissions involved with making lithium chemicals from natural resources every single type of resource every process pathway every technology you can imagine so it's been a heck of a lot of fun building that building that company as well and kind of along along the side when I when I have time I really like doing independent research and I really like writing so on my website if you guys ever want to go check it out I've published a ton of articles on different topics that I think are interesting and in those publications really what I'm doing is like kind of working through topics that I think are are true and are interesting or relevant to the energy transition in the lithium natural resource space mostly also what I do is make memes so if you guys fall if anyone follows me on twitter you probably saw my memes already and if you are in the market for more spicy lithium and hard tech memes like I'm a one-stop shop so you know just you know follow me on twitter and you'll you'll get plugged in anyways this is why we're here isn't it so you know september 2020 was not a very nice day in in the Bay Area I'm really I'm I've been working on kind of energy and climate and water related projects for a decade like my entire adult life and in the last you know probably two or three years two things have happened one acute severe kind of weather phenomena have really ramped up in a big way that I don't think anyone was expecting to happen so soon globally of course I'm sure I'm preaching to the choir here but but two people also started really caring about climate change like two years ago like the the the bankers decided we had to solve climate change in 2020 for whatever reason so now this is like an incredibly exciting space and anyone who was already working on it is probably you know up to their up to their eyeballs and work the they're doing um why you know why is it why should we solve climate change you know climate change I think of climate change as um as as kind of a great filter kind of phenomenon um you know climate change could really set us back hundreds or you know I mean dare I say thousands of years of in terms of progress on our planet as a species and I think that's really terrifying um so um so yeah so this is why this is why it's so important to be working on these types of questions and really trying to like solve actual problems I I see you know kind of just to start talking about the thing that I'm actually talking about today um I see working on decarbonization of the battery supply chain as really kind of like a like a 22nd century type problem like you know we kind of just got the world over over the the hump of electrifying transport and you know try to now make a lot of batteries um I use the word try because it is of course very hard but um but but going all the way back up to the mine and pulling out all the fossil fuel infrastructure all the way back up to mine up up to the mine is um is is is really like so much harder than building a battery pilot plant in in Fremont or whatever have you um like it is it is such a it is such a gigantic global like multi stakeholder task um and it's it it's really it's really just like a subset of decarbonizing industry in general right and decarbonizing kind of all of industry is really important because it underpins our high quality of lives right like it's us rich people in rich countries who are mostly driving climate change um so we need to be able to like produce the things that that we need whether it's materials or energy or whatever have you in a way that that isn't isn't making this happen anymore um so anyways this is a quote I stole from JB's trouble um from a talk he gave uh on campus in 2020 I thought it was a great quote so he said the embodied emissions of the materials that used to make batteries are significant and need to be understood if you were to power all your ED manufacturing using coal it would make no difference at the end of the day one thing we did when we built the gigafactory was to make it all electric there's literally no natural gas line so there's little to no local emissions at that factory when we when you weave natural gas to your facility it makes it much harder than chase it out the emissions all the way back up to the minor certificate too when we look at terawatt hour scales of production we need to make sure we are not creating unintended consequences as we go through the industrial shift that is why we are in the situation in the first place and we need to rapidly remediate that so in this presentation I'm going to use this quote as kind of a template for jumping off of um on each of these five different points um so so why why do the embodied emissions of value metals matter um they they are quantitatively objectively important um if you run an lca on a on a lithium-ion battery so so most lca's I've seen of a lithium-ion cell compared or kind of an electric vehicle on whatever grid compared to an internal combustion engine shows something like 50 to 75 percent decarbonization for the EV versus an ice vehicle um that's honestly like not as good as you would expect you you know I mean maybe other people have different expectations but um we're it's it's it's going to be a heck of a lot of work to to kind of overhaul all of our trans our transport infrastructure and it seems like kind of um kind of sad to be doing it for only you know cutting the CO2 emissions of transport by half um part of part of part of why the emissions of the kind of EV from from you know cradle cradle to grave or kind of wherever you want to cut off your accounting um is due to the the materials used to make the battery um and of course other materials in the car like aluminum and stuff um and something something we have found doing life cycle assessment of battery metals and enviro is that um the the embodied CO2 emissions of making these chemicals from different types of resources in different ways with different types of energy inputs can vary tremendously so there are lithium hydroxide products you can buy today that have kind of a you know nine x difference in their embodied CO2 emissions um that's massive so those those high embodied emission products kind of you know hurt the um the CO2 the overall CO2 mitigation opportunity of the electric vehicle um because we're emitting CO so much CO2 upstream and shifting the environmental burden if you will um so if you were to use like just kind of dirty you know honestly quite typically Chinese battery metals um versus kind of cleaner lower carbon ones the delta in a an mc811 cell of the embodied CO2 emissions would be 55 kilos of CO2 per kilowatt hour um at three terawatt hours per year that's equal to the the CO2 emissions of um Ireland Switzerland and Portugal combined and that's just the difference in the embodied CO2 emissions of battery metals not even the aluminum in the car um so that's really significant um and for context you know you might not have any data points in your mind about like what is 55 kilos of CO2 per kilowatt or high or low um there are battery manufacturers who say that they're going to have an embodied emissions of 10 kilos of CO2 per kilowatt hour in their batteries by 2030 um that's basically unachievable and it's definitely not achievable if they don't decarbonize their supply chains and there's there's kind of a um really interesting conclusion here that I've I've kind of more operationalized over the last couple of years which is to realize that really the power to choose those battery metals that goes into the EVs that the rest of us drive is in the hands of a very small group of people just a couple companies um so um you know helping people understand this is really important for the people making those decisions of what metals to buy this is okay so quote quote sub subsection number two um if you were to power all your EV manufacturing using coal it'd make no difference at the end of the day um this uh you know this this this I can't I can't think of any word besides meme because I call everything a meme now um this this meme here um is from a sustainability report of a Chinese lithium company um I find it very funny that uh if you compare 2018 to 2019 uh embodied emissions of their lithium carbonate equivalent lithium chemical product they um they reduced it 0.9 percent um and and the the bars look like you know at least five percent maybe almost 10 percent of the delta um and and this this meme was originally green and I removed the color because I didn't think it was appropriate to use the color green um for to talk about you know how much how much coal and how much energy that you seem to make the lithium chemical um the point I'm trying to make is that there's an absolute like massive quantity of re-washing happening in the space um and it's it's pretty troubling um does it go all the way to the top I don't know but um you know in in the lithium ion batteries supply chain there there's historically been a tremendous amount of kind of deception and angry washing so I guess I'm kind of used to it now but um you know I just find it really important to like make sure we are actually talking about things like accurately right that's been a big part of a lot of my publications and a lot of my work um you know Tesla does not buy lithium hydroxide from Australia like hasn't bought one gram of lithium hydroxide from Australia you know it all comes from China um and it all comes from that lithium producer in China who's burning quite a bit of coal um to make lithium hydroxide um I'll skip this because I don't want to go into it but basically I think that emphasis on minimizing CO2 emissions and decarbonization is that kind of has geopolitical um you know kind of tangents associated with it but you know but that's another day I'm sorry for another day um so the Nevada Gigafactory has no natural gas line once you once you start assuming you can use fossil fuels in a in a battery metal production process um it gets a lot harder to to decarbonize it if you're kind of walking yourself in right um so so it's really it's really critical that people who are who are designing and operating battery metal plants like lithium chemical plants for example like mines chemical plants etc um need to be very deliberate about the technical decisions they make to achieve decarbonization goals it's easy it's easy enough to to greenwash and lie in sustainability reports um but eventually you know I hope I hope at least that it doesn't work forever um and I and I hope at least that investors start to kind of take it more seriously and and I and I and I also hope that you know eventually regulators take it more seriously as kind of um as a kind of a securities violation to be honest right um but really really like instead of getting stuck in a by violating securities laws and having a whole you know a heck of a lot of problems for management um it's just easier to to instruct the engineers to try to minimize their fossil fuel use in their in their operation um and and and really make deliberate decisions um to to do that right um just one example of kind of now now I guess in a modern sense kind of a easy way to decarbonize heat for example is to use mechanical vapor recompression for steam and for thermal processing um which which is usual for for a range of thermal processes for converting lithium chemicals and making other battery metals because really really what it does is it makes sense to that instead of burning cold or natural gas to generate all of the steam that you would use in a in a chemical process you're now probably 90 to 95 percent powering steam generation with electricity which can be much more easily decarbonized than a fossil fuel the emissions all the way back up to the minor significant um the mining industry actually knows this so the mining industry takes decarbonization relatively seriously in a way that I don't think oil and gas does um and the reason why is because it is it is fundamentally possible to decarbonize mining so mining does not need to emit co2 in the process of extracting minerals from the earth and and refining them um a great a great example is electric mining trucks so a lot of a lot of underground mining is already electric um so it's not at all a new idea um to to electrify you know transport of ore in a mine um and um you know uh kind of some because some types of mining operations that currently are mostly powered by diesel will eventually be electrified whether that's through green hydrogen or or directly batteries and I think it'd be pretty cool if it was directly with batteries and you're mining lithium with a lithium-mine battery-powered truck it'd be kind of like the lithium was mining itself um so this this stuff's happening you know it could it could happen a lot faster um but um I um I like I just want to make the point that like you know the mining industry is taking this relatively seriously but there's still there's still a lot of work to do and you really have to you have really have to choose decarbonization if you if you want it so what happens when what happens when we scale up the battery industry um and um and and it's dependent on on coal or you know kind of legacy processes that are less efficient or whatever have you if you think about if you think about that delta co2 that I mentioned earlier between kind of a high embodied emissions bill of materials versus a low embodied emissions bill of materials um that that delta is almost as is basically as large as the entire rest of the uh of the co2 emissions associated with making the battery so it's it's huge just kind of purchasing decisions can make a huge difference um I um I've been kind of like screaming this from the Rooftopster the last two years and trying to kind of convince battery and EV industry stakeholders to to take this seriously um part of the issue here that I've noticed over the last couple years is that EV companies and battery companies um are not mining companies and they don't understand mining and chemical processing for the most part um and what that means is that kind of the EV industry and the battery industry which has actually relied on LCA for probably at least a decade to understand the co2 mitigation impact of making electric vehicles compared to ice vehicles has been dramatically underestimating the embodied emissions of raw materials simply because they didn't know kind of what data to collect really um and they didn't know how to kind of do diligence on their suppliers to understand if numbers were right or not um so the the result is that most database values for doing electrical assessment on batteries are are are extremely inaccurate um but um the kind of database values aside every single battery bill of materials will be different right so if I'm uh if I'm a tesla or volkswagen I could theoretically have completely different suppliers of lithium hydroxide and that hydroxide can be made in completely different ways so it's important to kind of take like an asset by asset supplier by supplier look at and value emissions and um and and kind of make sure that that priority to decarbonize and track emissions kind of goes goes all the way back up to the mine um so yeah so it's kind of shocking but like it it almost like doesn't matter if you completely decarbonize or completely carbon neutralized a battery manufacturing process um or even or even cell assembly is not even that important if you if you don't pay a lot of attention to the sourcing of your raw materials it gets completely you know dwarfed by the embodied emissions of those materials um yeah so I this was this was more geared towards the the the lithium industry but um really the the place that this comment was coming from um you know why why would um why would an EV company care about um you know lithium mining cfg emissions um the reason why the reason why I talk about this pretty often is because in the mining industry people used to not take upstream scope three very seriously so if I was using sodium hydroxide in a process for example I would not report it in my co2 emissions accounting um but that's um that's not really a complete way to look at things like you have you have to understand your upstream scope three at least um in in in kind of co2 accounting um processes um really quickly a couple little case studies so there are there are reasons to think that the embodied co2 emissions are making lithium chemicals for examples actually going up so in the last couple years there's just been a shift to making lithium hydroxide from from australian hard rock process in china and that is um that is typically higher co2 intense than um than chilean brine evaporation process for example but um but there's there's so there's there's reasons to think that that those embodied emissions numbers can continue going up because um because grade and impurity profiles of new natural resources that we're tapping into to make lithium chemicals are are kind of declining right as you develop and produce a lot of lithium from natural resource assets that are that are already existing you obviously can't expand them dramatically you have to tap into new resources um so if you have a kind of lower grade or higher impurity concentrations you need to plow more energy and more materials into the system to to extract that lithium and make a lithium chemical but um but there's lots of interesting ideas of how to reduce emissions as well geothermal lithium is one so this this is the idea of building a geothermal energy plant kind of combined and co-developed with lithium extraction to make lithium chemicals using direct lithium extraction process um so that's um those are super interesting projects the the two main kind of hubs for geothermal lithium historically has been southern california and the upper end valley in germany in france um but uh but there's activity going on all around the world on geothermal lithium um and then a second little mini case study to share with you is um the sedimentary clay natural resources that you can find all the way from the border of oregon down all the way into mexico all across nevada there are probably like hundreds of millions of tons of lithium carbonate equivalent stored in clays um across the west united states um there's a couple there's a couple projects that are that are in later stages of development like facar pass in northern nevada and ryleigh ridge in central nevada kind of close to where i am right now um these these these sedimentary minerals are processed using mostly direct sulfuric acid bleaching and there's there's there's one little issue when you do that which is that sedimentary clay minerals tend to have a lot of carbonate minerals kind of mixed in with them so if those carbonates are not removed you end up liberating quite a bit of co2 when you acidify the carbonates mixed in with that mineral this this is something i'm a little bit nervous about i i'm actually not working on many sedimentary clay lithium projects anymore because i i find this this problem um very challenging and semi semi-intractable especially for lower grade resources um so um this is you know this this really like underlines for me the importance of kind of understanding drivers of CO2 emissions in a lithium chemical extraction process and using life cycle assessment to um to make sure what really matters because one of these projects could be acidifying a whole bunch of carbonates and releasing you know 30 kilos of CO2 per kilo of lithium carbonate but but uh but have electric mining trucks and and call themselves carbon neutral or something like this um that would be of course completely deceptive um a lot of coal is still used in making lithium chemicals um i think it would be cool if we stopped using coal and making in lithium chemicals and all battery metals for that matter um i uh i i kind of want i want someone to like start a a movement or something to weed coal out of the battery supply chain because i think it's absolutely bonkers that this is kind of still going on in 2022 um so just to summarize uh you know if we're gonna kind of get past the great filter and you know be a more advanced alien species and and develop kind of a 22nd century um battery metal supply chain uh everyone has kind of a part to play so you know i've talked a lot about kind of the role that engineers and technologists have here um because i because i am one of those people um uh so you know won't go too much to detail there but um but i would i would i would also kind of really emphasize the importance of of investors taking this seriously and kind of investing some time and energy and understanding embodied emissions of battery metals um i think it would be fabulous if uh if if there were some kind of moratorium in investing in in high cf2 intense battery metals um i know that's totally impossible i mean no one's gonna do that but um but but obviously you know buyers of these chemicals do have some sway over over which chemicals end up there in their battery um and um and yeah i just i just think like kind of all all of these from stakeholders across the the battery supply chain have to be using life cycle assessment to understand the embodied CO2 emissions of their product um and make the right decisions about about decarbonization to to achieve real decarbonization um so yeah that's um you know that's that's what i would say about that so um so yeah that's it um that's all i have to say i uh welcome your questions um if anyone wants to chat later on um you know feel free to reach out you have my email here um i'm very reachable i might i might not have time to chat um but um you know uh feel free feel free to reach out and hopefully we can link up at some point so uh thanks alex for that great um talk regarding your work actually find myself wondering exactly what you're doing there but you probably can't talk about about that and about it as a classic rocker your theme could be could be uh the old who song don't be fooled again is that kind of if i get your drift here don't think you're yeah you're solving a problem that you might not actually be solving i um yeah no what am i what am i doing here so i'm i'm involved with a lithium brine exploration project so um you know i'm an engineer i've worked on like tech startups like living in a city kind of person um i i never thought i would be involved with such an early stage project like this like literally the problem set that we're working on right now is shooting like sound waves into the earth to figure out where salty water might be and kind of in what thickness is and and and how extensive in kind of x y and then how many different aquifers you can find all the way down and then um in in two weeks from now probably on valentine's day we're gonna we're gonna poke a hole and um that will be the first time that we'll know the actual geochemistry kind of as you go down into the earth which is really what matters you can take surface samples easy peasy but um we're gonna we're gonna have to drill about a kilometer deep to to understand you know where lithium could be um kind of going all the way down and start to understand how much lithium there could be in this particular um exploration target and to understand if we have higher enough grade to extract it economically and build an actual operation that could make lithium chemicals um uh that could go in straight into a cathode uh for a lithium battery so um so yeah it's it's it's crazy it's literally crazy you know like this company i'm involved with is investing hundreds of thousands of dollars millions of dollars actually to drill holes in a salty pile of mud to see if there is lithium there we don't even know how much lithium is there we don't even know if it's a project yet so um so you know i i think it's a lot of fun i like being out in the desert i think it's really fun being out here um but um but it is it is a truly like that like kind of unprecedented like exercise like not very many people have worked on anything like that before where it's so risky um so bonkers i mean it's just it's just yeah totally kind of crazy projects um but i find that a lot of fun so if anyone has questions i'll just bring you the microphone so we can hear you so i was intrigued i guess i see the solution to this problem being entrepreneurs like you on the one hand and other players um particularly struck the last few years as you indicated by the buy-in from the financial um community uh finance banking insurance and whatnot and i know even there are you probably know this um really recently rolled up investment funds that have a sustainability in that kind of way of valuing investments in their portfolio so the question for me is are you talking to those guys i could see them trying to do kind of um quick quick uh simple life cycle but not going as deeply to what extent are you communicating with that community already um so the uh the investor sphere is vast right it goes way beyond silicon valley it goes way beyond new york um there are so many people i could be talking to who i'll never be able to talk to who control significant quantities of capital um which is partly why i publish actually really is to try to scale up my my message of my work right um but um am i talking to them yes a couple so like you know one of my one of my lca clients uh is is building a um a spodium mineral mine in brazil and um she she has uh black rock is i think her her largest investor in in her company building the spodium mine to make to make spodium concentrate for conversion to lithium hydroxide and um black rock black rock has solicited co2 emission data from my client um and i've helped kind of you know essentially fill in the excel sheet of what black rock wants right so i've had some experience um kind of feeding data uh kind of down the chain to to investors um to help them understand kind of what my what kind of what my client's co2 profiles look like um i've got i've got on my to-do list for tomorrow or the day after um filling out a kind of a co2 emission profile for for a major european auto manufacturer um so um so i i i have been in contact with a couple of really i mean globally relevant significant names um and um i can tell you that really the most sophisticated folks are mostly asking like pretty much the right questions um so so most of the major european auto manufacturers have internal lifecycle assessment teams already they do they do they build their own lca models like it's like a a little academic department within within the the renause and the Volkswagen's of the world um and that's actually like i mean i find really interesting so i've been helping those types of folks better understand battery metal processing because well why we've been so successful doing life cycle assessment in this space is that uh is because i'm a chemical engineer we have mining engineers on staff we have geologists on staff like we're deeply technical in natural resources sector um and and kind of legacy lca practitioners don't have that so they haven't been able to um to catch up with us i guess um so um yeah sorry long-winded answer but but yes you know i am telling to those types of folks um i uh i wish they would um i wish well i mean i don't know i've been kind of surprised on the upside in the last like year two about the about capital allocation decisions to low carbon projects i have to admit in lithium at least um so i i don't know i'm pretty excited and optimistic to be honest but um but i i'm sure there's always more kind of permeation of these ideas into into the investors sphere that would be helpful um and um and yeah actually you know you know what one interesting comment to share before I stop talking um something i've something i've noticed people chatting about in in kind of in these topics of like natural resource decarbonization and and kind of investor kind of responsibility to investors of reporting sea of emissions and such is um poor performing assets like coal mines and things like that are being shifted into the private sector so private companies are buying coal mines and um and kind of natural resource assets that have environmental issues and then just continuing to operate them because they're not accountable to um to as many investors as public companies are and they're they're not as scrutinized so that's something i'm a little bit worried about actually um just to kind of hamper down my optimism um and then those private companies don't care about optics or having a cute sustainability report like they are just trying to make money um so yeah that's that's something to like kind of keep keep an eye on i guess any questions in the room here so i'll try another one so you started with this book with jayby strudel and teslan you got back into tesla know that tesla's off about a trillion dollars in market cap are you still working with him and if so what's the thing i actually have a friend who did with jb in the early eb versus internal combustion engine life cycle comparison and that was pretty advanced in the day maybe five seven years ago but not as advanced yeah talking about here so are you continuing to work with people like that um so i'm not no i just stole jb's quote that's a great quote i haven't met jb in person yet i've had a couple interactions with him um but um but i uh i don't know can i i haven't signed an nd i guess but um you know i was i was up at their offices like two two or three days ago chatting about lca just casually um so um you know i'm not i'm not working with them directly but i i understand that kind of sophisticated folks like redwood and and tesla and and the volkspeckants of the world are are taking life cycle assessment very seriously um principally because they actually do care about decarbonization for the most part but um it's it's also gonna it's also very soon going to be the law of the land in europe to have much more sophisticated carbon emission disclosures on on products sold in the european union um so they're going to be forced to take life cycle assessment very seriously um so i'm not yeah i'm less i'm kind of almost now i'm now over the last two years less worried about the teslas and the redwoods and the volkspeckants of the world um because i think i think those types of folks are doing a pretty solid job on on tracking their supply chain c of two emissions so yeah europe's europe's been big on this particularly on the finance issue you probably know the people running the european climate investor disclosure process were the same people that uk government hired to analyze the financial crisis in 08 and 09 and they were asked at the end of that what's the next big thing and this is what they picked essentially a financial disclosure regarding both uh transition risk and physical risk to companies that people are investing in i think i think you may again might might find some good allies there as well well i i just i think greenwashing should be a securities violation right yeah a lot a lie to investors is a lie to investors regardless of what it's about right no and especially if it's related to climate and environmental impacts that a company incurs in making its product investors need accurate information on that to make decisions um because of because of near future compliance reasons and you know just you know morals or whatever have you um so oh you know i mean oil and gas companies are having the hardest time of anyone right um they're they're really squirming but um yeah no i just uh event the amount of greenwashing in the world is is really incredible you know like you see things like um you see things like blue hydrogen right like i'm i'm very i'm very anti blue hydrogen i think it's an absolute like scam and just for context if anyone doesn't know like blue hydrogen so 95 percent of hydrogen today is made from fossil fuels right from from coal or natural gas um and um blue hydrogen is this idea of making hydrogen from natural gas and then capturing the ceo tune storing it in the subsurface um i see blue hydrogen as an opportunity simply to kind of retain the value of natural gas assets like natural resource assets in the share prices of oil and gas companies um that is by far the best explanation of why anyone would want to produce blue hydrogen um and um i'm i'm just like shocked by how how often blue hydrogen is kind of gets away with calling itself clean and you know there's mckinsey reports where you know blue hydrogen is labeled as as clean hydrogen and kind of lumps with green hydrogen made from water and renewables and um it's a crime really it's a crime because the ceo tune missions in um that the would result from making hydrogen that could be captured and sequestered um are not tracked today right so today when they make hydrogen they're just dumping it all in the atmosphere and um and the methane emissions in the in the well field of conducting that natural gas from the subsurface to the blue hydrogen plant or whatever plant you know whatever you're doing with the natural gas are also not really tracks very well so there's there's just no there's no there's no compliance reason to do blue hydrogen right um and blue hydrogen is just one example i mean there's lots of examples like that where it's it's so obvious that someone's gonna someone's gonna call a product clean or green and that's gonna end up being a disaster in terms of ceo tune missions um and a lot of folks are kind of like sleek walking into this um because they because they don't really understand like the drivers of ceo tune missions and they and they kind of overtrust companies who who just make kind of simple declarations right so yeah lots of lots of emotions about um about green washing and um and you know those topics um i take it very personally really yeah well hang in there i do i do think there may be some win wins in here but if we don't have the right accounting framework and full transparency we're not going to even see those so i do agree with that part of it there's no kind of simple way to do it it's it's equally hard to what you're doing to make sure that the bit that you allow is actually going to get you to a better place through the big transition yeah so any other comments in the room i think we can now adjoin oh yes sir stand one sec i'm trying to do it better for decarbonizing for decarbonizing the the like like lithium chemicals or decarbonizing what uh just like the battery making process in general i'm thinking more like specifically towards like EV EV car like EV cars well um i think my clients are doing things right obviously but i'm biased of course one uh can you say who they are john's asking can you say specifically i'm trying to think if any of any of my my my cathode or cell lca clients are public um one one that's public is um is named frayer uh so they're building a uh they're building an lfp they're building a cell plant in norway um which has a very clean grid so they're off to a good start but um but yeah so we're helping them with with lca on their on their cell manufacturing process um that's just one of probably five different cathode or cell uh companies that were where he was at the moment that are they're all that are otherwise confidential um i will um you know i hinted at one person uh one company that was making kind of tenuous claims and has um kind of tenuous expectations for their body emissions um and i might as well just name them because there's i i don't really lose anything in in not in in any of them um north vault in in sweden um north north vault is is the company that um while they're planning to make to make cells for uh for for european auto manufacturers kind of in the next five to ten years and um north vault has published a target of total embodied emissions of 10 kilos of co2 per kilowatt hour of storage in their cell um i was showing you like i was showing you kind of some of the orders of magnitude of of kind of kilos of co2 per kilowatt hour of raw materials versus cell manufacturing etc and um you know based on those numbers where probably the average right now is something like 75 to 100 kilos of co2 per kilowatt hour um for lithium ion battery um you know that's you know almost an order of magnitude higher than than their target and um i i'm not aware of of north vault publishing relationships with any low carbon battery metal producers or developers for that matter um so so today i've seen no evidence that north vault is doing anything to to promote production of low carbon battery metals and um and i i'm you know i just find that disappointing to be honest like i'm sure i'm sure they've kind of got their heads on right you know i'm sure i'm sure that they you know i they have an lca team so that i i know that they they must know roughly at least the kind of drivers of emissions and making their product but um but what i'm not seeing is their procurement team kind of putting the pedals of the metal and like you know signing actual contracts to buy actual low carbon metals and chemicals um so um so yeah that's just kind of like a disappointment name i guess um so who's you know you asked me who's doing it right um i think i think my clients i think my clients are doing things right um so you know hired me to do your lca's but um i'm joking joking um i you know if i had to like kind of go through the list i don't know if that's interesting to you guys maybe that's excessive but like most of them are like kind of doing it right but like not really um but most of them don't as i said a couple minutes ago like most of them don't fully really understand the embodied emissions of their of their cell bill of materials so it's impossible to do it right if you don't understand that um so um yeah i don't i don't know if anyone's really doing it like perfectly or you know kind of up to my expectations and that's okay um hopefully hopefully some some more of them kind of get there in the next couple years but uh yeah so it's a bit it's a it's a bit of a mess um okay alex at this point we're just about that time here so we'll transition to the up close personal student session thanks very much for sharing your story with you i think it's a very very important one going forward with the clean clean energy transition thanks again yeah thanks for having me uh yeah sorry again couldn't be there you know stuck out here in the desert but um you know again if anyone ever wants to chat you know feel free to reach out so you'll get a few right now cool okay bye