Welcome back to another OpenShift Commons briefing, and today, Chris is back with us. You've seen him before when we've also talked about Crossplain, and that was a few months back. And Crossplain has applied for graduation from the CNCF, so that's exciting news as well. And Chris just had a blog post go out that I will send to Chris for those of you watching on all the OpenShift TV channels. You can go open up his blog post as well, and I will let Chris go ahead and introduce himself and talk about, well, I will just let you go, Chris. Thank you. Thank you, Karina. So yeah, hi everyone. My name is Chris. Today we're going to be talking a little bit about Crossplain providers, as well as their recent OLM repackaging effort from the Office of the CTO. So who am I? First things first. So I'm a software engineer in the Red Hat Office of the CTO, more specifically in the emerging technologies team. I'm also a maintainer on the Crossplain sub-projects for provider AWS and the provider in cluster. I'm also a six storage member and maintainer on some sub-projects. So my Twitter is, is Chris Tao underscore if you want to follow me there. My GitHub is just Chris Tao. So our agenda at a high level, we're basically going to go over the what, where, when, why, who for Crossplain providers, as well as then the how for OLM repackaging. In between there, we're going to go over the main components to help provide some more context for folks who aren't familiar with Crossplain. So let's start with the what. So what is Crossplain? Let's quickly go over the project data at a high level. So the first main pillar for Crossplain is really provisioning. Crossplain allows you to provision cloud resources from within your Kubernetes cluster. It also allows you to manage the entire lifecycle. So not only creating, you know, cloud resources, but also then updating those and, you know, eventually deleting them if you need to. And lastly, the specific providers expose information on these external resources, right? So you can not only create but also observe these resources and observe their state over time. Right. So something happens to your, your RDS postgres instance. You will be notified on the custom resource through events. And so from the Crossplain docs, we have that providers are packages that enable Crossplain to provision infrastructure on an external service. Right. So again, the main goal of a provider is really to provision things outside of your Kubernetes cluster or your OpenShift cluster. So these providers bring CRDs or managed resources that map one to one with external infrastructure resources, as well as controllers to manage the lifecycle of these resources. So something key to note in that sentence is this notion of mapping, you know, one to one of providers really being focused on fidelity for these external resources. This is going to be a common theme. But what are providers really, right? Definitions are all well and good, but what does it mean? So providers are similar to operators, right? But the main difference is that, you know, for operators, we utilize OLM for providers. We use the Crossplain provider to handle installation and all other management, right? So updating, removing providers, that's all handled by the Crossplain operator. And so providers use a lot of the same tooling as operators, such as QBuilder, controller runtime, and controller tools. For this reason, a lot of folks in the community often consider providers to be an opinionated form of an operator. But again, the main difference is that providers are designed to reference some external resource, right? So for a developer, the really nice thing is you don't have to worry about our back deployment and all the bootstrapping since there already exists a lot of tooling for new developers and folks that want to create new providers. So the where. So where are providers located, right? So Crossplain providers are open source and, you know, most will be available on GitHub under the Crossplain or Crossplain contrib organizations. So some examples, we already talked about the provider AWS and provider and cluster. There's also, you know, the provider SQL, which allows you to orchestrate SQL servers by creating users, grants, rules, et cetera, all your favorite SQL resources. As well as the provider helm, which allows you to manage and deploy helm charts using a custom resource. And lastly, the provider AWS that we talked about. But there's also many others that are not listed here. My personal favorite is the provider pizza, which allows you to order pizza from Domino's using custom resources. And so they're all available under the Crossplain orgs. So the when. So when does it make sense to create a provider, right? When should you create a provider versus when should you use some other solution, right? So it really makes sense to create a provider if you're consuming external resources, right? We've mentioned this before, but if you have some external resource and you want to manage the CRUD lifecycle for these resources, a provider is a really good fit. Similarly, there's a really high emphasis on high fidelity, right? A really important focus. And so you create a provider if there are resources that you want to use are well-defined and granular, right? That doesn't really make sense to create a provider for managing abstractions, right? Since that's not the goal. And lastly, a special feature in Crossplain is this notion of a composition engine. And so the best, with an asterisk, the best way to utilize the composition engine is to create a provider that manages your resources, right? The reason that there's an asterisk here is it's technically possible to utilize any resource within a composition, but it's really recommended that you only use resources that are exposed by a provider. So the why, right? Why should you create or contribute to a provider? Well, so Crossplain is a CNCF sandbox project. As Karina mentioned, they're applied for a graduation, so hopefully they'll be in the next steps in the CNCF soon. Yeah, and so all the providers are open source so anybody can contribute. So this comes with quite a few benefits, right? Not only is their shared development and maintenance of common resources, right? If there's 10 companies and all of them want the ability to provision resources on Azure or IBM Cloud, they can all share a common code base to achieve that instead of creating their own bespoke solutions. And for cloud vendors, they can choose to expose their API in Kubernetes through a common interface, right? So this is not only good for the cloud vendors, but also good for users, right? I would prefer a standard interface or a common interface at least for creating resources on Azure, GCP, AWS, and Crossplain helps to make that possible. And lastly, there's a very streamlined development and consumption process, right? Crossplain handles a lot of the messy parts, so cluster administrators and developers can get started on doing the work they need to do. So why might you want to repackage a provider, right? We've talked a lot about the Crossplain design and structure, but why might we want to take a provider and structure it as a standalone operator? So one of the first issues has to do with proxies, right? We'll go into more detail about this shortly, but something that's really integral to the Crossplain operator is pulling an OCI image, right? So the operator itself pulls an image. And so this can cause quite a few issues when there's a cluster that's running behind a proxy. This is not like an unfixable issue, but it's definitely one more thing for administrators to note. And it's not immediately clear that this is how Crossplain is designed. Similar problem has to do with credentials. If you're using a private container registry, you'll need to supply credentials separately to Crossplain and the Kubelet, right? As you can imagine, this can cause issues with credentials going stale or diverging between Crossplain and the Kubelet. And lastly, and I think the most compelling reason has to do with resource limitations, right? Essentially, you might not always want to install the Crossplain operator and allow Crossplain to manage the entire lifecycle of a provider. With the OLM repackaging effort, you can choose to only install the providers you need and directly provision resources from those providers. So for example, I might just want to use the provider AWS to create S3 buckets. And so I can do that with the OLM repackaging. And I don't have to worry about all the other setup for Crossplain or if there's any other issues. So who is actually creating and maintaining these providers? Well, there's three main groups from my perspective. There's folks from the open source community. So as I mentioned, almost all of the providers that I know of at least are open source and there's many contributors and active members from the community. There's also contributors for organizations like Upbound, Red Hat, Squiz and Accenture. And lastly, there's engineers from organizations like Alibaba, IBM, AWS and Equinix that contribute to the development of their respective providers. And so let's get into the components of a provider and OLM operator. So let's do a quick primer again on the provider AWS. So this is one of the most popular Crossplain providers with support for dozens of AWS resources. And for our purposes, it's also a really great example of a provider to dissect. And then from the OLM side, we're going to be looking at the memcached operator, which is a common operator that's used in guides and examples like this one. And it's also a great example of the standard structure of an OLM operator. So what are the core components? Right, so let's take a look at the memcached operator. So we can see here, first we have our API folder. This is where we define all of the API resources that this operator exposes. Then we have the controllers directory, which contains all of our controllers and reconciler logic for the aforementioned API resources. And lastly, we have the config and bundle directory. Together, these allow us to describe all the metadata and deployment information for our operator. And as for provider, we have, we'll see the structure is pretty similar, right? We have the API's directory here, which just like the API directory contains all of our types that we want to expose. And then we also have our PKG directory, which contains all of our controllers and reconciler. So as you can see at this point, the difference so far is pretty minimal. And lastly, or I guess most importantly is we have a combination of folders here. We have the package folder, as well as our cluster and build folder. And so together these three directories kind of handle everything around deployment and packaging that we'll get into in a second. And so at this point, I think most people have noticed that the difference here is mainly semantic, right? But the last point that we talked about around deployment for providers is really key. And we're going to hone into that a little bit more here. So how do we actually then deploy a provider, right? There's no deployment, there's no cluster rule, no cluster rule binding service account, nothing that I can use that's clear here that I can deploy with. So this is something that's missing, right? And so let's take a quick aside on deployment. So as we mentioned before, within the cross plane model, you have to have the cross plane operator installed already. And the operator, the cross plane operator, exposes a few different API groups. So namely, the PKG package dot cross plane dot IO group contains a provider resource, a provider custom resource. And this is one of the most popular ways to install a cross plane provider. All you have to do is define the image and tag that you want to provision. So for this example, we're going to use the provider AWS image with the alpha tag. So this is straight from the cross plane docs. One of the other ways to install a provider is through the use of a configuration, right? This is installed the same way as providers, but configurations are designed to expose XRDs or composite resource definitions that we won't get into in this talk, but that's a whole other topic in itself. But users can install providers through the use of dependencies on this configuration. So we can see within the spec there's a depends on array where we can define the provider as well as the version. The minimal version that we want to install. And so what is the delta then, right? Essentially, this boils down to where are the missing components related to deploying the provider. So, you know, my favorite way to try and figure out something like this is just to start hacking away and break things down. So let's actually take a look at what our package contains, right? So with a tool like undocker, you can actually browse through the contents of our image without digging through the entire build process, right? I personally would rather look at the end result than take a look at every step of the build process and dig through a make file. So let's pull our image with the 0.18.1 tag. We'll run undocker, which allows us to unpack the image layer by layer. And then we can see that within this directory, the only file is a package.yaml file in the root of the image. This is interesting, right? There's no controller. There's nothing here that stands out. It's just a yaml file. So if we take a look within this yaml file, it starts off with a long list of the CRDs that this provider exposes. And then all the way at the bottom in the last, you know, 143 lines, we'll see there's meta.package.crossplane resource here. And at the bottom of that, we have within our spec an image referenced here, which is the provider AWS controller, right? So this is the actual controller that we'll be using. And everything inside of the package.yaml is basically just metadata and CRDs that we have to create. So what's the rest of the puzzle then, right? So based on this, we can deduce that there's two separate steps here, right? There's our actual metadata image, right, which is the provider AWS. And then there's the actual controller, which is the provider AWS-controller. And so it turns out, if we take through the crossplane operator code, that the operator parses the package.yaml, installs all the CRDs, and configures the RBAC for the provider at runtime, right? So it parses all the CRDs and creates our service account, our cluster role, cluster role binding. All of that happens at runtime when we try to install the operator, or when we try to install the provider. And so for brevity here, we're not going to go over the crossplane operator code base in detail, but there is a link in these slides if anyone is interested in digging around there. And so at the end, the operator, the crossplane operator creates our deployment using the tag referenced in the spec.controller.image within our package.yaml document. And so this is where that aforementioned issue we talked about a little while ago around proxies comes, right? So if the main crossplane operator is pulling an image, we have to make sure that it not only has the appropriate credentials, but that it can also even access the container registry we're using. And so this is kind of the crux of that issue. And so this is notable, right? Because OLM opts to define all of our RBAC and deployment at build time, whereas crossplane opts to do it at runtime. So let's get into the actual process of OLM repackaging then. So there is a project repository that we can use to help out with this process. Namely, there's the OLM repackage repository within the Red Hat ET GitHub org. And so if we clone the OLM repackage repository and examine the contents, we'll see quite a few files here that we're going to be using. So from the top, we have the Dockerfile and Makefile that we're using. So the Dockerfile just contains everything required for building the controller itself. And the Makefile just has all the targets we need to actually build our operator and build our bundle that we'll see in a little bit. We also then have our project.boilerplate.txt file. So this just contains some boilerplate for our project file. This is used by the GenProject script here at the bottom. And so this project file just contains a lot of metadata about our operator. Lastly, I guess we also have the config directory. So as you can see, this has quite a few subfolders, the contents of which has been omitted. But we know that the CRD directory contains all of our generated CRDs. The manager contains our deployment. Manifest defines what our CSV, our cluster service version looks like. And lastly, within the RBAC directory, we have all of our generated RBAC manifests, which is related to the GenRBAC script. So this script will create an RBAC.go file, which contains all of our QBuilder annotations that are used to generate the RBAC during the build process. So we use quite a few different tools during the repackaging process as well. Namely, we use YQ, which is a tool for querying different fields within our YAML document. The operator SDK CLI tool, which is used to handle generation and validation for deployment manifests. And lastly, we use Customize and Controller Gen. So the former of YQ and Operator SDK must be installed manually prior to repackaging, and the latter two, Customize and Controller Gen, are both automatically installed by targets in our make file. So not pictured here is Golang and Docker or Podman. Golang, of course, you need for actually compiling code and running different generation, and Docker and Podman are needed, of course, to build and push up the image. So what is each process then of repackaging? So part one is all the setup steps. So you have to clone your target repository. In this example, we're going to be using the provider AWS. Then we're going to set up all of our dependencies. So this was everything that we saw on the previous slide. And we also have to make sure that we have credentials and access from those credentials set up for a container registry like key.io that we're going to use here. And lastly, you have to clone the OLM repackage repository. And then copy the contents from the OLM repackage directory into the contents of the root of the provider. If that doesn't make sense, we're going to go over an example now. So in this example, we're going to clone the provider AWS, as mentioned before. And so we're just going to be using the master tag so we can get all the most recent changes. And then we're going to CD into our provider AWS and clone our OLM repackage repository. So my preferred way to do this is just to create a hidden folder. I like to use .work. And so once that's done, we can just recursively copy the contents. So this will override our provider make file as well as bring the config directory as well as our Docker file. So now we just have to set some environment variables. More specifically, our container registry user or organization as well as the operator image that we're using. So under our org, we're going to use the provider AWS tag or the provider AWS image and the master tag. So now we can just run Docker build and Docker push with our new image. So within our make file now, we're going to pull controller tools and controller gen, generate all of our or cogen all of our files that we need. And in one second, we're going to generate all of our CRDs. The next step is running the gen RBAC script, which generates our RBAC.go. So the actual contents of this goal file is a bunch of annotations basically within our within this file, we have comments which describe what the RBAC needs to look like. And then we'll just quickly build the image. So for me, that was all cached, but if you're building this yourself, it might take a second. And so we've gone through the build process for the operator. We've added all of our environment variables, run the cogen and push it up to our container registry. And the last step of the build process is actually creating our bundle. And although we've created our operator and that's been pushed up, we still have to create the bundle. And this bundle contains all of our manifests and metadata. And this is put into a single package that's installed by the operator lifecycle manager or OLM. So now we can start the process of building our bundle. So we're just going to have to set all of our environment variables again. So again, with the image, the user, the repository and the tag that we're using, we're just going to set the operator image again as well. And now we can run the gen project script. So this just defines, as I said before, our project file, which contains a bunch of metadata about our operator. Although this isn't strictly required, this helps to provide more detailed information for folks that are trying to install our operator. So this will finish in just a second. Good chance for me to grab some water. Yeah, so there we go. That's done. And now we're just going to run a few commands here to template in some values into our config directory. Of course, we can manually go through this, but it's easier to just run a command here. And now we're going to rename our cluster service version.yml to include the name of this operator. So we chose to name our operator provider AWS. So we're just going to make that change. And now we can actually create our bundle. So this again runs all of our cogeneration for our CRDs, as well as generating our RBAC again. So this will take another second. And so just to talk about something we did kind of behind the scenes. So when we templated in all these values into our config directory, this just updates a bunch of references and metadata. So our bundle has been created now. We get some warnings here, but no issues, luckily. And now we can set our bundle image, which is the image reference for what our bundle is called. So I like to follow the convention of image-bundle basically. This will effectively just be called provider AWS-bundle. And it will have the same tag as the provider AWS. So now we can just build our bundle, which is just a Docker file. And this should happen pretty quickly since we're just copying a bunch of files around. And now we can push that up. Okay, perfect. So now let's actually run the thing. Let's run the provider AWS operator. So again, here we're going to set our bundle image. And now we can use the operator SDK to run the bundle. So this is not the way to run the bundle in production, but for our purposes here, this should work. And so the operator SDK will start the process of creating our catalog source, our operator group, and subscription. So these are the main resources required to not only create a new operator within OpenShift, but also it allows us to create our operator group, which defines naming spaces that we can target, as well as the subscription, which actually requests the operator to be created. And so now we're just waiting on the cluster service version to be created. We'll see that it's pending, now it's installing, and it succeeded. And so just like that, we now have our operator running. So we can check to see that all of our CRDs are defined here with all of our AWS resources. So now we can get started with creating an S3 bucket or RDS database or anything we want. So that's all for this presentation. Thank you for your time. My LinkedIn and Twitter are here if you want to connect or chat. Or feel free to pin me on Slack. I'm on the Kubernetes Slack as Krish or the Crossfine Slack as Krish as well, I believe. Are there any questions? Thanks, Krish. Hey, Krish, do you see any questions in Twitch? All right, we do have a question. Why is the repackaging necessary? Yeah, so let's go back a few slides here. There we go. So there's a few reasons why you might want to repackage a provider. If you're looking to get started off the ground as quickly as possible, it might not be the right fit. From my perspective, one of the main benefits of repackaging a provider is you can choose to just install the providers you need and directly provision resources from those providers while building your own abstractions on top of them. All right, you might not necessarily want to utilize the Crossfine Operator or Composition Engine. You could achieve something similar with, let's say, Helm, for example. So that's one reason. Aside from that, the first two issues here, you can either work around these issues, and I believe there's also fixes that are inactive development to kind of handle these. So for example, one use case that I can talk about myself is another project that I contribute to is the container object storage interface project under the SIG storage. And so for that project, we essentially are trying to create drivers or provisioners for object storage vendors. So we could use the provider AWS operator as a driver for the COSY project. And that makes things easier for folks that want to get started since they just need to install the provider AWS operator instead of having to install the Crossfine Operator and handle everything through that operator. Yeah, and I guess, I mean, if you're strictly an OLM type shop, just running operators, you may not want to have another layer of abstraction from, you know, Crossfine, where you can just kind of repackage it and plug it in to work with all of your existing OLM workflows that you've already established. So I do have another question, Chris, that I don't know if it was clear to everyone, but so will this repackaging like the package, the repackage repo work with any of the Crossfine providers? Yeah, so not only will the OLM repackage repository work with any of the Crossfine providers out there. There's also an automated version of this process. So what I showed today was kind of the manual way to go about it, but under the Red Hat ET org, there is GitHub Action that can be used to kind of automate every step of this process and repackage a provider for you and push it to, you know, a target registry. So that GitHub Action basically goes through the exact same workflow here, except, you know, it can be automated and applied to any operator. So let's say, you know, you always want to use the most latest version of the provider GCP. You could set up that action to run on a schedule, let's say once a day, you know, at 12am. And it can, you know, repackage the most recent version of the provider GCP and push it up to, you know, your favorite container registry. So Krish, I have one more question for you. And it's a leading one. Does this Crossfine operator have you tested it up with OKD yet? Yes. So we've primarily been using OpenShift, but we haven't directly tried it with upstream OKD yet. But if you wanted to use, you know, OKD directly or Kubernetes, vanilla Kubernetes, all you need to do is install the OLM operator and then you can get started so that you're able to achieve that pretty quickly. So it's not limited just to OpenShift OCP. Yes. Perfect. And Krish, you've been working mostly with the AWS provisioner, right? And you're the main maintainer for that provisioner. Is that correct? I would say I'm one of the maintainers. I wouldn't say I'm the main maintainer. Yeah, but I've been contributing to the upstream provider AWS for quite a while now. I was mentioning that because I was wondering if you had any tips as you're, as anybody else is looking to help write the providers, what are some things that you've run into, some gotchas, some tips that you'd like to share? Yeah, so I guess from my experience, I think the biggest tip that I could have for someone that's looking to get started with contributing to either the cross-plane operator itself or any of the providers. I'd say the biggest thing is just to reach out and talk to people. So the cross-plane communities is I think one of the best open source communities out there. Folks are always willing to help out and answer questions. And there's community meetings that run, I believe, every Thursday now. So that's always a really great place to start, just to talk to people and get a sense of what there is to work on. And more generally, if you're looking to contribute to an open source project, I think cross-plane providers are a great place to start. Not only is it a good chance to learn the fundamentals of contributing to an open source project and also everything about Kubernetes and Go, it's also a good chance to learn about a cloud provider. You might want to learn about GCP or AWS or Azure. And you can use your experience from writing a GCP provider or resources in the GCP provider to kind of learn more in-depth information about Google Cloud itself. In terms of gotchas, I think the biggest thing is just staying up to date. The cross-plane project is still, I guess, it's more stable now. But when I started contributing, the project was definitely still in the earlier stages. And so there would be breaking changes once in a while and it would be important to stay up to date. Now that the project has matured quite a bit, I think the API is pretty stable these days. Thank you. Hopefully that answers your question. Yes. I'd love to see different repackaging scenarios and using OKD as a playground for that. I think that I'll take that to the OKD working group that's kicking off in about 30 minutes. So, yeah, hopefully we can get some testing done from the OKD working group on this and give you some feedback there as well. So, and that should be able to find a blog post if we get that done on OKD.io sometime in the not too distant future. So, yeah, again, really great work. It's wonderful to see the use of OLM and operators to repackage this. And I'm looking forward to getting some feedback on this and watching the journey that cross-playing takes in the CNCF too in the not too distant future. My only critique, Chris, would be we should have used provider pizza because it's lunchtime. Yeah, you can get everybody a slice of pizza. Yeah, it's lunchtime somewhere here. We're still drinking coffee on a West Coast. That's good. So, yeah, Scott and Chris, really, this is great work that you guys have been doing and I'm looking forward to seeing a few more providers out there. So, we'll get those tested and give you feedback from the OKD and other arenas and Karina as always. Thank you for organizing this and making this happen. Thanks again, Chris. And Chris, if there aren't any other questions, can you please see us out? Thanks again, Chris. This is awesome. Always love having you on. Thank you, Karina. Thank you. Thank you, Diane. Take care. Thank you for having me.


public means of public bike transportation on the city and the system is composed of of some stations and and these stations might have bicycles on them or maybe not so knowing where the stations are and how many bicycles are or how many spots to park the bicycle it's something that makes makes it really important to have awareness on the location of the stations and the availability so that was just for setting up the context so I'm hello I'm Luis Eskerda currently working at scraping hub and I'm going to talk about how I'm going to talk about how I managed to grow a project from it's a good okay how I managed to to grow a project from Barcelona to the whole world and for the people that knew what bike sharing is want to share this map with you this works with this this is a real-time map of the world this is all the all the systems I currently support on my project and every time I don't know if you can see it but every time there is an explosion on the map it means that somebody has just took a bicycle or or just left it on it's real-time ish so yeah as you can see some guy there just left the bicycle somebody took it there and and that's on the whole on the whole world or on the whole world in the as in city bikes can see so we can see the systems that are not on this map because they are not yet supported so what's this about in reality many years ago I had the not so novel idea of mixing up smartphones with these bike sharing systems there were some applications fortunately for for me in Barcelona there was no such Android application for the biasing system so I decided to create one and the first step that I encountered was that this information was not available anywhere so I looked on on the internet for you know busy API busy information and I that's the first time I I saw the term scrapping I will refer to the original library that inspired the project because if it was not for this library I would probably not be here and that's also what got me into into Python so the main idea was to to there was a website for the for the for the Barcelona system and there was a map and you had to take this information from from the JavaScript inside the map so it was kind of hidden hidden and then I knew why some apps that were around were really slow and it was because all the scraping thing was happening on the smartphones and at that time smartphones were not really that powerful so they were really slow they would break and they will crash so I created open biasing which was the first part of the project it had an open API and it was also an Android application that was I think it was a licensee to and that was for Barcelona then I realized that in Paris the problem was exactly the same even though the website was even more complicated to to scrape but I created an exact clone of the same project but I just rename it did open belly and then I realized that in it was also the case for doubling in fact doubling bikes was using the exact same system that belly was using so it was really easy to add it just three more lines of code and then I had another project then London started their cycle hard project which started with open information already that you could freely access and I decided I also wanted to be there so I created yet another clone of the same project and at this point it started getting really clear that this was not the best way to keep working that it's not easy to keep separate projects at the same time and then I decided to call it city bikes and it was it was the so sorry city bikes was was the the composite of all the cities that I started finding around at this point it was mostly the same Python library I had been using for the busy but just with more and more networks kind of struck together so it's it still felt a bit like like not doing a really good work on it but on the outside it was it was good there were many cities around and and so this is what city bikes is composed of the library was renamed to buy bikes basically this is the this is the main part of the whole project and on buy bikes people can what people or myself we add new scrapers for all these bike sharing websites and when you add it to buy bikes then suddenly it gets added to the API of city bikes and so basically buy bikes in in a sense it's a Python library that gives you access to bike sharing information but not dependent on my project at all but it's the actual library we are using to extract all this information then this is called giardo it's it's not really something it just the system I'm using to manage all the tasks and on the queue so all the scraping does must be must be a process it somehow and I just wanted to show a bit of the stack of the project not too much maybe the only novel thing is that this is using RQ instead of salary which is the main task processing system being used on the on the Python world and then the there is the API of the project fortunately for everybody in this room this part of the library does not have a logo so the API it so since the project started I mean I was not really intending to build an API but I knew that when you start sending information through the internet to a mobile device it's almost impossible to protect it so you better do it already open and then the API kind of happened already so it was already on the pressure night was the the most important part and in the end there is also the Android application which unfortunately I'm not spending much time anymore and and that's because of our realization I had that the Android application really does not produce any kind of of new information or new content whilst by using the API we are providing something that can be used to create maps so the main idea it's maps are not maps are useful representations of reality but they do not have the power of creating something new so and that's that's quite important to to realize because it's it's the point where you start understanding how wrong some city councils are a lot things and how things should be done in a different way for instance since publishing this API there is a people up for getting your bike sharing information there is a Google glass up there is city mapper which is if I know already they are using the city bikes API for Barcelona, Sao Paulo and two or three more cities I don't know but basically these are cities where they have problems accessing information so by providing it they can support it these are two apps one is GPL descents it the other one it's MIT but it's good I mean it's good to have applications that are open source or free software using the API because they are continuing the work I started and I don't really have to support all these many platforms there is even one app called city bikes I not even gotten in contact with with a guy but just decided to put the name there is an app for Windows phone so I don't I mean there was no way I was going to support any of these all of these platforms it will be it will be a waste of time and back to the growth thing I would like to say that it's been a good and calm and steady thing to do but and we hear many words around supplies like smart cities and open data and we are really content and so happy about all these terms but in the end they they do not mean anything if you are not actually providing what people needs so there are some cities are are starting to provide open data and that's and that's a good thing but they're going to be a lot of cities that maybe it will take 10 years for them to even know that they should be doing it or maybe they don't have any kind of of reinforcement of doing it so maybe they will never do it and that's where I met the evil part of the of the project it's called PPPs these are called as public private partnerships so basically this relation happens every time a company goes to offer a public service so a private company is going to offer a public service for a for a government or for a city council and so when the city council gets gets a contract to work with with these guys the information usually stays with the private part and not with the public part so somehow we could consider that these companies are blackmailing city councils for the information they should already own so let's say you are a city council you have just opened a bike sharing system and you say well okay I want I want to build an Android app and they cannot because they do not have the rights to access this information but the company obviously is really happy to provide them with an app that obviously it will be overpriced so I like to think that city bike somehow could be a key contributor on this on this right in the sense that the project kind of forces companies and city councils to to get it together so by providing a reality that it's not there anymore it's not still there sorry at some point they are forced it to to do it just because they they have no other option the information it's already on the net and people are using it there are many applications using it so they they they kind of are forced it to to release it or yeah so so I just read obviously I'm going to miss quote here's obviously the article does not have anything to do with even the theme of this presentation I think it's a it's a nice it's a nice quote so I think we should develop our software and choose our licenses for the future we want to see not the present that we are stuck with and this has been a major a major part of what the project is about so it's not just about going around scraping websites putting information out and everything so it's more about trying to create an image of how the world will look like if of this information was free and I'm not even talking about bicycles right now with just any kind of information that it's not sensible to user privacy that can be useful for anybody to to build things and I think that this map kind of could be that that metaphor I mean we are seeing the world we are seeing the people used bicycles everywhere and the only way to do it is is if you just release all this information so that's mainly that was it hello okay so now I have time for questions how I how I am on time pretty early quite good like and so we still have probably 10 15 minutes for 15 yep okay so what do you say if we add the Bilbao bike share system to city bikes yeah okay so I've been looking around it looks like they've been publishing this so this information it's open so let's just go to I mean I know not this one so they they released this information for free do what I found out that it's quite funny they also have an Android app that it's not using the open data feed but that it does provide a bit more of information but anyway for the sake of doing things the way it should be done you're just going to use the the public feed okay it's here it's in csa if be XML or WMS so what sucks less than the free XML I guess so just going to show a bit of how the deal how is it is or maybe how hard it is I don't know to add a new system to buy bikes so these are the the spiders and then there is another data folder that maps meta information to these spiders so for in the case of Bilbao because they are providing an open data feed that probably we're not going to be able to reuse ever because this is a format that they are only going to release for Bilbao I would have to create a new file we'll call it Bilbao and because I don't really know what I'm doing I'm just going to look at other fighters at the same time so we can there is a basic class that inherits from bike sharing systems so in this case it will be Bilbao I think this bike share system and we can get some some parameters we can add some meta information there so there are some things that are already being published on the API and they are important to if they appear here like what is the which is the company running it or maybe they have a license or something so this just goes here and goes well with it with the API by the way the so the idea of buy bikes is so basically you import buy bikes you buy bikes get some any kind of bike sharing system you want let's say busy you obviously have to put it somewhere and then you update it and suddenly it has stations on it so for stationing by seeing station spring station for instance and and and it's like that you don't have to do anything this is not a rubber around my API so if anybody here is doing data science or things like that in Python give the library a try because it's quite easy to use and it contains all the resources you need to to create whatever you want to do related to bike sharing information 10 minutes okay okay so I think it's better we we just go to question time if you are really interested in knowing how a system itself to the library we can talk it later or I can show it to you anyone who wants the microphone thanks for the talk it's awesome did you get into any legal issues with the with the companies that own the data that you're scraping did you get like a decision decision yeah I think I completely forgot to add that slide and it was a quite important part of it so for that I'm sorry to the audience yeah we've got into some trouble sometimes but the good thing is by by by publishing this information and and having all these small army of developers writing stuff for it the only guy that's going to look good but on the paper are always going to be there so usually they always try to avoid any kind of conflict there was a conflict with JC Rico in France where they send me a season this is letter for all the prejudices that the project was causing them because they were actually planning on selling all this information to any company that wanted to run analysis on it so may I don't know but the the men as of all the laws I was causing them was valued in around I don't know I think it was around 30,000 euros per year but then a new law passed by in in France where all public infrastructure or private companies providing public services are almost forced to publish this information and they even released a license for it so in the end it worked okay then there was another season this is from Brisbane there was another guy from a guy from Sweden that he had like a partnership with Clear Channel to be the sole developer of the app so I think they kind of made a deal with him that he would make the app for free in exchange he would be the only exploiter of this feed of the information and he I mean I exchanged some really angry words with him on Twitter and through email and everything and it really did not fix and so I just decided to forget about the city and it's not on the map anymore so when before I was talking about what's a map and what's information and what's create something so for instance if we go to this email which is the bike sharing system in Madrid right now they've gone as far away as creating a PNG map so scrappers will not get this information so yeah well no what would be easier is to break their Android app and actually access their API but the last time that happened hell broke loose and the many vulnerabilities were discovered on their systems so I think I'm letting this case rest for at least one year or two until they forget who I am but yeah these are been having the problem so far another funny problem I have had so far it's in getting people using my API to properly reference back the project because if not the message of the project gets completely lost and people just start assuming that this information it's already freely available when it's not we are going to create links to to create this database of information just to provide them with a source when there is not a source but it's really important that they say where the information comes from because if not just information it's there so we don't have to fix anything because it's already open and we are in the future so thank you for the talk my name is Vojtek it was really great to learn about the bicycles but your message is also that there is some extra data about the cities like the open data and other systems that provide not only information about the bicycle stations but you mentioned there is also some other piece of information could you like give me an example what else you could scrap from the data that is provided by city councils etc don't actually think I said that if I said that I was misusing my own words myself I said that I you mean well while I was pretending to write the the system for Bilbao I said something about the company and everything or no no no I mean like the data that you are presenting any wrappers is about many bicycles but what I understood is that the message of this is that this data should be open and sometimes the city councils are providing already some APIs about the cities not only about the bicycles so my question is okay is there any other like piece of information I don't know like the paths of the buses that you could track in this way and there is there is many people I mean there's lately there's been lots of maps showing thank you for the question there's been many maps showing real-time tracks of underground trains buses and everything and people love this kind of map so this information it's available I think and on the UK at least and in the US in some parts cities usually are just able to publish rubbish information that people cannot really create anything with like I don't know the public webcams of traffic so I so usually not there is not much to to get from from open data portals and that's that's a problem I in fact I've been discussing it for with them because they started really excited on on the hype of of the open data thing they completely believe it everything that happened and then nothing happened and nothing happened first because it was I second because there are companies really interested in selling big data solutions so suddenly they tell the city councils that they have a big data problem and that they need a solution for that and that's not true but still it wins good money and so in the end some city council have started just releasing different apps and we must consider that every time they release an app there is going to be public feed about that but what what I was trying to convince them it's they should really stop doing apps because they really suck at doing it so what they should be is just providing information and obviously not just that but every time they contract they have a contract with a private company there should be somebody there that knows understands something so they are able to get a more fair agreement that the other way they would not get I think it's just yesterday by looking on how I will scrape the well just integrate the the bike sharing feed from Bilbao I was looking for the name of the company providing the service and there was no way of finding it in the end I arrived at a PDF explaining the elicitation for the project so the company was there but the company it's not actually the company providing the service it's a consultancy company that it's sub contracting different companies to build the application to to provide the feeds and so it's it's really public projects are like these are they are really ugly and we are really know it so yeah hey have you tried having a look at open-street maps data to see if you can get locations for state bicycle stations from there and fact I tried to contribute back all the stations I have to open-street maps but the thing is I could only contribute the ones that come from a source that it's not liable of prosecution so what I mean really interesting right now in in OSM it's integrating their information usually I would I do not need the information from the whole world but just from the actual cities I have so I can get bicycle paths or I can get the altitude of the of the stations so maybe I can enhance somehow the API because right now the API the good thing is it's a cohesive syntax for a for a for an object let's say for a station and everything and that's what developers like they write an application for their city and suddenly they have support for the whole world and that's a good thing but in the end what my API is offering it's just a reformulation of the original information so what would be really cool would be to enhance this information and it's something I'm looking into thank you so have one minute for a really short question if there is any okay let's hi one technical question where do you run all the scrapers do you run on Amazon web servers or something like this so is it your own server somewhere in data center yeah I have to lean out virtual machines and then a dedicated OVH which does not have much CPU power but lots of this can run so I'm trying to stick to the rule of do it do yourself ever do you sorry that I want to do all the parts of it that's a problem because it's difficult but I really wanted to learn to learn so these different servers are interconnected in a VPN they also have tor exit nodes on the inside and then there is ready somewhere database somewhere and then there is the serving part which is done with NGX okay so let's thank the speaker again


You usually don't have a scenario that you are fighting for control of sentiment, of longer-term sentiment. It's not like the five-day moving average. Welcome to Access a Trader, the number one community for those who are committed to taking control of their trading in order to achieve success, profitability, and longevity. Thank you for joining us. Here's Dan Shapiro to help you find your edge, master your process, and own your future. Hey guys, good evening everybody. Welcome to another edition of the Access a Trader dot com nightly wrap up show. So last week we closed down 4% negative across all the major benchmarks. And what seemed to be an orderly back test just only a couple of days ago turned into something more. So guys remember we talked about, well now we're getting close to a very, very big area. If you watch the weekend update, we talked about the significance of the 50-day moving average. And let's kind of review here. I want to kind of show you guys something. So if you go back to here, before we broke the 50-day moving average, the first day, the first time they tested it, you could clearly see by just by the green candle, which means a higher close and open. The higher candle, right, the higher candle, they defended the 50-day moving average the first time around and then the bounce. And the next day, right, the next day they came and just didn't quite confirm the day, but they closed right on the 50-day moving average and ultimately everybody knows what happened. The following day they got below it and stayed below it for the next five months issuing a pretty steep sell-off. And that's kind of what would happen today. What happened today, right? We discussed that 303, 304 level on the cues on the weekend update and, you know, we talked about it on the weekend video. I thought the first shot, excuse me, we talked about it on the webinar, I forgot if I talked about it on the weekend video, but we talked about probably the first shot at the 50-day moving average was going to be defended, right? And that's exactly what happened this morning. If you guys remember, there was a bounce, initial bounce off that 304, 303 level. They started rallying, you know, rally, you know, almost four points off the 50-day moving average. So we knew right away that was the line in the sand. And now we talked about, well, now if we start building below that 304 level, that is going to be a very, very decisive blow to the bulls. And again, if you go back to the short history, once you give up the 50-day moving average, it's going to be a very, very tough time for the long bias argument to kind of make a stand. The problem with today's day was it was, I had a very frustrating day, okay? And the reason why I had such a very frustrating day was, number one, we gapped down pretty significantly overnight. So basically what happened was the majority of stocks that I liked, they were well below the pivot, right? Well below the daily pivot. And even the names that were still in play, they were catching up to their average true range very, very closely, especially at the opens. So it took away a lot of the value from us, right? But the most important part was I wanted to see what happened on the 50-day moving average. And number one, congratulations to all you guys who took that initial bounce. Somehow I missed it. Don't ask me why. But I missed it, right? So it bounced from 304 all the way up to 307. That was a beautiful, beautiful bounce. But what happened after is where the frustration kicked in. Usually when you defend the level, right? When you, for example, if you're trading Tesla and they defend, I'm just using a number, 885, right? 885, and that's a big, big order of rising support. When they test that level, it bounces. The next candle comes through, they confirm it. Here comes the price action back the wrong way. When you're trading the ETF side, it's a little bit different. So actually a lot different. And we saw today not once, not twice, not three times, but four times, right? The 50-day moving average being lost. All four times they set new lows on the day. All four times they got back above the 50-day moving average. All four times they started to rally, only to lose back the 50-day moving average, make another new low for the day. And this continued literally for about three, four hours. And I would use the word frustration, right? Again, you don't need to be down a lot of money to be frustrated. Sometimes the price action is just so radic. And again, I kept on saying this in the webinar today. You don't see a day like this often because you usually don't have a scenario that you are fighting for control of sentiment, of longer-term sentiment. It's not like the five-day moving average. People are controlled of a short-term sentiment. This is a long sentiment. Remember, once we broke here, we started a five-month cycle of declines. So this is a kind of a big deal. And the whole day, we literally saw over 404, under 404, excuse me, over 304, under 304, back and forth, back and forth, back and forth. At some point today at around noon, I was like, listen, I give up, right? I give up. I'm just not getting any type of clarity today. I'm really not. Every move is going like 50 cents at a time.  It's a 50-cent move, blah, blah, blah. But you can't have that the whole day. And this is one of those days that usually if you guys have been watching this update of all you guys have been in the webinar for the last 12 years or so, I'm pretty definitive, man. I'm pretty definitive. I kind of know what to expect of price action. And I'm telling you, I was sitting there today after 11 o'clock, after 11, 30, all this, and I have no idea what's going on. I wouldn't be shocked if we rallied all the way back to green. I wouldn't be shocked if we sold off another 500 points. I just wouldn't be shocked. You know what? I'm just going to leave it alone. And the one thing that I learned a long, long time ago, when you don't know what's going on. And again, over time, you're going to understand more and more things because you're going to see different scenarios. You're going to see you have so much screen time as you're kind of a go-to vice for a history lesson. But some days, man, and especially when you're fighting, remember, it's like a UFC match-up. And you're standing, both the fighters are standing in the middle of the ring, and they're just swinging on each other, one back, and they're just both connecting. And that's exactly what happened today. It was just super frustrating. Technically, we didn't get a lot accomplished, monetary. I took a small loss on the day. I just couldn't find an edge today, man. And that's what really, really frustrated me. But there is a silver lining to this, right? And that's the whole point. There is a silver lining. We always say sometimes the things that we kind of dissect throughout the day that don't show up on the scoreboard, meaning your monetary win or loss, sometimes you'll get better information. So if you have to take a little bit of step back, at least you're getting enough information, quality information that you can use for the rest of the week and especially go into tomorrow. The one thing that I did observe, just when the cues started bouncing, there was a lot of names. There were a lot of names that didn't rally with the cues. At any given point, you had Tesla not rallying. NVIDIA continued its breakdown from Friday's session. Microsoft continued to break down. You had Apple, right? Continue to move down as well. So you had a lot of names not rallying. Even Amazon, we saw when the market was rallying, man, they were physically coming for 120 weekly puts as the market was rallying, as the cues were rallying. So it really did give me a lot of kind of ammunition to at least it didn't help me today. It wasn't gonna help me today, but at least I could use it in the future. And the most important part was that data. So now we know a definitive area. We don't have to guess anymore. So tomorrow, this 303 level on the cues is everything, right? It's absolutely everything. But the question is, why do they hold it up today the first time? Again, I can make a case while here it is on 4-7. We also held the first time only to give it back the next day and the day after to start a new trend. But Matt Whitaker, a big shout out to Matt today. Number one, he identified the bounce perfectly this morning on the cues. But I think he made a great point today. And that great point was, well, the NASDAQ is being pulled down. But if you look at the S&P, S&P is still above the 50-day moving average and still has to test it. Remember, the one thing, that's a great, great point. The one thing that we always have to remember, that we want to see everything go down a tandem. So if the cues are underneath the 50-day moving average but the spies are still above, you can guess. You can have an educational guess and say, well, is the NASDAQ going to pull down the S&P? Or is the S&P going to pull up the NASDAQ considering everything is going back into gravity, the bet would always be the NASDAQ pulling down the S&P. But I think he made a valid point. If the S&P and the NASDAQ would have both tested today at the same time, at least we knew going into tomorrow's session, we knew, hey, if both confirmed the 50-day moving average, you're going to have a pretty violent sell-off. Because it's not going to be just regulated to the NASDAQ 100. You're going to have the whole mass of spread of names going through financial services, consumer cyclicals, whatever the case may be. But I still think that's a great, great point. I still think the S&P have to have their own day, their own line in the sand, to test all the way back down to this 390 level. Again, maybe it happens tomorrow, but just keep this in mind. They don't have to break down tomorrow. It could be two, three days, or the market could just put in this kind of a bottoming tail to start rallying up. Because remember, at the end of the day, we are still above the 50-day moving average. So going into tomorrow, again, yeah, look, of course, there's some names that look OK. Look at Cardinal Health. Look at Cardinal Health. Standing on a nice little channel here. Oh, the stock, EV. What was it, EVTL? No, wrong symbol. Hold on, let me give you the right symbol. It's not EVTL. It's, let me give you the right symbol. It's ENLC, right? Like a name like ENLC, right, that looks interesting, right? There's a couple of names that do look interesting on the upside. But at the same time, look at the semiconductors, right?  AMD, as I forgot who said it in the webinar, he goes, AMD only testing the 50-day reclaiming and losing it 17 times today. Yeah, that's true. That's exactly what happened. Now we have a definitive line in the sand and AMD. So you have, NVIDIA continues to be weak. AMD continues to be weak and very, very close to losing its 50-day. You have microchip. Again, another semiconductor looks exactly like AMD as well. You have Zoom, right? I'm shocked they didn't take down Zoom yet. Zoom blew up on earnings. And it's just kind of building a bear flag. Again, maybe it goes tomorrow. Maybe it doesn't. But the point is, again, the value is definitely still to the downside. The only question is, I would, first of all, I would love to see a little bit of a gap open tomorrow just because the gap down today really destroyed 99% of our setup. So maybe that's where the frustration kicked in. And maybe I started just kind of getting a little more anxious. But there's certain days, and I tell you throughout the year, it's OK to turn around and go, look, I have no idea what's going on. I don't know how many times they need to test, retest, reclaim, lose the 50-day moving average. Today was one of those days. I just buy 11, 1130. I got super. I don't easily frustrated, but I got frustrated, right? I got kind of frustrated. And I say, you know what, let me just move to the side. I don't care what this market does the rest of the day. Let the market do the heavy lifting. And if you see how the queues closed, they did close within 40 cents of the 50-day moving average. It's tomorrow the day. The bears are kind of getting their knife and fork ready to kind of eat. We'll see. We'll see. Again, we can't control the overnight futures. We can't control any overnight news. But the only thing we can control is our risk, right? Risk tolerance, risk exposure is absolutely everything. And if you don't feel comfortable, and that's exactly what happened to me around 11, 1130 today, Eastern time, right before lunchtime, if you don't feel comfortable, just get the hell out of the way. It's not your last day of trade. Clear your head, right? Again, you don't need to lose a massive amount of money to clear your head. Just clear your head, right? There's certain days you lose your edge throughout the year, right? This was the day for me. Again, you had a fist fight, literally a fist fight, right on the 50-day moving average. It was Rocky versus Ivan Drago.  Again, we'll see what happens tomorrow. We have our game plan ready. We know the bottom of the channels here. Is this tomorrow the next day that the day that bears finally prevail? Let's see, right, to be determined. Guys, have a great night, everybody. Stay blessed, right? Stay calm. Again, you don't need to trade every single day. It was a perfect example of it. And with God's help, tomorrow we'll get a little bit more seamless action and have a terrific day. Take care, everybody. Have a good night.