Hello and welcome everybody to the OpenShift Commons briefing this week. We're really pleased to have with us today Alan Nam from Google. He's the product manager for both Kubernetes and Container Engine, as well as Clayton Coleman, who is the lead architect for OpenShift, which as many of you know, up streams Kubernetes and we do these weekly podcasts out to the OpenShift Commons community and then we post them afterwards for recording for viewing purposes on YouTube and elsewhere. So we're really hoping to have a very interesting presentation on the present and the future of Kubernetes. As we all know, a recent release went out the door for Kubernetes and there's lots of good pieces and parts that got added in, and then there's a new roadmap coming up for the next release. So we're going to try and dive into those aspects. We'll do Q&A through the chat in BlueJeans. So if you type it in there, everybody is on mute except the speakers. We'll try and answer the questions in the chat, and if they don't get answered, we will then reiterate them after the presentation, and we'll have a conversation and do Q&A verbally afterwards. So without further ado, Alan, thank you for joining us and I'll let you introduce yourself and take it away. Thank you. So I'm Alan Nam. Product Manager here at Google focused on Kubernetes and Container Engine. So I'm going to be focusing on the present state of Kubernetes, and then I'll be handing it over to Clayton to talk about the future. Before I get started on the present, I always like to spend a few minutes talking about how we got here, and I think it's important to add context to the conversation. So the path to modern containers really started at Google back in 2004, when Google engineers contributed much of the code in the Linux kernels that supported C groups and namespaces. You guys have heard this probably over and over again, but everything at Google runs on containers. We spent up 2 billion containers per week. Through our experience with containers and developing our internal system called Borg, using this 10 years of experience, we basically took all the best practices associated with managing containers and developed a framework that we open source called Kubernetes. So Kubernetes really at the end of the day, lets you take a cluster first approach to application development, where the focus is on the application and the resources that they need. As I mentioned, it was really inspired by Google's experience. It supports multiple clouds, bare metal environments. It's 100 percent open source written and go. At the end of the day, it really enables application developers to focus on the application, not the machines. Kubernetes GA back in July 2015, that was when we announced our 1.0 version, and since then the adoption rate has been phenomenal. Looking at this graph here, you see the number of commits since January 2015. Our latest version which I'll be talking about in a few minutes, 1.2 had 5,000 commits associated with it, and it had increase of 50 percent unique contributors, which is just phenomenal. All of this is really driven by the community. We are in the top 0.01 percent of all GitHub projects. There are 800 plus unique contributors, and over 1200 external projects today that are based on Kubernetes. This slide shows you just a small subset of the companies that are using Kubernetes today, and as well as all of our great community partners that are contributing on a daily basis. Thank you for making this happen. I'm going to provide a two-minute accelerated overview of Kubernetes just before I get into the 1.2 features just to make sure we're all level set. Containers, Kubernetes manages orchestrates containers, and developers typically find three key benefits with containers, and those are portability, repeatability, and abstraction. Portability and repeatability come from the fact that each container image includes everything that's needed to run the process that it represents. But the big win really is around abstraction. By abstracting the app from the operating system and the machine, you get clean separation of concerns between application development maintenance, infrastructure deployment, and operation, which really translates into agility for the developer. You start with containers, you containerize your application, but then you have challenges around how do you scale, how do you update, how do you manage the hard part of building distributed applications, and that's where Kubernetes comes in. Kubernetes really takes an opinionated view around managing containers, and we have these primitives that we introduce, which are pods, replica controllers, services, labels, and so on. A pod is really a unit of deployment. It's a collection of one or more co-located containers and volumes, containers in a pod, share process network, and inner process communication namespaces. So an example is, we like to recommend that when people run applications on containers, a container should do one thing and do one thing really well, but there are certain cases where containers have similar behaviors and share things. So for example, you look at a web application and a log roller. Rather than put both of these in the same containers, because you may have to scale them independently, why not put them in separate containers? But at the same time, you want these containers living close to each other so that they can share resources, and that's where a pod comes in. A web-servering container and a log roller can live within the same pod, share the same IP address, share the same volume, and ultimately act as one. And a pod, all the containers in a pod actually live and die together, which makes it really good from state management and so on. Pods have labels. A label is really a key value pair that's used to identify the role or what this pod actually is doing. So for example, I could have a label called roll and say this pod is a web-server or this pod is a front-end. And then that basically is used to represent all the various objects that you then create using what we call label selectors. So in this particular case, a replication controller, which is really a spec that you create to really tell Kubernetes the desired state of your pods that are currently running. So the replication controller ensures that a specified number of identical pods are running at one time. It can be used for scaling both annual and automatic for doing health management, for doing rolling updates, et cetera. So a replication controller takes a label selector. So for example, I can say I'd like five replicas of pod that have roll front-end and it'll go off and create five replicas of that pod and then ensure that the actual state is always equal to the desired state. And you can do that for many pods. So you can have multiple replication controllers that represent pods that are front-ends, pods that are mid-tier, pods that are back-ends. You basically define that spec and that desired state and you submit it to Kubernetes to ensure it's actually implemented. Now that you've created your replication controller and you have this application that's running a number of replicas, how do you expose it to the outside world? Well, we introduced this concept of a service really which is an abstraction and defines a set of pods and how to access them. So there's a stable IP address and a corresponding DNS name that's generated associated with that particular replication controller and it's abstracted using a service implementation. Now requests coming in externally can discover this application and be routed automatically to those replicas that are running across the cluster. And in this particular case, I can leverage my label selectors to specify which replication controllers I'd like to expose. And when I put all this together, I now have this microservice oriented architecture where my application is composed of multiple tiers that scale independently, that are exposed independently and can communicate together. So what's new in Kubernetes 1.2? The first thing that we introduced was what's called multi-zone clusters. Previously with Kubernetes, your application was limited to a single zone. With multi-zone clusters, we've actually made it possible for you to have your application span multiple availability zones. So basically you create nodes in each of the zone and then you submit a replication controller spec with the number of replicas that your application needs and those pods get distributed across the different zones. And now your application is highly available because it's distributed across multiple failure domains. Multi-zone clusters is GA as of Kubernetes 1.2 and it's fully automatic support for both compute engine and AWS including on Google or container engine service. It requires some manual configuration in on-prem and other clouds but we're actually working on that. Every node that's created within a zone gets a label that associates it with that particular zone which makes it possible for the Kubernetes scheduler to schedule the pods in those particular nodes in the appropriate zone. The second feature that we released in 1.2 and it is in beta in Kubernetes 1.2 is this concept of a deployments object. Prior to Kubernetes 1.2, when you're doing a rolling update, it was very much a client-side action where it was imperative in nature. And Kubernetes doesn't like imperative things. We believe in declarative, we believe in being able to create server-side objects that actually have a controller associated with them that ensures desired state and actual state are equal. So we introduced concept of deployment which really takes and enables you to define an object that specifies how you want to update your replication controller. So rather than say I want to submit a new replication controller that has a number of replicas using this particular pod, I can now define this in a spec called a deployment object, submit it to Kubernetes and Kubernetes will constantly maintain this object and ensure actual desired state are equal. And then you can, if you want to make changes from a rolling update perspective, change your application, you can now submit to this update that deployment object and Kubernetes will pick up the changes and roll the updates through. The other feature which has been around but we actually went beta with it in Kubernetes 1.2 and is also available now in Container Engine is this whole notion of daemon sets. So suppose you want to run a pod on every node prior to daemon sets, it was, you know, you could do it but it required quite a bit of administrative overhead associated with it. So now you can run a daemon set command, deploy daemon set, have pods deployed on every node in the cluster as well as leverage a label selector to deploy these pods on specific nodes or specific, exactly specific nodes that match that particular label. Horizontal pod autoscalers went GA in 1.2. So for those of you that aren't familiar with horizontal pod autoscaler, this is different from node autoscaling. This is leveraging metrics like CPU utilization to actually have your pods autoscale within your cluster. Today we only support CPU utilization but we're looking at adding more and more metrics including custom metrics, which is an alpha in 1.2 but in 1.3 you'll actually start seeing more metrics that you can do horizontal pod autoscaling around. Config maps are something that we introduced as well in 1.2, so they enable you to really bring in configuration files, for example, into a pod namespace and allow the pods to be able to pick up configuration information in a, you know, in a deterministic fashion and manage these configurations through the Kubernetes API. So this actually improves the administration of the application in a big way so that you're not having to leverage other means of persisting data to manage the configuration for your actual application that's represented through the pods. We also baited ingress which is our object that abstracts away some of the complexities associated with routing incoming traffic to backends using layer seven load balancer. So with ingress you can basically define rules and say I'd like requests coming from here routed to this particular backend. You can use the HTTP host header or define URL paths that map the incoming requests to the appropriate backends within your application. We support HA proxy, engine X, the native load balancers both on AWS and Google Compute Engine. Implementations are in progress. We also in 1.2 announced support for SSL, which is great. So this feature is beta in 1.2 and we're continuing to invest in areas to abstract away that some of the complexities associated with URL path mapping and so on. Network isolation is another feature that went alpha in Kubernetes 1.2. This is the ability to restrict pod to pod traffic across namespaces. So you can have certain pods live in various namespace in a particular namespace and say, I don't want any type of traffic flowing between these two namespaces, these pods can't communicate with each other. So this is something our customers have asked us for and we've introduced the alpha in 1.2. Scalability has been a top ask for a lot of customers prior to Kubernetes 1.2 we tested, internally we tested with 250 nodes, with 1.2 we've now tested with 1,000 nodes. So we support that and this represents up to 30,000 pods within that cluster. And this again, this is what we've tested. There is nothing stopping someone from going in and spitting up 2,000 nodes, we just have not tested against that. So we are continuing to invest in this area, from 1,000 we want to go to 2,000 and higher and higher. And this is all based on customer feedback. Some of the other areas around performance is really optimize the IP table cube proxy path. So there's really no measurable hit from CPU throughput or latency across that path. We've optimized the Qubelet with a 4x reduction CPU memory and then some other optimizations here listed around binary encoded API, caching and parallelization and scheduler that we're planning for the 1.3 timeframe. So those are the main areas that we've seen we've invested in with 1.2. And at this point, I'm gonna hand it over to Clayton to talk about the investments we're making in 1.3. Great, thank you, Alan. So can everyone hear me? Absolutely, yeah. All right. Okay, so as Alan covered, there was a lot of great work in Kubernetes 1.2. And I think you'll notice a theme as Alan was alluding to that we're continuing a lot of that work into Kubernetes 1.3 and the 1.4. One of the key goals for Kubernetes 1.3 was to have a very predictable release cycle. And so that release cycle being predictable helps ensure that we can deliver these features, get feedback and then move on. So for 1.3, there was a bunch of goals, but I boiled them down to really the three that I think are most relevant for people who are interested in what's coming and as well as people who are looking to make the decision to move applications onto Kubernetes. And those three short-term goals are to support the more diverse application workloads. A lot of the buzz in the container space has been about being able to take pretty diverse applications and be able to run them inside of containers. But the next step is really important is do those application workloads actually run well in clusters and making sure that Kubernetes has the tools to enable that? Performance is a huge goal. The better the cluster performs, the less we have to worry about when it comes to can we scale to meet the most demanding needs. There's just a lot of opportunities to improve the overall responsiveness and as quite a few people are famous for saying performance is a feature. And finally, it's not just about running the workloads and it's not just about supporting larger workloads, but it's also about being able to understand what's actually happening in the cluster and for the cluster to be even more, even better able to react to the changing conditions and keep those application workloads healthy. There's a pretty massive roadmap for features that we want to add to Kubernetes, capabilities that come from some of Google's long experience in the container and cluster management space, as well as other contributions from the wider scheduling and cluster management communities like Mesos, which we have an opportunity to set a direction and continue to improve how well those applications run. A feature that has received a lot of attention that we are targeting for 1.3 is Petset. And the name is somewhat of a working name, but the basic idea being that it's pretty easy to run a scale out cluster of a bunch of independent containers that don't have to talk to each other. If the containers can come and go, you can make a lot of assumptions around load balancing and trying to keep those pods up that makes it fairly easy to say, yep, we can run 30 of these, give or take five or 10. But when you flip that around and you say, well, it's not just scale out horizontally scalable stateless applications that we want to run. There's a whole other class of applications out there in the world, which are stateful, which have certain needs regarding network identity and the ability to join clusters that we feel is extremely important to support at a fundamental level in Kubernetes. So the work that's coming in Kubernetes 1.3 will be to deliver an alpha version of Petsets. And a Petset is a lot like a replication controller or a daemon set in that you define a template for what you want your pod to look like. But instead of the replication controller, which is optimized to create multiple copies of your application, but each individual copy isn't very special, the Petset is intended to ensure that each of the pods that gets created has a unique identity that's stable over time. And so an example would be if you're running a ZooKeeper cluster, you need at least three members to have a quorum to be able to remain highly available, even if one fails. You need persistent disks so that ZooKeeper can store any of the cluster state. And if one of those instances dies, you want to bring back an instance that actually matches not just the volume, but also the network identity of that ZooKeeper instance. So in the drawing down below, you can see I've just done a very simple, you've got one member, two and then three. The Petset makes it easy to ensure that you have three instances of ZooKeeper running, but it also does a little bit of extra work to say it will work its hardest to ensure that there's only ever one of the first ZooKeeper instance and that the first ZooKeeper instance actually knows that it's ZooKeeper one as opposed to ZooKeeper two or ZooKeeper three. And there's a ton of edge cases in this space generally. Our goal for Kubernetes 1.3 is to set a foundation and then enable much more sophisticated cluster management inside of the images so that people can realize that goal of they can have a pre-canned ZooKeeper cluster that they can scale dynamically, perhaps, that can recover free failure and that is in general a low touch operation for administrators and that this applies not just to things like ZooKeeper, but also things like databases, which want to ensure that there's only a single instance running of MySQL master when it dies and comes back. The IP address or the DNS name of that, MySQL cluster doesn't change. For instance, if you have slaves or replicas built around MySQL that even though the individual pod comes and goes there's still something representing that unique idea of the MySQL master. And right now there's examples that are going up if you follow the link to the pull request on this slide. We're working through a number of examples and we expect to continue that throughout the one for timeframe. Along with petsets, one of the scenarios that we think is really important in Kubernetes is being able to take applications you've containerized and adapt them for the Kubernetes environment.  For instance, from a vendor or there's somebody else in your organization that's providing an image. And there's some initialization you need to do or you want to wait for a number of other services to be available before you actually start your containers. We are adding the init container concept as a way of giving application developers a tool for separating out responsibilities. So the easiest example is waiting for a dependency to be ready. So each init container gets a chance to run before the rest of the containers in the pod are started. So if I have a pod with three application containers and two initialization containers, the first init container will run to completion. If it succeeds, the second initialization container will run. And if that succeeds, the rest of the containers will be started. On the other hand, if the first init container fails, it will block the startup of the second one. We won't actually ever start either the second initialization container or the regular application containers until each one completes. And this will give application authors and administrators tools for splitting out responsibility. So you can have an image that has all of your tools for waiting for service dependencies or talking to a central service discovery registry and registering yourself or downloading config or secrets from external configuration sources to do that before the application containers start. And because these are just regular containers in the pod, they can share volumes. So you can write data to desk that your application containers can then use. And this, we think of this as an enabler for a number of types of features, including Petsets, because the initialization container can help take some of the load off of the EtsyD image or the ZooKeeper image, for example, by doing initialization on disk of what the desired configuration would be.  Our goal is to get more feedback and to look at some of the needs of both Petsets and regular applications, and then to use that to guide where the init container concept goes. Another feature coming in cube one three is the ability to run recurring jobs on the cluster. Today, there is a job concept, which as Alan mentioned, has been graduated into a beta state, I believe. That job is the idea of taking a, again, taking a pod and running it until at least one succeeds or at least n succeed. The scheduled job is a job that is run at regular intervals. So the scheduled job actually is a template for another job, just like a replication controller is a template for a pod. A scheduled job is a template for a job that is run. We're using the crontab syntax to start for anybody who's familiar with that.  And the goal will be that you'll be able to easily set this up from the command line in both Kubernetes and OpenShift to be able to say, I want to set up a container to run at the specific interval and to be able to customize what that container does. So as an operation shop, if you want to run recurring tasks nightly or if you want to run them on an hourly basis, for instance, to go and clean things up on the cluster or to run a number of checks on the cluster, that should be much easier to accomplish once scheduled jobs land. And finally, I think the fourth of the biggest application-focused features that's coming in Kubernetes 1.3 is as we've talked about really from the beginning of Kubernetes, a goal is to be declarative about the system, which allows a number of config management on top of the system, as well as making it easier to reason about how all the different components of the system can work together. And a pretty important concept that underlies a cluster is it's not enough just to have the ability to keep applications running, you want to minimize the impact, you want to minimize the impact to the cluster when applications are restarted or minimize the impact to the application when the cluster needs to take action. And the concept in that's coming in alpha, Q1.3 and the timing on this is a little uncertain. We've got some of the basic API objects, but many of the components of the system may not be taking advantage of it in 1.3 is to have a central place for components to both check whether a particular action would be impactful to an application in a way that would violate the admin or user's desired policy and also a place for the cluster to record the impacts of restarts and nodes going down on the application so that future interactions from that component take that into account. So an example might be an application that defines that it needs three nines of uptime. And the time period that each month that it can be down is recorded and tabulated if an administrator wants to go drain a node to be able to take it out of operation and then bring a new node in place and there happens to be a component of that application running on there, the admin will be able to check in today, we'll be able to check in through the tool on whether this would violate the availability of the application. And the goal is as we move forward to make more and more components of the system automatically take disruption budgets into account, which really is the first broadening of the step around policy. It's not just about I wanna have five of these running, but I want to ensure that no matter what across the course of a week or the course of a month or a year that I'm meeting some sort of goal for the service whether it's SLOs or SLAs, depending on who's providing the service the cluster is working with the application author to make sure that the right application or to make sure that disruption is minimized. And this also sets up a number of other elements of the cluster that will be able to more intelligently react just by virtue of having a central place to aggregate, how much disruption and applications willing to tolerate. So there's a ton more application features coming, but ultimately if you can't run things effectively that tends to, if something's slow it doesn't really matter how many features it has. And so performance is a big focus for us. One of the biggest changes that will start showing up in the cube one three timeframe into Kubernetes one four is the next version of SED which is an evolution of the work in the current SED two version to be a much more powerful key value store. The core OS SED team and some of the folks within the Kubernetes community have worked closely together to take a number of requirements and pieces of feedback and use that to guide the evolution of SED. Some of the easy wins will be the underlying storage model of SED will be much more efficient. Keys and values will be treated slightly differently. Values will be stored on disk so the memory requirements of SED should in general drop. We'll be storing binary values on disk instead of text values. So this opens up some other optimizations I'll talk about in a little bit for efficiency and improving the amount of data that we can store in a single SED instance. There are, there's a new feature coming that we don't plan to take advantage of in cube one three which is the ability to do very limited read write transactions but there are a lot of performance optimizations in Kubernetes around scheduling as well as around the efficiency of some of our core operations for updates that will allow us to do, this feature will allow us to get some very significant wins as we add new features in the future. The watch function which is really, it's a fundamental SED concept, the ability to connect to an SED server and to see all of the changes that happened since a very precise time in, is anyone familiar with the Kubernetes API as we depend on this feature a lot? Watch is a fundamental property of the Kubernetes API as well which means it's easier for administrators and developers to build integrations that can react to things that happen on the cluster in a very efficient low latency manner. The watch function in SED three has been turbocharged and we are expecting to see some pretty significant efficiency gains which will mean that both the cluster will get more reactive. The latency between the time a pod is scheduled to when it starts we'll see a lot of wins in that space as well as just in general increasing the number of watches altogether so that it will expand the range of integrations that people can build to the Kubernetes cluster to allow us to support even more control and integration points with third party software. And the timing on the SED three official GA release is roughly congruent with the one three timeframe but I expect that there'll be some soap time that we want to put just to make sure that we can recommend this as a production configuration for folks in the cube one three timeframe. A second big optimization is as Alan alluded to is being more efficient about how we store data in SED as well as how we transmit data back and forth between all the components in the Kubernetes cluster. Kubernetes talks to itself a lot so the Kubernetes nodes talk to the API servers the scheduler talks to the API servers controllers talk to the API servers and any wins that we can get and how quickly we can take API objects and convert them onto the network and then back off the network has a multiplying effect through the rest of the platform. So we are planning on introducing a serialization format that is an optional format alongside JSON for the majority of the Kubernetes objects for inter cluster communication but for people doing integrations we will probably publish a protobuf spec that people can write against if you wanna use any of the existing protobuf implementations in the many different languages out there. Generally it's for efficiency we're kind of hedging our bets a little bit we wanna make sure that we're gonna use this internally for storing an SED as well as inter cluster communication we're probably gonna hedge a little bit and recommend people be a little bit cautious about using this as clients just so that we have enough time to make sure that we get the right model in place. We're actually generating our protobuf schema from the internal objects and so there's some fiddly details we don't want people to be disrupted. So expect to see a lot more about this over the next couple of releases. And finally on the performance front there's just a whole host of optimizations that lots of different people from a lot of different companies and individuals has been made to the cluster already. Just some real big wins across the board so we're very optimistic about Kubernetes 1.3 we're not gonna give any early numbers but it's very promising about the improvements that have been made and even better those performance improvements are gonna allow us to keep adding features and keep improving the ability of the cluster to run itself. I'm gonna jump through a few of these remaining slides but along with better applications and more performance we really do want to make the cluster easier to run and manage. Federation which is the idea of having an API that looks a lot like Kubernetes available for talking to multiple clusters and having very specific use cases that lets you replicate pods across a set of clusters in a consistent way as well as having services that allow you to globally access those is targeted for Alpha for Kubernetes 1.3 there's a bunch of great work going on. It is a pretty big feature so I'm gonna hedge a little bit and say we expect to get all the foundational pieces in Kubernetes 1.3 but some of that will trail into Kubernetes 1.4 and we're gonna take baby steps on this the goal really is to enable people who have clusters and multiple geographies to easily replicate applications across a wide range it will not be the full set of Kubernetes features magically being replicated across every cluster and that's where future work will focus. Notification will help ensure that over committing the cluster and getting better resource efficiency as possible so for those familiar with how Kubernetes handles pods today depending on what resources you request you get put in a quality of service tier the three quality service tiers are best effort which is you take what you can get burstable which is pretty much the normal scenario for anything that tends to want to stay on one machine and get a certain amount of CPU and memory and then guaranteed which is the highest tier and guaranteed are the most important services in your infrastructure the ones that can tolerate the least disruption the work that's being done in Kubernetes 1.3 will look for conditions on the node that due to over commit there's too much memory or too much disk being requested and then to kick pods off the nodes starting with the lowest quality of service tier best effort and then moving up to higher tiers is necessary I believe we have discussed kicking off burstable but not guaranteed off of a node and this sort of reaction eventually will hook into disruption budgets but there's also other integrations into the scheduler so that you get booted off a node and the scheduler doesn't just put you back on the same one when the replication controller spins that up so there's a number of iterative steps we'll take over the next couple of releases but this will allow the cluster to be much more reactive and nodes to be much more reactive about their health cascading deletion is something that's been requested by quite a few people building user experiences around Kubernetes as well as users who want to centrally manage want to easily manage the cleanup of the diverse number of resources that can compose a Kubernetes or OpenShift application cascading deletion will be a way of on a pod or replication controller, deployment, service, route any of the concepts in both Kubernetes and OpenShift to declare who your parent or owner is and when the parent gets deleted you get deleted as well so for instance in OpenShift we're looking forward to this feature it'll allow us to define templates that your parent service might be your front end but if your database if there's no point to your database sticking around if the front end gets deleted you can define a relationship between the database service to the front end service and when you delete your front end service it will actually go and ensure that all the rest of the resources are cleaned up so we look forward to taking advantage of this to give regular end users who are creating and then deleting apps and then kind of an iterative flow to be much more efficient network policy was something Alan described a little bit of policy is getting a little bit more sophisticated we're hoping to get some of the first API objects that represent those more sophisticated policies into Kubernetes 1.3 and a number of the network vendors including OpenShift SDN are hoping to take advantage of this as quickly as possible in OpenShift SDN we have the multi-tenant SDN plugin today we'd like to transition to that so that it's just a native Kubernetes that uses the native Kubernetes policy object and then to layer on top of that a lot of the advances that we've been discussing in the network SIG around improving the ability for you to isolate individual namespaces or even components within those namespaces to give you the security policies that you want on your applications and again there's so much to there's so much coming and being worked on across such a huge range of collaborators I couldn't even fit all of them in but the pod and node metrics API and Heapster becoming more official so that it's something that you can rely on and expect it to be stable over time is something that's being worked on the downward API for resource limits so that pods can get access to the amount of memory or CPU that an application has available Sec Comp support which finally merged into Docker quite recently will allow us an even deeper control over what applications can do with regards to syscalls which in OpenShift allows us to deliver a much better multi-tenant experience closes get another potential avenue of attack if you want to have multiple tenants on the same machine a lot of usability improvements in cube control AWS cloud provider has gotten a ton of other work and then in the storage side we've been squizzling some things around to make them easier to maintain and I would expect in cube 1.4 to actually see some of the really exciting stuff with cube storage classes start to come into play so we'll have more of that when we start talking about Kubernetes 1.4 So that is all for me, Diane. All right, well that was enough and that brings me to probably my first question is nobody's talked about dates or timelines for the 1.3 release do you have a guesstimate or a time when that's supposed to be out the door? So our goal is to start freezing the code and the next week and a half that'll be the kind of traditional slush period after that where features are supposed to be in and we're trying to finalize the details the rough schedule has it about the end of June and so just based on we're trying to get a little bit better about how rigidly we hit that date so you might wanna hedge your bets a little bit but I think we're looking at the end of June and early July. We did have one question which takes us back a little bit to the disruption budgets asking how to measure the impact of an application is it based on quality of service level or just the number of restarts and uptime and that was Elon and if you'd like to follow up just unmute yourself, here we go. Hi, this is only, can you hear me? Yes, we can. Hi, I was just curious about when you talk about this disruption budgets how do you measure the impact of applications? Are you talking about certain quality of service level or just the number of restarts or time of uptime? The focus in the initial stages is on uptime and the availability spread of the application so a more available application the impact of downtime would be measured appropriately I think because it's very early in this the goal is we do have a number of discussions and long-term plans to greatly improve the number of things that a disruption budget could take into account the focus initially will be on uptime of a particular pod from the time it goes ready to the time that it goes into graceful termination. I see, thank you. All right, and Mark is asking about any updates to identity and access management in Kube coming? So I did have one more slide. So there's a number of features from OpenShift that we've been working in the Kubernetes community to get upstream with participation from others with Google and CoroS in some cases who were helping to move those features upstream. So Alan mentioned deployments and ingress which actually have parallels to OpenShift's deployment configs and routes and we think of those as being places that we're gonna continue to evolve. In Kube 1.3 there are a couple of core goals so pod security policy which is the equivalent of the OpenShift security context constraints. We'll go into Kubernetes as alpha. It will not be enabled by default but our goal is is to, over the course of the 1.3 and the 1.4 timeframe to get that in a state where by default on a Kube cluster you can start up a Kube cluster and expect to have users be constrained to whatever the default policy would be. For instance, not being able to access host paths or not being able to access the host network or run privileged, et cetera. And then OpenShift authorization which is the roles and role bindings. There's a, Eric from CoreOS is actually in the progress of helping move that into Kubernetes. And the goal would be that over time out of the box in Kubernetes you'll gradually see the full OpenShift RBAC systems in Kubernetes and we're incrementally dripping it in so that we get, you know, soap time and testing and we can take it as an opportunity to kind of do a V2 of authorization to add in some of the things that we would, you know, we've learned from running an OpenShift RBAC for the last year. There are a couple of other authentication related features but I would say that the progress on pod security policy and authorization would be the biggest ones. Great. There is another question from Alec. What's the plans about HTTP2? That is a great question. There is some enablement going on right now. Tim St. Clair from Red Hat and a few others from his team have been measuring and impacting on the API server side to enable it for the clients to reuse connections between the masters and the servers. In terms of applications running on the cluster, you know, anybody should be able to take advantage of that. There's been some discussion of it for Ingress. So any of the native HTTP2-capable Ingress controllers would be able to take advantage of it. And then the last holdout really is the exec and port forward APIs in Kubernetes leverage the speedy protocol. And we've been working with the Golang maintainers of the HTTP2 library to ensure that we can reuse the HTTP2 library in order to move those protocols forward. So there's a lot of movement across the board. I would expect it to move into Q1.4 as well or to happen over the Q1.3 and Q1.4 time frames. All right. Well, those are the questions that we have in the chat. Does anyone else, is there anything else you want to add, Alan or Clayton? No. Oops. I'm good, actually. Here comes one more question. Any audit log improvements in version 3.1 there's still no audit log? So one of the things that is happening in OpenShift is we did a quick initial implementation of an audit log for that at least on paper is targeted for OpenShift 3.2.1. There is a discussion about we'd like to move that into Kubernetes and there've been a number of discussions. We were actually somewhat waiting to get the OpenShift authorization work upstream and that gives us a little bit more of a foundation to include information along what the user can do, what their effective role is. But I would expect to see something like that in Kubernetes and OpenShift very soon. So Alex, there's another follow-up from Alex. So in long-term, HAProxy will be removed from OpenShift, he's asking? No, so the plan for, and this is more of an in-progress thing, as we add features to Kubernetes, we really wanna do it in a way that minimizes impact to OpenShift users. So for instance, routes, we worked upstream on Ingress to ensure that Ingresses were a little bit more flexible. There is still ongoing work to improve routes, including features coming in 33-like AB routing. And our goal is over time to converge the API objects, so it's a seamless transition for users. So for instance, if you're using routes or deployment configs today, our goal would be for that to be seamless for you and to be able to take advantage of Ingress without any impact as those features are enabled. So as an example as well, Kubernetes deployments are simpler and implemented in a slightly different way than OpenShift deployments. So they don't allow you to run hooks or to run custom deployment processes. Where our goal is to just keep adding those features to Kubernetes until there's parity. And then to end users, you can keep using deployment configs or you can keep using deployments and there's no impact one way or another. You can, they will be seamlessly interoperable. Cool. So there's another question. What is the suggested way to install OpenShift Origin on top of existing Kubernetes cluster? That might be a topic for another whole conversation. I think that's a good question and it's a tough question. A lot of the features that OpenShift provides, there's features that run on top of Kubernetes like deployments and builds. And then there's features that are deeply integrated into Kubernetes like the security work around security context constraints, the security controls around builds and services that prevent malicious parties in a multi-tenant environment prevent malicious parties from going in doing things that are unsafe on the cluster. And one of our goals over the one three and one four releases have been where possible we're upstreaming those features and the sticky bit really is it's totally possible to run OpenShift on top of Kubernetes today but you do have to give up some features in order to do that. And we're kind of looking to converge over the next couple of releases so that all of the core capabilities for security are possible to be layered on top of a cube cluster so that even if they're not in cube directly you'd be able to enable them without impact without having to reinstall Kubernetes. So it's on the roadmap but it is an evolving target really just to be able to ensure the level of security that we want to guide. There's a lot of hook points that aren't quite totally flexible yet in cube. Chris John is asking are there any improvements in AB testing support? I think that's very... Yeah, from an OpenShift perspective we plan to add support in routes for AB services. So the first pull request is actually just opened to OpenShift on this. So a route points to a service today and you'd be able to specify additional target services and give those percentages or weights and have that be reflected by the HA proxy router. And we have discussed a little bit of that around ingresses but I think it's not gonna be a one three goal because we had already locked in cube one three before we started working on the feature so it will eventually come to ingress as I suspect. Well, we're almost to the end of the hour and I hope everybody found this as helpful as I have and we'll repeat this I think after the next release doing present and future as a format is actually quite good so you can figure out where we're actually at and where we're going. This has been incredibly helpful, Clayton and Allen. So thank you all for coming. We'll be posting this recording up very shortly on the blog.openshift.com and also on YouTube so you can look for it and I'll post it to the mailing list for OpenShift Commons. If you have further questions, you can post them to the OpenShift Commons mailing list and we'll send them along to the Clayton and Allen and the other folks. Clayton, maybe one last thing I would ask is if someone wants to get involved in the Kubernetes project or what is and or to have another feature added in what's the best way for them to connect? So what we've tried to do and maybe Allen can speak to this as well, what we've tried to do is be a little bit more formal about the beginning of each Kubernetes release, working and soliciting feedback from the community, identifying people who might want to contribute features and trying to shepherd them through the process so that we can ensure that even if you aren't yourself able to directly contribute a feature that your voice has heard and you'll be able to focus, you'll be able to get input in. The Kubernetes SIGs are a big part of that and then it for people who do wish to contribute features, we've been working a lot to try and streamline how easy and how approachable it is to understand kind of the process and to make it more of a community owned road map, not just a community influenced road map. Again, I'll second that as well. The SIGs are a great way to participate with the community. We have SIGs that happen on a weekly basis so go ahead and check out the SIGs and if there's one that's interest you join and if there's a particular SIG that you feel isn't represented or a particular topic area, please suggest and we can look at starting up a new SIG. Perfect, well, I think that's a great way to end. Again, I'll thank you both for taking part in this comments briefing and we will have it uploaded shortly and we'll grab the slide decks from both of you and have those available as well. So thanks again and we look forward to redoing this again after the next release.


QuickBooks Online 2024. PayPal bank feeds import data to QuickBooks. Get ready and some coffee because we get work done on time with QuickBooks Online 2024. First a word from our sponsor. Yeah actually we're sponsoring ourselves on this one because apparently the merchandisers they don't want to be seen with us but but that's okay whatever because our merchandise is is better than their stupid stuff anyways like this CPA thinking cap for example CPA thinking CAP you see what we did with like with the letters and this CPA thinking cap is not just for CPAs either anyone can and should have at least one possibly multiple CPA thinking caps why because based on our scientific survey of five people all of whom directly profit from the sale of these CPA thinking caps wearing this CPA thinking cap without a doubt according to the survey increases accounting productivity tenfold yeah at least yeah apparently the hat actually channels like accounting energy from the quantum field ether directly into your head allowing you to navigate spreadsheets faster it's kind of like how in like the matrix when neo learns kung fu or at least that's what the scientific survey saying so get one because the scientific survey participants could really use some extra cash if you would like a commercial free experience consider subscribing to our website at accounting instruction dot com or accounting instruction dot think of it dot com here we are in our quick books online bank feed practice file we set up in a prior presentation opening the major financial statement reports as we do every time the reports on the left hand side within the favorites right click it on that balance sheet to open a link in a new tab right click in the P&L profit laws otherwise known as the income statement to do the same as well as with the trustee TB trial balance if you don't have that trial balance in the favorites you can search for it let's tab to the right close the hamburger change the range we're going from 010124 tab 033124 tab selecting the drop down so we can see it on a month by month breakout and run it let's tab to the right close the hamburger again change the range 010124 tab 033124 tab drop down so we can see the months and run to refresh one more time tab until the right close in the hamburger and then we'll change that range 010124 tab 033124 tab selecting the drop down months and run it almost forgot on that last one that last rep very barely got that one out let's go back to the balance sheet we've been working on our bank feeds and we've been first connecting the checking account with the bank feeds and then we noted that you could connect the credit card account so we connected the credit card account down below now other things that you can connect to the bank feeds could include other checking accounts savings accounts and possibly even brokerage accounts if they're with your financial institution where you might be investing say in stocks and bonds for example this time though we want to connect to the PayPal which will act kind of like another checking account so let's go over a quick little history of the PayPal note that PayPal used to be able many people used it to facilitate electronic transfers more easily than on their other banking system possibly being able to facilitate transactions with other institutions and possibly even you know in other countries and that type of thing from a content creator side of things many people used something like a PayPal as an intermediary in order to collect funds which would be more secure possibly with on their website or payments from third-party platforms and in that sense then PayPal was kind of an intermediary platform to help facilitate the transactions and then but more and more these days PayPal is being used to make normal payments as well so if you're making payments with PayPal more and more it's going to look like a checking account so how you're going to set up PayPal within QuickBooks might depend in part on how your what your flow looks like in terms of how you're treating PayPal is it just a source to facilitate deposits from other platforms that you need and then you're just going to transfer that into your bank account as soon as you get the money or are you using it more like a checking account and are if you are having more advanced uses of it are you using it in more difficult transactions possibly with other currencies where you might have fees related to the transaction and do you want to break out those fees all right so why is that important well let's go back to the first tab and let's go to the transactions and then we're on the bank transactions so if we look at our checking account let's imagine a situation where we're just going to use PayPal as a tool for us to collect money from possibly from like our website or possibly from another platform that wants to pay us through the platform of PayPal rather than going to it to an electronic transfer to our bank and all we are doing then is taking the money out of PayPal as soon as it goes there and putting it in our bank well if that were the case then we might not need to connect the PayPal right we might say do I need another PayPal up here to connect to the PayPal well if you're just using PayPal to say someone sent me money YouTube sent me money let's imagine they sent it to PayPal for whatever reason then I'm going to wait till it hits PayPal and then I'm just going to transfer it into my checking account then you might you might just say well it would be easier for me not to connect the PayPal because I'm just going to wait till it connects to the checking account something similar to the stripe here which which is a similar kind of intermediary platform and I'm just going to record it as revenue once it hits my my checking account that would be the easiest thing to do so if I had my own website for example and people paid me through PayPal sometimes right if if all the money that's going through PayPal is from my website and I'm just using it as a tool to facilitate the transaction which is a secure format of the transaction and then I take it out of PayPal as soon as I get it and put it into my checking account then everything that is deposited from PayPal I can assume came from the transactions for my website and I could just record it as revenue once I see the transfer into my checking account you see how that might be the easiest kind of format in that type of scenario however once you start using PayPal you might start using it for other things if you're paying things out of PayPal like you're paying expenses out of PayPal for example and sometimes that's just an easier thing to do sometimes that's an easy way to connect your your online payments then it's more likely that you're going to need to connect it to another bank feed if you could or use the app which we'll talk about shortly because then then you're going to want to pull in all the transactions that you're paying out of PayPal so if you're making normal monthly payments out of PayPal then you probably want to treat it like a checking account or if you get paid from multiple places let's say YouTube your website and you know another platform all pay to PayPal and then as you get the money in PayPal you transfer that money to your checking account well then it gets a little bit difficult to categorize the payments if you want to break out the payments that came from YouTube versus your website versus the other platform then it might be a little bit more difficult to to differentiate that if you transfer it from PayPal into your into your QuickBooks with one lump sum right so that might be another reason why you would need to break out the PayPal as it as its own as its own checking account that you would connect directly to the bank feeds or use the app you have a similar situation with like Stripe and other types of payment platforms if they're working as an intermediary platform the question is like if Stripe was facilitating transactions from my website do I really need all the transactions that are coming from the website in QuickBooks most people would probably argue no it might be better to use that to use your website detailed data for Stripe to see the transactions there and just pull in the end result meaning once the Stripe hits my checking account I'm just going to record it as revenue if I have a website I don't need to try to track all of the sales to all the individual customers that might not help me all I really want is to be tracking the enough to have their them on my email list right so I can send them basically emails I don't need to see every transaction in QuickBooks because that will log down or slow down possibly the system it's more detailed than you need in QuickBooks would be the general idea all right so let's imagine that we're going to connect PayPal and we think that we need to have PayPal as like a checking account system with the bank feeds there's two ways that you can basically do it you could add PayPal with the normal bank feeds so if I link to a bank for example then I can search for PayPal and they have the PayPal connection here so you'd have to choose you know the appropriate PayPal but you've got your your PayPal connection possibly this way and down here here's the here's the PayPal.com right here and then they have a PayPal app so the pay if you go to the app store if I close this back up and I open up the hamburger and I go down to my apps down here and we search we go to the search field I'm going to close this up and I search for PayPal then we have the PayPal app here and the PayPal apps I think this one only has 15 so this one has the 2006 71 so I think that's the main one let's just take a look at that I'm not going to connect the app but then you can look at the you can watch the review on the app here and basically the app seems to basically act similarly as the bank feeds however the PayPal transactions could get a little bit more detailed when you look at them because they might charge fees for the transactions especially if you have different currencies of the transaction so they can get a little bit complicated if you deal with multiple currencies for example and you have fees related to the transactions so if you just use the normal connection of a bank feed I think it's going to be a little bit more streamlined that it might not possibly break out some of the added items such as fees and whatnot and and just record you know the net amount whereas I think the app is an attempt to do a similar thing as the bank feeds but give you more detailed information so you can break out some of the transaction in more detail now there's pros and cons to either of those methods right because just like the question as to whether I should connect PayPal or not if I've I've if I have a very simple system and I don't need to connect PayPal it would be easier not to if I just deposit it into my bank and I'm only using PayPal to collect the deposits I might not connect to PayPal because it's another step I don't need the app is a similar issue possibly right because it's going to give you more detail if you don't need that more detail then the app might actually be more complex than it needs to be so you might just go with the normal kind of bank feed but if you want to break out that more detail if that becomes important because the fees are messing up your flow or something like that the the app might be a better solution than connecting with the bank feeds so you could see here it's got three stars two thousand six seventy one on the rating you can look at some of the ratings down below so you've got some you know happy and some non-happy ratings to see which of those would be the best way to go so I'm going to close this out and I'm going to go back over here now we're going to imagine we connect kind of like the bank feeds in that we'll have an upload just like we did with the checking account so I'm going to go back into the transactions and we'll upload another card here for basically PayPal transactions so it will act like us to us as though another you know checking account connection that we would have so remember that the general process when you connect a new account is you can link to it here you can search for the account and then it's going to ask you to add an account for it and then you can set up the actual general ledger account the general ledger accounts are over here so let's actually do that first so I'm over here and I'm going to say let's add another account new and then I'm going to say let's put in place a bank type of account and it's going to be a we'll just call it a checking account and I'm going to call it PayPal PayPal and so there it is to do it and so I'm just going to save that and so now we have our PayPal account if I select the dropdown you can connect to the bank here or we can go again back over here now that we have the account set up and we can connect to the bank and then tell it to go to that account or we can upload from a file which is what we will do I'm just going to make some mock transactions over here in Excel that are coming that we're going to imagine coming from PayPal date and this is going to be the amount and then the description so let's say on one three to four let's say we had three thousand come in from three thousand from YouTube and then the bank feed data would have you know their bank feed jargon and what not within there let's say on three seven two to four we also had another one thousand one thousand come in and we'll say that that comes in from our our website so let's let's put it in from let's just say it's an online platform so we'll just say it's an online platform we'll just make it generic and this would be two four and then let's say we have on one fifteen another one thousand two fifty from our website and then we'll put some numbers behind it for the bank jargon and then let's say that we made some payments this is going to be one fifteen two four one fifteen two four now the date it's formatting funny or different let's do it like that and then so then let's say that we had some money going out on one twenty seven two four let's say we now we have money going out of two hundred and thirty two out back again out back stay house and then so something like that and so then so so let's go ahead and save this I'm going to say save it to do do do right before I save it I also want to make a couple more here I'm going to say let's say on oh two oh one two four we had a transfer out of here for let's say four hundred dollars that goes to the checking account and we'll practice our bank transfers here and let's say on two to eleven let's say two fifteen two four we have another transfer going out which was let's say four hundred and eighty dollars to the checking account and let's say on two twenty one two four we have another two hundred going out to the checking account and let's do a couple more just so we can practice our transfers on two twenty five two four we have another five fifty going out to the checking account okay so that should be enough to practice our transfers let's go ahead and save it and then I'm going to say file save as and we'll save it as a CSV file comma deliminated file and so okay and then we'll save it like that and then let's go into our bank feeds and I'm going to add another card so I'm in the bank transactions selecting the drop-down I'm going to upload a file upload it it's going to be to do a there's the one five one it's the PayPal file upload that one continue and this time I want to select the drop-down and not have the checking account but the PayPal account that we set up so it's going into the pay if you didn't have the PayPal account you could add it as you go but we added the PayPal account so then we there we have it is the first row a header it is one column that looks good this is the format of the date month month day day year year year year dates lining up description lines up amount those are our headers that are pulling in so let's continue and everything looks good so right the YouTube's an increase online platforms an increase our it should be out website but our websites an increase out back these are all decreases and in the transfer so that looks good let's continue and say so QuickBooks will import that's what we want you to do and then it says import completed boom it's been done all right sweet so we're gonna say then so here's gonna be the information so it looks just like you know the checking account now because we're using it more and more like a checking account and so now you can see that if if we is it is am I on the there's my PayPal this is the there it is it looks so much like the checking account I was in the checking account didn't even know it just like me being a poet and not knowing it you know so but any case here's the so so now you can see that if we waited till these cleared the bank these items are also gonna be on the checking account side and but but if when they clear the bank they're just gonna come from PayPal let me show you that I'm gonna say save file save as to do and I'm just gonna make this an Excel file so this is gonna be number two and then I'll put this is gonna be for the checking account so let's say on the checking account I'm gonna have these same four transactions but they will be positive you know how I can change that easily just a Excel trick for you I can copy that and then paste it special and then subtract boom now it's positive copy paste and then delete that oh fancy and then I could say from PayPal or PayPal transfer right boom so I'm gonna save that to do it and then I'm gonna file save as as a CSV file you will have these files if you're using if you have access to these so a CSV file these valuable files and then hold on it says do you want to replace it as a CSV file bank feed number two I'm gonna say it let's say make it number three I thought it was an Excel file before let's save it number three and then okay and then I can go back in here and this time I'm gonna put these into the checking account so I'm gonna say as though they float into the checking account upload from file upload and here's the new the newbie is number three this time why is it up here instead of down there I don't know I think that's the one I think that's the one and then I'm gonna say continue put this one into the checking account boom it's gonna be yes columns one date formatted date date description description amount amount movie be in got two letters for you be in be in so this is going in from PayPal when we transfer it so that looks good let's do it and see if it works so it's gonna transfer four of them import complete all right then that's what I'm talking about all right so then we can see down here that of course as it hits the bank it's gonna be paired to what happened over here on PayPal because it sees them both it's not matched because matched if it was if it was matched that would mean it's matching to a transaction that has already been in that has been imported already these are both in bank feed limbo neither one of them have been connected to our actual financial statements yet now if they if I didn't have PayPal then what would happen I wouldn't have this tag and I would only see it come through the bank over here on the deposits which I could just record is revenue when they come through however I don't have the detail I can't see what the revenue was for this way it might not matter if there was only one source of revenue and I knew it was all coming from YouTube or something right that I could just call it YouTube revenue and I wouldn't need the PayPal account but in our case we had multiple revenue sources here so that might make it worthwhile to connect because now I can distinguish the PayPal from my website because that will come through the bank feeds on the PayPal side and because I'm actually using PayPal to spend money then I don't know if Outback Steakhouse takes PayPal but but if I'm spending money on PayPal then I'm kind of using it like a checking account and you would think I would need the more detail in there and again if you were doing transactions that were in another country or something where PayPal there's going to be fees for the currency transfers then then you might you might have more detail with regards to the fees where if you wanted to break out those fees in their own account it might be worthwhile to have another app right that could break out the fees which may or may not be useful to you you might just say I don't really need the fees are kind of maybe in material with regards to the be nice to break them out but maybe it's more complexity in the system than I need so so so that those are the trade-offs that you have to kind of think about so we'll record these transactions and think about those bank feed transfers in essence between the two bank feed files and future presentations as we do we'll see the difference between using an expense form to record a transaction versus a bank deposit form versus a transfer form and what we've already kind of seen that with a similar form which is the pay down credit card form all of these forms could be used to record the same kind of transaction going from PayPal to the checking account for example and so we'll say why would one be better when would you want to use one versus the other what's the difference between those forms we'll get into that in more detail next time nothing new has happened to our our data on the balance sheet income statement or the trial balance which is just the balance sheet on top of the income statement as of yet because this data has just been pulled into what I call bank feed limbo PayPal bank feed limbo that is and we still need crucial information before we can pull it in here as with normal bank feeds the one piece being the mandatory piece and account we need to assign it to an account and we'd also would like to assign it to a customer or vendor whether it be deposit or a decrease


From around the globe, it's theCUBE with digital coverage of AWS re-invent 2020. Sponsored by Intel and AWS. Welcome back to theCUBE's coverage of AWS re-invent 2020. I'm Lisa Martin. Joining me next is one of our CUBE alumni, Brett McMillan is back, the director of US Federal for AWS. Brett, it's great to see you, glad that you're safe and well. Great, it's great to be back. I think last year when we did theCUBE, we were on the convention floor. It feels very different this year here at re-invent. It's gone virtual. And yet it's still true to how re-invent's always been. It's a learning conference and we're releasing a lot of new products and services for our customers. Yes, a lot of content as you say. The one thing I think I would say about this re-invent, one of the things that's different, it's so quiet around us. Normally we're talking loudly over tens of thousands of people on the showroom floor, but great that AWS is still able to connect in such an actually an even bigger way with its customers. So during Theresa Carlson's keynote, wanna get your opinion on this or some info, she talked about the AWS open data sponsorship program and that you guys are gonna be hosting the National Institutes of Health NIH, sequence read archive data. The biologist and main former gets really excited about that, talk to us about that. Cause especially during the global health crisis that we're in that sounds really promising. Yeah, it very much is. I am so happy that we're working with NIH on this and multiple other initiatives. So the secret read archive or SRA, essentially what it is, it's a very large data set of sequence genomic data. And it's a wide variety of genomic data and it's got not only human genomic data, but all life forms or all branches of life is in SRA to include viruses. And that's really important here during the pandemic. It's one of the largest and oldest sequence genomic data sets that are out there. And yet it's very modern. It has been designed for next generation sequencing. So it's growing, it's modern and it's well used. It's one of the more important ones that I think it's out there. One of the reasons this is so important is that we know to find cures for what human ailments and disease and death. But by studying the genomic code, we can come up with the answers of these or the scientists can come up with the answers for that. And that's what Amazon is doing is we're putting in the hands of the scientists, the tools so that they can help cure heart disease and diabetes and cancer and depression and yes, even viruses that can cause pandemics. So making this data, sorry, just making this data available to those scientists worldwide is incredibly important. Talk to us about that. Yeah, it is. And so within NIH, we're working with the NCBI. When you're dealing with NIH, there's lots of acronyms. And at NIH, it's the National Center for Biotechnology Information. And so we're working with them to make this available as an open data set. What this is important is it's all about increasing the speed for scientific discovery. I personally think that in the fullness of time, the scientists will come up with cures for just about all of the human ailments that are out there. And it's our job at AWS to put into the hands of the scientists the tools they need to make things happen quickly or in our lifetime. And I'm really excited to be working with NIH on that. When we start talking about it, there's multiple things the scientist needs. One is access to these data sets. And SRA, it's a very large data set. It's 45 petabytes and it's growing. I personally believe that it's gonna double every year, year and a half. So it's a very large data set and it's hard to move that data around. It's so much easier to just go into the cloud, compute against it and do your research there in the cloud. And so it's super important. 45 petabytes give you an idea. If it were all human data, that's equivalent to seven and a half million people or put another way, 90% of everybody living in New York City. So that's how big this is. But then also what AWS is doing is we're bringing compute. So in the cloud, you can scale up your compute, scale it down. And then kind of the third leg of the store is giving the scientists easy access to the specialized tool sets they need. And we're doing that in a few different ways. One, the people who design these tool sets design a lot of them on AWS, but then we also make them available through something called AWS Marketplace. So they can just go into Marketplace, get a catalog, go in there and say, I wanna launch this. It launches the software, it launches the infrastructure underneath and it speeds the ability for those scientists to come up with the cures that they need. So SRA is stored in Amazon S3, which is a very popular object store, not just in the scientific community, but virtually every industry uses S3. And by making this available on these public data sets, we're giving the scientists the ability to speed up their research. One of the things that jumps out to me too is it's in addition to enabling them to speed up research, it's also facilitating collaboration globally because now you've got the cloud to drive all of this, which allows researchers in completely different parts of the world to be working together almost in real time. So I can imagine incredible power that this is gonna to provide to that community. So I have to ask you that, you talked about this being all life forms including viruses, COVID-19, what are some of the things that you think we can see expect this to facilitate? Yeah, so earlier in the year, we took the genetic code or NIH took the genetic code and they put it in an SRA-like format and that's now available on AWS. And here's what's great about it is that you can now make it so anybody in the world can go to this open data set and start doing their research. One of our goals here is go back to a democratization of research. So it used to be that, for example, the very first vaccine that came out was a small part of the vaccine that was done by a rural country doctor using essentially test tubes in a microscope. It's gotten hard to do that because data sets are so large you need so much compute by using the power of the cloud we've redemocratized it and now anybody can do it. So for example, with the SRA data set that was done by NIH organizations like the University of British Columbia, Cloud Innovation Center is doing research and what they've done is they've scanned the SRA database. Think about it, they scanned 11 million entries for coronavirus sequencing. That's really hard to do in a typical on-premise data center who's relatively easy to do on AWS. So by making this available we can have a larger number of scientists working on the problems that we need to have solved. Well, and as we all know in the US operation warp speed that warp speed alone term really signifies how quickly we all need this to be progressing forward. But this is not the first partnership that AWS has had with the NIH. Talk to me about what you guys, what some of the other things are that you're doing together. Yeah, we've been working with NIH for a very long time. Back in 2012, we worked with NIH on what was called the thousand genome data set. This is another really important data set and it's a large number of, again, sequence human genomes. And we moved that into again, an open data set on AWS. And what's happened in the last eight years is many scientists have been able to compute about on it. And the other wonderful power of the cloud is over time we continue to bring out tools to make it easier for people to work. So whether or not they're computing using our instance types, we call it Elastic Cloud Compute for EC2, whether they're doing that or they were doing some high performance computing using EMR, Elastic MapReduce, they can do that. And then we brought out new things that really take it to the next layer level like Amazon SageMaker. And this makes it really easy for the scientists to launch machine learning algorithms on AWS. So we've done the thousand genome data set. There's a number of other areas within NIH that we've been working on. So for example, over at National Cancer Institute, we've been providing some expert guidance on best practices to how you can architect and work on these COVID related workloads. NIH does things with collaboration with many different universities, over 2,500 academic institutions. And they do that through grants. And so we've been working with the Office of Director and they run their grant management applications in ERA on AWS and that allows it to scale up and work very efficiently. And then we entered in with NIH into this program called STRIDES. STRIDES is a program for not only NIH but also all these other institutions that work within NIH to use the power of the cloud, use commercial cloud for scientific discovery. And we started that back in July of 2018, long before COVID happened. It was so great that we had that up and running because now we're able to help them out through the STRIDES program. Right, can you imagine if, let's not even go there, I was gonna say. But so okay, so the SRA data is available through the AWS Open Data Sponsorship Program. We talked about STRIDES, what are some of the other ways that AWS is supporting STRIDES? Yeah, no, so STRIDES is a wide ranging through multiple different institutes. So for example, over at the National Heart, Lung and Blood Institute, NHLBI, I said there's a lot of acronyms, NHLBI, they've been working on harmonizing genomic data. And so working with the University of Michigan, they've been analyzing through a program that they call TopMed. We've also been working with NIH on establishing best practices, making sure everything's secure. So we've been providing AWS professional services that are showing them how to do this. So one portion of STRIDES is getting the right dataset and the right compute and the right tools in the hands of the scientists. The other areas that we've been working on is making sure the scientists know how to use it. And so we've been developing these cloud learning pathways and we started this quite a while back and it's been so helpful here during the COVID. So scientists can now go on and they can do self-paced online courses, which have been really helpful here during the pandemic, and they can learn how to maximize their use of cloud technologies through these pathways that we've developed for them. Well, that education is imperative. I mean, think about all of the knowledge that they have within their scientific discipline being able to leverage technology in a way that's easy is absolutely imperative to the timing. So, let's talk about other datasets that are available. So you've got the SRAs available. What other datasets are available through this program? Yeah, we have a long wide range of datasets that we're doing open datasets. And in general, these datasets are improving the human condition or improving the world in which we live in. And so I talked about a few things. There's a few more things. So for example, there's the Cancer Genomic Atlas that we've been working with National Cancer Institute as well as the National Human Genomic Research Institute. And that's a very important dataset that's being computed throughout the world commonly within the scientific community that dataset is called CCGA. Then we also have some datasets that are focused on certain groups. So for example, Kids First is a dataset that's looking at a lot of the challenges and diseases that kids get. Every great thing from very rare pediatric cancers to heart defects, et cetera. And so we're working with them. But it's not just in the medical side. We have open datasets with, for example, NOAA, National Oceanic and Atmospheric Administration to understand what's happening better with climate change and to slow the rate of climate change within the Department of Interior. They have that Landsat database that is looking at pictures of the Earth, satellite pictures of the Earth so we can better understand the world we live in. Similarly, NASA has a lot of data that we put out there. And over in the Department of Energy, there's datasets that we're researching against or that the scientists are researching against to make sure that we have better clean, renewable energy sources. But it's not just government agencies that we work with. When we find a dataset that's important, we also work with non-profit organizations. Non-profit organizations are also, they're not flushed with cash and they're trying to make every dollar work. And so we work with them. Organizations like the Child Mind Institute or the Allen Institute for Brain Science. And these are largely like neuroimaging data and we've made that available via our open dataset program. So there's a wide range of things that we're doing and what's great about it is when we do it, you democratize science and you allow many, many more scientists to work on these problems that are so critical for us. The availability is incredible but also the breadth and depth of what you just spoke, it's not just government, for example, got about 30 seconds left. So I'm going to ask you to summarize some of the announcements that you think are really, really critical for federal customers to be paying attention to from re-invent 2020. Yeah, so one of the things that these federal government customers have been coming to us on is they've had to have new ways to communicate with their customer, with the public. And so we have a product that we've had for a while called AWS Connect and it's been used very extensively throughout government customers and it's used in industry too. We've had a number of announcements this week. Andy Jass made multiple announcements on enhancements to AWS Connect or additional services. Everything from helping to verify that that's the right person from AWS Connect ID to making sure that that customer gets a good customer experience to connect wisdom or making sure that the managers of these call centers can manage the call centers better. And so I'm really excited that we're putting in the hands of both government and industry a cloud based solution to make their connections to the public better. It's all about connections these days. Brett, I wish we had more time because I know we can impact so much more with you but thank you for joining me on the queue today, sharing some of the insights, some of the impacts and availability that AWS is enabling the scientific and other federal communities. It's incredibly important and we appreciate your time. Thank you, Lisa. For Brett McMillan, I'm Lisa Martin. You're watching theCUBE's coverage of AWS re-invent 2020.


and welcome. Today, we're going to be looking at extracting attachments and messages from an email. So, when you send someone an email, it goes to their email server. It's actually stored on that server, well, depending on the server setup, but in many cases, you have what's called an inbox file or a mailbox file, which is the plain text file. And emails are completely plain text. So, I have an example here. I have an email file that has an inbox file that has three files in it against plain text. The name and extension doesn't really matter for our uses here. But, if I list out here, you can see that I have a file called toot, which is short for tutorial since this inbox file specifically for this tutorial. I can go into this file and you can see what it looks like. It has who it's from, who it's to, the names, the servers, the timestamps, blah, blah, blah. If I move down and I search for attachments, you can see that we have here, it says it's an attachment. Let's go down to the next one for a better example. So, it's plain text, but it's format, a specific way. Each email has its own little, little unique tag here. In attachments, it will tell you what type of attachment it is. It's an image. It's a JPEG. Here's the file name. It's an attachment. And it's encoded in base 64, which is at least how I know. I've always seen it like this. So, if I've talked about this in the past, base 64. Base 64 takes any file, whether it be binary or not, and it converts it to a plain text file. So, if you have a picture, a music file, a PDF, anything that's a binary file that isn't plain text, it will convert it. And that's what this, all this is right here, all these letters, it, that is our image. This is our JPEG, but convert it to plain text. And that is how attachments are sent through email, through their plain text. And as you can see, an image takes up a lot of text. And this makes it very easy to transfer data different ways because, you know, all computers accept plain text. But how do we take that out and decode it? Now, you could write your own script, which I've done this in the past, where I'll look for the base 64, extract that out, run it through base 64 to put it in a file. Then I have to look and find the file name, so make sure I put it into the file, so it's not just randomly named and it has the right extension, if it's a JPEG or a PDF or whatnot. And that's how an email file is set up. But how do you, without having to write all that scripting yourself, extract attachments? This is very useful. For example, I have filters that filter out different emails to different mailbox files. And then I have scripts that automate that when an email comes in, it does something. For example, the United States Postal Service has a service where they will email you every morning. They will scan all the outgoing mail, the letters, and they will email it to you with an attachment with a picture, with a scan of your snail mail letters. So I know what's going to be in my mailbox before it gets here. And that way I can know, oh, it's an important check coming, a bill coming. I know. And then if I go out to the mailbox when the mail comes in, it's not there. I can report it as stolen. It's also just nice to know what's coming later today. And as a side note, criminals have been using this because whether you're using that service or not, the United States Postal Service offers it to you and it's very easy to sign up. But if you don't sign up, it allows someone else to, with just a little bit of information about you, sign up for you. And then they'll start receiving your emails and they can then know ahead of time whether you're going to be getting a check or something important or they can have stuff sent to your house and they know when it's coming and then go out to the mailbox and get it before you do. So it's advised, even if you're not going to use this service, that you should probably sign up for it to help prevent criminals from using it. Hopefully they come up with a better system to prevent that in the future. Anyway, that's one option there. Another thing that I use it for is where I work. When we get paid, they send us our paychecks through email as a PDF, just the stubs because I get direct deposit. And I have all my paystubs filtered out to a certain mailbox file. So all my paystubs are in one mailbox file. So it makes it very easy for me to pull that mailbox file instead of having to go in and go, open this email, download attachment, save to here, go to the next email. I can write a script that automatically can pull those out every when they come in or if I have a mailbox file, pull them all out at the end. So that's what we're going to be looking at today. There's actually two programs I'm going to show you today. I'm going to show you one and then in the next video, I'm going to show you another option that has some features that actually are a little more advanced for this. But I have, again, this one mailbox file. So what I can do is I can use a package. So if I do apt, actually I look for aptitude when I'm searching, use your package manager, you can search for a program called mpack. And it is right here. And if we do aptitude show mpack, as you can see with the I here, I already have this installed. But once you install it, it says right here mpack and I don't know if that's mumpack or I'm going to say mumpack sounds better. Our utilities for encoding and decoding respectively, binary files from mine mine, people may follow the way I was saying this in the last video, which means multi-purpose internet mail extension, format mail messages. So basically it's for encoding and decoding them. So what you can do here, now that it's installed, I can say mumpack and I can point to that to file. And if I didn't already mention too short because it's the mailbox file for this tutorial, tutorial to not like a fart. Anyway, and it went in and it found a an attachment called text, my file dot text. And if I cat that out, you can see that it's the text file, which is great. But we already looked at the fact that there's more than one attachment in this, there's more than one email. And that's the thing that has the drawback for mumpack is that it's going to find just the the one, the first one in the email or is it the last one? No, I think it's the first one. So here's the thing when you set up a mail server, you have options to have all your emails going to one inbox file or have a directory where each email is in an individual file. So you have a folder and for a certain mail address and every email that comes in goes into its own mailbox file. It will still look like this, but for each message and we're broken down. And unfortunately, mumpack doesn't support that multi message multi attachment thing, it's just going to find that first one and attack and extract it. So you can manually split up the files if you want with a script that wouldn't be too hard. But I'm going to show you another program in the next video that will just extract everything will strike the messages, the text messages. And actually, if I give mumpack another option, so another option, if I run that again, mumpack on that file, you can see it says this file already exists and it gives it on the same file name with dot one. If I run it again, it's going to give it dot two. If I give it dash f option, it's going to tell you that that file already exists, but it overwrites it. It doesn't create a new file with an extension. Another option, if you were to look into the man page for mumpack, you can see that there's this dash t option, which writes text file parts of multi part messages as a file. So you might think, oh, so we have multiple emails in here. That's the option for splitting this up. And no, what that's going to do is it's going to, if I show you here, let's go ahead and clear the screen. We'll do dash f for force override and t for that split option and give it that. And it's going to create three files. And what it's creating here is it's taking that first email and splitting it up into its three sections, but it's still ignoring this last two emails or any other email in there. And the different parts are when you send an email, if it's formatted, and a lot of email clients do this and a lot of people frown upon it, is if you have to do formatting, bigger text, color text, it's actually sending it as HTML. And a lot of clients, even if you don't put that in there, it's still defaults to HTML. And a lot of people don't like that because it increases the size of the file for a number of reasons. And also if you have a client that doesn't support HTML, such as I use mutt, which is a text-based client. And by default, it's if when I get an email, if it's an HTML format, I'm going to see all the HTML tags, it makes it hard to read, but I actually run it through a script that strips out the HTML and formats it somewhat right for me. But when it sends it as HTML, it also sends a copy in plain text. So when I get it in, I'll have a plain text copy and the HTML copy. So now you've doubled up on that, but then you also add the tags, which also increases the size of the file. But that's what it's doing. It's split up the plain text, the HTML, and that attachment. And if I list that, you can see they're all here. So I can cat out, again, you know, parts of this message, part two will be the HTML. But again, it's still only seeing that first email. In the next video, we'll look at another program that will rip out everything from this single mailbox file. So again, you have options, M unpack, we're great if all your emails are going to separate inbox files. But if you're having them all go into one inbox file, which is fairly common, you're either going to have to write a script that splits it up for you and then runs it through M box, or I'm sorry, M unpack, or use the next script. So watch my next video. If you're interested in doing that, I think if you're watching, please visit filmsbychris.com. That's Chris of the K. There is a link in the description to my page and also link in the description to patreon.com forward slash metal x 1000, which is a place you can go and support me on a monthly basis if you can. If you don't like Patreon, because a lot of people are having issues with Patreon, having looked into why political reasons, I would assume you can also support me through PayPal. There's a link if you go to filmsbychris.com again, Chris of the K link in the description support section. There's a link there to support me through PayPal. And as always, I appreciate your support, watching, liking, sharing, subscribing, commenting, all that stuff. Have a great day.


We're building new tools to accelerate how scientists can discover new materials. Everything is a computing platform these days. Not just our phones, but our watches, our cars. As computer chips become ubiquitous, we really need to make certain that all the materials that are used are sustainable as possible. We are trying to use advanced simulation, including quantum computing, AI, deep search and advanced automation to help us develop better and more sustainable materials that are used in the production of computer chips. More sustainable materials and manufacturing processes will enable more sustainable computing, which will enable a more sustainable planet.