That's around. Cool. Okay, thanks for coming along, everybody.  So first up, I'm going to talk to you about Ffemoral PKI. Explain what one is. Tim is going to talk to you about our implementation of the Ffemoral PKI ac Tom yw'n gweithio'n gweithio'n gweithio'n gweithio'n HPE Helion OpenStack. First up, what is an ephemeral PKI, and why do I want one? So, today everyone seems to care about cloud security. It's the number one barrier to cloud adoption, it's what most of our customers worry about. And effectively, we need to make cloud secure. So I'm going to start covering very briefly why you need a PKI at all and then drop into what an ephemeral PKI is and why one of those is better and you need one of them more. So, typical cloud deployment has got loads of services. You've got Nova, Swift, Glards, et cetera, et cetera. Traditionally, you'd separate them into security domains, look at some sort of filtering, some sort of monitoring between the security domains, maybe using physical separation if you're particularly cautious, maybe using VLANs, and this works great on paper. But then you come to add in all the other services you need to make a system work and you end up with a huge number of things leading to talk to lots of other things. This doesn't work out so well, so the next step is your security guide tells you to turn on TLS because everyone knows you just turn on TLS. And this makes everything confidential, secure, more cryptos has got to be better, right? Unfortunately, all the supporting services also need TLS, although there are some exceptions. Turn all of that on and it's all happy, except TLS needs some supporting infrastructure of its own. Most TLS deployments use X509 certificates to identify the TLS endpoints. Typically, you use certificates on the server, but in some cases, you use them on a client as well. And the certificates are obtained by submitting a CSR to a registration authority, which handles decision-making, issuing decisions, and then the certificate authority handles the actual signing of the certificate. During the process, a server owner would submit a CSR to the RA. The certificate administrator will look at the fields in the certificate and go, do these meet our policy? Are they sensible? Is this server owner trying to do the right thing? And if it matches, then issue the certificate. The certificate authority also provides a bunch of revocation functionalities so that if the administrator decides he's made a mistake and shouldn't have issued that certificate, or if the certificate becomes compromised, he can then revoke that. However, there's a few issues associated with this. Firstly, revocation is largely theoretical. There's a couple of mechanisms, a CRL and OCSP. Certificate revocation lists are just what they sound like. They're a list of certificates that are revoked. Unfortunately, it becomes largely unwieldy in real life. I don't think any operating systems ship with CRLs for quite a while, and they need to be maintained and kept up to date. Secondly, you've got OCSP, Online Certificate Status Protocol, which is implemented in a bunch of web browsers but in almost no client libraries whatsoever. So neither of these things work particularly well in service-to-service architectures. Secondly, bulk certificate refresh is pretty difficult. If you end up yourself in a harbly situation where you need to refresh all the private keys in your organisation, ideally by yesterday, it's very hard to figure out which ones are still in use. You better hope your PKI and your CMDB are very tightly tied together, otherwise you'll have roughly a week's spending figuring out which certificates are still in use and which ones aren't. Another problem is that manual certificate administration is not 100%. People make mistakes. I have issued certificates incorrectly, and I'm the person who wrote certificate policies, so I should know what has been issued. If your certificate administrator says they've never issued a certificate incorrectly, they're probably either mistaken or lying. The worst part is they may well have not noticed they did it. And finally, certificates expire. Everyone knows that certificates are going to expire, but sometimes it still catches you out. And it quite often results in a 3M phone call asking why something's gone offline. So, a fair more PKI. We've taken an alternative approach to PKI and are focusing on using passive revocation rather than active revocation. We deal this with short-term certificates, which are typically valid for 12 to 24 hours. Passive revocation works with certificate expiry, so once a certificate has expired, then it's no longer regarded as valid. Clients more or less without exception support this and apply strong checking to expiry dates, even if they don't do any revocation checking at all. Clearly replacing the private key is a client decision, but when the certificates are being bounced every 12 to 24 hours, it's not a massive challenge to replace the private keys as well. And at this point, you've got a poor man's version of perfect forward secrecy. So, the nice thing about passive revocation is there's no need for CRLs or ACSP. You simply wait for the certificate to expire. In the event a certificate needs to be revoked, you just wait 12 hours and don't issue it another one. Now clearly maintaining this is not something that a certificate administrator could do. We have probably 1,000, 2,000, 3,000 certificates running on our public cloud at any moment.  Some of them aren't. Can't tell you which ones. And no certificate administrator could replace 3,000 certificates every day. So at this point, we built a rules-based certificate issuing and management process called anchor. The ephemeral PKI uses a rules-based decision engine which applies a series of validators to enforce the certificate policy. The certificate policy is roughly the same as a certificate administrator would use when he's making a manual decision, except it's enforced automatically. So it's enforced 100% every time. And finally, it's a stateless system. So it's easy to deploy in high availability, which has some nice benefits like you can deploy it in silence. So if I don't want my Nova nodes to trust my Swift PKI, then I simply don't install the trust anchor for that. And a compromised Swift certificate couldn't be used to impersonate the Nova nodes, say. I'm going to hand over to Tim who's going to talk to you about the anchor project, which is our implementation of this. Thank you, Doug. Okay, so my name's Tim Kelsey. I'm a security engineer working mostly upstream for HP. And I'm going to talk a little bit about anchor. Okay.  One of the disadvantages is that there's a lot more requests, a lot more certificates needed to be refreshed a lot more quickly. So anchor is our ephemeral CA implementation and an automatic registration authority. It issues very short lifetime certificates as part of the public key infrastructure. And the key points there are that it has no active revocation mechanism at all. It relies entirely on passive revocation. It is itself ephemeral. It's stateless, which means it's very easy to deploy in HA configurations and what have you. And it has no additional deployment overhead. You don't need to add OCSP responders or maintain databases and what have you of who's got what certificates. You just issue them and basically forget because they're going to expire and you rely on your policy to not reissue that certificate. Okay. So anchor itself is a fairly recent project that evolved from an internal component that we developed at HP. It's since been open sourced, released under the Apache 2 license and is available on Stackforge. The project itself falls under the auspices of the newly formed security project. This was formerly the open stack security group and the vulnerability management team who have now merged. So this is one of the projects which now lies under that project. So we desire anchor to be a good open stack citizen. So we have a very strong focus on our CICD gate tests. We're aiming for 100% test coverage. We've probably got about 98% currently. So we're well on the way to getting that. And we've also integrated a tool called Bandit into our gate test. Bandit is another security project undertaking. It's a tool for scanning Python code automatically and flagging up potential security problems. It's kind of like a linter but with a specific security focus. We actually have a talk later this week about Bandit specifically. So if anyone's interested in that, come along. And finally, we've tried to make strong idiomatic choices for our libraries, independence and bits and pieces. So there shouldn't be any surprises to anybody who's familiar to working on any open stack component when looking at the code base. So functionally anchor falls, it breaks down into four main functional blocks. We have a REST API, as you'd expect. We have an authentication system, the decision engine, and then finally the certificate issuing system. So our REST API is built upon Pekin, as you would expect. It's very simple, very minimal. We actually only have a single endpoint slash sign. This is an example of actually using a very simple request. I'll show it on screen there. We have four basic headers. You provide a user to indicate who's requesting the certificate. We have a secret field. This is used by the authentication module. I'll talk about that in a moment. We have the CSR. That's an X509 certificate signing request. And finally, a field to indicate how that CSR has been encoded. We only support PEM at the moment, but that's a future flexible thing. So once the REST API has ascertained that all the expected fields are available, the next thing is the authentication module. So Anchor actually ships today with three different authentication modules that can be configured through a JSON configuration file. We have a very basic shared secret module. It just uses a preconfigured shared secret stored in the configuration file itself. It's very basic, not particularly secure, but it is self-contained. So if anyone wants to try out Anchor or run it in a test environment or what have you, it's the easy choice. In addition to that, we have a keystone-based authentication module which uses a keystone token. And this has the advantage that it can pass through roll information into the decision engine. I'll speak about that in the next slide. And we have an LDAP implementation which passes through group membership as opposed to keystone rolls. So assuming the authentication module says, yep, everything is great. We move on to the next step, which is the decision engine. The decision engine is our automatic registration authority, effectively. So as Doug mentioned, it's built out of a series of rule chains which are built from composable validators where a validator is simply a Python function. We didn't use any domain-specific language or anything complicated here at Python. We all know it and it does the job quite nicely. So effectively, the CSR is presented along with any extracted authentication details to each of the validators in each of the validation chains. The validator will run. Either it will exit cleanly, which indicates that the check passed or it will fire an exception to say, yes, something's wrong. And in the event of an exception, we bail out and report an error to the user. As each step in the validator chain executes, we emit appropriate log events to say if it failed, if it passed, why, what have you, for the purposes of forensics and auditing. So anchor ships with a whole bunch of these validators. There's a lot of interesting stuff. Conceptually, they fall into three sets. We don't really make a hard distinction in the code. They're all just functions, but you can think about them this way. So at the lowest level, we have CSR sanity checking. These just make sure that all the right punctuation is in the right places in the CSR. It's just syntax, really. Right number of fields and no duplications and bits and pieces. So above that, we have validators which enforce the security policy. These check that the FQDNs are correct, that they resolve to appropriate IP addresses and everything is as you would expect it to be based on whatever security policy you have. And then finally, and perhaps most interestingly, we have a number of validators which encode expert knowledge. And this is specific information about how anchor is being used and how your actual cloud deployment is configured. So, as Doug mentioned, you could silo off an anchor instance to provide certificates for your nova nodes. In that scenario, you might have a naming convention and you know that all of your nova nodes have FQDN that matches a certain pattern. And you can encode that into one of these expert knowledge validators. You may know that all of your nova nodes live in a certain predefined IP block and you can verify that the provided FQDN resolves to an IP within that range. And you can do various sensible things about making sure that the appropriate roles are present and will have you for provisioning nova resources. So you could write very specific rulesets for individual bits within the cloud deployment and use a siloed version of anchor to enforce those. So assuming that the decision engine said everything is great, this is a valid and reasonable and correct CSR, then the next step is actually issuing the certificate. So this actually is probably the most straightforward bit of the whole thing. It's very basic. We don't really write our own crypto code here for obvious reasons. We're using PyCA cryptography, which is an excellent, very high quality library, which wraps through an FFI interface open SSL in various other back ends. So we actually needed to add a little bit of X509 code of our own just to wrap the FFI bindings exposed by cryptography. Some functionality was missing though. Hopefully that's changed since I wrote this. And yeah, we am its appropriate audit logs to say, yes, we issued the certificate for this person on this date and so on and so forth. So, okay. So that's basically a sort of whistle stop tour of anchor. It's a very early project, as I said. So we've got a fairly extensive roadmap of bits and pieces that we're going to be working on going forward. Here are a few of the highlights. So documentation isn't exactly where we'd like it to be today. So we're going to pretty much straight after this, actually.  We obviously want to replace this bit of X509 wrapping code that we have and call directly into PyCA cryptography. So a Barbican plug-in would be a really great thing to have. Barbican, as of the kilo, I believe, has certificates operations now. So it would be great to be able to use anchor as a backend CA for those operations. We'd like to add a KMIT-based HSM interface layer so that we can use a hardware security module for the certificate operations rather than the locally available OpenSSL library or whatever. We'd also like to put it in the PyCADF library for NYSER audit logs and various bits and pieces that you get from that. More validators will emerge as we explore additional use cases. We've talked about a message queue model for getting certificates so you can use a queue to issue requests and get your certificates. And some of our profiles would be great for hardling the whole thing. So those are all things that we're working on. So if it's peaked anybody's interest, then we'd always welcome patches, comments, imports, whatever would be great. So that's all I have. I'm going to now pass the floor to Tom who will talk about how we actually use this thing. Thanks, Tim. Hello, I'm Tom Cowan. I'm on the compute team for HP's Helium OpenStack. I recently will be looking at using TLS in our deployment and getting TLS everywhere using anchor. So we've been running anchor in production now for about a few months since our 1.0 release. So I'm going to run through how we've done this and the lessons we've learned. So this is all based on a triple O deployment in 1.0. So there's already work done by the community to get some TLS in triple O. So we based our initial design around what was already there. So this is the existing architecture where you get client connection coming in to the control plane. So the control plane is just like management node running all the API servers, all the interesting stuff like that. So connection comes in to the virtual IP address, the VIP, which is assigned to one of the nodes in that cluster. And this is hands off to the load balancer and then off to the local service around the IP address. So we used that and we added a bit into the gap. So now we've got the client using its native HTTPS coming into load balancer again. But this time handing off into the TLS terminator. In this case, we're using S tunnel. The reason we use this rather than having the services doing their own native termination is because they're not really designed for reloading the certs every 12 hours. So we tried to implement it natively, but none of the libraries are really there to reload certs on the fly. The Python libraries don't support that yet. Some services do, but we can't went for this blanky approach using this TLS terminator. So this is a more physical diagram of how the connection is flowing through our cloud at the moment. I was at HA Proxy load balancers across to tunnel and then tunnel strips off the TLS and puts plain text across to the local host. So there's no unencrypted communication across our cloud at all. So talk about a few of those components. So we've got S tunnel. S tunnel is configured to listen on a port and then it needs another port to hand off to the unencrypted connection. So it's really easy to configure. Works really well, and obviously you can reload certs on the fly. You just have to send a sik up to it. The last little gotcha we found was a really bad bug in up to 5.9 where occasionally it would just prematurely close the connection. So we're having a glance. Image downloads just fail completely randomly. Pretty easy to harden as well. Make sure you disable SLV2 and V3 unless you want to get poodled and make sure you choose your cipher suites. We've got a load balancers at HA Proxy. First thing though is it doesn't really know about TLS at all. It's running at a later four. So it just hands off TCP packets to the services below. The thing we did have to change about it was the health checks. So the health checks pretty much just to make sure that the services underneath are alive and they're still running. Initially before they were using the default HA Proxy checks which were just socket connections to see the service still running. But now we've got this layer of S tunnel in between it. We have to do full HTTP checks now which has improved the availability of our cloud, actually. So we've configured that to be much nicer. We've actually made some better checks as well for grabMQ and MySQL to check the partition and stuff as well. So deploying anchor itself, really simple. It's based on PCAN which means you can just make a virtual amp, pip install your requirements, make sure you're apt against all your other requirements and then run it up on new whisky or whatever whisky-based stack you want to use. We did that in triple O with an image element and then we configured that with heat. So heat just adds all the information you need about the validation rules, how long you want your certs valid for and that'll go in and configure it as it works really nicely. Along with deploying anchor, you need some other things. You need NTP. Because we're issuing certs every 12 hours, there's a small gap when you replace a cert whereas if the nodes are out of sync, you can get a node completely unavailable because the client doesn't trust it. So when the client connects to a server and the server has a cert which is issued in what the client thinks is the future, then it'll be like, oh, I can't do this. That's invalid certificate. So make sure you've got the timing on all your nodes really close. We did have the option of backdating the issue date of the certificate but we didn't want to do that hack so we just made NTP really tight on our cloud. Another problem with deploying anchor was you have to have everything behind TLS. So that means anchor as well. So if you want to get your certs for your other nodes, initially you have to have your control plane nodes come up first. So we've got this sort of ordering. So the control plane node has to get first from the local host, kicked it down onto action again so it can be accessed from the rest of the cloud and then you're up from there. Obviously, before anchor gets up, none of the other servers can talk to each other because no one has a cert yet. So there's been some quite important ordering there. So the anchor client is the client that has to retrieve the cert from anchor, pass in all the information, create the CSR, also has to check the expiry. So when we want a new cert every 12 hours, we want to check when it's about to expire, when we need a new one and then generate CSR and talk to anchor. We also always get a new cert on reboot which is a check we added because we're finding out that when a node was coming back up from a reboot sometimes, it wasn't in sync with NTP because NTP sometimes takes a while to catch up with what the real time is. We always get a new cert when we reboot, make sure it's valid for another 12 hours and continue. Also we've got some other actions we need to do when we get a new certificate such as kicks down onto life again, reload the certificate for the TLS terminator. So initially we looked at using cert mug for this but it was a bit too bulky, it has quite complex configuration for it. It just wasn't simple enough for what we wanted. So we just went off and used some cron and bash. We're carrying checks every couple hours with cron. That just calls out to an open SSL command you can see up there which just repars the date. Check it right if it's... It's about to expire in the next hour. We'll get a new certificate. Generate a new certificate... Generate a new certificate is our talk to anchor using that curl command and anchor will talk back to us, hopefully give us a certificate and we can carry on. So some of the problems we faced deploying this one of the big ones was clustering traffic. Clustering traffic is really hard to deal with ephemeral PKI. Both Rabbit and MySQL have clustering traffic with TLS but it's not really possible to do that natively with ephemeral certificates mainly because they're neither in support reloading certificates while having a cluster up. So MySQL Galera clustering actually requires all the nodes in the cluster to have the same private key and certificate. So it doesn't make sense at all to choose a private keys... ephemeral private keys there and refresh it. So you'd have to reboot the cluster as well. It's just got really messy. A similar sort of thing for Rabbit. We did talk to the Rabbit team. I think there's making progress on being able to refresh certificates as you go along but as of the moment it's not there yet. So we do use long lived certificates for these but we can't use ephemeral PKI unfortunately yet. So configuring open stack services was a bit pain also. There's so many different flags and protocol settings you have to do in all these different config files. So it took our team a couple of weeks to really find all the little config values we had to get and set them all and then eventually we got TLS everywhere which was great but it took a while to find them all and we had to patch a few client reapers as well. So I think there is some work being done upstream to unify all these options so it's coming but we had to find it all. The monitoring as well was a bit tricky. That's because when we get the certificates for the nodes the certificates are signed for the virtual IP address on the control plane. That means the monitoring wants to go directly into the node. It can't validate the certificate for the actual node itself. It has to validate for the IP address or the virtual IP address. So we have to turn off cert validation for monitoring which isn't a big deal but it's nice to have. So some improvements we're looking at doing in Hedeon 2.0, Hedeon Albus stack 2.0. We want to have a more layered termination so where we saw HAProxy not aware of TLS. We want to have it aware of TLS and then re-encrypt pass-on and that gives us a lot more flexibility of how we control that. We also want to have multiple IPs supported in the subject old name. That means you can have the cert valid for multiple IP addresses or addresses. We're also looking at doing more side deployments so per service certificates which are really hard on us a lot. And also having a look at having the services listed on a Unix socket. So rather than having the service running on the network you can keep it completely local and have external hand-off to the Unix socket which would be really great. So here's a few references for you. Our cathead is a project I've been working on locally which kind of solves the rather new in front of bash for cert tracking and cert retrieval. It's a bit of Python code which does some bit smarter scheduling and talks directly to anchor extendable. So check that out if you're interested in that sort of stuff. Obviously you've got anchor in stack forge, security project and everything else. Thanks guys, if you've got any questions I think we've got some time. Hi, thanks for the talk. So a quick question about the cron with the curl. Is there any the cron job with the curl? Is there any plans to use a configuration file or allow us to use a configuration file instead of the username password so it doesn't make it into the process list? For the anchor client sort of stuff. Yeah, I think that project's working out at the end there. Our cathead is a lot more configurable, a lot more open and we've got some plans to put some time into that to make it. So it's definitely possible to do that for the configuration values in that rather than the cron job. So yeah, definitely a good point. Okay, thanks. A couple of questions at the conceptual level. You're using username and password to authenticate initially. So if someone hacks the username and password and therefore gets the certificate, what do you do for the next 24, 48 hours? That's the first question. The second one is someone or an entity genuine that's got a certificate and it's about to expire in say five minutes. And then it actually wants to authenticate and do something that's going to say last ten minutes or longer than the life of the certificate. Okay, so please stay near the mics just in case I need to. So the first question was regarding the username and password used to request the certificate and what happened if someone compromised that username and password? So first of all, you should be deploying this in production using Keystone or LDAP rather than the static username and password which gives you some level of protection there. But the idea is that you should tie your validators down quite tightly and the validators are completely user-controllable so you tailor them to your system and they should be built so that only a person who... sorry, only a node that meets those criteria can request a certificate and it'll only be able to request a certificate that meets the criteria of validators. So you could only issue a certificate to Nova rather than Google.com, say. Obviously it's the issue that if you log into the CA and you could compromise the certificate that's the same as every other CA out there. So this doesn't particularly change that risk profile. I'm sorry, I've forgotten your second question. I think I can just answer that one. So you say if the certificate expires during the connection, is that what you're... So I don't think that's really a problem because the only time of validators is when it initiates the connection or during a renegotiation of the connection and typically services don't ever issue renegotiations and you disable a renegotiation. So, yeah, I don't think that's really an issue. Even if you expire while the connection's going it should just be fine. To add to that, we should have mentioned that we have it configured and we generally recommend it to be configured that if your certificate expires in, say, four hours time you request a new certificate and then roll that in place early so that you're not running right up to the certificate window and then replacing it. No, but you said replace with LDAP. I mean LDAP is still username password, isn't it? Yes. So this is where the validators come in. So it should only grant the node that is requesting a certificate, a certificate for that node or similar nodes in the area. You shouldn't be able to request a much broader certificate. Right, but does it actually check that the request is coming from the node or is it just purely the username password that is sufficient to prove that it's from that node? So it will do things like a reverse lookup on the IP that the certificate is being issued for and check that against the IP address the request was submitted from. So if you've submitted a request from 1.1.1 and you're requesting a certificate for Google.com it will notice that your 1.1.1 isn't the address for Google.com and then log it, raise it to your Arc site and administrative should investigate. Right. So you've got some checks in, but ultimately the issue is if you find out after a certificate has been issued for whatever reason that it was issued wrongly you are effectively screwed for that amount of time until the 24 or 48 hours run out. Correct. But that's no different to using CRLs where you're screwed for as long as it takes them to product. That's true. Sort of following off one of his questions is there interest in adding support for different authentication such as GSS API or getting a remote user header from Apache? Absolutely. And the other question is can you add extra fields to the certificates that you sign? So to answer your first question and your second question was adding extra fields on to the certificates that get issued? Yes, so except custom fields and you can assemble those in. Okay, thank you. I should say as well, sorry, that you can add validators quite easily as well so if you want to add custom validators the idea is that we will acquire a live as small usage scenarios emerge. Hi, you mentioned at one point you had an issue with image downloads intermittently failing. What was the cause of that, did you say? That was to do with a bug in STUNL up into STUNL up to 5.9 so it was just prematurely closing the TCP connection before it actually needed to be closed. Thank you. Is everyone in? Great. Thank you.


Hey guys welcome back skits on episode 12 topic today is raster images and in my opinion the best raster image format is bitmaps Hot take but uh, I don't like these other formats. They seem to either They the compression is not worth it in my opinion you can save you know six bytes here with the PNG Seven bytes here with some other format Meanwhile gta 5 is a hundred gig install. So it's like what's the point? Bitmaps just work just simple header simple rgb array very straightforward format to work with I'm also going to talk about different rasterization techniques for different geometric objects lines circles Triangles things like that and we're going to implement some basic algorithms for that purpose and just as a little preface or I guess Post-fists Here is what our result and it's going to look like All of these geometric elements are generated entirely in x86 assembly No libraries nothing in C. Everything is literally every instruction being executed Was written by us with the exception of the syscalls that are used to write files on Linux, but yeah, that's just what we're doing today. So again line circles other shapes Pixels themselves filling different parts of the domain. That's also also opacity. You kind of see here a Translucent fill all these things will be covered in this video if I interest you stick around if not, I'll see you in the next video so Let's begin So what goes into a bitmap file? And so we're going to stick with this windows bitmap Obviously, there's like other types of bitmaps like on Linux There's different file formats that are used for Programming and things like that and see but it's not very portable in my experience I haven't said anything online that can render even like on Firefox. You can't render these Linux You know specific bitmap formats But anyway a bitmap file has BMP file should say has Two different sections actually maybe three sections. There's actually two headers Just combined into one here for simplicity's sake and then there's the actual RGB data So it's not too dissimilar from the elf files that we're going to be that we've already created in this series In every single video. So those files have a header as well as Instructions in the data section down here. So it's a pretty similar concept and there are many different variants for bitmaps But we're going to stick to this windows bitmap before which is just one type of Combination of parameters that is understood by different software when you open a bitmap and we're going to use 32-bit color in ARGB format. So A is the opacity R is red G is green B is blue and so that's one byte for each of those things and That's good because One thing about bitmaps is you have to have padding in your RGB Data array at the end of every row, I believe but thankfully because 32 bits is four bytes There's no need for padding because you're padding to I think like a four byte boundary. So there's no reason to do that Okay, so with that out of the way Let's talk about the actual header itself And so I ripped this from Wikipedia You can check this out yourself or some other resource, but this basically is kind of the headers required for this Windows bitmap before variant and So basically the first part of the header is just a bunch of garbage nonsense Indicating that it's a bitmap file and then down here this section here I believe this number is how that these are Softwares like gimp and whatever that's how it detects what version you're using. I'm pretty sure that's how it works And so we're gonna use this Windows bitmap before version. So that size is 108 bytes Anyway, there's a bunch of stuff in this DIB header Which is a function of which variant you've picked and some of those things are like the again You can see here the size of this so it's The size of the image Some bunch of information about the number of bits per pixel planes This is if you wanted to print everything out how how much pixels per unit length and color information things like this masks for RGB and alpha channel and Other stuff like that not really too important. In fact only four things are variable in this file as far as we're gonna Use it. We're not gonna change the number of we're not gonna change the variance for this Bitmap file we're not gonna change number of bits per pixel not gonna change number of color planes That's all gonna be fixed. We're only gonna change the size of the image And so all these four things here. So this orange one is the width the width of the bitmap in pixels The light blue one is the height in pixels This down here would be the size of the raw bitmap data. That's just gonna be width times height Times number of bytes per pixel, which is 32 bits or four bytes of 4 wh bytes And as well as this top thing here, and this is the size of the bitmap file itself the whole file header and Pixel array information so again all these four things are just a function of The width and height of your image So yeah professor that is very easy correct Now the actual pixel array data is pretty simple to understand basically You just have you know Bits that indicate different colors and the way that the counting works is that you start at the bottom left of the image and Count to the right and then upwards so your pixel zero is at the bottom left Pixel 7 in this 8 pixel image is at the top Right and you kind of can see in this a RGB format that You know this first pixel here is Blue, but it's not fully opaque so you can see it's light blue and then pixel one is Here it's green and not fully opaque For example, let's say this second glass pixel here pixel 6 that is red and fully opaque. That's this one And so yeah, basically that's how the pixel data is encoded and because of ending this you can see it's kind of backwards here but You can see that familiar a RGB format if you just look at the kind of a double Definition of this 32-bit value so that you kind of can see it's a RGB instead of b g r a Just because of how things flip with ending this Okay Pretty simple stuff, right? That's why I like bitmaps. It's very simple So we're going to make some functions here to help us draw bitmaps The first one is something called write bitmap and that's just going to basically take Your array of pixel data, which is going to be pointed to by register RSI in this case as well as the dimensions of your image data and Write that out to a file descriptor So this is going to write the headers basically going to do everything That you know the professor was talking about was being so easy I'm going to write this header out and a fill in these slots based off image width and height and then it's going to Write the actual pixel array data at the very end So this is going to be a function that we're going to use to write the bitmaps to a file Next We have a set pixel this function is just going to turn a pixel at a certain position to a certain color So you're going to pass in The pixel array data start address in RDI You're going to pass in the color in ESI and here We're going to talk about things in terms of the 32 bit register because everything here is 32 bits The colors themselves are 32 bits. So that's why this register ESI which includes You know inside it the the color of the pixel that you're setting That's why it's 32 bit register but in addition you notice that the Even the width and the height of our image have also been set in a 32 bit register and the reason for that is Because you only have four bytes into which to encode this information So if you're pixel with it only fit in four bytes, you only have four bytes divided in so yeah That's why those values are 32 bits Okay, so yeah, it's a pixel. That's a pixel color get pixel Basically queries a pixel color. That's basically This tool if you have paint you kind of query You know colors of different things I can query the color of this yellow and use yellow. It's like the Color picker tool, I guess it's called Okay That's what get pixel does it just gets the pixel color at a certain location We have set line set line is going to draw a line for us It's gonna rasterize a line between a start point and an endpoint Set circle set circle will do the same thing for a circle It will rasterize a circle with a given center and a radius and then Set fill that will fill a Boundary so you have some kind of boundary shape. It will fill that with a certain color like for example this Fill tool on the top left here. I can just fill this boundary with orange and you can see that whole boundary filled with orange so that's gonna be one function we have and Yeah, so I remember well, I should say that all these things here are using a different positioning scheme So typically when you look at Images you should think about the pixel at the top left to be Pixel zero zero and actually if we open up in Gimp, you can see an example of this So if you if you look here Kind of on the top you can see the rulers here on the top of the image Basically pixel zero zero is at the top left and I can go all the way here and I can actually look at which pixels This is pixels zero zero. This is pixel five two So I kind of can see if you look down here at the bottom left the position within the image. Anyway, if you recall that's not how the Array data was defined the array defines pixels zero at the bottom left so we're going to have to convert between the pixel array definition of pixel zero and the Coordinates that we're used to using to identify Pixels for example pixel six even though it's called pixel six in the image array data. This would actually be pixel zero to or sorry to zero in Inconventional understanding, okay So now to access a pixel That's what I'm just talking about. So basically you can see here We have some image. I guess it's with w w equals one two three four five six pixels and height five pixels h And so based off that understanding the bottom left pixel is that address? offset zero You know zero times four bytes per pixel, but one pixel above pixel zero is W pixels away. So that's w times four bytes Similarly, this pixel here is 2w times four bytes and this pixel here is 2w plus one Times four bytes. That's the offset from the start or address of this array So the question is how can we relate this? addressing to an x and a y position and it's pretty straightforward all you do is If you're measuring from the top left, which is this case we are this pixel, which is zero one two three zero one so pixel three one You can just basically compute the number of Basically image rows between this pixel and the bottom, which is I guess h minus y minus one multiply that by the width of the image data w and then add x which is just the horizontal offset multiply that entire expression by four bytes and that would be able to point to a given address based off a given x and y position in an image that is W by h pixels in dimension a lot of words not very much meaning just a pretty straightforward explanation there okay now talking about some rasterization, so There's one very Common way to enter a good way to rasterize in this case a line and you can look up the press and ham You know algorithm or the midpoint algorithm. They're pretty much one of the same And basically the way it works is in layman's terms You know see you had a line like this line here in red with a it says it well I say a gentle slope downwards You step through all values x Between x zero and x one So this would be x zero and this would be x one Step through all values x Occasionally incrementing y which goes from y zero to y one if the pixel Below is a better approximation in a pixel above. What does that mean? Well basically here? So if this was your start pixel in in blue here You have two options for your Next pixel either this one on top or the one on the bottom and we intuitively as humans can say well one on the top is a much a better pixel to pick but the question is Will it always be not always you can't always pick the pixel to the right because you know that would eventually deviate from Eventually deviate from your line quite a bit as you can see and so you have to be able to pick and know When am I supposed to be moving down? You know is my is my directory supposed to be you know like this Like how am I supposed to be drawing this? You know is this the right pixel or not? Is this the right pixel? How do you know and so that's what we're talking about in this rasterization? slide Okay, let me fix that really quick and so yeah, there's two candidate pixels here the top one in the bottom one And so the question is how do you tell which pixel to choose? Well the way it works is in this midpoint algorithm or Breslin ham algorithm you evaluate based off the relative position of this Line that you're trying to plot and the midpoint of the next pixel and the idea is If the line passes above This midpoint so if the red line is above this green midpoint You color in the top pixel if The line is below the green dot you color in this pixel That's the entire algorithm So if you can do it into to understand that you're well on your way You can skip this part of the video if not there's some math that can go into it And even if you're curious about the math we're going to discuss it right now And so yeah, they basically you're going to increment why based off that so if this was the right pixel to pick You would not be incrementing why right the wide value is the which row you're in basically and If you were below the screen dot you would be incrementing why right from here to here as an increment in why? So yeah, that's how that works So here's the math that goes into this So Basically It's not that hard. It's pretty straightforward. Just take D Then the letter D to be the vertical distance that the line is let's say below and you could inverse this if you wanted But I'm just saying let D be the distance Below the next midpoint. So how far below this red line is relative to this green dot So in this case, we're actually above the the green dot. So our distance is negative D and then take dy and dx which are just the relative Kind of slope components of this line. And so in this case, I've just taken the entire section here. So let's say X zero is here and x1 is here Well, then dy would be to and dx would be six So our slope is one-third right what I say it was dy over dx so that's two over six or yet one-third But we don't really care so much about the actual division. We're going to be tracking things in terms of the effect that these components have on the distance D and So you can manipulate this so basically the green dot represents the slope of one half or yeah one half And so because of that you kind of can break apart this dy over dx idealized value And you know multiply both sides by dx subtract everything to equals zero and your d expression looks like this so D equals dy minus dx over two that is the Equation that represents how far below the red line is from the green dot and again because Everything here is a ratio. We really and we're going to be comparing things relative to zero We don't really need to have a an accurate Scale for this as long as we have the I guess all the matters of the sign. We don't care so much about how Magnified this is and so yeah, you could take the actual dy over dx value Or you could scale it by ten or by a thousand It's not going to make a difference for the algorithm to come And that's going to be important in a second But yeah, so basically the way it works is if D is below zero or let's say equal to zero that means that Your your red line is above the green dot meaning choose the upper pixel which means For the next pixel in your list of pixels that you're drawing Increment X which means go from here to here, but you're not going to increment Y So you're not going to go from here to here and The net last thing that you do is you have to adjust D which D just keeps track of your relative distance between the red line and the Subsequent green dot because you're always going to be looking at the midpoint of the current pixel and comparing relative to that And so if you are at this Future location. Well the midpoint now for this pixel is Let me draw it in orange is here And at that point you're now below that That midpoint and so you would then follow the second Possibility second option here So that's if D is greater than zero meaning that the red line is indeed below the midpoint you choose the lower pixel So in this case you would be Picking this pixel. So that would represent an increment in both X and Y So you're going from here to here that's incrementing X here to here is incrementing Y and At this point you have to again keep track of your relative distance. So in this case Well, I'm really sick of it before I say that above this back here this adjustment that we made for the Relative distance basically because we incremented X the line Overcross that one pixel is expected in our distance evaluation here to decrease by dy That's why we subtracted our guess we added dy to our Track of the distance value because we're measuring distance below the midpoint in this case For the second pixel that we're talking about here. We're both going to be doing that. We're going to be Dropping by dy But we're also going to have to accommodate the fact that we've dropped in dy And so now our next midpoint is actually below and so Let me draw that here if we if we're eventually going to picking this pixel which is the next pixel to go through the Corresponding midpoint is actually here. And so you have to make another Accommodation for that in the distance value long story short. You just increment d by dy minus dx And so things of note to this is that basically this particular algorithm, which again, this is the entire algorithm Basically, you just keep progressing through your image going from your initial X zero Y zero to your Your target X one Y one just incrementing X every iteration you can see here both options involve increment in X and Then you're sometimes incrementing Y Based off whether or not you need to based off what we talked about before So Yeah, that's that process and you should repeat this process for every pixel and clear out of pixels in the x direction Now that works in this particular example for slopes that are less than one Moving from left to right. So you have to be able to either in Pre-processing pre-processing or post-processing you have to be able to either Reverse or reflect that algorithm to handle all cases. So you may not be going from left to right You may be going from right to left. So you may have to invert your or switch your X zero Y zero and your X one Y one just flip them around and You might need to instead of going at a very soft gentle slope of less than one Maybe go in a greater than one in which case Just have to basically switch in this algorithm Everything to do with Y With everything to do with X just flip them around and the algorithm will work. So basically. Yeah, that's how it works the other thing to note is that This factors here this divided by two factor and this one half here and all this different stuff It's not really ideal when it comes to integer math And so you can scale everything here by a factor of two to get rid of this division Which would also mean you have to scale this by two and And this by two etc But that's fine. We can do that Okay so this is a simple algorithm for Rasterizing a line and there's other algorithms for rasterizing other shapes like Circles and conic sections ellipses things like that. I won't talk about them here that though this video would be like two hours long and so just go on Bing and type in press and ham Circle drawing algorithm or midpoint circle drawing algorithm and you'll see it bunch of derivations for those sets of things Or if you're curious look into the code because we do have a circle rasterization function that you can just see how that works in the code Okay, now the last thing I want to talk about was flood fill interestingly enough this tool is Called flood fill and so if you want to you know help blossom here Fill the heart with the color pink Well, we're gonna use flood fill to do just that and the way that works at least I think the simplest way to implement that is using recursion Which may or may not be good, but it's very simple to implement and so let's talk about how this works So here is the entire algorithm so basically you start somewhere inside the heart inside the boundary and Record the initial color of that pixel. So I picked this pixel here initial color of that pixel was white So now it's we want to make it pink want to make this whole heart pink. So our initial color is obviously white So the first thing we do well is we call flood fill on that pixel And here is the flood fill Function, this is the entire algorithm So first thing we do is we check if this pixel is outside the boundaries So let's say we picked the pixel over here.  What did I do? Let's see you pick the pixel out here Obviously that pixel is not in the boundaries. So you're gonna get This pixel is actually this pixel here, right? If you think about it, that's the pixel to the left of this one But above and below you have an issue right this pixel down here and this pixel up here You're gonna maybe get a seg fault You're gonna get some memory accessing issue if you try to access these pixels So just if if you're trying to flood fill outside the boundary Don't just don't do it. So it says if outside the boundary return Next thing is if the pixel is a different color from the original also return. So let's say you picked You know your color is supposed to be white if if the pixel that you're on happens to be red or orange or green Stop you're not supposed to fill this pixel You're only supposed to fill pixel pixels of the same color like you know her headband here, you know filling this Fills all the pixels of that red color, but does not fill any of the black pixels in the boundary, right? Okay, that's these first two conditions. So it just says don't don't bother with this pixel if these conditions are true Then all you do is you've colored this pixel. So if you picked this pixel to start with That's the first step you've colored that pixel great now it says a Call this function in the same function on the pixel above the pixel below pixels to the right and pixel pixel to the left and So in this case, we're going to be calling flood fill on the pixel above this one So now we're here. So the question is is this pixel in the boundaries? Yes, is this pixel white? Yes, in that case color this pixel Now at this pixel we're going to flood fill Pixel above pixel below pixel to the right to the left in which case we flood fill this pixel and Then we fill this pixel and then this pixel on this pixel and then when we're here We're going to flood fill this pixel But the question is is this pixel white the answer is no and So we're going to flood fill this pixel right or I guess it would be this this pixel and Then this pixel and then this pixel or whatever it happens to be basically you're just going to keep calling flood fill on All the pixels around the pixel that you started at in this recursive Format until every pixel is exhausted out of your stack Very simple to implement look at this. It's a very short little function like this. Okay, cool Put that out of the way that pretty much covers the basics of rasterization So we'll look at the code so I have two functions actually two examples here one is just the basics of bitmaps That's just gonna be making a simple bitmap from pre-existing binary data that you have in an inner race somewhere already That's just gonna write a bitmap file And then we have a example B here, which is all the basic drawing operations that I explained previously So drawing pixels probing colors drawing lines strong circles custom shapes as well as filling different regions with the flood fill algorithm and So with that Let's see the code So the first thing I want to look at is All the functions so these are all the functions that I talked about before and this is just the implementations And these are all available on the Repository so you can take a look. This is the let's start with the set pixel function This is the most simple of all the functions. This just sets a pixel out of certain position and you can see This implements that kind of mathematics that I talked about before Let me show you Basically this type of accessing mathematics and this is relevant for both the set pixel and the get pixel functions That's all that's going on in this example here As well as the get pixel function here It's the same mathematics Then we have the right bitmap function all this does as you can actually see down here I have Pre-encoded the entire bitmap header and the only thing that you have to Supply into this header is everything where it says reserve So this says you know reserved for bitmap width reserved for bitmap height This says reserved for a size of the map file, which again is a function of width and the height and then down here This is reserved for the size of the bitmap array itself So all those things are just passed into this function and this function writes both the header as well as the entire bitmap pixel data to a File descriptor which you've already opened With some other method Okay, that's right bitmap Only three other functions that we have one is set line So this set line function implements that midpoint line rasterization Discussion and you can see here. It actually uses the set pixel function as a Dependency it calls that to set each pixel and then we have the set circle function Which is a similar algorithm for rasterizing a circle. I didn't discuss the details of that But you can look it up And then the set fill function this implements the flood fill algorithm that we talked about before so you can see here all it does is it kind of saves the initial pixel color and then Jumps or actually calls this loop and then this loop just implements those conditionals I talked about before are we in the boundaries? Are we the right color to fill and then all it does as you kind of Can see here it calls the flood fill recursively on the pixel above on the pixel below On pixel to the left and to the right and that's the entire algorithm. So with that out of the way That's all of the functions that we've implemented for this particular set of examples And now I'm just going to show you What we have so in this directory, which you'll find on the SWHUB suppository We have two examples one is the basics of bitmaps the other is all the drawing operations that we talked about before So let's go into the first example That's just the basics of bitmaps So if I uh If I run this well, let me show you the code first. Let's open the code So all this has here Is just a few includes it has includes to open a file To close a file, which you don't actually need by the way. You can skip this one Um, and then the function that writes bitmaps. In fact these Elf executables don't even have a print buffer. There's nothing to support printing in these Executables because we're not printing all we're doing is writing files using a syscall And so here you can see we just have basic functions to open files close files and in this case Write a bitmap to a file Okay So the first thing is we open And slash create the bitmap file. You can see a previous episode talk about file creation and opening Then we write a bitmap And then we close the file and leave the executable. And so what are we writing? We're writing a Well writing to a file the script that we previously opened and we're writing an image Of a size four by two pixels And down here is the actual image array and this was the same array that I discussed Here And so if we run this Close out of this run this You'll see I now have this uh bitmap Windows bitmap file and I can open that in gimp And we could see how that looks you can see it's a very simple image again It's just four pixels by two pixels if I zoom in you can see I have You know a set of opaque and not so opaque blue pixels green pixels red pixels on white pixels Just as we intended so our right bitmap function works and we understand the inner Meaning of this of the bits that define the colors in this array Okay That example was pretty simple now the more elaborate example. So Let's go into example b In this Well, let me run it first so you can see what we're going for Just to refresh your memory. So if I open this Resulting bitmap file in gimp you can see here Couple things that we've got So this function is going to and I'll explain it, you know when I go through the file The first thing it does is it writes this blue pixel It generates this blue pixel on a completely transparent background Because remember if the entire array is initialized to zeros that means the entire array is black And completely transparent So yeah, so basically it drops a blue pixel right here on a completely transparent background Then the next thing it does is it basically queries this pixel color And it adds 100 green to it And it writes that pixel Diagonally from it So we're going to call basically get pixel on this pixel Add green to it Which would give you a basically a cyan pixel that we draw next to it here Next thing it does is it draws this white line So it plots with this set line function a white line from This pixel down to this pixel Or ultimately ultimately you could draw it from this pixel down here up to this pixel. It would work all the same And the next thing it does is it draws this green Circle here. So that uses the set Circle function and it will take in a center of the circle in terms of x and y position As well as a radius of the circle, which is the distance from that center to the outsides And then it will draw that green circle The next thing the function does is it Fills this region with orange Pretty simple that uses the flood fill algorithm Then I draw this triangle This triangle is just a series of black lines that we've rasterized with the set line function So it draws all three boundaries And then it flood fills the center of this triangle with yellow Then I draw this blue quadrilateral thing here again with four calls to that set line function And then I flood fill the center of that with in this case a transparent red background And then lastly I flood fill everything else around this with pink And you can see it It stopped at the boundaries Otherwise I would have had a segfault or something right so yeah Everything around these images everything around these graphics these shapes Is flood filled with magenta Okay, so that's kind of the visual what we just did and now I'll show you all the calls to functions inside this assembly file So in this case again, I have the The previous functions from the previous assembly file So I have file open file close as well as write bitmap As well as the exit function, but I also have all the other geometric rasterization and other routines that we developed so we have Set pixel set line set circle set fill and get pixel And so here I'm just going to go through the process that we have in this file just so you can Know when you take a look First thing we do is we open and create that file then we set that first blue pixel Then we grab the color of that blue pixel here with the get pixel function Then we set we change that blue to cyan by making it 100 more green as you can see here with this Add an instruction then we are making the pixel diagonal from that cyan Then we are making that white line here. You can see I'm passing in x zero y zero x one and y one And the initial color or guess the target color is All f's which is white Then we make a green circle here. You can see I passed in the You know, I'm passing in every single time the dimensions of the Image as well as the target color in this case. I'm passing in the center of the circle both x and y as well as the radius Then I'm filling that circle with The color in this case orange Now I'm drawing the black triangle in this case. You can see I'm drawing one line from coordinates 40 25 to 38 42 another line here from different coordinates And a third line here in different coordinates and all those coordinates should make a triangle Then I'm filling that triangle with the color and now I'm drawing it in the same way that blue Quadrilateral with four calls to the set line function here And then I'm filling that quadrilateral with Um a transparent red color here. You can see the first value is 7 f. So we're 50 opaque And then lastly I'm filling the background to magenta Then I'm writing the bitmap out closing the file and ending the program so again that generates a Bitmap dot bmp which happens to be Around uh 12k really and then we are uh That bitmap is the one I showed before that's just this graphic here So with that you now understand hopefully the basics of What a bitmap is header the actual data itself understand the basics or at least how to Um understand the basics how to look up the basics what to search for rasterization of Lines and circles and other geometric entities and how we can turn those kind of algorithmic implementations Into actual colors on a screen with the you know the bitmap file itself So with that I think we're done. I want to thanks uh, you guys for your time your attention And your nice comments, and if you want to hang out we have a Top secret discord server check the blast link in the description, and I'll see you there


Hi, so I'm Michael Horyszek, but call me just Michael because if you are not from Czech Republic and Want to pronounce want to try to pronounce my last name. It will be sounds very funny like I am now talking English I work for since now to that Which is company making web apps for local people like email search maps and we are Actually good that we are trying to Open our maps to open to the world because when I made a planned trip to Pyrenees, it's actually More detailed net then the Google ones. So We do web apps. So we have to test them somehow And he does that and therefore I'm standing right here to share our knowledge some best practices some tips and Point out some pitfalls So how can we test websites the easiest thing is Unit test and call some API. This is my test and You should really should does it Does test like this Because it's very mandatory, but it's not enough for web apps because you have to test Some requests and their spawns you can do that as well Flask and Django Supports it. It's it can be very simple like this piece of code but still It's not testing of the of the page because they know what is there are a lot of client sites script so you have to test also JavaScript You can do that. This is for example, Yasmin how's it called and Again, it's a unit test and Say Now you you have you can test the client side and Backend side, but you have to somehow connect it to make some integration tests of full web page that it's work As for the user seats. So how can we do that? You can do that with Selenium Selenium automates browser. That's it. This is what documentation says and How does it look like? It sounds pretty hard, but actually it isn't it looks Like that. It's very simple code. You firstly import something then you Open a browser in this example. It's from then you get some page It's our home page for example then print title of that page and After that just quitting closing the browser I'm sorry for these comments. It's pretty dummy and I don't like these comments in code So I will not say it again in this talk and what can Do Selenium more it can travel the web it can you can't Check some attributes on the elements you can click on them you can check some cookies you can make skin shots and That's it and I Don't I will not talk about it because I you can treat it in documentation of Selenium. I want to Talk about some pitfalls and Some best practices how to test the pages with Selenium. So first pitfall is a Forms it's very hard This code isn't simple and it's just so does That's type Selenium in search field and the submit that form and that's it and you can see it's five lines it's a lot of code and It's hard to find what actually does and you have to think about a lot of things like When you have on client side some on change event, it will not be fired when you send keys to that input Because when you send keys focus is still here. So you have to press stop like in this example To lose that focus and fire on change event Also, you have to think about all inputs like checkboxes selects text area and and more and every input have to be Be written differently Also, there is bootstrap which has shiny checkboxes which hides actually to a real checkbox and Instead shows some another element and you have to think about it. That's a lot of think you have to do So I made it simple With library web driver wrapper, which is on pi pi you can use it for Python 2 and Python 3 and it's well documented and Code like this one can be very simple just find some form and pass dictionary with keys that are names of these inputs and the value are values Pythonic types like there is some Text if it's a checkbox you can just pass through false and doesn't have to think about it That it's some bootstrap checkbox or that you have to click on it. That's library will Do it for you So very simple Another default is exceptions. Yeah, I can show you some example. I forgot it. Sorry I have some code here you can see that it's what What slideshows and when I run it It will come closer Open our home page print title and quit That's it. Sorry exceptions So when when you can find some exception with some element, which is not here On that page it will raise no such element, but It will not says where or which one so I made it made it simple because for example when you have For example, our search engine or Google wants to be it doesn't matter when you go to home page You see some form you fill fill in something click submit and then you want to check that some results are on the next page and Selenium will say is just no such element, but I want to know that it's on on which page because is there a bug on that page with results or is there a bug with page with form and That page is not fairing the good action With this exception you can see that the result count is not on home page So yeah, it's not here because you have to be on a different page So I know that the bug is not on page with results, but on the home page another pitfall is status codes or headers There is a big discussion on web braver Backtracker and it's closed. They want to feature a lot of people once feature To check these attributes, but you cannot because they says that It shouldn't be there It's just API for browser and if you want some headers or status codes and and so just We use some library for requests That's true, but if you want to make some owner request you have to Know that link you have to know if it's get or post or another You have to know if You have to pass the cookies and you have to pass if it's some kind of form you have to Get these attributes and it's it's pretty hard. It's about ten lines of codes to make it done and I I made it and put it to that library so you can just get the element of Some of some link and just download file or page and you can check actually what is there Another way is my colleague did that that he run a proxy and checks status codes and headers and and more from proxy because you can do that with so on state element exception it's if you will write some Selenium code and or some tests with Selenium you will find it pretty soon because when you do with some something with dome on the client side with Shavascript and These elements change or disappear and Shavascript completely changes dome of You don't have In in in Python you have just reference to that element in dome and if something changes if just exchanges it or if you go to another page if there is somewhere direct or something you don't have this element Anymore and you have just reference to something what doesn't exist. So that in that exception is just Get me some element which he has IDQ and send some keys just keep to make some changes and When you send keys again to that element state the element exception, it's It's something How to say You you don't want to reuse some elements. So and if you do that you have to be very careful because it happens a lot and Actually, it's easy to fix it just You know you see the difference in in that Example is under very bad thing and it's time sleep You don't want anything like this in your test code because What if it it will take a longer time it will fail because it will not be You want to wait for some action, but it will not be there. So it will fail For that there is Wait machinery in the selenium you can use it like like this, but you have to wait for elements A lot because there is a lot of JavaScript and you always want to wait for something. So I made it simple again Just wait for element and that's it. I Have Proof to I have some code here you can see This code from from slides. So just a little bit longer and It opens our home page pass pass something Then The fire some autocomplete then it won't so first element in that autocomplete so what was found or autocompeted and then show a text and send another keys and Show what's there later If I will run it It will fail Now you can see that it failed still element reference exception so If you want to fix it You have to just for example you slumped out or whatever to where you saw that's getting off element and Always get the fresh one If I will run this this one It works but You could see that there is time sleep to which is which waits for two seconds and you could see that action was Much faster and it doesn't have to wait for two seconds. So you can make it with wait for element and It's almost same just a little bit shorter and it works as expected You can see the change and when I run it it will be much faster about four seconds So use that another thing is Searching elements by text Selenium again doesn't support it But it's also commenting because You want to you will insert something to database then you want to show some Table or some Contacting for something and you want to check that That information is here and you doesn't care about in which element or whatever So you can't search by that It's implemented by expat which are very gate. You can't make a lot of things, but I I Would not recommend it use it a lot The first line is so very simple Example how it's done, but it's more complicated than that and It's very slow You can see that in one run, but if you use it a lot of times It will be very slow now and it It's not good for maintaining test with expat. It's better with IDs and classes because it's it's much much easier because if you use classes and Someone changes a template or whatever Elements around and still these classes are moving with these elements You don't have to change a test code ideally, okay We we know how to manipulate browser, but how to write test Well, you can use web-diver test case which is seen that library as well and And Why is there because it's it implements a lot of cool things like It creates driver for you and you just Implement to get the driver method like in this example. No default is Firefox and you can say with Constant if you want Open new browser for every test or test case or all tests If you will reuse browser because it's very slow to open a new browser window Then you have another things to keep in mind Which are for example that in test you can click and Suddenly some alert will show up and When that happens it's blocking like for user when some alerts show up user can't make anything just click on okay, just somehow dismiss it and Same thing is with Selenium. So when there is alert you have to dismiss it and if you will Try to do something else you will get just exceptions and when some Unexpected tests on failed test will open some alert. It will Stay there for forever to the or no forever, but to to the time when you close the browser so you have to close it and That test case does it for you in teardown and other things are Windows because see your web page can open a new tab to some other page some link to different word and Also, you want to go back to your web web app Which you are testing so again, this this case does it for you and It does a more thing Because how you want to check out if your page works just perfectly and For example, if there is some 500 error or something else like access denied or whatever you want to know about it and This test case After every test checks that if there is something like this by default it just looks for classes error and you can't Change that and you can see that in documentation Also, there are Error messages because your app can show some error message to user for example if user wants to insert another user with same username or It's some registration formula or whatever and there is already that username registered It should show a error message and you want to Check that test failed because you want it to insert it and if it's it's not done You want to know about it and you have to write a lot of code in every test To check that everything is alright, so I made some decorators and And That this case always check if there is some error or error page and if there is something it's That test will fail and if you want that it should be access denied you just Put the creator there and if you go to admin it should Shows so that's the night page and that's it. You are done. You don't have to always check everything also if there is some form and you want to register another user with admin and You want to check that it will shows error message again You will just put the correct expected error message and so which one what should be there and If there is this extension, it's okay. There is not it's not okay, and if there is different except error class Error message. It's also the felt test So it's very easy to test things like this and there are more decorators like this also for info messages if you want to test that after test after Registration there is some info message that everything was fine and you can look in or whatever You can test that as well another things I Was in JavaScript as I was showing you here you can test the JavaScript some yes mean unit test but When you run JavaScript in browser in real app, it's There always can be some bug which is not covered in unit test. So Just put these codes to your page and Selenium always checks for these errors And if there is some unexpected error in JavaScript again, it will fail. I can I Have the example for you It looks like that. It's very simple just go to home page and To test one for searching one for some telescopes and When I run it it works You could see that I Used one instance per test. I can also use one instance for full test case or all tests and if I run it It will be much faster because there will be just one browser Also, I can show you Some error You can see that this element is is not in that page and that's correct because it's very not here Also, I Love Pytest also does web drivers. So you can Write tests with Pytest with this library So that's it It's very simple To write tests with my web driver wrapper But there is still one more thing and it's so you don't have to run it in your laptop With open browser you doesn't want to always look at it and you want to run it on Server without x server it can be done on the beyond You can install By virtual display it's Python clients for backends x server backends Like one of you you can pick one of them which are shown in slide and For example, we are using x3 of B and we are happy with that For example XVNC are good that you can connect to it to it So if it's failing you can connect and see what's happening so it's nice and Use usage is very simple just start some display and so start it and run your test and then stop it I Can't show it to you that it works because I don't have any virtual x server on my machine here on the own server but it really works and And I would recommend you to use Very big size of that display because there is one pitfall that if you are using some Fixed elements it will mess up because Selenium will always scroll to that element you want to work with But if there is some fixed element It will just scroll to that element and if it's fixed something above it will fail to because That click action or something else would fire Would save another element There For you if you have some fixed elements you want the biggest size you can Another things are Selenium server You don't have anything because as you could see I just run browser directly, but If you want something more you should Check out slanium gird. It's written in Java. So be careful with it and It Allows you to run it in more browsers with more operating systems also some mobile devices and It will be faster It works That you have some master Selenium server and there are a lot of nodes on different type of Machines you will just run somewhere windows and the beyond and Android and whatever and Install Selenium server there Configure it like note and on some master you will register it and With your code you will just connect it to That master and you will say Which browser you want? There in this example for example is Firefox and when it's it's the same like with Chrome just you put remote there and So you want Firefox for example and that master will find node where is Firefox and you can say also that you want Firefox on the BN or that you want Chrome on the Android and so that master will find that node and if it's there It will run it and if it's not you will see you some exception So that's it. So if you like it, so check out the documentation. There are a lot more things then I was saying and Thank you. Thank you very much Michael. Do we have any questions we do? Hello If I use one instance per test case Can I have something like a setup and tear down methods for each test? because You can and it doesn't restart the browser, but and it is You have to think that if some tests messing up something You have to fix it in teardown Like like in the some another test if you will run some unit test and these unit tests Well unit test and it will Write something to database It's some it's same here. It's when you click somewhere and it will do something It's messed up and you have to fix it in teardown, but you can use it Normally as you are used to thanks Go back one slide, please Thanks for the talk You said you had to put Big configuration big resolutions when you're using the display servers like xvnc and other stuff Is that really mandatory or is it usable to? test Responsive design you can test responsive design If you will make it smaller, it's your web page will adapt. That's all right I'm just saying if you have some fixed Elements, and you can you can't do it. It's the page. It's broken. It won't work Yeah, okay. Thank you Hi, thanks for the talk. Could you comment on? Architecture of your test suite when you have a big application Are there some things that you definitely need to test or some that you shouldn't test? Can you comment on that? Well, it's a question for another 40 minutes, but There is always a 8020 that 80% of Work is in 20% of code. So Focus on what your users do mostly and test that If it's okay, if you this question is answer is good for your answer question. Excuse me Should you for example test your styles? size styles CSS your CSS is possible like the how the whether you get the I I don't think so. Well, you you can check which Values of CSS styles are there, but I wouldn't recommend it Thanks, I have one more question Can you make something like automated screenshot comparisons with this? For example start the test case do screenshots at every step remember it somehow and Then compare it every time you run the test case Is there a tool to automate that? Well, I forgot one thing. Thank you for question that when test fails and you use my Test case for unit test or my tools for pie test it will automatically make a skin shot and When you see in Jenkins that it's broken you can just check out skin shot It's very faster than the run it on my computer and check out what's what's happening Making screen shot is very simple. You can just driver that make screenshot, I think and so just pass a parameter where you want to store it and that's it and Then you can Write your code which just compares these images so you can do that very simple But I don't know anything about About any library which is doing that Any other questions? First thanks for the talk and which version of Selenium are you using with your workbench is to the Selenium yes the version. Yeah, there are two versions of Selenium 1 and 2 You don't want to use Selenium 1 because It's called Selenium and Selenium 2 is actually called web driver and Selenium 1 just Calls some javascript so it's not really user actions with Selenium 2, it's web driver and all browsers implemented it inside the browsers except Chrome with its binary Alongside and You call real API of browser and it's It's It's faking almost real user actions so Selenium 2. Okay, thanks very much Michael


The article discusses a method for correcting climate model data for systematic deviations from observations in climate impact modeling. The method is based on transfer functions that map the distribution of simulated historical data to that of observations. The bias correction preserves the warming signal and applies it to future projections. The proposed methodology is a modification of the transfer function approach used in water model intercomparison project, WATER MIP. The article highlights the potential and limitations of the applied bias correction, including its ability to represent trends and long-term means, but difficulties adjusting variability that may affect small-scale features or extremes. This article was authored by S. Hempel, K. Freeler, L. Warsawski, and others. We're article.tv, links in the description below.


This video will walk you through the process of generating LIDAR surface models from point cloud data using ArcGIS. One of the confusing things about ArcGIS is that LAS files do not show up automatically in ArcCatalog, so under Customize ArcCatalog Options, we can go into the Files type and we can add the ability to view LAS files within ArcCatalog. In the File type window, simply enter the file extension. You don't need the dot before the LAS, give a description, and then finally choose one of the fancy icons you have in your Windows system folder. Simply refresh your view in ArcCatalog, and now you'll be able to see your LAS files alongside your other geospatial datasets. If you want to work with LIDAR data in LAS format within ArcGIS, you first have to create an Esri LAS dataset container. To do this, just go to File, New, and choose LAS dataset. Here I've created a new LAS dataset, and I'm just going to rename it LAS dataset. Now that you've created the LAS dataset container, you'll want to add LAS files. But before you do that, go into the LAS dataset properties and check the box for store relative path names. This will allow you to move your LAS dataset and associated last files to different locations without worrying about breaking the path names. Now click on the LAS Files tab, click on the Add File button, and browse to the location of your LAS file. In this particular example, I've just got one LAS file called Point Cloud, so I'll go ahead and add that to my LAS dataset. Now let's do some data exploration by moving over to the Statistics tab. Statistics are not calculated automatically, so I need to calculate them by clicking on the Calculate button. Once the process is finished running, you can see that I have information on the returns, the attributes, the classification codes, and the classification flags. This type of information is going to be really important for me as I seek to make various LiDAR surface models from this single LiDAR Point Cloud dataset. What's great about this LiDAR dataset is that the contractor classified ground points, meaning that we'll be able to generate a digital elevation model that strips away above ground features like trees and buildings. Finally, it's a good idea to confirm our coordinate system so we know what units we're working with. Now that we've populated our LAS dataset, we can move over to ArcMap and visualize our LiDAR data. To begin, I'm going to load in some imagery as a reference, and then I'm going to load in the last dataset. By default, ArcGIS will display the points according to their elevation, but don't be alarmed if you don't see the points when you're zoomed out. Due to the limitations of ArcMap, you'll only see a red extent rectangle, but when you zoom in, the points will be displayed. To most effectively work with your LAS dataset, you'll want to make use of the LAS dataset toolbar. So I'm going to right click in the gray space above my other toolbars and access the LAS dataset toolbar. It's easy to adjust the symbology for the Point Cloud. In addition to colorizing the points by elevation, we can also colorize them by class or return number. The Profile tool is one of the best ways to develop a deeper understanding of your LiDAR Point Cloud data. You can access it from the LAS dataset by simply clicking on the LAS dataset Profile View icon. Then you're going to draw a line, click, and define a rectangle. This launches the Profile View window. The Profile View allows you to view your LiDAR Point Cloud data on a horizontal plane. You can zoom into specific locations. In this case, I've located a building along with some adjacent trees. Now I can change the symbology from the LAS dataset toolbar, and then if I click on the Refresh button in the Profile View, it's updated in the Profile View window. I'm making adjustments to the symbology to better understand the classification and return numbers in my LiDAR dataset. I've concluded that we have enough information to generate both ground and above-ground surface models using this Point Cloud data. I first want to create a raster surface model that consists of only the ground returns. This is sometimes called a DEM or Digital Elevation model. To do this, I'm going to have to create a LAS dataset layer file. A LAS dataset layer file can contain only a subset of the last points, so I'm going to use the Make LAS dataset layer tool. The input is going to be my LAS dataset. I'm going to give a name for the output layer, so this is not a new file, it's just a layer file within my ArcMap document.  The first thing I'm going to do is choose my class codes. I'm going to choose class 2, which if you recall corresponds to the ground returns. I'm going to keep all the return values and simply click OK. You'll notice that a new layer file appears in my ArcMap table of contents called Ground. These are the points only corresponding to the ground returns. To create a raster surface model from these ground returns, I'm going to have to use a geoprocessing tool under Conversion Tools to raster. I'm going to run the LAS dataset to raster tool. The input last dataset is going to be that ground return layer that I just created. I'm going to create an output raster and I'm going to call it DEM, Short for Digital Elevation Model. I'm going to add the .img extension to put it in UDAS Imagine format. UDAS Imagine format is a great format for large rasters. I want the value field to be elevation. This means that each pixel value will have the elevation value, but I'm going to change the interpolation method to triangulation. It's been my experience that the natural neighbor interpolation method works best. Now as we scroll down to the bottom here, you can see the output data type is float is fine, but I want to change my sampling conditions. I want to make my sample value, which is my cell size, specified by the sampling type to be 1.5. This is a little bit coarser than the point spacing for the dataset, but it will make my dataset a little bit more manageable in term of the size. I'm going to click OK and this will start the raster generation process. So while this is running, we'll do a quick recap of what we just did. We created a layer file of only the ground points, and then we fed that into a rasterization tool. We're creating a new raster in which each pixel is going to have an elevation based on the ground returns, and it's going to have a resolution of 1.5 meters. Now that we have our new raster dataset called DEM, Short for Digital Elevation Model, we probably want to go in and change the symbology and apply a nice color ramp to make it easier to interpret. Hillshade layers can really improve our ability to understand elevation data. So we'll create a hillshade layer using the image analysis functions. We can open those under Windows Image Analysis. This displays the image analysis toolbar. First, I'm going to select the raster that I want to process, the DEM, and then under Processing, I'm going to go down to the hillshade function and choose the appropriate color ramp. I'm just going to use gray here, and then I'm going to click on that Shaded Relief button. This generates a virtual hillshade layer on the fly, so instantaneously. I'm now going to slide that down below my DEM layer. I'm going to select both of them, create a new group layer, and then finally, I'm going to go into the properties of that DEM layer and adjust the transparency to around 35% so that the hillshade layer appears underneath it. Using the digital elevation model, we're able to clearly make out topographic features such as a small stream here that would otherwise be difficult to detect using imagery alone. Our first step in creating a surface model that represents above ground features is to create a new last data set layer consisting of only the first returns. Once we have the first return last layer, we can go into our Jew Processing Results tab and open up the last data set to raster tool that we ran to create the DEM. This ensures that we use the exact same settings. We simply replace the input last data set and change the name of the output raster. We'll call it DSM for Digital Surface Model, and then click down, confirm all the other settings have remained unchanged, and then click OK to generate the raster model. Once we have our new raster surface model, DSM or Digital Surface Model, just as before we'll adjust the symbology by applying a nice color ramp and then use the image analysis tools to generate a virtual hillshade layer to improve the interpretability of this raster data set. A digital surface model not only includes elevation for the entire terrain, but also the features that are above the ground surface such as buildings, trees and utility lines. Thus it's the type of layer you want to use for analyses such as line of sight. However, it's not particularly useful for measuring the height of individual features, and that's because it does not have the height relative to the ground. In order to do this, we'll need to create something called an NDSM or Normalized Digital Surface Model. You'll notice that in our DSM that we have here, there's a bit of an elevation gradient as we move from the left side of the screen to the right, meaning that elevation is increasing from west to east. Once we have our DSM and DEM raster models, it's very easy to generate the NDSM or Normalized Digital Surface Model. We'll simply enter an expression into the raster calculator. We'll take the DSM and subtract the DEM. The end product is going to be a new raster that has the pixel values with the elevation above ground. We'll navigate to the output raster dialog and call this new raster NDSM and give it the .img extension. Just as we did with our other raster surface models, we'll go in and adjust the symbology. Because this raster represents the height above ground, it's likely you'll want to adjust the stretch type to a min max. Furthermore, you'll probably want to edit the high-low values. You often get some spurious values, so it's a good idea to start at zero and then to enter some cutoff point for your vertical threshold. We're not going to bother generating a new hill shade for the NDSM. We'll use the hill shade from the DSM as from a visual analysis standpoint it will work just fine. So we'll simply copy and paste it and rearrange the NDSM so it's over the copy of the hill shade DSM layer and then adjust the transparency accordingly. With the NDSM enabled, we can use the identify tool to get the heights of some of these buildings. The heights return to our meters because that's the vertical units used for the LIDAR dataset. In this video, we showed you how to use ArcGIS to generate LIDAR surface models. We used the ground returns in the LAS file to produce a DEM or digital elevation model. We used the first returns to produce a DSM or digital surface model. And then we took the DSM, subtracted the DEM to create an NDSM normalized digital surface model in which each pixel represents the height above ground.


I think I need to start simplifying the paintings for these 24 watercolors of Christmas season. Oh my goodness, I painted a sugar glider this time. I had to go look up what a sugar glider was after I got a request for one, because I don't know that I've ever actually seen one, but I gave it a pink sky and started getting out my small brushes to do all the little ears and all the little features on this critter. I kind of want to paint one bigger now because there's lots of little fuzzies going on that I'd love to explore more, but in a tiny painting like this, I had to use my itty-bitty brush and I'd kind of like to get a little looser with it, but I used a big brush to add some gouache snow. And there we go, my sugar glider.