Hi Welcome to the euro BSD con Talk on transport layer security. I'm Michael Lucas. I'll be recording this talk so that it can be delivered best But I'll be live to take questions afterwards So what's in this talk TLS is huge And so I'm only going to touch on basic core that you need to know today and An overview of TLS how certificates work how they're revoked and invalidated How CSRs should be handled today and ACME? Mostly we'll be focusing on web servers This talk is based on Open SSL 111 or Libre SSL and I tried a few different platforms with all of those There is a newer open SSL just now, but rule one of open SSL is do not break the command line So everything should keep working TLS is often thought of as web server security and That's true as far as it goes, but there's a whole bunch of security things that TLS simply doesn't do TLS does not add security. It doesn't block intruders It doesn't keep credit card secret and it doesn't stop password theft What it does do is it encrypts traffic between clients and servers It identifies servers clients or both That's it. Nothing more Now TLS has been around for over a quarter century now And it's gone through several revisions Mostly illustrating that any cryptographic protocol developed in-house will fail Netscape Wanted the web to become an e-commerce platform that meant it net needed transport layer security So they developed this thing called secure sockets layer in-house the Once it was exposed to the real world they hurry it up and rushed out version 2 and then version 3 and Succeeding versions followed a little more slowly and We've been on TLS 1.2 since 2008 or so More than one cryptography expert told me that TLS 1.3 is the first version that wasn't rushed and has a proper cryptographic design When we're working on the internet Very often we fall back on pastel's robustness principle Which is to be conservative in what you send and liberal in what you accept This means you may have web servers that offer old versions of HTTP You may have a database engine with an older query support SMTP so long as you're not allowing and relaying spam you can pretty much get mail through Older versions of DNS still work And the fact that we can do this is glorious and it is one of the internet's advantages that Let us have the world we've built The robustness principle does not apply to TLS There's something called a downgrade attack if I can get say between you and Amazon and Convince Bezos to speak to your client using SSL version 3 I can capture and decrypt all the traffic The two of you exchange this gives me your username password and credit card numbers or I could change information In transit to completely alter your order or direct you to my website instead You must disable and Refuse to accept weak encryption What does weak encryption mean? Well all versions of TLS Accept 1.2 and 1.3 are known to be breakable and are officially deprecated Supporting older versions of TLS or worse SSL threatens the integrity of your network and Then in early 2021 the US national security agency strongly discouraged many common TLS 1.2 configurations and The NSA has this distressing tendency to predict future cryptographic attacks Now we cannot disable TLS 1.2 on the general internet yet of one thing there is no FIPS compliant TLS 1.3 browser and if if your site is meant to be accessible from Environments the US government considers secure you're stuck on TLS 1.2 I'm sorry, mr. Postel. We still love you, but TLS is one of those examples that demonstrates the limits of the robustness principle now There's a couple things I want to make a point about cryptography It's a very specialized skill with many pieces and there are Very few sysadmins or SREs who really understand it Cryptography is a great happy with lots of math If you're not doing the calculations and studying the innards of the algorithms You don't really have an opinion on what works and what doesn't maybe your Opinion People like me we have no choice but to trust the cryptographers or a standards body like FIPS. This is why FIPS and similar standards exist. I Strongly discourage Almost everyone from hand-tuning ciphers and algorithms because we're not equipped a key part of modern sysadmins skill is admitting what you don't know and Don't engage in occult cryptic IT Really what we need to know is protect your private keys revoke your certificates at any excuse and to use open SSL's high cipher list Now the key problem of cryptography is deciding who to trust We generally have two trust models the web of trust that you see in PGP and certificate authorities the web of trust Requires educated users Which immediately rolls out the general public? Certificate authorities are easiest for users TLS is about end users. That's what it uses a CA Verifies the identity of people hosts and organizations Each has a trust anchor certificate of its own and it uses that certificate to issue certificates for hosts and users declaring that we have audited this person and Or this server and they are who they claim to be and Client software comes with a bunch of anchor trust anchors That it trusts to sign other people's certificates There are six major trust anchor bundles Microsoft Apple Google Mozilla Adobe and Oracle Now different software uses different trust anchor bundles and not all certificate authorities are in all trust bundles Let's look at what comes with open SL SSL for an example The trust bundle is installed per install and per Unix if you are using Free BSD versus sent us the trust bundle you get may be slightly different Most Unix is use the Mozilla bundle One important thing is every Operating system has its own way to manage the certificates in its trust bundle Always read the documentation because if you just dump a certificate you want to trust in the directory It'll get overwritten by an upgrade now. There is an open SSL dash CA file option that lets you Trust a particular search in a particular command if you're testing Now anytime we talk about trust bundles there's an overriding question of who are these people? If I look at the Mozilla bundle there are entities there like the Shanghai electric Electronic certification authority Who is that? Do I need to trust them? Should I trust all these governments? They need us to trust them, but do we need to trust them? This is the ultimate Question of the certificate authority model. I'm not going to belabor it any further in this talk But it it is a legitimate concern So how are these certificates used through something called the chain of trust here? We have a very simple chain of trust there is a company called Gilder that we trust there's root certificate and They have signed my certificate. This is the document and this is the diagram you will find Dating back to the 1990s. It's Very simple, and I'm only in it's very obsolete. I'm including it simply to say Certificate validation is finding a path between the certificate and a trust anchor What's a little more realistic now is something like this where? the root CA has its certificate and it's signed and Intermediate certificate and that intermediate certificate in turn signs yours now If you're running an application your application must provide the intermediate CA certificate It's often called a chain file Without that your application will error What's a little more realistic now is This tree of trust where you have multiple CAs signing multiple intermediate certificates So that if any one of the certificates further up the chain is Revoked declared invalid or cannot be verified You still have a viable chain of trust between a trust anchor and down to your certificate If you have this you must include both intermediate certificates and This is where software often goes wrong Open SSL not long ago imploded when a trust anchor expired and it couldn't validate the other route and It gets more complicated than that sometimes you have trust anchor sometimes you have intermediate certificates that get promoted to trust anchors Your software must always be updated forever so How would you use open SSL? It's the foundational TLS toolkit of the internet and it gets lots of flak for being complicated. The problem is It's complicated because cryptography is complicated. This is hard stuff So basically you run open SSL some sub-command and then options for it the sub-command sets the feature you're using options configure your task For example We've all troubleshot a problem by using telnet or net cat to connect to a port It's a vital skill. It doesn't work well on a TLS protected port because we can't decrypt it by hand But you can use Open SSL's s-client and connect to a port and talk to it here My web server has a redirect. I wanted to be sure it was still in place But web browsers cache stuff even when you want them not to So I used open SSL s-client to connect to that port and make sure that the redirect was still in place So let's touch on certificates quick TLS certificates are really X dot 509 certificates. This is the IT use standard For digital identity certificates. It's built on ASN one, which is the standard used by SNMP and telecom firms This is the same object tree as those protocols use Just a different part except when they steal from each other You don't really need to know ASN one except to recognize when you see a stream of numbers that This is a bit of ASN one your software doesn't know certificates also pillage the x dot 500 standard That you see from LDAP and that's where you get these labels for parts of identifying information every label has an ASN one identifier and Certificates expire usually every year or every 90 days As certificate has assorted components the certificate signing request or the CR CSR is A form basically please sign me that you send to the CA and once you have the certificate The CSR is irrelevant. It includes your public key and it comes with a private key file The private key is the secret for your certificate whoever has that Can's pretend to be you Other certificate file is assigned extract from the CSR that the CA sends back to you And you combine the certificate with the private key to authenticate Whatever happens protect your private key Now certificates can come in a few different validation levels Domain validated means that the CA verifies that the requester controls the host and Then there's organization Validated and extended validation That perform deeper audits To most of us the important thing is all of them provide the browser lock icon Do you need one of the higher validation levels? Only if you are a bank or financial institution that is bound by regulations to do so Certificates have a couple different algorithms these days There's RSA the older standard we all know and presumably love 2048 bits is the current standard. It's been around for a long time and it's we know it's pretty solid ECDSA is a newer standard. It does not use key length, but instead uses named elliptic curves It's roughly as strong as our essay we think but it requires less computing to validate It's also not been around as long and hasn't earned the same trust in General use ECDSA if you are targeting mobile devices Now let's talk about common names, they're the most widely recognized part of a certificate It could be an email address a user ID first and last name a device serial number Up until the year 2000 CN was used to store host names This is now deprecated and I suspect many of you don't know that Uh, it's incorrect Despite that some software expects to find a host name in the CN People are working to actively exterminate this If you are using or looking for or inserting host names in CN deliberately for your application, please stop and Why a CN can have a maximum of 63 characters Host names can be longer than that. I mean, I have this domain You keep using that word. I do not think it means what you think it means calm There is no way I can get WWW dot dot that domain name in a certificate The use of host names in CN is holding back improvements to TLS The common name is not checked against restraints Uh and Those constraints are important as we'll See a couple of them and the replacement is called a server alternative name or San Now you can view the contents of a certificate Including the certificate chain identified entities extensions and so on Such as those certificate subject alternative names Here I've deliberately pulled them out of a certificate and there are a few of the domain names on my X509 cert for my site There's also a nifty thing called a wildcard certificate Where you can put the certificate on any host in your domain or subdomain that sounds great except You must put the private key on all the hosts If one host is compromised everything is compromised And if your intruder is sneaky, you will never know The save is used is for stuff like dev ops where you're constantly deploying and removing machines in this dynamic cloud Environment, uh, I don't do that thankfully, but that is where people sensibly use these now we uh Yes, we still have to we're still recording uh revoking certificate if you think your certificate is compromised if someone has stolen the private key If you think someone has stolen the private key revoke the certificate The certificate authority lets all the clients know by two different ways One is certificate revocation lists and the other is the online certificate status protocol um If your certificate authority offers revocation Insurance as in they'll give you a new search for free if the old one is compromised Or if you're using acme with one of the free ca's revoke on any suspicion now Firefox and safari distribute lists of revoked certificates to their clients And chrome filters the revocation list chrome users see Uh, I run a small business. I revoked my certificate chrome users did not see it If your business is in a country that google doesn't pay much attention to You must assume that chrome users will not see the revocation either Uh protect that private key now The certificate authority lists that they distribute uh This file contains a serial number of all unexpired revoke certificate the certificate revocation list or crl The difference between that and the online certificate status protocol is that ocsp lets you check one certificate at a time rather than downloading a whole list And OCSP has this neat feature where it provides a certificate of validity It says for the next seven days.  Well, the simplest way is a short lived certificate You can get a name constraint certificate that lets your organization sign certificates In its own domain and you could give those certificates an expiration date of like five days And just deploy them through your automation regularly There's something called dain which puts key uh certificate key fingerprints into dns Uh, that's popular in email not so much on the web You can yell at your vendors chrome And tell them to fix their revocation model You can deploy browsers or forks of browsers that enable secure features You can store private keys in hardware security modules But This is the big problem with tls right now even as it's become more Common it's gotten harder to revoke a certificate So how do you get a certificate? You create one of those certificate signing requests csr as we talked about and they contain All the information that the ca must validate Now open ssl provides a friendly dialogue that gives you a csr. You've probably all seen this And It puts the hostname in cn It supports only one hostname This script has technically been obsolete since 2000 and people are still writing tutorials using it They're really trying to kill it. Don't use it Use a configuration file To create your csr or generate it wholly on the command line. It's not that hard Now It's possible to reuse your previous year's csr to renew your certificate This is horrible practice The csr is tied to a private key Are you certain that that key did not leak? um Have the key standards changed since last year csr's are free. They can be scripted Every bsd has worked hard on their random number generator Don't waste that effort and we are not so close to the heat death of the universe that entropy is a limited resource Make a new csr and a new private key every year Information that goes in your csr includes uh What you want to validate if you're getting a dv search you only need host names All the organization information only needs to be there if you are getting a higher validation level You put all of this information in using x.500 And before you start gather the official names, it's easier to find out the formal name of your company Before the audit then after And you create a configuration file much like this. This is for a dv certificate. So the only things in it Uh our host names. I also set the number of bits and key file names Uh some software chokes on a certificate without a cn. So I do put one in here Uh, but the real list is under alt names at the bottom And you run a comparatively straightforward open ssl command to generate the key and this Gives you a file with the csr and a private key Uh as set in the configuration file Now the hot new thing in tls is acme Which people all know as free Certificates as someone who paid a hundred dollars for a certificate in 1996 This was a big deal Now Let's encrypt certificates expire every 90 days. This is to encourage automation And acme basically works by Verifying who controls the host or if you prefer who controls the dns of the host And this is a automated process where the client Uh asks for a certificate The the ca sends back a change to be made to demonstrate control of the host The client makes the change And the ca acknowledges it and accepts the csr And boom you get your certificate There are three kinds of acme challenges Uh hdp o one is put a file with this content on your web server dns o one makes a change in the dns And al pn o one makes this change in a tls protocol itself What should you use? Uh for a single server htdp o one if you want a wildcard cert Most ca's make you use dns o one And if you're using load balancers, etc al pn o one is your friend When do you renew these certs about two-thirds of the way through the lifetime? This gives you four weeks to resolve any transient issues You can schedule it in cron to run weekly and most clients handle this just fine Now there's a bunch of acme clients out there. One of my favorites is open bsds. It's clean. It's simple. It just works It's also hooked in with uh liber ssl And that makes it a little difficult to get across some other unixes My favorite cross platform one is dehydrated And it's simple enough. It runs on a raspberry pi without trouble Now one of the problems of tls is you can set it up And it looks like it works, but you can miss that you have bad stuff enabled very easily ssl labs provides a free public tls checker now The catch with ssl labs is it only works on public websites and tests might set off intrusion detection So if you're in a very secure environment Check with your security staff before testing your web server If you want to run such tests Privately test ssl.sh works beautifully not as pretty gets you the information Now i'm almost out of the time I set for recording. So i'm going to touch on a few different topics quick If you do nothing else after this talk get your web servers up to tls 1.2 and 1.3 only There are other features then you can look at hsts Let's your web server declare to clients that this web server only speaks https and should never speak unencrypted http that will help those clients evade downgrade attacks CAA records are dns entries that let your organization dictate which ca's may issue certificates for your domain Certificate transparency lets you see who's issued certs for your domain If you want to see what goes wrong with tls Badssl.com Gives you all kinds of sample failures that you can test your clients against And finally there are name constraint ca's which let you sign certs for your own domain if you are If you are subject to regulatory requirements and you need a whole bunch of certificates Running your own ca and getting a name constraint ca certificate may save you a lot of money and a lot of outside headache So That is the end of the recorded part of the talk tls is huge, but i'm sticking around for any sysadmin level questions And of course there is my book tls mastery that is out. Thank you for sitting through this and please Say hello Howdy folks if any any questions, uh I think we can probably read the the chat or Probably we can unmute people as well Yeah, people are talking about acme.sh uh They're they're not really transparent with some of their decisions Which kind of discouraged me from using them. You mentioned, um tls testing sites Uh one one site that i've been quite fond of is hardnize.com Run say a large number of tests on whatever domain you you feed it Oh, i've not heard of that one so, um I think it's free um um I think uh, I think it's free free service, uh, but the Uh, if you want to run a number of tests on your own domains, uh, you need to uh register, I think But it was actually actually quite quite useful It even uh, this detects your tls for smtp configurations and point out the errors to me Yes, that would be useful In general like anything else You want multiple testing tools from multiple providers Because no one of them is complete Yes, there's a test ssl dot s8 sorry Morgan asked are there any comprehensive testing tools which are not a web service test ssl dot sh It it is a shell script. It provides everything that's in ssl labs It is uh Uh, it's perfectly fine And it's all command line How could I talk about running my own ca? We don't have many questions here yet. So I'll I'll give a uh A quick bit on that Running your own ca is one of those things I recommend like building your own firewall at home Everybody should do it at least once You will learn a huge amount by doing it Then uh You can start by using Open ssl has a ca command That is perfectly fine for educational use but is not suited For production or for exposure to the internet uh It even includes things like an ocsp responder Which uh Is very educational to to run in debug mode and see exactly what happens in those conversations If you are looking at running a ca in Your company's production environment Uh, then You really want a a better tool kit than raw open ssl Something that say has database locking And has an ocsp responder Written by people who actually write internet facing demons routinely and are uh And who habitually write software that can withstand the internet But I I would encourage you to do it at least once just to learn how all of this really fits together Okay, what what else have we got? Yeah more acme Katie server Okay, tia I have never used tls with a storage server So I would say one check the documentation Two according to the standard everything should Take server alternative name A host name in cn has been deprecated for 21 years now Here here in where I live the deprecation is now old enough to drink vote and get married uh So hopefully It you it recognizes server alternative name Let's see tested blog using ssl labs and got a single line nothing really suspicious I suspect uh, if if nothing suspicious shows up in your log with ssl labs I suspect that your ssl configuration was clean and Not much made it to the server because I I have spoken with people who had poor server configurations and uh pointed ssl labs at their host and Got warnings from intrusion detection and some of them got blocked Okay, we have some folks typing I suppose Do I have any experience with snmp over tls any future in that area? I've played with it some while working on the snmp book um It works It's fine Uh, it has it definitely has a A higher overhead than just raw snmp I don't think that the industry is going to widely adopt it Uh Simply because every vendor seems to have their own snmp replacement And wants it to become standard So Any thoughts on tls and open vn open vpn with a private ca The my main Concern with these softwares that include A ca for generating Well, I'll I'll pick an open vpn because it was mentioned The documentation is great on getting you up and running And it's not so great on how to renew the certificates So Be sure that you know how to renew certificates And maintain this ca over the long term Uh before You deploy it I've talked with more than one person who tried to figure out how to renew certificates And wound up wiping out their existing ca and having to reissue everything in a hurry Which is Well, it is very secure because getting rid of the old private keys Uh reduces the the attack surface, but it's certainly not convenient. Let's see How does quick compare with tls? Oh There are I'm here at the front of the room and people are going to call me the expert on this and I am I have done some reading into it, but I think we have People better qualified to answer than me in the audience, so If you have a better answer then Please put it in the comments. Thank you. Yes. Yes, you broke me. It's early. My caffeine hasn't kicked in If I was actually in vienna, you would see me with a with a cup of tea in each hand Guzzling Hoping to be awake at some point Isn't it too risky to place your certificates in one place? That's a uh, that's Is it risky to have all of your certificates and provided by one ca? Yes, it's also risky to have all of your certificates provided by multiple ca's Choose your headache What I do find useful is um Or or what I suspect will become useful are caa records Which state which ca's may issue certificates? For your zones Let's see can tls be used in peer-to-peer environments. Is it a good idea? If you are willing to maintain your own ca and issue client certs all around By all means do so If I was to architect tls from scratch, what would I do? I would become a truck driver Okay, and we have an answer to quick versus tls there so technically the time slot is over But i'm i'm gonna hang around for a bit and after this i'll be uh Hanging out in the the chat room or the hallway track If you want to come by and talk And yeah, there are there are many applications that use tls for just one thing or another And actually if you're deploying something now or if you're creating a protocol or an application Starting right off with it only speaks tls 1.3 is not a bad idea um What do we have in store if security flaws are found in tls 1.2 or 1.3? well 1.2 The nsa has already said please avoid these algorithms So I think that we will find cracks there and Multiple cryptographers have told me that tls 1.3 Is the first version of tls with a proper cryptographic design It took a while to get actual cryptographers In the tls design so I suspect that tls 1.3 is Pretty solid But it does have room for extensions expansion and alteration So it from what I have read It's I don't want to call it future proof, but they left some space For fixing problems Okay, I think that's about it Thank you for attending


Ac mae'r troiigau i'r ddweud ym mhwynt yn cael ei hunain, mae'r rhaid o'r cyhoeddur ar Eurwyr, arall o'r rhaid gennym, a mae'r ddweud yw wedi'i ddiddordeb Dm25. Janis Barath Farchas, ddod ddod. Mae'r rhagos o'r referenda wedi'u bod yn gwneud o'r Lleidio Lleidio a ddod o'r oponence. Ond mae'r rhaid o'r cyhoeddur o'r ddod o'r cyhoeddor yng nghymo i'r ddod. Y rydym yn y gyrfa i. Is it not, is that the two official campaigns, both the remain and the leave campaign are enfantilising voters in a rather cynical and just astonishing fashion. We on the other hand rely on investing in reasoned debate and unity. rydyn ni'n meddwl siarad ar gyfer. Fy rwy'n meddwl y gweithio ddweud, rydyn ni'n meddwl i ddodgi'r strategaidd. Rwy'n meddwl i'r economaidd, rydyn ni'n meddwl cyfryd yn meddwl y strategaidd ar ddweud yn gwybodaeth y cyfrifredig yn ei dweud. Fy rydyn ni'n meddwl y gweithio ddweud, mae'r ddweud yn ei ddweud yn fwyrofodol. Yn ymgyrch, ymgyrch yn ei ddweud a'r bobl yn ysgolol, a'r bobl yn ysgolol, Mae'r ffordd yma, gyda'n gwneud hynny, rwy'n dweud bod yna yw uned i'r ddechrau sydd yn dysgu, sydd Amtoni gyda i'r ffordd yw'n mynd i'r ddweud yw'r Dymogradd yw'r Dymogradd yw'r rhiannol du i'r unio eu ddysgu. Mae'n ddysgu i'r union eu ddysgu eu ddysgu i'r unio eu ddysgu i'r unio eu ddysgu i'r Dymogradd yn ddysgu i'r Dymogradd Y Bryddon yn ddiwedd. So we are here to present the radical case for keeping Britain in the European Union, the internationalist case for doing so, the case for rejecting the beggar thy neighbour logic of both the British establishment and the Brussels establishment. Our campaign engages honestly with all sides of the argument. So when I hear Brexiters say that sovereignty is paramount, I agree with them. Sovereignty is an odd to be central. But the fact that the British people must never settle for diminished democratic sovereignty does not mean that leaving is going to restore sovereignty to the House of Commons. Voting to leave the European Union will only benefit a national oligarchy, a national ruling class, which is particularly keen on ruling over the British people completely and utterly and democratically. Consider the complaints about the overage of the regulatory environment mechanism of the European Union of Brussels. We agree it is important to keep a check on bureaucrats luxuriating in the power of their unelected office, but leaving the European Union will not change anything in this regard. Britain's establishment will never allow Prime Minister Boris Johnson to leave the single market, even if the voters on the 23rd of June vote to leave the European Union, and so there will be no escape from the European Union's regulatory framework. On migration, which is the other rallying call of the Brexit campaign. I think we should be concerned that the undisputed net benefits of migration are very asymmetrically scattered throughout society. And yes, public services in certain parts of Britain are indeed strained, leaving many with a feeling that they've been marginalised in their own country. However, this feeling is not caused by migration, ladies and gentlemen. It is merely correlated with it. The reason public services are failing is the rolling austerity that cloaks a vicious class war against Britain's poor, a class war that was being waged since the 1980s, a class war that would be waged even if the borders remained hermetically sealed. Indeed, we all know that, don't we? Without the labour, skills and dedication of migrants, staffing the National Health Service and other public services, those public services would have collapsed already. And lest we forget, turning that native poor against migrant labour is a variant of the old divide and rule trick that the British establishment honed ages ago to dominate the empire. Today, the establishment uses the same trick to dominate the domestic natives to hide austerity's effects and to deflect anger towards the other, the migrant, the foreigner. Friends and opponents ask me very often how is it that, given my complete failure last year to impress upon the European Union the importance of treating my countrymen and women humanely, I have the audacity to stand in front of a British public, asking them to stay in the same European Union that crushed us. Why don't I agree with those who argue, like Tariq Ali, my good friend and comrade Tariq Ali, that speeding up the European Union's fragmentation through Brexit is not such a bad idea? Well, I answer with a question. Will the European Union's disintegration cause progressive Democrats to rise up across Europe, to empower their parliaments, to usher in the forces of light and hope, and foster a harmonious co-operation between Europeans? Not likely. Yesterday I read on a left-wing site, a British one, a depiction of our campaign of the event today as left-wing turncoats fearing change. I have news for those comments. We crave change. We are working for change. We are campaigning to tear up the state score, but to ensure that change is progressive. We have to embed Britain's democracy in a broader surge of democracy that runs throughout the breadth and the width of the European Union. This is why here I'm about to sign the London Declaration for a Social Europe, a Democratic Europe, a Dynamic Europe, a Peaceful Europe, an Open Europe, a Sustainable Europe.  Now my message to my British friends is don't be like him. Don't stay in Europe just to complain. And my message to you comrades is after this event we must all go out there and campaign, campaign and campaign to bring out the vote that will cause us to stay in the European Union with enthusiasm for a common cause, which is nothing else than taking arms against the sea of troubles and by opposing end them.


In this lesson, we will learn how to test for the presence of transition metal ions and the ammonium ion. A flame test may be used to test for the presence of group 1 and group 2 metal ions, as well as for the copper 2 plus ion. To test for transition metal ions, we can add aqueous sodium hydroxide to the unknown solutions. The color of the metal hydroxide precipitate formed can help us determine the identity of the transition metal ion. General equation for the reaction is metal ion plus hydroxide ion gives a metal hydroxide precipitate. Addition of sodium hydroxide to a solution containing copper 2 plus ions creates copper 2 hydroxide, which is a blue precipitate. Iron 2 hydroxide is a green precipitate, which slowly becomes reddish brown upon standing. Iron 3 hydroxide is a reddish brown precipitate. Zinc hydroxide is white. Here is a challenge. Why does iron 2 hydroxide go from green to reddish brown upon standing? Please pause the lesson to think about this and resume when you are done. Iron 2 hydroxide goes from green to reddish brown due to the oxidation of iron 2 plus to iron 3 plus. Hence, on standing iron 2 hydroxide oxidizes to iron 3 hydroxide. Some non-transition metal ions that form a precipitate upon the addition of aqueous sodium hydroxide are the calcium ions, the aluminum ion, and the magnesium ion. These resulting metal hydroxide precipitates are white. Further addition of sodium hydroxide will dissolve aluminum hydroxide. Calcium hydroxide and magnesium hydroxide can be compared using a flame test.  To test for the ammonium ion add aqueous sodium hydroxide and warm in a water bath. Ammonia is formed and its presence can be detected using red litmus paper which turns blue. Alternatively, a glass rod may be dipped in concentrated hydrochloric acid and held near the mouth of the reaction vessel. Ammonium chloride in the form of a thick white smoke should form. The color of the metal hydroxide precipitate produced can help us determine the identity of the metal ion. Most of these metal ions are transition metal ions. The ammonium ion can be tested using the addition of aqueous sodium hydroxide which forms ammonia.


From Hollywood, it's time now for Johnny dollar. Oh, thanks Hi Pat Johnny Sure did and I'm getting ready to leave for South Bend right now in the company of a beautiful charming lovely Charming lovely girl named Bonnie Lamar That's right Thomas Renee Lamar's daughter Telegram for her arrived at the same time I received yours Didn't show her my wife. No, she doesn't know yet that you think it might be murder And every week day night Bob Bailey in the transcribed adventures of the man with the action-packed expense account America's fabulous freelance insurance investigators cause truly Johnny dollar Expense account submitted by special investigator Johnny dollar to the Universal Adjustment Bureau Hartford Connecticut attention. Mr. Patrick McCracken Following as an accounting of expenditure is during my investigation of the Lamar matter or Was it murder? Expense account item one. I'm calling it item one Pat because it's really the first tab on the Lamar case Previous expenses here in La Jolla were charged against the Jolly Roger case Expenses for the vacation you promised me and I'm now so rudely interrupted Item one nine dollars sixty cents for that long distance called a Pat McCracken in Hartford Vacation same as I was trying to take and tell me something Pat As a claim already been filed on Lamar's million and a half dollar policy Well, and how'd you know about his death so quickly? Oh sure none of them for years good man So was he actually there when Lamar died? Look, you keep referring to him as the old man. Just how old was he the doctor's name on her telegram was Wilson You know his first name. No, I don't okay. Yeah. Well, that's flattering. Okay. It looks like I am but tell me something Yeah What makes Larry think the man was murdered our plane leaves in about an hour Pat. That's the one thing I don't want it at all. I hung up leaving Pat to ponder over that last remark Why Larry Comstock that I was coming and finish my packing for the trip back east When I finished I paid my bill at the fancy motel and all I can say is thank goodness It was on expense account and I knocked on the door of the cottage next to mine. Hi, Vani Any way I can help you more than you have You've been wonderful Ranging the flight back for me for us taking care of the things here Johnny that's right for us. I'm making the trip with you, but you I thought you said Hartford, Connecticut And your vacations all the vacations all over Wouldn't be any fun for me to stay around After this all done and South Bend is along the way I'd feel better if I kind of took you home rather than let you make the long trip alone under the circumstances Maybe I have some business or something to attend to there Darling, I don't know what easy You made it so wonderful when daddy couldn't get here these last few days and Now that this terrible thing has happened Stick by me this way. It's the only way I'd have it you You're so wonderful Come on now. Come on. Get your things together. I've called for a cab to the airport in San Diego. Thank you, darling. I Love you for this Sure. I can't say I exactly relish thoughts of the flight back east much as I hoped I could be of some small comfort to the girl Much as I genuinely wanted to Such things can be pretty rough particularly in this instance But I am an insurance investigator And in a matter of this sort a million and a half dollars at stake the possibility of murder Well, it's up to me to suspect everyone whether I like it or not Yeah, I sometimes think it's a pretty rotten racket to be Johnny sleep on a sleep You'll need all of it. You can get before you have to face things at home. I wasn't sleeping. I was just thinking Being so thankful that you're here with me Honey, I wired ahead for a hotel reservation. What? Yeah, I'm gonna stay in South Bend a few days You're wonderful. Oh, no, I'm gonna be honest with you. I I also wired a friend of mine Well a fellow with whom I do business now and then so I Well anyhow, I'll be there for you for a few days and maybe more and as long as I can be of any help to you It's funny. Hmm You know, you still haven't told me what you do Well Don't worry about that now, but I'm curious Tell me It'll give us something to talk about Did you wire anyone at your home about your arrival? Yes Harris in the butler Johnny. Well, how about the doctor who telegraphed you? Yes, Dr. Wilson to honey Wilson Edward T. Wilson Not tell no no now you you stop talking and try to get some rest I'm going back to the lounge in the tail section so that you'll have nothing to do but get some sleep Then you won't tell me no tomorrow Sleep no, I can maybe very rough. I Felt like a traitor to her Well, we landed in Chicago at 10 a.m. And took a cab from the airport to the Lamar home on the outskirts of South Bend. I Never before realized that the big industrial city with all its huge dirty sprawling factories had such a wealthy residential section and The Lamar home on parody Lane was one of the most impressive of all Set far back and what must have been an acre of well-kept lawn In addition to Harrison the butler we were met at the door by the housekeeper cook upstairs and downstairs maids and a couple of other servants all of them obviously in deep sorrow over the passing of the master of the house and May I most humbly for all of us express our deepest sympathy in this hour of this It's all Thank You Harrison Thank You all I'm going to my room and we'll call you when yes miss This is mr. Dollar. Mr. Dollar.  You know where to reach me. Yes Johnny, thank you And now to get to work whether elected or not I took the cab to the townhouse dump my bags Then back to Chicago in the office of Lawrence Comstock try mutuals representative. He was waiting for me Well, Johnny, you sure walked into something this time thick one Larry You don't know you don't know the half of it the million and a half of it You gave Pat McCracken back in heart for the idea that Lamar's death might be murder. I think it is I really think it is Johnny. Why Tom Lamar was one of the best friends. I ever had Should have been your commission on the insurance. He was carrying was enough to set you up for life Oh, no, Johnny, don't talk like that Tom was a good friend of mine quite aside from business I sold him his very first policy years ago And he was just a bookkeeper for Atlas processing company earning $70 a week and when he married delice release his wife Who died five years ago? Oh that policy was only for 2,500 straight life So, you know how little my commission was on that but I liked him I saw that he had a spark about him and with the proper kind of encouragement he could go places and he did Yeah, so I understand I understand the Lamar metal products is a really big thing general metal fabricators just bought them out Oh, yes, and Tom was getting all ready to retire Spend the rest of his days having fun gall fishing winters in California and summers in Minnesota that sort of thing and Taking care of Vani his adopted daughter Yeah Kind of worth taking care of too Hey I know her Larry Met her in La Jolla, California.  Why I didn't oh Yes, of course the family doctor Ed Wilson. I should have realized he sent a telegram to Vani to the same place You were in La Jolla She's a wonderful girl John You're telling me But Larry Yes, something you told Pat McCracken back in Hartford has led him to think that possibly Thomas Lamar was murdered John Johnny in the years. I've known Tom Lamar. Yeah, I've not only known him, but I've known his family Well and much of his affairs personal as well as business Well, his wife Delice I would have married her long ago if I'd been able to get to the point Larry Oh, yes, of course and his daughter La Vaughn Vani I wish she'd been my daughter my child. Come on Larry. Come on get out. She's a wonderful girl. You said that oh Yes, of course Well, there were things in her past Vani's past that even her mother and later her father didn't know about But I did for heaven's sake man get to the point You too. Yes me too Yeah me the confirmed bachelor take them and leave them have fun forget to make a big Listen Johnny now listen carefully Dr. E. T. Wilson Ed Wilson an old friend of mine as well as Tom. Yes It was Ed who made the last insurance examination four months ago Thomas Renee Lamar wasn't better health than you are after all he was only 59 and he'd lived a careful life taking good care of himself Well, go on we were sure of his physical condition sure of it. That's why I let him add to his already large policy Larry you've told Pat McCracken and you've admitted to me that you think Thomas Lamar was murdered. Yes, John Because of one man who the one man who shared his Confidences business in person. Yeah, who was closer to him than even Ed Wilson or me well who one man who alone could be sure of Benefiting by Tom Lamar's death. Oh look Larry that bush you're beating around is getting bigger and bigger. It's so simple John So just scourgingly simple. All right. All right, Larry. All right taking any way you like I'm here for two reasons because I'm a sign of this case and because of Vani Yes, I know now Who is it you suspect? The man Vani is really in love with I'm sorry John Now here's our star to tell you about tomorrow's intriguing episode of this week's story tomorrow some stuff I didn't want to hear but I had to Join us, won't you yours truly Johnny dollar It's truly Johnny dollars starring Bob Bailey is transcribed in Hollywood It is produced and directed by Jack Johnstone who also wrote tonight's story Be sure to join us tomorrow night same time and station for the next exciting episode of yours truly Johnny dollar Roy Rowan speaking


Hi, I am Gustavo Ortiz with Cal-O-E-S that all overnight temperatures are impacting much of the state. Cal-O-E-S is coordinating support to counties in opening warming centers to protect vulnerable Californians. On the 10th anniversary of the Sandy Hook School shooting, Cal-O-E-S and other state leaders honor the victims through action in promoting gun safety and California gun violence restraining order effort. And finally, give your family the gift of peace of mind this holiday season. To prepare for a disaster, sign up for alerts and start getting ready now, visit listoscalifornia.org. To see more from us, go to news.cal-o-e-s.ca.gov and follow us on all our social media platforms.


Hi everyone, Joe for jaspyscasebreaks.com coming at you with the Chargers, Bengals, Dolphins, Eagles, Cowboys, Random, Number Block, Randomizer for Pick Your Team, Number 9. There's the instructions on the Number Block if you're not familiar with it. Big thanks to Jay, Don, Scott, and Roy for getting into this one. There are the numbers right there. Let's roll it and randomize it. Randomize it. Two and a four, six times for names and numbers. One, two, three, four, five, and six. Two and a four, six times. We got Jay down to Don. EA, Roy, what's going on guys? How are you? Two and a four, six times for the numbers. One, two, three, four, five, and six. And final time, we got three down to eight. There we go. All right, so Jay, Jay with three, Scott with six, Don with seven, and one, Roy with nine, Don with zero, two, four, Roy with five, Don with eight. Let's sort this by column B. Some good numbers for Don. Good numbers for everybody if they hit. They're only bad numbers if they don't hit. We've got to do one more of these Number Blocks and then we're in Brakesville. Take a look at jaspyscasebrakes.com and let's get to that limited football. I'll see you next time for that one. Bye-bye.


What's up guys, today we're going to be opening an epic Destiny Hero Plasma 10. I believe this from the year 2007. It's going to be pretty awesome because first of all, this is an upside downturn, which probably makes it more valuable or something. I don't know if it does or not, but we're still opening it. And inside we have elemental energy, power of the duelist, cyber dark impact, strike of Nios, and force of the breaker. A lot of you guys always ask me, open GX. And I always get burned by GX. But I was like, you know what, we'll open this 10, just for you guys. Let's see if we can pull a crazy ultimate rare out of five packs. The chances are not high. Let's see if we can do it. They give away for this video. Statue of the Wicked First Edition from Ferranic Guardian. Giant Trunade with shifted set number right there. And Eurya Uriah, however you say it, Lord of the Ceiling, Lord of the Ceiling, Lord of Searing Flames. All you have to do is like this video, be subscribed and let me know what your favorite GX set from this 10 is. There's only five of them. You got five choices. All right, we're also going to open up some OTS 13 because I have a few more packs left. And I want to pull a Book of Moon ultimate rare again. We did pull one or two, actually. I think we pulled two. Bahamut Shark. And I've had these packs laying around. I'm like, you know what, it's time to open these. It is time to open these. Some of them, you know, we opened some in the, what's it called? The Not Raging Battle. I always say the Power Tool Dragon Tin for Kevin. Now we're going to open up the rest that I have. Here we go. Deep Sea Diva. That's a nice super rare. Can we get an Ulti? I kept one pack for the collection. Just in case maybe one day we open every pack again. Just in case. So I kept pulling. So that might have an ultimate rare in it. Who knows? Oh, Crystal Wing Synchro Dragon. But yeah, if we open every pack again, it's going to be epic. It's also going to be way harder now because everything is even more expensive. Spiral Mission Rescue. Can't go you defender. Spiral Mission Rescue. So far, only super rare. Not looking too good. Spiral Mission Rescue. Is that three in a row? What is this? Three straight. Spiral Mission Rescues. That is super weird. Very, very weird. All right. All right. Can we break that streak? I'm guessing no. Yeah. Generator token. So far, nothing. We're just striking out big time. ABC Dragon Buster. Okay. Not a bad one. It's better than Spiral Mission Rescue. Uh-oh. I just completely annihilated this pack, but I can't get the cards out. Floodgate Trap Hole Junk Speeder. I like how it skips right to the thing. All right. Two more packs. Looks like we are not getting too much out of your spiral gear drone. All these packs. No ultimates. Not doing too good. Lady of White in White and Bahumut Shark. All right. So we got all super rare. So that's okay. We saved our luck for the 10. This is what we're here for anyway. We're here for the Destiny Hero Plasma. Can our promo not be damaged? I've never opened this 10, so we can add this to the 10s that I've opened, which is a lot at this point. We've all opened all 2002. All 2003. Okay. This is not ripping. We've opened a couple of 2004 at least. We've opened some 2005 with Dark Magician Girl. Yeah. That was some stretchy stuff. The 2002 ones just rip off super easily, probably because they're like a million years old. All right. We opened it up. The value has been annihilated until we pull a crazy ultima rare. Then the value will return. Then the value. I'm still not even sure, honestly, that these are probably like, I'm not even sure if Unlimited has retail packs, but I probably do. And these are probably them. Okay. Look at those beautiful packs before we do that. And we check out this this pocket. Excellent. Let's just check out the pocket guy. Let's go ahead and do it. New tactical evolution. The dawn of a new dueling era. So this was 2017, I think, right? Yeah. The more you... What does that say? Okay. Let's move in. The more you got, the stronger you get. I like it. I like it. Yeah. Good. Yeah. Buy more cards. All right. Rainbow Dragon, Justoni and Emperor, Frost and Flame. You know, cool stuff like that. Collect all 25 must have Yu-Gi-Oh! GX cards. Oh, cool. So I like how your Eye Lord of Searing Flames is on the Shadows of Infinity when you have Raw Vi-L on the cover art. Okay. There he is. All right. There he is. Oh, this is actually really cool.  That is, that's pretty cool. It's kind of like the McDonald's packs. I remember getting that.  And then I got it. So it was pretty, pretty awesome. All right. Destiny Hero Plasma. Oh, guys looking pretty good. Yeah. He might have a little dimple right there or something. Overall, pretty nice condition, I'd say. Pretty nice secret rare. I think this guy's actually, we're like 15 bucks or something, which is actually pretty good for a kind of a random promo. All right. Let's put this back. Let's get our five packs out. Oh, set our promo back there. Reassemble the tin. All right. The good old hole in the front. All right. So now we have Force of the Breaker. We already talked about this, but Strike of Nio. So you don't get to open that too much and I definitely don't. So is this Cyber Dark? In fact, it looks super like weird to anybody. It looks like really yellowy. Normally they're like super dark. I don't know. Power of the Duelist. Is this like European or something? It might be European actually. And elemental energy. That's a good thing because European doesn't have hobby retail either way, even if it's first edition. So let's go ahead and let's do Strike of Nio's last because that's a crazy set. So is elements or energy. Let's do, I don't open much Force of the Breaker. So let's go with Power of the Duelist first. I think probably I don't know. It is actually the supers and ultras are really cool, but the retail stuff is not not that cool on this side. Let's see if we can pull something crazy. I'm going to move these all the way. All right. Flying saucer. Moose. Moose. Oh, elemental hero. Nio's. What was this guy's? A new elemental hero. That's right from Nio Space. It's a classic. Wonder Garage. Oh. Wonder Garage. Baby Sarasaurus. Oh, Nio Space and Dark Panther. We got a Super Rare. Right off the bat. That's pretty cool. Crop circles, fake hero. We got an upside down alien hunter and brainwashing beam. That's pretty cool. Of course, we have to sleeve up the new elemental hero, elemental hero Nio's. So far, not a bad start, not a bad start. Cyberdark and back.  Can we pull an amazing Cyberdark Dragon Unlimited? Could you imagine that after pulling that ultra or that ultimate rare? Stratismodian. Instant Fusion. Okay. Nice card. Man Beast of Aries. Accumulated Fortune. And Iris, the Earth Mother. Countercleaner. That is creepy. That is creepy. Alert Queen Level 3. Barrier Statue of the Stormwinds might be worth something. I know that some, a couple of the Barrier Statues are worth something. Can't remember which one is which. I think Stormwinds is actually, I feel like that's the good one. And Instant Fusion. Okay. Pretty good comments out of that. Okay. Force of the Breaker. Yeah, the, not too much in Odawa this set. The Doomfire Dragon, I think is the name. And then the Harpy's Queen, I think. Ultimate Rare. All right. Wildfire. I feel like they're missing something that's pretty good out of here. Let me know in the comments. Crystal Beast Amberman. Yeah, the Crystal Beasts are worth pretty good amount. Crystal Abundance. Lucky Iron Axe. Okay. We're looking for luck. But we got a Lucky Iron Axe. So that's not exactly what we're looking for, but that's okay. All right. We've got some big packs right here. We have Elemental Energy and Strike of Neos. Like, Elemental Energy, the Ultimate Rares are crazy. They're crazy good if you get the Ultra Rare Ultimate Rares. Yeah, it's very confusing. When talking about GX, it's just, people are just getting into the hobby. They're like, what in the world does that mean? So if you're interested in hobby retail, just check out my video up there. It will probably help explain it. All right. Bonding H2. Come on. Give us something crazy. Scar, Scout of Dark World. Reborn Zombie. Rapid Fire Magician. Oh, Hydra, get on as a comment. Thought it was a foil at first. I was like, wait, they have a Champion Pack version too? Okay. I've actually pulled Rapid Fire Magician Ultimate Rare in a video before. All right. Last pack. So far, we've gotten a Super Rare and our promo and then a couple old comments. So not exactly the best, but that's okay. This was a birthday present at one point. So I did get it for $0, which is great. So thanks, Mom. Thanks for buying me a Yu-Gi-Oh-Tin at my age. Okay. Synthesis Smesp. Spell, not smell. Okay. Electric Virus. Destruction Cyclone. Flashbang. Okay. Hit the like button for a lot, guys. Let's see what we can do here. We got a Super Rare. Great Shogun. Sheen. That's actually a really cool one. That would have been a crazy Ultimate Rare pull. Great Shogun. Sheen. That's super cool. Super Rare. Puppet Plant and Glomaw. That card's annoying. All right. All right. Let's see what we got. We did okay. We did okay. We didn't do the worst you could possibly do. So we opened five packs and out of five packs, we got Instant Fusion. Common. Berry Statue of the Stormwinds. Elemental Hero. Nios. Great Shogun. Sheen. Super Rare. Neo Space and Dark Panther. Super Rare. Actually, pretty cool Supers. And the Secret Rare promo. So we definitely did not make our money back. So but I hope you guys enjoyed this video opening some GX classic packs. Maybe if you have some, hit me up. Maybe I'll buy them. They are kind of expensive for what you get.  If you want to support me, make sure you guys check out my TCGPlayer and eBay affiliate links below. I appreciate all you guys that do that. And that's it for today. I'll see you guys later. Peace.


what's up y'all I'm back with the biggest tip that I've given out all year this is a one-play touchdown against cover three and cover four so you use the previous player the New Orleans Saints gun tight offset mini scheme to get them to start running more covers get them in cover three and cover four and you can use this play to take the top off the defense we sneak it in you always want to put your fastest player in the slots in this offense and that will allow you to use this deep corner out that's found in this place Saints drive out and it's also finding PA seems this deep corner route will allow us to burn cover to honestly all the cover for beaters I've seen online on websites on YouTube this is the best cover for beater in the game and there's no question about it so that's what we're going to go against here we're going to go against a cover for drop out of a big dime 146 you see here what you need to do is put a baby out route on X that's the only adjustment make sure you run it to the wide side of the field the baby out route will hold down the zones don't ask me why the baby out routes hold down the zones I don't understand but they do you'll see it'll hold down the deep blue on the corner and allow this deep corner route to run into a huge hole for a big play big play touchdown against cover four which should be designed to stop big play touchdowns and that's why people run it you see a lot of people just sit and cover for and think it's safe this is the fastest most effective bomb I had all year verse cover for all the deep polls take a long time to develop you need great pass protection this one happens fairly fast that's all we're going to do we're going to take Brandon Coleman put him on an out route he will hold down the deep blue zones and then because of the angle of this corner route it's not too shallow it's just perfect go run away from the safety so because of that you don't have to worry about either the corner or the safety and that's why I think it's the perfect cover for bomb to mix in with the northern Saints gun tight offset many scheme the other two people like to put on drags and in routes or hitches or stuff like that so you can pepper the middle of the field while they're running back to try to cover this bomb that you show them that you can score one play against max coverage this is great weapon to have and I hope you guys enjoy it and add it to your game and you can improve this northern Saints scheme so if you liked this video if it helps you beat cover four and cover three please hit the like and comment I'm going to start a new mini scheme once I get the 300 likes on this video I'm gonna start a whole new mini scheme I really appreciate y'all coming by checking out this tip hope it takes your man game to the next level check back soon