The- the hell? I don't even think the guy knows what's happening. Hehehe. Hello everyone and welcome back to another episode of Secret Kaiju Game, which will remain nameless. And as you can see at the bottom, I have 26,000 fricking points. Yes, I have been grinding this because I am so aware that this game can disappear. Mechadinosaur. Because it has the best fricking weapon or ultimate I have ever seen. Also it can fly and I kind of want to test out how that works. I am tempted for legendary type though. Oh, okay. Oh, we are. We are here. We are in the middle of the battle. Right, I have no idea how this guy works. I think spaceballs fly. Yes! Woo! Okay, I can only fly for so long. That's interesting. Right, so it's like a sprint. Uh, uh, don't. I will come through. Hit you. Hehehe. We didn't get away from me, dammit. Right, let's check out my moves. So one is, one's a laser that takes a long time to fire. Cool, cool, cool. Two, two's a hit. Right, okay. Karate chop and two's a karate chop. Number three. Okay, three is a good one. Very quick as well. Four. Right, I don't know what this thing does. I've noticed this a lot. I'm wondering if it protects from beams or it adds a shield. I'm not too sure. Oh, there's a dude over there. Okay. Oh, is this guy going to join in? Don't join in, dude. Don't. Oh, yeah, here he comes. Oh, no, he's ganking up. Uh-oh. He's like, hey, dude, can you help me? He's like, oh, you want to help me? I will help thee. Hold on. Let me do my laser. There we go. He's like, ah, stop it.  I thought we should fight him. Okay. Well, oh, I did a thing. There we go. Losers. Three is the flame. Oh, wait, wait, wait. What? I don't think so. Karate chop. Oh, I'm good at this. I'm good with this guy. I'm a laser. I do a weird thing.  All right. Let's do this then. Shoo. Thunderbolt. Wait, wait. Can I do my ultimate? Yes. This looks awesome. A rainbow lasers and missiles. Looks so good. I got my first kill, sweet. Karate chop. Shoo. Oh, God. I thought I blocked. I guess I did not. Uh-oh. Uh-oh. This is not looking good for me. Oh, God. He hit me. Shoo. I will fly to you. Oh, wow. I'm not very good at this. There you go. There we go. Okay. Okay. Okay. Yeah, that's good. That's good. That's good. Get a few good hits in here. Oh, no. I've got to weigh up my options a little bit here. Oh, God. Oh, wait. Did I just? Oh, I did my laser beam. I didn't realize I did that. No grabs for you. Okay. Oh, that didn't work. There we go. I might be terrible. Oh, God. Oh, hello. I'm coming. I'm coming. Give me a second to get back.  There we go. Two. There we go. Three. Get wrecked, son. Oh, my God. I'm doing so good. Okay. Not that bad. I've run out of moves. I literally have nothing. Oh, my God. Okay. Okay. Hold on. Hold on. There we go. Okay. We need to burn him. Wait, is he running away? Oh, God. Oh, God. Oh, that. Okay. Okay. I'm trying to block. He does not. He does not like fighting in the water. He does not like fighting in the water. Will you fight me? Will you get back here? I don't have a head. Oh, God.  Oh mitzvah. He's really good. I fought him before Just fell over. Oh, why do I keep spawning here? It's like no you this is the only place you may spawn. Yeah, welcome. I Make a Godzilla wait. Oh, oh now you have a now you want to fight I was just about to take on him for you know third partying, but now you want to go, huh? Yeah, yeah, how about that huh you can do run you're gonna. Yeah, I thought you were do you want to take on this then? Oh a little bit of block very good very good Yeah, I saw that coming. Yeah. Yeah. Yeah, you want to go again? Oh god. Oh, okay. Okay. That's cool Right. Let's burn you And then it's for sure. Oh god. Oh, all right then all right. All right, all right That's not so good One and then I think we're doing good here. We're doing good trades. It's gonna do tail. Yeah, so that coming. Yeah, yeah, that's great That's great. Oh, okay. Okay. Oh, I need to do my two two.  I got burning feet.  Yeah, there we go.  Yeah, yeah Hold on. Let me do this Is he just he's just picking on him now. This is not cool, dude All right. Well, I'll give you that one for free, but you will die now Or not. Hold on give me a second Oh, you think you could get away. He's almost dead as well, but I'll finish him. I'll get him He's like, oh god. He could fly No Right, right. Let's burn him And slam slam. There we go. There.   I'm so sorry. I know you're trying your hardest. You really are. Let me lazy He's out Boaty, you did it. Ah, that's the thanks I get for trying to help you. Huh? There we go. There's some burnage And then who are you are boaty? You're going down son. Oh, okay. Okay. That's fine. That's fine Yeah, yeah, you might do this, but what about amazing laser beam? Yeah It looks so cool You want to fight, huh? You can do your tail. Yeah, that's all I'm here. Ah, here. Ah Another two. Oh god. Nope. Uh, right. Didn't stun him. Didn't stun him.  I'm stuck Let me get out. Let me get out. Let me get out. Let me get out. Let me get out. All right. You don't want me to get out That's fine. We can go.  That's cool He's trying his best now. Oh god. Yeah, he's gonna get me his way quicker his guard break Oh god, uh, let's do it. Yeah, I guess so Hahaha No, you don't I've lost me abilities No, you don't Yeah, back in the water His only weakness. No, what? Let's burn him Just melting his face That's brilliant And nothing you can do either. He's like damn it. Oh, what? Oh, that's not cool. Now is it? Hey, what a laser? Uh-huh. Yep. Yep. Yep. Mm-hmm. Yeah, I know You know what? I didn't have to come in here, but I felt like it was necessary. Oh, that's not good. That's not good. Yeah. Oh, well done, dude Why always here? Wait, what are you doing? Oh, I'm doing well. Oh, there we go.  Well, I can't even see what he's doing, but I am apparently being wrecked here Right. That's it. You're getting burnies. There we go. Come on. Just die already. There we go. Take care Whoa, it was a lot harder than I thought when you get stuck in the water is this guy. It's not so great The worst things to fight. I think in this game Or is Eva Eva and then the lava thing because they just have so much speed and maneuverability They can just time everything you're gonna do move out the way and do something to counter it I feel like there needs to be a slowness to it Oh, oops. Oh, sorry. Oh, no. I got excited again Sorry, sorry Oh It was an accident. Okay. I didn't mean to go for you. Damn it. Why am I always spawning here? Oh, okay. That looks cool, too Yeah, I still prefer my beam though. My beam looks awesome. Wait, are you gonna take us on? You're not.  Don't do it Oh, yes as well All right, then what are you gonna get lasered? There you go. There's your laser. How about that? Do you want to go? All right, then it's on That's fine I'll just do some hot feet towards you hot tomato hot tamale There we go. Get away and then block it. Oh, yeah, just in case he did actually hit me there Stabbed inside.  No Okay, okay. I'm seeing this guy's pattern here There we go. You're gonna growl. Yeah, he's gonna growl. Okay. We're gonna do burny feet after this Burny feet And then cry chops And if he's close enough too, oh he grabbed me. That's okay. That's okay Okay, we're gonna do our laser. He's pretty much just standing there ready to be hit Try just uh That's what it was. I thought it was beef a beam Oh my god, that looks so cool Another tip. What's your tip? Yeah, tell me the tips. So hold on. I just want to see this guy's laser Oh, look at that man. That looks awesome.  Ow, ow, ow. Is everyone is everyone just going for him? Oh my god Everyone's just killing him. Right by turn Laser! There we go. Now hit him Got him, got him, got him. There we go. Laser him too. Just for good measure Ow. Oh, he's definitely dead. Phew. Phew Oh He's got himself in the fray He's instant death. All right, let's go. Moose. Well done everybody. Robot or roose Robot team All right, pure pure. There we go. Let's f him while he's away That's a good combo. I like that one and then oh, we can't want him. Okay, never mind Oh, you try to step on me. Oh, yeah her block it. Okay. Let's just do our ultimate while, you know All right, this guy doesn't block. Oh, he does Oh, never mind. He doesn't block when I'm doing this though. So even better And then three of them. Oh, no, that's four. That's that's the opposite of what I wanted to do Okay, okay, that's cool. Whoa. I've got my laser back already That's awesome. Oh god. Okay, ready go go go go go go Yeah, no one likes goji when it does that because it just hurts everybody.  All right, let's laser them Yeah, to me to you to me to you Let's go Fly right, we're gonna drop on him. We're gonna drop on what one move should we do? Should we do three drop and do three? Well, where is he? All right, there he is Am I three? You know what I think Out of all the kaiju I've played so far This I prefer out of all of them. It has the maneuverability if I want to get away from something That's or catch up to something mainly if it tries to run away from me It has a good amount of lasers quick guard breaks quick jabs to combo with other things And the booty ability like the three is really good. I think this guy's a really well balanced dude, but you want to fight Okay, what do you do? All right then Oh, okay, you want a hug? I get it. I get it.  Oh, okay Uh, I was gonna say don't tell me the game's gone. Wait, how are you swimming like that? God the hacks never end for this thing. Jesus. Look at it No, but I can fly that high I don't even think the guy knows what's happening He's literally just flying around He could actually control it can actually hit him in the sky. Oh, I can How'd you fly dude? I want to learn Teach me Oh I could laser in the sky Oh, look at the missiles go Oh my god, this must be such a rare thing I'm gonna I'm gonna try and see how high I can go and wow you can see them for absolutely ages There it is The heart of mount fuji feel like my legs should rotate The hands don't feel like enough propulsion to really push me forward Hello Okay, I soon be the only put oh no, there's a Godzilla over there. Oh, we could do an epic fly battle. Oh, I can just breathe I've got an incoming call. I could just drop down and do this I'm gonna get you No All of the awesome nurse come back here think I won't be able to get you in here. I can get you in here This guy just does not want to die. He's like, please no, please. Oh no. Yes Yes, there could always be one flying Titan And I it's Yeah Oh Go then we are the gods of this new earth. Hi big laser Oh, you'll go you'll fight now, will you? Oh, yeah. Yeah. Yeah when they all help. Uh-huh. I see how it is Now you'll fight. Uh-huh. Oh wait, am I on the floor? Oh, you knocked me down, dude Oh, we're so close to dying here It's literally next hit dude Oh, wait. Oh, yes. He has no idea that I'm not almost dead. Oh now you're afraid I'll see how it is There can be only one headless night in the sky And it's me. I think we're gonna leave that here I did not expect to be going through this But hey, oh You never can tell well when these games and videos are gonna go But if you enjoyed the video leave a like until next time. Well, I'll see you guys later Oh, bye-bye


Good morning and welcome to this public meeting of the consumer product safety commission. Before we start, can I confirm that all the commissioners are here? Commissioner Feldman. I'm here. Thank you. Thank you, Mr. Trump. I'm here. Good morning. Good morning and Commissioner Boyle and I am here as well. Thank you this morning. CPC staff will brief the commission on draft supplemental notice proposed rule making regarding information disclosure under section 6 B, the consumer product safety act. Or the 6 B regulation, this proposal is a continuation of the work commission started in 2014 to update our section 6 B regulation. Those who are unfamiliar section, 6 B of the CPSC, which is our authorizing statute. Places of strict restrictions of how and when CPSC can discuss a manufacturer's consumer products. Before the agency can tell the public about a product, we generally must negotiate with the manufacturer over what we say and when. This can prevent CPSC from issuing timely warnings about dangerous products and put Americans at risk. We're the only federal safety agency with such a statutory restriction. And while I believe the best solution would be Congress to step in and appeal the the statute and put us on par with our sister safety agencies. As long as we have 6 B in place and raise the law, the commission. Can and should do better to ensure that our regulations are as consumer friendly as possible and don't restrict us beyond what the statute requires. I look forward to working with my colleagues to review this package and make the proposal available for public comment. In a moment, I will turn this meeting over staff so that they can brief us. Once they completed the briefing, each commissioner will have 10 minutes to ask questions and staff multiple rounds necessary. As a reminder to my colleagues, if your questions address the agency's legal authority, other legal advice, please hold them until closed executive session has been requested to be held later in today, following the public briefing briefing. Us today is Amy golden attorney in the office of general counsel. Also, with us are Austin, general counsel, most of the handshift, excuse me, assistant general counsel and Alberta mills commission secretary. Thank you and without, I'm going to turn the microphone over to miss Colvin. Good morning, chair and commissioners. My name is Amy Colvin and I'm an attorney in the office of the general counsel in the division of federal court litigation. I will brief the commission today on the draft supplemental notice of proposed rule making to update 16 CFR part 1101. Information disclosure under section 6 B of the consumer product safety act. Next slide. So, section 61 of the consumer product safety act states prior to its public disclosure of any information obtained under this act, or to be disclosed to the public in connection there with. The commission to the extent practicable must provide manufacturers for private labelers with advanced notice and a reasonable opportunity to submit comments on the proposed disclosure. If the manner in which such consumer product is to be designated or described in such information will permit the public to ascertain readily the identity of such manufacturer or private labeler. And section 3 of the consumer product safety act defines manufacturer to include an importer. Next slide. Section 61 of the CPSA requires the commission to take reasonable steps to assure that the information the commission intends to disclose is accurate. And that such disclosure is fair in the circumstances and reasonably related to effectuating the purposes of this act. Next slide. In disclosing any information under section 6 B, the commissioner may and upon the request of the manufacturer or private label or shall include with the disclosure any comments or other information. Or summary, there are submitted by such manufacturer or private label or to the extent permitted by and subject to the requirements of section 6. So, that was an overview of the requirements under section 61 of the CPSA in the next slide. I'll talk about the updates that the consumer product safety improvement made. So, next slide. The consumer product safety improvement act of 2008 amended section 6 of the consumer product safety act. First, it shortened from 30 to 15 days. The total period for manufacturers and private labelers to have notice to receive notice and have an opportunity to comment and information the commission proposed to disclose. Second, CPSA eliminated the requirement that the commission publish a federal register notice when the commission makes a public health and safety finding to shorten the time for providing notice. And finally, CPSA brought in the statutory exceptions to section 6 B. Next slide. So, the commission published a final rule interpreting section 6 B in December of 1983 in November of 2008, the commission published a final rule amending the 6 B regulation to reflect the CPSA amendments. And in February 2014, the commission published a notice of proposed rulemaking to amend the 6 B regulation. The 2014 MPR was based upon principles contained in the commission's fiscal year 2013 mid-year review and operating plan adjustments. The 2014 MPR proposed to modernize the 6 B regulation to account for the significant improvements in information technology that had occurred since the regulations initial adoption in 1983. The 2014 MPR also proposed to streamline the regulation to align it more closely with the text of section 6 B, including with respect to protect protecting information filed by firms under section 15 B of the CPSA. The commission received 24 comments from the public on the 2014 MPR. Next slide. The draft supplemental MPR retains many revisions that were originally proposed in the 2014 MPR with smaller note change. It also supplements the 2014 MPR based upon comments that we received and or to further modernize and more precisely align the 6 B regulation with the statute. Finally, the draft supplemental MPR responds to comments on the 2014 MPR. So, the next set of slides will be will provide a brief overview of some of the proposed revisions. Next slide. So, the first set of revisions appear at section 11, 1.11 B where the draft supplemental MPR proposes to add 3 categories to the list of information that is not subject to the requirements of section 6 B, 1, because it does not constitute a disclosure to the public. These categories were initially proposed in the 2014 MPR and based upon comments that we had received the draft supplemental MPR proposes some minor revisions to some of the categories and also provides examples for clarity. The first category that the draft supplemental proposes to add to 11, 0, 1, 0, 11 B concerns reports of harm that are posted on the publicly available consumer product safety information database, which is otherwise known as saferproducts.gov. So, section 6 A F 1 of the consumer product safety act specifically excludes reports of harm from the provisions of section 6 B. And under the draft supplemental MPR is proposed approach, the commission may release for identify information contained in a report of harm that is provided that is posted to saferproducts.gov without providing notice under 6 B 1. If the commission does not characterize the information contained in the report and the commission's use of the information is accurate and not misleading. So, for example, the commission could state that saferproducts.gov received 15 reports involving a particular manufacturer's lamp, but the commission would have to provide 6 B notice an opportunity to comment. If that same release then also warn consumers to stop using the lamp because of a hazard or contained other information that would constitute a public disclosure for which 6 B 1 notice would be required. The supplemental MPR also explains that the commission will continue to provide 6 B notice before disclosing reports of harm that are not posted on saferproducts.gov. Next slide. The second category of information that the supplemental MPR proposes to add to 11, 0, 1, 0, 11 B involves information that the commission has already disclosed in accordance with section 6 B 1. This was initially proposed in 2014. I'm sorry. I'm sorry. Let me go back and talk about publicly available information. The second category of information is publicly available information. So commenters noted that publicly available information may be misleading or inaccurate or biased and that the commission's reference to that information can imply that the information is verified or accurate. So the draft supplemental MPR proposes a modified approach based upon the comments that we received. And so under this proposal, the commission may release or identify information that is available to the public provided. 1, the commission clearly indicates the source of the information to the commission does not characterize or delay new information and 3, the commission's use of the information is accurate and not misleading. So, for example, the commission could state that it is aware of a particular newspaper article that references 15 reports involving a particular manufacturer's stroller provided the commission credits the source. It is reasonable to attribute integrity to the newspaper. For example, the newspaper follows journalistic standards and the commission's description of the newspaper's report is accurate and not misleading. And then the final category of information that we propose to add is information that was not previously disclosed that in context is not disclosed materially more or materially different information about a consumer product than what the commission previously disclosed. So this approach simply expands upon the commission's current practice of disclosing a summary of a press release that was initially issued in accordance with section 6 B1. So, here, for example, a commissioner could relate in a speech, the findings regarding manufacturer A's blender that appears in a CPSC staff report that was previously subject to section 6 B1 and posted to our website. But the commissioner could not discuss other staff findings that are not in that report unless the commission already provided the firm with notice and an opportunity to comment. Next slide. The 2014 MPR proposed requiring that the commission communicate electronically with firms and providing 6 B notices commenters overwhelmingly supported this proposal, but they did see clarification and some of the procedures that we would use. So, the draft supplemental MPR continues to propose that to the extent practicable, the commission will use electronic transmission of 6 B notices to avoid delays inherent in other methods. In response to comments, the supplemental MPR clarifies that if electronic transmission is not practical or the commission cannot confirm electronic receipt of the notice, we will use take appropriate steps to provide notice using other means, including delivery. Via US mail. Next slide. The draft supplemental MPR proposes 2 changes with respect to firms comments. So, section 6 B1 requires the commission to include with its disclosure of firms comments or other information or summary thereof. Only if the firm requests inclusion and such such inclusion is permitted by and subject to the requirements of section 6. The 6 B regulation currently, however, requires the commission to disclose comments that a firm submits in response to a section 6 B1 notice, even when the firm does not request disclosure. So, the draft supplemental MPR proposes to conform the regulation to the statute. It also requires that any requests for withholding must be made in writing. Next slide. The 2014 MPR proposed requiring the manufacturer of private label or to provide a legal rationale such as a statute of regulation to support withholding its comments. Because disclosure when it's not requested by a firm generally would become discretionary for the CPSC. The draft supplemental MPR instead proposes that the firm must explain any request for non disclosure. Next slide. So, section 6 B1 requires the commission to provide notice to the extent practicable there. The current regulation at 11 and 1.26 B provides examples of instances where it not be practical for the commission to provide 6 B notice. The draft supplemental MPR proposes to add 2 additional examples to the regulation. And this is a new proposed revision that did not appear in the 2014 MPR. The 1st example is when the commission has been unable after a diligent search to obtain the contact information for the manufacturer or private label or of the consumer product to which the information pertains. And here, this example kind of just represents what happens now commission staff conducts searches in internal databases and other sources to to find the contact information for particular firm. But there have been occasions where staff has been unable to provide or to find the contact information and in those instances, the commission has been unable to provide notice. The 2nd example is when an extraordinary circumstance necessitates the immediate disclosure of information to protect the public health and safety. Well, the commission simultaneously proceeds notification of the manufacturer or private labeler. So, for example, in a holiday weekend, the commission might become aware of a serious hazard involving a new consumer product that is associated with the holiday. And the commission attempts to contact the manufacturer without any success in this instance, the commission could go out with the disclosure and immediately notify the public of the hazard, but also wait for response for the firm. Importantly, how I just wanted to know that the commission would still take reasonable steps to assure that the information it intends to disclose is accurate and that disclosure is fair the circumstances and reasonably related to effectuating the purposes of the act. Next slide. So, this is a new proposed revision that did not appear in the 2014 NPR section 61 requires the commission to take reasonable steps to assure prior to disclosing information. That the information is accurate draft supplemental NPR proposes to add as another reasonable step that commission could take to assure accuracy instances when the commission staff relies on a statement made under oath. Or a similar statement enforceable under penalty of perjury, such as a sworn affidavit that yields or corroborates the information to be disclosed. Next slide. The next revision involves attorney work, so a product and attorney client privilege information. This revision was proposed in the 2014 NPR. So, in general, section 61 requires the commission to take reasonable steps to assure that disclosure is fair in the circumstances. The regulation currently at 1101.33 B3 provides examples of disclosure that generally would not be fair in the circumstances. So, the supplemental NPR proposes to delete the current example of releasing legally privileged information that has been provided to the commission intentionally. We believe that firms waive privilege when they provide legally privileged material intentionally to the commission. But if a firm inadvertently provides, submits privilege information, we will treat that information in accordance with inadvertent disclosure. Next slide. The final set of changes involves information containing the section 15 B report. So, in addition to the requirements of section 61 section 65, the consumer product safety act imposes further disclosure limitations and information submitted pursuant to section 15 B unless certain exceptions apply. The current current regulation at 1101.63 states section 65 does not apply to information independently obtained or prepared by the commission staff. The draft supplemental notice of proposed rulemaking proposes 2 revisions to this section at 1101.63, which we propose to designate as 1101.63 B next slide. So, the proposal proposal states section 65 does not apply to information independently obtained or prepared, which is language that is in the current regulation. And here's the new language or developed through subsequent investigation and verification by the commission, any member of the commission for any employee agent or representative, including contractor of the commission in a official capacity. And so this, this is a new proposed change that did not appear on the 2014 NPR and this change simply reflects current practice and recognizes that information that may appear in a 15 B reports such as contact information for consumers who. Report incidents to a firm or the names of retailers and distributors that the commission can contact those consumers to obtain additional information about the incidents or contact the retailer. To obtain additional information about a product that the retailer sells, even when the underlying information appears in a 15 B report. Next change next slide. I mean, so this is this is a change that was initially proposed in the 2014. NPR and we made slight adjustments. So, section 65 does not apply to information that's already available to the public, including, but not limited to information appearing in a company's press statements, websites, FAQs, product user manuals, sales materials, SEC filings or other public statements or documents. And so this is just to represent that section 65 is additional limitation on disclosure of 15 B information. Does not apply to information that is already available to the public, such as the price of a particular consumer product, which the public can find by walking into a retail store or conducting a search online. Or the sizes or colors of a particular product, which could be found simply by looking at a retail display. But depending on the information proposed for disclosure section 61 notice may apply. So that concludes my presentation. I'm happy to answer any questions that you may have. Thank you, Ms. Dolbin. I appreciate that. Appreciate the presentation as well at this point in time. We're going to turn to questions from the commissioners. As mentioned, we'll be 10 minute rounds, multiple rounds, if necessary. I'm going to start with myself. So, under the proposed revisions information is Raven may develop the public through sources, other than the commission press or so it's not covered by 60 as long as the commission indicates the sources of the information. And the commissions use the information is accurate, not some sleeting. How does this type of. Proposed revision address concerns that this type of information should be covered by 60. So, Congress applied the 6 B procedures only to the commission's public disclosure of information. So, public disclosure just by definition includes only kind of material that is newly available that that we are releasing and it's we're making known something that was previously unknown. And so when we view public information, we don't consider it to constitute a public disclosure under the act because we are not. Presenting new information, it's already out to the public, but we do recognize that there could be fairness issues in particular those issues raised by commenters that. Information that's publicly available could be misleading or inaccurate. And so the draft supplemental provides other protections to ensure fairness. 1st, the source of the repeated information must be identified. So the name of the newspaper or the scientific journal 2nd, the commission must provide the information itself without characterizing it. Or adding any kind of additional information that would constitute a disclosure and trigger the requirements under section 6 B. And then finally, the reference, the commission's reference to the information must be accurate and not misleading. So, a follow up to that, and then how do you assess the argument that the commission's repetition of information that's already in the public domain somehow fundamentally changes the nature of the information. And so, CPSC is legally required to go through a 6 B process. Right. So, we 1, I'm sorry, I miss has been careful to give her answer to your last question in terms of what's already in the document. And now we're probably crossing over into. An assessment of legal arguments that might that might be made by commenters. So, if we'd say that for a closed session, please. Okay, um, in it, um, so, Miss Colvin, can you explain the origin of the proposal that the release of public information should be accurate and not misleading and how that might work in practice? Um, sure. So, when we were considering referencing publicly available information, we looked at section 6, a 7 of the consumer product safety act and this section contains the retraction requirements. So, it requires the commission to publish or retraction of inaccurate or misleading public disclosures. And while section 6, a 7 doesn't technically apply, if we're just pointing to information that's publicly available, because it's not a public disclosure, the principles in section 6, a 7, certainly are should be considered guiding principles to both the commission and staff. That, you know, the accuracy of the information that the commission that the information, the commission releases is accurate and that we avoid misleading the public. So, the draft would require each commissioner and commission staff to look at the standard in 6, a 7 and ensure that they, when they release information that is publicly available, that their use of the information is accurate, not misleading. And would certainly be available to assist in any assessment and to provide any guidance. So, will the pros changes affect commission's treatment of confidential commercial information or financial information? Absolutely not section 6, a of the CPSA still applies and has very strict requirements on the commission's disclosure of any confidential, commercial or financial information or trade secret information. Firms are still able to submit comments that contain that information or to comment on proposed disclosures and claim that the information that the commission intends to disclose or the information, their comments contain confidential, commercial or financial information. They just need to submit the claims in accordance with the commission's FOIA requirements and the FOIA regulations specifically at 10, 15.18. Well, thank you. I'm going to reserve the rest of my time and defer to the other commissioners at this point in time. Commissioner Feldman. Thank you, Mr. Chairman, and thank you, Ms. Colvin and Mr. Schlick, our general counsel for all the work that's going into this and for the presentation today. I know that this is something that's been on the agency's books as a pending rulemaking since as early as 2014. And I'm glad that we have the opportunity to talk about it today. Ms. Colvin, I did have a couple of questions that I did want to raise it when we're talking about the disclosure of information that's already publicly available when we're talking about disclosing media reports that are that are already. Out there and published it would seem to me that the commission's disclosure and distribution of that information necessarily involves some sort of assessment on our end. Of the, the integrity of those media reports and is that accurate? Is that an assessment that we would have to make? Well, right, there's, we would certainly look at the integrity of the source. So, for example, does the newspaper follow journalistic standards? Does the scientific journal, do they have a peer review requirement? So, we would certainly definitely consider that and determining whether or not we could point to a particular publicly available information. Okay. I mean, in other contexts, we've relied on studies that haven't been peer reviewed, but we're talking about a different context, right? I mean, we would just look at the standards that the particular publication uses. So, you're right. I shouldn't say that it needs to be peer reviewed, but we would definitely. Okay. That's concerning to me. I appreciate you putting a little bit more meat on the bones of how we would sort of begin the process of assessing the integrity of a particular media outlet, but that sounds awfully close to content moderation and that's not something that we have much experience with, nor is that, it's a very dangerous road to go down. I think that gives me a significant amount of pause. But for more than that, it's troubling to me, you know, whenever we put information out from a particular source, the proposition that that stands for almost certainly is going to be sort of attributed back to CPSC as our own. How do we go about avoiding that? The proposal states that we would 1st point to the source of the information and clearly indicate the source. We would not provide any kind of characterization of the information or provide any new information that would constitute a disclosure. And then we would make sure that our use of the information is accurate, not misleading. So, for example, we could state the New England Journal of Medicine just published a report on manufacturer A's product. Okay. Yeah, it's not a hypothetical concern though. You know, recently and under prior leadership, our Office of Communication published recommendations. For example on how deep to bury beach umbrellas. We took information that came from an outside source. It was the community publication in Ocean City, Maryland, that made the recommendation of burying a beach umbrella 2 feet. We published that. There's no independent CPSC testing. But we sort of regurgitated media information that was already publicly available. And then on multiple occasions, watching this story and this recommendation get regurgitated. This 2 feet recommendation was parroted back as if it were a CPSC recommendation and it wasn't. It was from a community newspaper in Ocean City, Maryland. So, again, not a hypothetical concern. It's a real concern. And it's a concern that I have and I think glad that we're putting this out for publication, because this will be an opportunity for us to receive comments about how best to proceed on this run. I did have one other question that I want to be mindful of the time. In the presentation, you talked about instances where they're there. We're noticing comment isn't practicable. So this would be staff's proposals to expand 16 CFR 1101.26 B. And the staff proposal, as I understand it would allow the commission to act in extraordinary circumstances that would necessitate immediate disclosure of information to protect public health and safety. While the commission simultaneously pursues notification to the manufacturer or the private labeler. My question to you would be, who makes the call about what constitutes an extraordinary circumstance? Would this be sort of delegated to staff to pursue that? Or is that a call for the commission to make? You know, that's a really good question that I don't have the answer to, but I'm happy to get back to you on that. I appreciate that miss Colvin. Thank you very much. I may have additional questions, but for the time being, I'm going to yield to my colleagues. Thank you. Thank you, commissioner. Commissioner Trump. Thank you. So like the gag rule puts inappropriate and dangerous restrictions on the agency's ability to timely share. Critical information on safety with the public. And people can be hurt and killed as a result of this inappropriate statutory burden. I know that Congress is considering the sunshine and product safety act to correct this. And I hope it passes. But this agency, I think what we're recognizing today is that the agency's also historically put additional burdens on ourselves on our ability to communicate through overly restrictive interpretations of the act. And I agree that we should change that wherever that's appropriate. I'm not sure we go as far as we need to with this proposal. And so I would love to hear great ideas through a robust comment process. I do have just a couple of specific questions on the proposal. So in, in the proposed draft section. 1101.11 a 3. Would read the information must be obtained generated or received under the acts to be disclosed to the public in connection there with. Does the word generated appear anywhere in the statute. The word generated does not appear in the statute. Okay. And then in 1101.11 be 5 and be 6. We exempt certain information from 61 requirements, specifically a report of farm on safer products dot gov. And information that has already been made available to the public through sources other than the commission provided that we clearly indicate the source of the information. And the commission's use of information is accurate and not misleading as you just went over how that process would work. But in the preamble, I think we create some ambiguity and how we describe those exemptions preamble states that we can release information without 61 notice. As long as quote 1, the commission does not characterize the information contained in the report. And to the commission's use of the information is accurate and not misleading. We already have an obligation there's 6 b 6 and and 6 b 7 to not release information that is inaccurate or misleading. But the part about the not characterizing the information could be an issue. As it's undefined and I think that could be broadly interpreted. Does the word characterize appear in the statute. Yeah, it does not appear in the statute. I think to remove some ambiguity there and and retain our ability to put information in proper context when we talk about it. We may need to change to that characterization part of the proposal to add some clarity and hopefully we can discuss that through the comment process. Last question that I have here for for fairness, we allow firms to include their comments. We give them the opportunity to provide their comments alongside ours and really certain information. Many times they want to do that, you know, they want to be able to explain or put things in their own context. But in this package, we develop a lot of attention to the idea of firms objecting to their own comments being released with our release. How often do we find firms objecting to their own words being made public. I don't have specific data, but in my experience, and this is just my. Experience reviewing comment letters, the majority of firms request that their comments be withheld. And do what kinds of issues do they raise as to to justify not wanting them to be released. Typically firms will state that their comments contain confidential commercial or financial information that is protected under section 6A to. And the commission's FOIA regulation. They also. Claim that release of the information would violate section 61 because, for example, they provided information. In connection with settlement negotiations. And so they would object to the release of that information. Is it ever that they would just simply be embarrassed by some of the arguments they make to try to keep their statements private. I have never experienced that or seen that in my review. I have no further questions. Thank you. Thank you, Commissioner. Commissioner Boyle. Thank you, Mr. Chair. Thank you to the staff who worked on this. Package, I actually don't have questions. I appreciate the opportunity to take a fresh look at the regulation in light of the need to modernize and modern communications. I think it's a good idea that we do so. And I am looking forward to the robust comment period that I am anticipating. So I appreciate the work and thank you. Thank you, Commissioner. We can go to a second round of questions. Anybody has any. Commissioner Feldman. Yes, I don't have any so good. Commissioner Feldman. Thank you. Miss Colvin, we heard some discussion today about the information publicly to put that information. In the proper context. And I understand why that's that's important because. Serve consumers better to have actionable information that's appropriately contextualized, but at the same time. With what I'm reading in staff's proposal, which is talking about.  What's allowable under 6B that that's already publicly available that in order to do that permissively that we would need to. Keep that sort of without, without putting any additional context on it without sort of comment from the agency. So any, any thoughts from you or guidance? How do we put information in the proper context? But at the same time, make sure that we're not adding. Any sort of additional commentary or thoughts from the agency characterizing that information one way or the other. So, I think I look at it 2 ways 1st. If we're simply referencing or responding to reporters inquiry, did you see newspaper access article on. This particular product and we acknowledge. Yes, we are aware of this article on manufacturer a stroller. That in that context, that is not that cannot constitute a public disclosure. So section 6B and the requirements wouldn't apply because we are not. Why wouldn't that be a public disclosure? Okay. Because the definition of disclosure, a public disclosure is really releasing new material that was previously unknown. And so we're acknowledging that there is a newspaper article out there. That's information that it's already out there and the public is aware of it. So that's my kind of the 1st response. The 2nd is just framing it and to make sure that. We are providing accurate and not misleading information is the steps that we will take to 1st, make sure that people understand we are not the source of the information. We are aware there is this New England Journal of Medical Study or sorry, New England Journal of Medicine study and the commission is aware of it. Number 2, we don't say anything about the information contained in the report. We don't kind of, we don't provide our own interpretation or explanation of the information. And number 3, we don't provide new information. So for example, we would not be able to state. Here's this newspapers article about 40 incidents involving a particular manufacturer's bike. And by the way, the commission is aware of an additional 50 incidents. And that instance, that is new information that we are proposing to and I will use the word disclose that would require 61 notice and opportunity to comment. I'm curious then why why this would be drafted so narrowly that we would be responding just to newspaper articles that we've put out but putting that aside. I'm curious also. How this plays out when the commission puts out a release with with the kind of information that we're talking about today. You know, if we put out a press release character or sort of stating with no additional information what we're learning in a particular article that's that's out there or a medical journal or what have you. In my experience and in communications that's almost certainly going to generate a significant amount of public and immediate inquiries about sort of the nature of the statement. Why we why we made that statement what we believe it to mean. You know, what would the. Guidance be and I'm not necessarily looking for legal guidance, but but how does this play out how would our office of communications handle the media inquiries that are almost certainly going to follow what what what what restrictions would they be under. Well, I think that they could. State, for example, if there is a huge report on a great increase the number of incidents involving particular firms product. This would be a way for the commission and in particular. The office of communications to state consumers go go to safer products dot gov and please report your incidents without saying anything else and that way we have the message out that if there's an incident if you've experienced an incident please report it on safer products dot gov. And I think that's what we've been doing already and that's kind of consistent communication and messaging that the commission does right now. I think that makes sense when when we have an OCM that staffed by competent communications professionals and I've got a great deal of faith in the team that we currently have in place right now but that that hasn't always been the case. So, you know, when you've got a director of communications that that is a professional and disciplined in able to stay on message in situations where that's not the case which we've experienced at this agency. I think that's where I have some concerns. So again, I think this is glad that we're putting this out for comment. I'm I'd be curious to see how the public responds to those concerns in particular. I have no further questions, but again, I want to thank you for the work that you put into this and thank you for the presentation today. Thank you, commissioner commissioner Trump. Did you want a second round? No, I'm I'm sad. Thank you. Thank you, Mr. Boyle. No, thank you. Thank you. Thanks to the staff for this informative briefing and to all the commissioners for their active participation. We look forward to hearing from the public and putting this out for notice as well. With that, this briefing is adjourned. Thank you.


Hello, everyone, and welcome. I hope you all are having a good week here at KubeCon. Today, we're going to be talking about DoorDash as migration from stats D to Prometheus with over 10 million metrics per second. Before we get started, I wanted to just do a quick poll of the room. How many people here work at companies that use stats D? Okay, what about Prometheus? All right, and what about both? Okay, all right, try to catch some people in the middle of their migration, right? Cool, so my name is Ben Raskin. I'm a Solutions Architect at Chronosphere where I work on customer enablement and onboarding. Chronosphere is an observability company that focuses on cloud-native companies. Prior to Chronosphere, I was a software engineer at Uber working on M3, an open-source metrics platform. Hey, everyone, I'm Emma. I joined DoorDash for about three and a half years ago. After graduating from Carnegie Mellon University, now I'm in our Co-infrajecture Observability team and I help the whole organization to migrate from stats D platform to Prometheus based monitoring. Yeah. Cool, so just a bit of an agenda. We're first going to look at some of the challenges that DoorDash faced with stats D. We'll take a look at how Prometheus resolved those. Next, we'll talk about the client-side migration effort from stats D to Prometheus. Next, we'll talk about enablement, sort of how we got the end users and engineers and service teams up to speed. And then finally, we'll do a bit of a retro looking at some of the learnings and key results of this gigantic task. So we wanted to begin by talking about some of the challenges that DoorDash faced with stats D. So a lack of naming standardization and limited support for tags makes it hard to give context and meaning to underlying stats D metrics. So we can see here, we have two metrics here coming from two different services. They're both tracking the same thing, the number of page views. But unless you have intimate knowledge of these services and the metrics that they're producing, it's hard to know that these two metrics are in fact tracking the same thing. We'll see in a bit with tags and labels in Prometheus, it's much easier to give context to these particular underlying time series. Next, the number of metrics scales with user traffic in stats D. This means if the number of user requests or traffic to the overall business goes up, the number of metrics go up. Not necessarily the cardinality or unique number of metrics, but the total volume. This oftentimes requires the need for an aggregation tier before actually storing the metrics in the time series database. Otherwise, these metrics can grow exponentially. Next, because stats D is sent over UDP, there's the potential for packet loss. This is particularly problematic, especially during high traffic times when the server has the opportunity to get overwhelmed and therefore you could be dropping important data points. Next, the way that stats D reports counters is by deltas. The biggest issue with delta counters is there's no way to interpolate missing data points. So once the server drops the metrics because of packet loss or some other reason why you would lose these particular counters, there's no way to actually estimate or interpolate those missing data points. And finally, the lack of histograms in stats D requires the need to pre-compute percentiles for latencies. And the biggest issue with pre-computing percentiles is that when you wanna aggregate them at query time, it's mathematically nearly impossible to get an accurate representation of the aggregated view. So why Prometheus? Based on the issues and pinpoints we experienced in the stats D platform, we set some new observability requirements and principles. First, we have a strong preference of using open source. The advantages includes firstly, it is more cost efficient compared to building and maintaining everything from scratch by ourselves. Secondly, in terms of the integration, we can make use of the open source data formats and open source query languages. Lastly, using open source can prevent vendor locking, which means that we can consider self-hosting in the future. Second, we want more governance and control on the whole monitoring system. We want standard conventions. For example, the standard for common text and metrics naming conventions. The common text can be used for alerts and dashboards, especially for a dashboard, it can be used for grouping and filtering. Also, we want to improve the governance and control on the call side. If we have ability to break down the usage by team and service and organization, then we can set quota and also really meet the team and the service level. The last requirement is about self-service and productivity empowerment. So for new services, we want the metrics can be automatically discovered and collected using the current metrics pipeline. We don't want extra works. Also, we want to automate the process of generating basic dashboards and alerts. This is very important to accelerate the migration and onboarding process. So why did DoorDash choose Prometheus? So first of all, Prometheus has emerged as the dominant standard for open source metrics and it aligned well with DoorDash's organizational strategy and requirements. First of all, the simple fact that there's a ton of community support out there really helped alleviate a lot of the pressure on the central durability team. Since now, end users could go to Stack Overflow and GitHub and all these other resources online to get the answers to their questions as opposed to having the bottleneck on a central durability team. Next, the query language, promql, has become the standard with regards to querying time series data. So again, this really helped onboarding of new engineers. Since now, we have this standard query language and it's not proprietary at all and so you can bring those skills right to different companies. The tag-based metric ingestion format allows you to give context and meaning to the underlying time series and there's several advantages of this. One of the advantages that I wanted to highlight with regards to DoorDash is now with a tag-based model, it's super easy to see how many metrics each service or each team is sending to the platform. And so this makes it really easy to do cost accounting. Next, because Prometheus is a pole-based system, this allows for client-side aggregation so you no longer have this worry of your metric traffic scaling with your actual business or user traffic. Next, strong support. There's strong support for the more fundamental tools in DoorDash with Prometheus, things like Kubernetes and Envoy, they really play well nicely with Prometheus. And then finally, Prometheus has obviously native support for histograms, such that you don't need to pre-compute percentiles and you can accurately aggregate histograms at query time. And then lastly, Prometheus reports counters as cumulative counters or running counters. And I wanted to dive into this point just a little bit more, just talking a little bit about the sort of the visual representation differences between delta counters and running counters. So we see here with delta counters, when you lose a data point, it's lost forever. There's no way to know what that data point might have been. With running counters or cumulative counters, because they're monotonically increasing, you can apply a special rate function in PromQL to interpolate or estimate what that data point would have been. Okay, let's see what we have done in the client-side migration. For the metric generation part, I will first introduce the custom metrics, then the PromQL six-porters that we have used. Lastly, it's about short-lived jobs. For custom metrics, including the application and the service metrics, we suggest service owner and the microservices to use PromQL's native libraries to generate the metrics. One interesting case that I want to share with you guys is that the Python client with a multi-process pattern rather than a multi-thread pattern. As we know, the PromQL's native client presumes a multi-thread pattern, which means the metrics can be showed within the workers. However, from the multi-process pattern, we discover that the metrics can be incomplete or require custom implementation with unsorted LPS impact. So to overcome the performance penalty, we use that spotter for this part of metrics. Also, we provided some internal libraries for the developers to use and for their flexibility. For example, we provide the internal library to generate HTTP and GRPC metrics automatically. Also, the JVM PromQL six-porters is used in .h Docker based images to export the JVM metrics. For some other cases and metrics, we also use the open source exporters. The community of PromQLs is so big and it provides and maintains so many used for open source exporters. For example, the AWS Cloud, which is metrics, there are many two exporters. The first one is the official PromQL six-porters and the second one is YC exporter. The difference of the two exporters is the different APIs used in their co-functions. And we choose the latter one because it have a better service discovery mechanism and also have less load on APIs. Also, we use Stasti exporters and Stasti is still alive in our monitoring system. There are many two cases that we use Stasti exporter. The first one is that for the legacy system that can not be migrated to Prometheus, we use Stasti exporter for that part of metrics. The second use case is, as I mentioned before, for the Python client with a multi-process pattern, we use Stasti exporters to cover the metrics from my two Prometheus and then start forwarding them to the backend. Also, we use JVM exporters, node exporters, Kafka-like exporters and the PG Bouncer exporters. These are many used for the infrastructure and the platform related metrics. For the short-lived jobs that cannot be scrapped by the metrics collector, we use the Prometheus aggregation gateway with a small modification for that part of metrics. To better control the metrics, we have some standard tags for all the metrics. The common tags include, first, the service, which is the service name. Second, the app, which is the application name. One service can include multi-different apps. Also, we have Kubernetes cluster, which is to indicate the cluster that the service is running on. And environment label is used to differentiate the production and the staging environments. Also, we have the SAP environment, which is used to differentiate the cannery sandbox with other environments. The region and zone are AWS technologies and they are used to indicate the region and zone ID that the service is deployed in. Lastly, we have COSR region, which is used to analyze the metrics COSR attribution. As you can see in the dashboard, there's a screenshot which show one of our dashboard. In the header, we have the common tags in the drop-down list for filtering. Yeah, next is about service discovery. Service discovery is used to find and discover the jobs or services for scrapping. It is done through Kubernetes annotations, which tell parameters what endpoints to scrap. The central of the ability team created gold standard Kubernetes manifest template, which the service team can use. At all that should we have the service template, which is to help the developer to generate the Kubernetes manifest automatically. So after the service owner or the service team added an annotation, which means their jobs or services is ready for scrapping. We also deployed in the metrics collector as agent in the Kubernetes cluster, which means that for every node, there will be one metrics collector to scrap all the metrics on that node. Also for a service that are not deployed in the Kubernetes environment, we deployed the metrics collector as sidecar with the service, which means that the metrics collector will collect the metrics from that service automatically and forward them to the backend. The default scrap options are also defined by the COT team, which means that the scrap options including like scrap frequency, scrap timeout, and so on are this kind of configurations are included in the service collector. Also we populate the standard labels in the metrics collector for all the metrics, the standard labels as I described in the previous slide. So during this migration, it also coincided with hyper growth at DoorDash, mainly accelerated by the pandemic. And so cost control was a huge issue for the central observability team. Fortunately with a Prometheus based monitoring system, there's a bunch of different mechanisms that can help you reduce cardinality, control costs, and improve performance. So I wanted to talk about a couple of these different mechanisms. So starting with relabel rules. So relabel rules are a native Prometheus feature that allow you to drop labels or metrics client side. So typically when you're using some of these open source exporters that Emma talked about, there's often time metrics that you don't need. You don't ever query them in dashboards or alerts. And so you can safely drop them client side. The next two mechanisms, roll up rules and mapping rules are features of M3. M3 as I explained in the beginning is an open source metrics platform that was developed at Uber. And it actually acts as the underlying metrics platform at Chronosphere. So roll up rules allow you to pre aggregate metrics before actually storing them in the time series database. So again, just as with roll up rules, oftentimes with common exporters, there's usually labels or tags that you don't necessarily need. And oftentimes these labels are very expensive. They have high cardinality and you typically don't need to see, you typically don't need to break down your metrics by these particular labels. A common one is instance or pod ID. So with roll up rules, you can safely aggregate them before actually storing them in the time series database. Mapping rules allow you to define the storage policies for each time series. So this means you can down sample a subset of your metrics and store them at different resolutions. So let's say your default scrape interval is 30 seconds, but you have a subset of metrics that you wanna store at one minute. What you can do is you can use mapping rules to do that. Lastly, we have recording rules. So recording rules allow you to pre-compute expensive promQL expressions and then store those results in the time series database. So all of these greatly improve performance, especially at query time, since now you're querying fewer and fewer time series. Cool, so I wanted to talk about enablement. I think this was probably one of the more interesting pieces. DoorDash obviously has a lot of engineers, a lot of teams. And so enabling the users on Prometheus and teaching them about promQL was one of the, I would say, probably more difficult but interesting aspects of this migration. So the first thing that we needed to teach users is how to best use the Prometheus data model. Because it's a tag-based model and most of the engineering team was used to statsD, which is a node or path-based model, we really needed to sort of teach them about all of the best practices. And one of the main things was teaching them not to overindex on the metric name, but instead use tags and labels to give context and meaning to the underlying time series. PromQL, as I mentioned earlier, is the Prometheus query language and it has become sort of the standard in the industry for querying time series data, especially tag-based time series data. And so we had to teach the engineering and service teams not only about sort of like the syntax and how to actually write these queries, but we also had to teach them about the quirks of PromQL. Two of these that I wanted to highlight are how you query counters because with a running or cumulative counter, you need to apply a special rate function and oftentimes one of the benefits of the rate function is that it will actually interpolate or estimate missing data points. And so we had to really sort of pull back the covers a bit on the rate function and especially with the more sort of technical or power users of the system. Next, we had to talk about sort of how to query histograms versus timers. So with histograms, as you'll recall, you do all of the percentile calculations at query time and you do this through specialized PromQL functions. Next, not everything needs to be built from scratch. Because we now had a standard data model throughout DoorDash, what we could do is we could rely on open source dashboards and alerts and recording rules that we could just pull into the system and Teams could just get all of that visibility right out of the box. Even with service teams, we could build gold standard dashboards that other service teams could use as models for building their dashboards. Lastly, the DoorDash central observability team decided to manage all of these resources through Terraform, so managing all the dashboards, alerts, aggregation rules, recording rules, things like that. And so we had to come up with a way for Teams to easily onboard all of their different assets. And then, of course, we had to give them the tools to make it easy to do so. The final part is about timeline results in the learnings. Let's say our timeline of the migration. Our journey began in Q1 2020. That's the time that we decided to move away from Stasti into our Promq tiers. And then, in Q2 2020, we started the migration of infrastructure and the Kubernetes metrics. And it finished in Q3 2020. During that phase, we migrated all the Kubernetes and infrastructure metrics, including the Seattle weather metrics, and notice pointer metrics, and also the AWS metrics. As I mentioned before, we used a lot of exporters as them. And we deployed the metrics collector as the agent in our Kubernetes cluster. For the services that is now deployed in Kubernetes cluster, for example, like EC2 instances and also ECS clusters, we deployed the metrics collector as a sidecar with a service to scrap their metrics automatically. So starting from Q4 2020, we started migration of service and application metrics. We finished all the metrics, dashboards, and alerts within one year for all the service teams. Then in Q4 2020, we deprecated the original Stasti pipeline and we only used some Stasti spotters and kept a lightweight of the Stasti pipeline, yeah. Cool, so just taking a look at some of the numbers of this massive migration. So we started with more than 7,000 alerts, 1,500 dashboards, and metrics for over 130 services. Needless to say, this was a massive collaboration across all teams and engineers, especially MS team, the central observability team. One year post migration, which also coincided with hyper growth in DoorDash's business, we now have over 27,000 alerts, 2200 dashboards, we're ingesting over 15 million metrics per second. And then post aggregation, we're persisting just over 10 million metrics per second today. So a couple of the learnings that we took away from this migration, the first one, switching from percentiles to histograms is tough. This was not only was this a client side change, where we had to sort of teach engineers about buckets and which buckets to select, the appropriate number of buckets, things like that. But we also had to teach them about how to actually query it through PromQL. And also, obviously along the way, we wanted to teach them about some of the advantages. So now we could, now engineering teams could get accurate aggregations across instances. Next, the instance label is a important concern for overall volume. As I mentioned before, the instance label is one of those, is one of those super high cardinality labels that unfortunately gets added to every single Prometheus metric. And this is to keep track of the unique metrics for the cumulative counters and histograms. Unfortunately, with rollup rules, we can pre-aggregate that instance label away and then safely store a lot less cardinality in the time series database. Yeah, one of the important learnings is that Prometheus is now friendly with short-lived jobs. So we use Prometheus aggregation gateway for this part of metrics. And it could be dangerous because it's kind of like push model and that is something that we want to avoid. So we just limited the method to some special cases. The last learning that listed here is that we really need to automate the monitoring onboarding process for teams. So in all that, the steps are like, first we provide the service teams with weak keys which document how to generate metrics using the Prometheus native clients. We give them some examples to follow. Then after the service owner and the service team add the annotations in the Kubernetes manifest which means their metrics, their service and the jobs are ready for a scraping. After that, we provide them with some Grafana dashboards, Grafana templates for them to use and copy from. Also in dollars we provide the alert tools for the service teams to use. They can use that to generate the alerts automatically. And we provide the alert template for some common alerts. For example, like the Kubernetes container level house. We provide the alerts to monitor like the Kubernetes container restart count CPU and memory utilizations. And also these metrics, yeah. Awesome, so yeah, thanks for coming to our talk. I think we'll take a few questions now. And if you wanna talk to myself or Emma or anyone from the Chronosphere team, we have a booth here at G15. I think we're giving away some pretty cool prizes today. We also created a video game just for this conference. So you should stop by and see if you can see if you can beat our high score. Yeah, so my question is that you mentioned there are 2,200 dashboards. Why you maintain that many dashboards? Because you think that you have to standardize or have similar set of dashboards? Yeah, so I mean, I think a lot of those, I mean, I think the original number of, I think it was like 1,500. I think a lot of those were probably still just kicking around. And a lot of the Prometheus-based ones we probably don't have as many, which obviously you can tell it's probably about, 600, 700 or so.  I'm not sure exactly how many engineers there are, but I would guess probably in like the thousand-ish.  But yeah, I don't know if Emma has any other comments about that. Yeah, I think we have lots of dashboards there, and yeah, in different buckets, and for each team, like we have some like, Kubernetes container-level metrics for them to use, but they also keep their own copy, yeah. Yeah, hi, for someone who's just starting on this migration journey, what would be some mistakes that you would recommend that we can avoid? I saw the learnings, but would love some more detail there. It's a good question. I would say, I mean, I think the biggest thing that, and Emma could probably talk a little bit more from the DoorDash side, but I think the biggest thing that we found to be very helpful is really working with one or two, sort of like, you know, pilot teams, to sort of, you know, kind of get that model out there. You know, especially around, I would say, like the use of the Prometheus clients. I think having, you know, having a couple services, right, that are using each of the different, like, Prometheus clients, right, like maybe have one Go service, one, you know, Node service or something, and sort of, you know, getting them sort of fully across the line at the beginning, I think that can really help, because that, you sort of learn, right, a lot, you know, in that journey, and then other teams can sort of, you know, tack on and follow that. Yeah, so from my perspective, I feel like we really need to set the common text as soon as possible. Yeah, it's difficult to change, like if you have other set of common labels. For example, like we have the cost, oh, we have a cost origin that we really want to get rid of, because we want to use a service app, these two combinations to find a service uniquely, yeah. Yeah, so basically you provide the whole standard, kind of like templates and everything, but it's actually just for the first generation, right? So like, if the goal standard changes, how do you like ensure that all the teams actually follow the updated goal standard that you've provided? For example, like let's say that the cost of origin is like a new tag. How can I make sure that all the team actually gonna make it inside the tag as well? Oh, actually for example, like the cost origin label, they are generated from our side automatically. So in the metrics collector, we have like some shell script to get like AWS and AWS region and zone ID and also the service name automatically set for them. So they are not able to change that and we publish all these labels for them automatically, yeah. Yeah, I think that can avoid some mistakes, especially some spelling mistakes from the service team. Yeah, I think what we found and honestly to this gentleman's point, I think having a central observability team sort of take lead in this migration effort was super beneficial, because they sort of have a view right across the company and they can set those standards in place early on. And so obviously things are gonna change right in the future but the more that you can eliminate the unknown at the beginning, I think the better. So I think in general, you really wanna look at what your scrape interval is. I think that's like a pretty good standard to look at. So if your scrape interval is 30 seconds, having jobs under 30 seconds, probably not gonna work because those jobs will complete before Prometheus or the collectors can actually scrape them. So I think it, I mean, yeah, it really depends but for me if I'm just gonna say like a blanket statement, I would say like the scrape interval is probably a good sort of boundary. No worries, I heard. Yeah, the question was, do we have plans to correlate metrics, time series data with traces, specifically around open telemetry? So at Chronosphere, we actually just, or last year we launched a tracing product where we do just this. Yeah, I don't wanna say like, I can't really talk about DoorDash, but yes, we do have plans and we do have customers actively using that. And using it to correlate, right, between metrics and traces. Thank you. Thank you.


Hey guys, we are here with another episode of Artie Trio. This is our mood board for the month, mood board number 17. Can you believe it's 17? We've been doing this over a year. Wow. All right. Anyway, Bea did this one and I really find the combination of sort of gray-ish, green-y, neutral tones and some blues and yellows are really interesting. As usual, of course, because I do consider myself primarily a painter, I picked out paint colors and then like the rest of you, I'm sure you have a bin of scrap paper and collage things because we do do mixed media that seem to just never get put away. So despite the fact that I have a growing bits bank full of scraps and teeny tiny bits and paper and I went to the basket and pulled things out of the basket. They need to be put away or used up anyway. So I thought we'd do a combination of paint, maybe some ink, papers.  So I don't know where we're going. As usual, I have absolutely no plan. We are going to start off with gluing this over here, so I have it handy and I don't lose it. Then we're going to get started. Doing something I think on this black background and using the black as one of the paper as one of our colors. So let me fast forward through my process and we'll see where we get and I'll be back. Okay, as usual, I made my notes on this side. I had no plan as I told you at the beginning of the video. I thought this was going to go somewhere else and it didn't and this is what came out. Is it my best work? But I do like what happened. So it's pretty interesting and it was fun to play with the paint and the collage and the pens and the ink and those things I don't get out nearly often enough. Anyway, I'd love to see you experiment with this month's mood board and I'd love to see the unexpected places that it takes you. If you decide to do that, I would love to have you tag me in the post or the video. The best place to do that is in the Arte Trio Facebook group. The link for which is in the video description along with the other two videos from my other two teachers, Mike Deacon and Bea Grob. And let's see what they did with it this month. I'm sure it looks nothing like mine. It usually doesn't. My videos with the mood boards usually go off into left field somewhere. This one did too. I mean, again, I like it, but I didn't. Yeah. Not sure where she came from, but okay. Alright, that's it for this week. I hope that you guys get something interesting out of this that it gives you the freedom to experiment and to just go with it. And if it works, it works. And if it doesn't, it doesn't just close the book and put it on the shelf and you're good to go onto the next one. That's it for right now. Leave any questions, comments, or concerns down below. If you do create something and you do just decide to take my advice and go with the flow and see where it takes you. Again, I'd love to have you tag me in the post over in the Arte Trio Facebook group. If you make a video on it, do let me know that you do that. I will go watch it. And the most important thing, go out and have a great day. Do something nice for yourself because you deserve it and I'll see you later. Bye guys.