Hello and welcome. My name is Shannon Kemp and I'm the Chief Digital Officer of Data Diversity. We would like to thank you for joining this Data Diversity webinar. I'll have trust issues with my data. Can I really change that? Sponsored today by Precisely. Just a couple of points to get us started. Due to the large number of people that attend these sessions, you will be muted during the webinar. For questions, we'd be collecting via the Q&A panel. And if you'd like to chat with us or with each other, we certainly encourage you to do so. And just to note, Zoom defaults the chat to send to just the panelists, but you may absolutely change it to network with everyone. To find the Q&A or the chat panels, you can click those icons found in the bottom of your screen for those features. And as always, we will send a follow-up email within two business days containing links to the slides, the recording of this session, and any additional information requested throughout the webinar. Now let me introduce to you our speakers for today, Julie Skeen and Paul Rasmussen. As a senior product marketing manager with Precisely, she has over 25 years of experience working on solutions for customers in data intensive industries. She focuses on understanding customer needs and ensuring Precisely's data quality and data observability solutions are aligned with those needs. As a product manager at Precisely, Paul is passionate about helping companies tackle their data challenges by focusing on data integrity, analytics, governance, and data ops. After receiving his Bachelor of Management Information Systems at the University of Texas in San Antonio, he accumulated extensive years of experience in the engineering, consulting, and product management industries, leading to his strong technical background and business acumen. And with that, I'll give the floor to Julie and Paul to get today's webinar started. Hello and welcome. Thank you, Shannon, I appreciate it. Before we start, let me take a moment to introduce Precisely. We are a leading vendor in the data integrity space. We have decades of experience in data quality, helping some of the world's largest companies get the most from their data. We have 12,000 customers in 100 countries worldwide, serving 99 of the Fortune 100. We work with customers from a variety of industries from telecommunications insurance and financial services to retailers, manufacturers, and many others. We partner with leading technology companies to jointly solve challenges for our customers. So we've seen organizations working to implement various initiatives to meet today's overwhelming business challenges, while competing in an uncertain economy and complying with complex regulations. So one of those initiatives include risk management, understanding your customer better, finding ways to leverage AI and ML, companies are looking for ways to leverage their data. It's likely no surprise that 83% of CEOs want their organizations to be more data driven. So investors recognize the need to leverage data in ways that allow them to make better decisions and drive more efficient processes. CEOs worldwide are pushing their organizations to be more data driven to ensure the success of these critical projects. Knowing that doing so is only possible with high quality trusted data. So there's a gap between what businesses want to do with their data and what practitioners within the organizations feel is possible. Less than a third of practitioners believe they are leveraging their data to drive their actions. Even fewer practitioners believe they can completely trust their data. So organizations who are closest to the data feel that they can't trust it for various reasons. For instance, they can't get it fast enough. It's trapped in complex legacy systems, not available when and where it's needed. And it's not as fresh as the business demands. They don't understand it. They don't understand the lineage of the data. They don't know how it's used by the business. And they don't have accountability around data changes. Hey, Julie, that don't understand it. This is what's really interesting is I've heard this from a lot of customers lately. It was kind of really stuck out of me when they say this because it's something you assume that everybody understands the data. But in one example, I had a customer add a new third party partner data feed and they introduced, it was a currency code. But the data engineer that was building the analytics on top of that didn't understand that there was a new currency code introduced and that currency code really had some downstream impacts that could have been avoided had the data just been understood. So it's just really interesting how don't understand it is really a modern problem. Well, thanks for that color commentary there. I appreciate that. So just to finish out this slide. Other ways that teams don't trust their data are when it's full of errors or non standardize. Or maybe they don't have the context to use it. They might be lacking additional internal or external data that could provide the context needed for decision making. Also, sometimes they don't know when it's going to break and downtime comes as a surprise. Anomalies unexpectedly impact the business downstream. And the true root cause of data maybe is never even identified. If you're experiencing any of these situations, you're not alone. As Paul said, every day we talk to companies who are struggling to meet their data driven business goals because of issues with their data. They might be dealing with problems of how to fit operational data into analytical use cases or managing desperate data sets to get a better view of where the best data lies and struggling to understand where the data is coming from that drives critical decisions. There's even some famous technical organizations that have struggled with this. And a couple of just very public examples were NASA in the late 90s, they took a 125 million dollar hit when they lost the Mars orbiter. And it turned out the engineering team there was responsible for developing the orbiter use English units of measurement while NASA use the metric system. That's definitely an extreme case of not understanding the context or source of the information. Another example that was very public was when PayPal had to agree to pay the US government $7.7 million when it was found that nearly 500 PayPal transactions worth almost $44,000 violated sanctions that ban US companies from doing business with individuals or organizations on a blacklist. So for those of you are thinking well I'm glad we're not trying to put a straight spacecraft on Mars. The cost of dealing with these types of trust issues can still be quite substantial and take on many forms beyond the direct costs associated with fines or destroyed spacecraft. There's the obvious direct cost of being PayPal having to pay $7.7 million of the government over a compliance issue but there are other types of costs that you need to take into effect and take into account when you think the nature of trust and how you come to trust your data. There's a cost of preventing. For example, quality costs can be understood in terms of both avoidable and unavoidable. So cost to prevent includes the cost of building traditional defensive measures into software and analytics products through QA requirements development exception cases training etc. There's a cost of finding issues once you've discovered that you have a problem and endless search for trying to understand where the problem is coming from and why it occurred this time can be very time consuming. There's a cost of failure when those issues aren't addressed, and they do get into the final product. And they start to undermine the confidence in the process that you're setting up, or in the reports and dashboards you're putting together to help drive the internal decision making. So, with this whole is probably a good time for us to think about what we mean about trusted data. Paul, maybe you can chime in on this. What do we mean when we talk about trust in data? Sure. So, let's talk about the path that trusted data first. So, you've got trusted data and that what we've seen the typical progression really in the market and other customers goes like this. And we're losing the pyramid to kind of reflect this at the bottom of the pyramid, we have the collected data. And we have a lot of it. You know, we all know data and related products have enormous utility and it's been a gold rush to collect that data. We've seen in the companies grabbing as much as they can with intent to figure it out later. You know, there's the whole phrase, there's the golden, then there are hills and data is the same way. But this also assumes the collection of this data is timely and that they're appropriate aggregate and granular levels. And really, the game changer on this data collection a few years ago was the really the mainstreaming of new technologies such as the Internet of Things, commonly referred to as IoT, along with instrumentation and digitalization efforts. And really, especially with the rise of the popularity of temporal data, you know, data representing data states over time. And next on next on the pyramid is quality data. This layer represents data that is really usable, reliable, it's based on consistent and objectively measured criteria. Here, here we're making the data trustworthy for trying to avoid the scenario where user finds, you know, a single issue in a data source like a data warehouse, and that assumes the entire data source is untrustworthy. You'll also notice that the size of the level is getting smaller, representing less data meeting this criteria. On the third layer is discoverable data. Discoverable data is data that is coherent based on various degrees of data literacy documentation and user education. It's really the enablement of users in your organization to know what exists, the credibility, quality, context and the status of that data. And lastly is the accessible data. And it's really the goal, getting the right data to the right users. It represents the ease in which users can obtain usable data products, metadata and documentation, a secure and cost efficient manner that meets compliance, confidentiality and regulatory standards. And what's really interesting to me about this chart is very similar to one we've seen and used over the years with respect to critical data assets. Essentially, it's the same Pareto principle applied to data, whereas 80% of the issues come from 20% of the data, and it posits that the most productive use of resources is to make the most used data a priority. To give you a real world example of this, for a mid-sized company may have 20 columns in its data catalog. So those 20,000 columns, only only 400 of them are typically used to drive the business. That means only 2 to 4% of the data is being used by your organization. So, Julie, what do you need to make this happen? Alright, so to achieve trusted data, you need data integrity. Data integrity is data with maximum accuracy, consistency and context for confident business decision making. This requires a holistic view of your data. To dig into this a bit, having consistency in the data means that people can reliably trust the type of data every time they access it. Accuracy means once you have the data, does it match what you'd expect? And there's no data missing, improperly formatted or otherwise erroneous. Context means you understand how that data relates to other data and how it applies in the given specific business scenario you're trying to use it in. Data integrity brings these three dimensions together and applies them equally across the data being used for that specific purpose. Everyone is on a journey to continuously improve the integrity of their data, better understand their business, and ultimately better serve their customers. We've learned from our customers that there isn't a standard linear journey to data integrity that works for everyone. And that the days of large corporate initiatives are a thing of the past. Customers have told us that their business and IT teams are working more closely together than ever, jointly identifying the specific scope that delivers meaningful business impact. And as a result, they tackle data integrity through distinct projects that give them business value, no matter where those steps fit into the journey, and then plan their next move. We've seen that organizational needs are changing. It used to be that organizations had data management projects that were led by IT teams and focused primarily on responding to issues. They would target operational use cases and look to improve efficiency and effectiveness. They typically address data that resided on premise at the organization. Now we're seeing proactive data engineering with data engineers embedding data quality within data pipelines. There's a huge focus on analytics, including artificial intelligence and machine learning use cases. In addition, companies are now migrating and centralizing data in the cloud using cloud data providers such as Snowflake and Databricks. These organizational trends have a significant impact on how the data quality needs of the business are addressed. We see additional involvement from the business teams, not just as participants but jointly leading the initiatives. Business teams need to ensure that data can be trusted. And they are now an active participant in the resolution process. These processes can now become more automated with the deployment methods and with built in intelligence. This intelligence can be used to guide users in building the appropriate rules to perform sophisticated matching processes and by monitoring the data. This observation process can provide proactive alerts to users about potential issues so that the issues can be investigated and resolved before any decisions have been made based on that data. All of this can provide a more seamless user experience by leveraging semantics and metadata as part of the process. In addition, because companies are migrating and centralizing data in the cloud, they want to ensure data quality validations occur where the data resides without taking it out of the centralized location in order to apply data quality rules. This requires data quality processes that can run natively where the data sits. So now we're going to move on to talk about some of the common themes that we see in successful data quality programs. Paul, you want to go ahead and start off. At the top of this list, leadership via is incredibly crucial that management understands the importance of data quality. And you do this by getting them to understand the costs, speaking their terms, costs, both known and hidden, which we've already talked about. And once you gain agreement on the data quality, the data quality is real and a costly problem, identify specific real world examples and remediations. Next we have institutionalizing data quality and the associated mindset. It's another common theme we see in successful implementations. We see a mindset that everyone in the organization can contribute to the quality of the data. Everybody can be able to view and understand the data. It's lineage, it's overall status and even contribute to fixing or correcting the issues with the data. That's another hallmark of organizations that are successful in implementing these data quality solutions. Next on the list, metrics and measurements. Everybody has varying degrees of what we consider good data. You need to come to a common understanding of what the definition is of good data and what fixed data would look like, preferably at a quantitative level. And the key at this juncture is really get everyone actively involved in the process of identifying truly problematic issues and making that entire process a positive experience. Really about taking the next one. Yep, for systems and automation, more and more people are relying on automation to take complex problems, audit and execute on those on a regular basis. Automation becomes key, but also the ability to build decision making into those automation processes to help alleviate some of the tasks from manual intervention. Investing in tools that can monitor your data will create automated alerts, can raise alerts that can be proactively resolved. And last on the list, openness and interoperability. Openness and interoperability is a mindset and it's really an absolute necessity in the modern world. I like how one of my customers praised it the other day, they called it liberating the data. As a Conan point, as a reference point, let's try to recall what it used to look like when you wanted to use a new application a few years ago. You were dependent on an operating system version, arcane and manual deployment methods and infrequent release cycles. Now with that advent of cloud and software as a service, applications are simply immediately usable. So let's take an example, an email application. You know, you just, if you want to use one, you just pick one that works from you, works for you on the cloud, start using it. The same applies for data. If you want a data set, find a trusted one that works for you and simply use it. So to put simply, really it's to find early and automate, standardize, measure, reuse and educate. So let's look at each one of those in a little bit more detail. So the first one, find early and automate. I'm really basing this on a fundamental tenant that states good data increases customer satisfaction and improves profitability. So in other words, your data serves one of these three purposes. It increases or decreases profitability, improves competitiveness or enables regulatory compliance. So let's take for example, let's assume your customer finds an issue related to your data. This is going to be your most expensive outcome. The next scenario is that you find the issue before your customer, which is good. This is definitely going to be cheaper than your customer finding the issue, but still pretty expensive. And lastly, the least expensive option, your system prevents the issue from occurring in the first place. That's where you want to be. And this goes beyond just systems. It's a paradigm shift so that your entire organization is educated and empowered to improve your data. So what does this look like in practice? Let's take a look at what typically happens at most organizations in their data products delivery process. You've probably seen this in your organization. You have a data supplier. This is the source of the data on the far left. It could be a purchase data set, an internally sourced data set, such as a customer list, marketing data, invoice, or an order, or even an order. So what happens is it starts off as a single data set, has numerous processes performed on it across the organization. For example, it might be transformed, enriched and published to the cloud. And this is just a simple process in the real world. What starts off as just one data set may end up being copied and modified numerous times with thousands of transformations. Essentially, it mushrooms across the organization. And then some other process takes this data and consumes it. And then a de-factor issue is found in the data. Eventually, that issue is reported to IT. It gets prioritized, assigned, analyzed, fixed, tested, deployed. And eventually, maybe days, weeks, or potentially months that fix gets remedied in the source system. And to make matters worse, data in most companies is not single use. This process repeats itself after every iteration of new data. So now let's look at an alternative model. In this model, we're simply implementing time-tested improving data quality improvement methods. And you find these same techniques in almost every quality management technology or methodology. For example, there's TQM, which describes a management approach to long-term success through customer satisfaction, as well as a host of others, SBC, SIGBA, and Laney. And a basic premise of all of those techniques is for a manufacturer product or developer product, is each step is monitored using statistical methods and rules. In this example, there's an issue in process step number three. If you cut the issue prior to step number four, intermediate it, you've effectively, proactively prevented the issue in the final consumed data product. And so why isn't this being done for data? It's because it's traditionally been time-consuming and expensive for complex data products. The good news is technology has caught up for data. Let's look at the next one, standardizing and measuring. And that's what the approach in the prior slide, standardization and measuring, is an important part of your data quality initiative. We've seen customers have great success using rules and scores to accomplish those. So scores essentially define what fixed data would look like. Some examples are a quality score. And so when you look at a quality score, you're looking at how good is my data? Which data is failing the criteria? Some examples would be, I used to feel null, blank, has a semantic been identified? Do the records have outliers? Is there a corresponding business logic applied? And so those are really categories you'll see traditional. When you hear data quality, you'll hear people talk about it as the dimensions, accuracy, completeness, consistency. You know, there's lots of variations on those or individual organizations use their own. The second are the governance scores. And these answers the question really, how well am I governing my data? Stewarding and governing my data? Questions like, is there a business owner assigned? Is the data steward assigned? Does the asset have a descriptions populated? Is the status certified? Those are really trust scores. Can I trust my data? Those examples would be, how many data sets are being used that consume this data? Is there a rating system across the organization? Is the governance score high? The governance score, but my quality is low. The key at this juncture is to get everyone actively involved in the process of identifying truly problematic issues and making that a positive experience. To kind of give you an example of those, I'm going to show you a system for storing those and displaying them. So we are seeing a system here. This is a data catalog. We've got the individual assets. You can see there is a data set called customer cleansed in the sales schema. We'll have a governance score and a quality score. So looking at the quality score, we're going to see individual rules on the far left. We can see if the emails parse, the phone numbers parse for the dates standardized a score and then the history of that score. So for example, we can see the highs and the lows. Are we progressing? Are we getting better with the data? Are we getting lower? Are there alerts on the data? What's the typical patterns been of my data quality? From here, we can see how that was calculated. So everyone has the same understanding of how those scores are consistently applied and the definition of the score. So that's just a quick example of the repository or capturing data quality metrics and showing trends over time. All right, Paul, thank you. Appreciate that. As you can see, having a clear view into the quality of your data can help drive the conversation around how to best address quality issues. The foundational view of the data via a catalog or governance system is a great kickoff point by providing a common place to understand sources, context and lineage and other factors that can help you better articulate what issues exist and the extent of those issues. It's also a great place to help level set how the business defines quality and prioritize what actions should be taken first. Again, the idea isn't to tackle everything all at once, but to focus on business needs and objectives to drive an ongoing process of improvement. Bit by bit, step by step. As you can see, there's an overlap where in order to get a clear view into the trustworthiness of your data, you need to look at it from the perspective of both governance and quality. And cataloging gives you a foundational view of your data and becomes a great starting point by providing a common place to understand sources, context and lineage and other factors that can help. You make better decisions on top of that data and better articulate what issues might exist and the extent of those issues. Data governance provides a place to help level set how the business defines quality and prioritize what actions need to take place first. The idea isn't to tackle everything all at once, but to focus on the business needs and objectives to drive an ongoing process improvement. Working with customers, we have found a lot of different definitions of data quality and the associated capabilities. This can include profiling row management and data validation and the types of scoring that Paul discussed between quality and data governance. Another important aspect of data quality is the complex task of ensuring contact and address data is valid, including geocoding addresses and adding enrichment data to provide context. This can be done in batch, real time, or as part of a manual entry into a CRM or ERP system. Data quality can also include making sure that data is consistent across systems and not duplicated, enabling a single view of critical business data. In our increasingly real time world, we also need to ensure that we are considering data quality as data moves between systems and utilize machine learning and AI to quickly identify anomalies and outliers in the data. Paul? The effect of data quality on AI. We all know AI is out there. That's here to stay. Plenty of statistics out there. 67% of companies indicated that they are integrating gen AI into the overall AI strategy. That was in October 23. By 2024, by the end of it, 60% of professionals will use their own AI as part of their duties. This is the top of the already existing AI in the organizations. They know their existing traditional AI functions that are used for categorizations and predictions. They're really across the three different themes where the effects is, one, AI is changing the business. This can be partially attributed to the massive improvements in AI's accuracy. The more accurate AI is now compared to just as recent as three to five years ago. And it's both cyclical and exponential in growth. The better the data is trained on, the more accurate the predictions result in the more AI usage and more data. Another effect is the cloud. The cloud is a serious enabler of AI, especially as cloud becomes omnipresent in all of our personal business lives. Again, it's a cyclical relationship. The more data on the cloud, the more data for AI to learn from, the more AI usage. And finally, companies will need to catch up with data quality and integrity as AI governance becomes a requirement. We're at the early stages, but we're already starting to see inquiries about this. odds are your organization has already sent you a policy on your AI usage. So let's take a couple of examples here. First one. The data quality. High data quality saves data scientists time and improves ML accuracy. One of our resident data scientists has a great phrase that I like to repeat. ML models are an abstraction of the data they are trained on. So the quality of a model is reflected in the data it's trained on.  At this point, the science and technologies for data analysis are already quite powerful and well proven. But how much time and money is being wasted? The answer is a lot. And that's because 80% of a data scientist's time is really cleaning up that data, collecting it, and transforming it. And, you know, an estimated 54 to 90% of machine learning models don't make it into production from the initial pilots. You know, that's a lot of waste there. So let's look at example number two. Example number two would be ML ops. Essentially, data changes over time. So do your models. You've probably heard the term ML ops and data observability. If you haven't, ML ops are the people, process, and technologies involved with the continuous delivery of AI in your organization. And data observability is the process of monitoring your data for anomalies based on historical patterns. So one of the key components of ML ops is this concept of freshness, and how data freshness impacts the relevance of your model predictions over time.  They don't capture the trends that are present in recent data. In other words, you know, ML models tend to expire. For many cases, the old training data can even add noise when used with models with new data. A couple of examples there for freshness. Take for example, forest fires. ML models which use forest fire data from 20 to 30 years ago, you know, may not capture the patterns which are emerging with the present climate. And for data drift. So take for example, you've got a sales report that's using predictions based on an old product list that doesn't have new products listed. So as you can see, as your data changes, so does the quality of that data. Thanks, Paul. Appreciate that. So as we start to wrap up, let me talk about an example use case of how trust and data impacts your business. Does this use case sound familiar? I need to drive our new marketing campaign based on lists I get from outside vendors and internal sources. But how do I know if the lists are any good? What could be causing trust issues here? How do I judge the quality of the data? Could you establish quality checks that need to be run before the data is onboarded? Do you need to monitor for regular updates or freshness of the data? Do you need to establish measures for outside data providers to keep track of their performance over time? These are all things that might help address those trust issues. Am I getting a new data or the entire data done? In many cases, you don't always get just what changed from version to version of the data. You must still comb through the data to determine what was old information, what I already knew, and what has changed since the last time I saw this data set. So being able to match against and create a process that reconciles and deduplicates that data against your known context is a critical transformation step. Also brings up the trust in the data. Is information missing? Am I getting bad addresses? That's where contact information quality becomes key, particularly for high cost marketing campaigns like using mailers where address quality can have a big impact on the overall cost of your campaign. Is it fresh and accurate? Is also an area where you can build some measures around improving the trust. And then there's IT issues, monitoring for continual updates, ensuring that I'm always getting the freshest information once I start to automate these processes. Am I continuing to see the quality that I expected when I initially set up the process? Having the ability to observe those pipelines, to intelligently look for anomalies and changes that might alert you to potential problems, become something else you can do to improve the overall trust that you have in your data. As we discussed earlier, everyone is on a journey to continuously improve the integrity of their data, better understand their business, and ultimately better serve their customers. And as a result, they tackle data integrity through distinct projects that give them business value, no matter where those steps fit into the journey. And then plan their next move. And not surprisingly, that means they want solutions that give them the freedom to make those choices. Data integrity is a journey. It's continuous. And it requires best in class solutions working together to deliver value to the business. That's why precisely offers the modular, interoperable, precisely data integrity suite, which contains everything you need to deliver accurate, consistent contextual data to your business, wherever and whenever it's needed. This enables you to start wherever you are on your data integrity journey. This is a suite of services that are SaaS based and built around a common catalog to help you better understand your data. And from there, identify issues, understand the overall quality of the data, define business rules and scoring metrics against which you can better monitor and understand that data, and build remediation processes to improve data accuracy. It also adds context with geo-addressing and spatial analytics, and even taps into a library of data enrichment information to add additional outside context for the data that you have within your organization. All this is built on a foundation of intelligence and a common data catalog with agents to enable use with data both in the cloud and on-prem, addressing a wide variety of data management needs with one integrated solution. And with that, I'm going to turn it back over to Shannon. Julie and Paul, thank you so much for this great presentation and demonstration. If you have questions for them, feel free to put them in the Q&A section and just to answer the most commonly asked questions. Just a reminder, I will send a follow-up email by end of day Thursday for this webinar with links to the slides, links to the recording, and anything else requested throughout. So lots of questions coming in here. So diving in, can you please provide the name date of the IDC report you cited on early on in slides four and five? Yeah, I would need to look that back up to find that, but I can provide that to you, Shannon, if you want to. That'll be great. Yeah, and I can send that up in the follow-up email. And can you expect quality data when there is no data integrity and force between disparate datasets? I'll take that one. You can expect it, but you'll probably be let down. The, you know, really data integrity between disparate datasets, that's a definition of disparate datasets, right? Is a disparate dataset because it was from a different data source or is it because of the lineage and the parentage where it started off from the same original dataset, but it got transformed so many times before it got to the consumed dataset. I think you can't expect if you're not doing any data integrity or quality, right? If you're letting everybody run their own rules and do their own transformations and then expect them to be consistent, you're probably going to be disappointed. Really, the key is to get those consistent rules applied. Now, you may have two disparate datasets that have the same rules applied to them. You know, common validations, reference list checks, validity checks, and then you can absolutely expect the quality if you're running those same rules on those disparate datasets. But if you're not doing that, then yeah, it would be very difficult to expect high quality data if you're not, you know, doing any quality on them. Thank you. So many great questions coming in. So how are the data quality scores calculated? Does the application calculate or is it a direct feed from another source? Several different methods. You know, in our product, we support the feeding them in from other sources or calculating them. And how they're calculated is really going to vary based on what type of score. So when we showed you DQ scores, you know, you have a couple of different ways. You can introspect each record and say, you know, does it have a pass-fail criteria? And then at the end, you may say, 80% passed, 20% failed, or you may use some higher level metrics like, okay, is this completeness rule? 30% of the records were complete. So then you're kind of using an aggregate score based on those as well. And now for scores, for example, data governance score, you're not actually looking at the data. You're looking at metadata about the data. So as a business owner been assigned, well, that's a binary. So you can't necessarily say yes or no if it's yes. And so that's a metadata about it and it's calculated directly. So the answer to your question is lots of different ways that it's calculated depending on how you choose to implement it. Do you have any advice? How can I build a daily process cycle to find data quality issues and how to know where to search and test? So to build a daily process cycle, it really is using a tool out there that's purpose-built for doing that. It's going to find your data quality issues, both known and unknown. And then to do the kind of nowhere to search and test data catalog, data catalog and governance is really the way modern users and organizations are communicating those data literacy use cases where they're trying to enlighten the organizations about what data is out there. You know, socialization, who's using the data, ratings, comments on that data, what rules are being applied. And those are things that you'll see in a data catalog or a governance product. Perfect. And is it a good practice to monitor data quality on the data sources or just as part of the pipeline process? Lots of opinions out there, but really the industry has moved to that moving away from the process. We used to see ETL or enterprise or extract, transform and load. And so that transformation and quality was being done at that middle stage. And that for a big part of organizations has shifted to ELT, especially for cloud data sources, data warehouses, where the data really is extracted, it's loaded into the target system and then users and actually are running those transformations after it's been loaded versus doing it within the process. And then one of the downsides of doing it on the process is what if somebody uses that data before it's hit the pipeline process. Now all of a sudden, they're gonna get the same bad data because they were unaware that rules were run in that pipeline process. So they're not capturing those same rules. But I would say either in the source or in the target is preference. And Paul, there's probably one more dimension to that, which is if you're getting external data, right, then you would want to make sure that you're checking the quality of that before you add it into your own data source. Absolutely. Now what you see mostly on the pipeline process is its transformations. And sometimes those are more difficult to do on the source of the target. Sometimes you do those transformations as part of the pipeline process. Thank you. So we have a Q&A process that requires manual audits because we have to review documents to compare them with the data that has been entered into the system. Do you have any suggestions on how to automate that process? So that, okay. So I think what they're saying is right, they're manually entering it. I mean, one thing you can do is have like a front-end validation. So for example, I'd mentioned front-end validations that can occur on like CRM and ERP systems. So that as someone's entering it, you're also making sure that that quality of that data is high quality and they're not entering things that you know right off the bat can't be correct like an invalid address or things like that. Paul, do you want to add to that? Yeah, I think the key to that question and I need a little bit more detail is the document. So when I hear documents, I'm assuming you're referring to unstructured data. And that's where we're seeing a lot of advancement in data quality. So you can have, you know, you have your answers or your reference data or those baseline metrics and then you're able to use AI to read that unstructured data and answer questions about it. So without knowing more about the data sources, that's kind of my interpretation is you're looking at how to parse that unstructured data in the documents to answer questions against a known baseline. AI is kind of a long way in doing that. Indeed. So is there a breakdown of cost of poor quality by industry, say retail, banking power, et cetera? Absolutely. So if you're in those industries, you're probably on some of those publications. We, precisely, we've got a strategic services organization that focuses on industry data quality within industries, happy to engage and talk to some of those folks, briefly to reach out to us and we can get you pointed in the right direction for those, but absolutely there. I've seen them in the past. I can't tell you off the top of my head which publications, but they're definitely out there. And going back to the topic of AI, you know, so using AI for data quality, what about AI hallucination or even AI bias? Absolutely. Absolutely. Absolutely. Absolutely. Illucinations, are you seeing things in the data that don't exist or bias, but that's where we're talking about the temporal data, the freshness, you know, it's key that you have fresh data that's up to date and of high quality. That's what's kind of the points of the whole AI for data quality, essentially that without high quality data, your AI is erroneous and going to lead you down past that are incorrect. Thank you. And we've got a few minutes left. So, you know, I'll keep the questions coming in as many as I can hear. So if your data quality scorecard alerts you of a problem, where is the best place to do corrective actions in the source pipeline? I think we already talked about that one, didn't we? The scorecard alerts. That's the same. The corrective act, I think the corrective actions was asked before, but essentially the source, the source is generally the best place, or if you don't have access to the source, you know, typically you see those transformations done in a pipeline if you don't have access to a source. So a lot of depends on that one, but as a general rule, fixing the data at the source is going to be the best place. Think about a manufacturing process where you're building bicycles. If you're building bicycles and you have a defect on the welds for the frames, you know, if you find that defect at the time you've shipped the bicycles, that's going to be incredibly expensive to remediate versus if you discovered that issue, that defect in the manufacturing process at the very beginning at the source, you would alleviate all of those costs associated with remediating that and preventing it. Thank you. Yeah, it's a slightly different angle on the question there, but you know, I noted that the scores were weighted, quote unquote weighted. Could you explain more about that and maybe why certain decisions were made at different weightings? Sure, so weights are typically very subjective based on your organization. You know, there's some weights that are more out of the box.  That's your confidence, right? Your 98% confidence. It's an email address. And then the aggregations, right? You can aggregate at a, or you can weight at an aggregate level. It's really subjective about how your organization and the data, what that data is and how you want to score it. Scores, scores for their, vary between, you know, how good is the data? You know, is it 70% good? Is it 100% good? And then very binary, right? That'd be value X. So it's really going to vary on what data you're looking at and what rules and how your organization is improving that quality. Very nice. And do you see weighted score by dimension for aggregate scores? For example, certain dimensions impacting aggregated scores more than others? Absolutely. Yeah, I mean, some notions of validity may be more important than completeness. Maybe you need to validate. It's a valid address is much more important than if you have an address. It might be for completeness. So definitely weighted by dimension is important. Nice. And that is all the questions in the queue for right now. I'm just going to give it a couple of minutes here. But Julie and Paul. Just so you know, I'm sorry. There's one question in the chat looks like too. Oh, so the glossary. So how do you see integrating or leveraging a business term glossary in data quality? Yeah, so from that perspective, we see that data catalog as the foundation. And so you're cataloging everything you're doing quality on. And then that catalog is also integrated with the business glossary. And we see that it's very important, right?  You know, for example, a valid address, a valid shipping address is different than a valid marketing address. And it has to be 100%. What has to be, you know, anywhere from zero to 99%. So adding that context of a business glossary is gaining importance and become very important for data quality. That's where we talk about the context. Where is this data being used? How is it being used? Answering those questions. Perfect. And thank you both so much. Thank you for watching that. I am in the chat. So that's a great question. Thanks to everybody. And as you can see, there's a great QR code up there for you to learn more. I love it. Also, precisely be presenting on their data quality products in our April demo day, specifically focused on data quality. So look forward to that as well. And just a reminder, again, I will send a follow-up email by end of day Thursday for this webinar with links to slides, links to recording and anything else. Thank you both so much. Thanks, everybody. Thank you. Thanks, Julie. Paul, hope you have a great day. Thanks to all our attendees. Hope you, likewise, have a great day. Thanks, everyone. Bye-bye.


Hello hello can you hear me can you see me just turned off my fan so we'll wait a little while if you're watching this after the fact check the description below there should be at least a few time codes there to help get you through the video um otherwise we'll sit tight we're waiting kind of for all of the pipes to get the video through and we'll start saying hi to folks as they pop into the chat oh i got a whole new cat cam and it doesn't work well i don't think it's the camera's fault i think it's um i don't know hi biata hi samuel thanks for hopping stopping by hopping by stopping by um yeah so we'll wait just a little while and oh i'll get or get myself in the right discord chat channel hello dcd if you'd like to join uh discord you could go to the url adafru.it-discord i changed my mind about what we're doing i don't know people notice that i changed the titles and stuff it means that the uh the newsletter entry might be a little wrong but that's that's what we get for adding it before we actually do the stream because never quite know what i want to work on i decided that the usb host stuff i really needed to hear back from tack and i didn't didn't overnight so i'll wait for them chat in french is cat hi feda2 hello david in the discords yep so on the left here we've got discord up here and then this is the youtube chat i guess i could put the re-stream chat but it's usually pretty quiet so almost weekend trying to tan the corner mark says good afternoon slash evening everyone hoping the storm gets me and near me gets closer so i can see if my lightning detector works that sounds exciting we rarely have um rarely have lightning here it's like we've got some folks hi gordy welcome welcome welcome dcd hope camping was good it was it was good it's a lot of work we don't camp that often so and it was a new tent so it was and traffic and lots of lots of new new experiences it was fun to see arie the my son the toddler enjoy getting he got dirty within like the first 10 minutes but he enjoyed it which is good and uh i think we'll we'll have worked out some of the kinks uh for next time so yeah that'll be good all right so uh i'll get started here and hopefully we'll get more people as we go although it's august and people are doing stuff um hello everyone my name is scott i work for aida fruit on circuit python um if you don't know what those things are aida fruit is a hobby electronics uh open source hardware company out of new york um where they manufacture some of the boards you see and design them all and then uh also fulfill your orders from aida fruit dot com so if you want to support me uh support them by going to aida fruit dot com um circuit python is a is a re implementation well it's based on micro python which is a re implementation of micro of python for micro controllers micro controllers are little tiny computers such as the one under this lid here on the esp uh good morning unexpected maker um yeah so it's meant to be super low cost uh low power computers that are easily accessible and that hopefully a good way to learn how to program them so uh that's what aida fruit and circuit python is you know i didn't actually look at my checklist but i think besides the fact that i didn't make a note stock um which reminds me uh thank you to dcd for making taking time codes for last week really appreciate that um yeah i'm hello mostafa on linkedin um so i was planning on i in the title originally said that i was going to do usb host stuff and i i do have a a couple issues that i want to file on tiny usb i i tried to bug tack on the like aida fruit slack and uh he didn't get back to me overnight um tack is in vietnam so it's usually a i work during the day and tell him the things that i have questions about and then um and then uh he'll get back to me overnight but he must be busy or doing something so didn't hear from him uh overnight so that's why like changed my mind i was like i don't actually know which direction to go with these things so um um i decided one one task that did get asked about on the stream last week or the week before that uh is something that i've been in so much in the weeds of usb hosts that i feel like doing an idf update so um so the idf is like the stk that uh expressive provides for those chips um and uh we're currently on for something and then they changed the number of things for 5.0 and they've had 5.0 and 5.1 and uh so a community member micro dev who does some really good esp work has uh a pending pr that is a little out of date now um but does update circuit python to work with 5.0 and uh we want to do this update to 5.0 and actually 5.1 which is now out for circuit python 9 because the idea with circuit python 9 is that um we can change a bunch of stuff and then give it some time for people to test it and confirm that you know we haven't broken too many things while we do it uh sorry i don't have a sneeze button although i could mute myself um samuel says i love circuit playground express you guys are amazing i just got four blue fruits great glad it's working um i was just i was at a hardware meetup yesterday and there was a couple folks that were doing circuit python so that was really cool to hear what their projects were and what their experience with circuit python was so it's cool to yeah i'm starting to get out and and all that which is it's fun to chat with people about it i thought maybe you'd be able to see cap nope um oh fed a two has a good question would this new idf make it easier to to support bluetooth on the esp 32 s2 and s3 um unfortunately i don't think so i think the bluetooth stack is still pretty much the same um and it has the same i would assume it still has the same constraint that makes it kind of hard the the the main thing holding up esp bluetooth support is that nimble which is one of the stacks that the idf supports um it doesn't it assumes that you're you create all of the the servers like the services that you present as a bluetooth device that you create those before you get going whereas in circuit python like will create one for the workflow and then later on you may uh want to create them in the user code and uh it's not really designed for that and so there's like some tweaks that would need to be made to it um which makes it a little hard and uh thank you to david for the reminder to talk about circuit python day so circuit python day is a day we do every year where we try to celebrate circuit python um and catney's been putting together a great schedule for next week um it's a week from today uh circuit python day is and so it's basically streams like the whole day i think um i haven't actually looked at it so this stream um the deep dive will actually be uh a little bit later than it normally so instead of starting at two next week we're going to start at two thirty i think um and let's just pull it up on the blog and also it shrines i yeah and unexpected maker says no bilia on this too uh yeah but general like i'm just thinking about the whole family circuit python so there is a new circuit python 823 release that just fixes some protomatter stuff so here's the schedule cp is circuit python magnetron fricando but i said that somewhere near correct yeah and the discord server is uh the url ada fru to it slash discord uh microprython have stopped on id f502 as they have lots of bilie issues on five one at the moment interesting yeah lots things need different things um i don't think bilie issues would would stop us but it that's a good point and maybe we do maybe we do want to update to 501st um and then move to five one later okay so here's the schedule for circuit python day so you can see john park's gonna kick us off at 10 eastern which is seven my time um and then special 3d edition of hangouts so a lot of the shows that happened during the week are getting shuffled to friday next week so this is like 3d hangouts is usually on um wednesdays but now it'll be on uh friday next week uh beeps and boops with synthio which is going to be super fun out with paul cutler is at 11 a.m there's a circuit python day game jam stream with phoma guy who's also a deep diver um so check that out matrix portal message board build with maker melissa and then uh 230 eastern a circuit python day chat with jeff dan and catney hi porky good to see you after a long time hey uh yeah i'm happy to be back um special edition john park's workshop at four eastern which is normally Tuesdays or Thursdays i forget which day john does that um deep dive uh this what you're watching now will be at 5 30 eastern which is 230 pacific and then we'll be followed by show and tell and ask an engineer after that um so just a heads up that we're gonna start a little bit later next week but there's gonna be lots of stuff on on friday for circuit python day okay so we have this open so what i thought we'd do is uh there's kind of like two major things that i want that we want to do so when we're when we're shifting circuit python to another uh kind of like major version number so from eight to nine for example um we want like that's an opportunity for us to to really destabilize things and also change some apis if we want to uh the main reason the main forcing function for going from eight to nine right now is that we are planning on updating well we're working on updating and by we i mean dan dan is working on updating the micro python core uh that we use in circuit python um updating to uh updating to micro python 1.19 which involves an mpy format change so um they don't change that major version number with mpy versions um but we we do in circuit python that's why when you see bundles and libraries and stuff you see like 6x, 5x, 7x like we we build versions of npys for every major version of circuit python uh even if actually even if you uh can actually use them across boundaries we just want to teach people every major version you've got to get the npys that match because sometimes they won't work across versions um hello dylan oh and gordy says jp's product pick of the week is on tuesday and jp's workshop is on thursday i'd expect a maker says 4.4 to 5.0 is a big change 5.0 to 5.1 is a simpler challenge i think micro python is on 5.0 too now but there are ps ram issues where ps ram only uses half the amount available interesting and bill says he has ptsd from debugging pystack issues on uh esp yeah i'm glad you can make it dylan thanks thanks for tuning in um okay so what i thought i would do is if we look in pull request here and we actually look down here to the second oldest one we can see it's update esp idf to version 5.0 and this was done by micro dev in september of last year so it's been a long time coming but we did decide 823 is out but not for the pico pad from main to 823 yeah so sorry if you've ever watched these before you know that uh i do get a little distracted so david's question is like there's a board there there's a board that was added after 8.2 was released and um so they're on main but not on 8.2 those ones uh if they if we do want to get them on the stable release they would have to be um they would have to be put into the 8.2 branch um which yeah is a little bit annoying but um that's kind of the world we're in right now and it's also a little bit of forcing function for people to use the latest for newer boards like main is really not that destabilized yet but um you know we're hoping to get like this idf update in and then also um also the microfiber merge okay so um this is update esp idf to the release 5.0 branch it's from september and we can see that like there's a bunch of milestone changes um but we need idf 5 yeah and as we turn the corner to 9.0 alpha this is in february and it's august now it took us a while to get to we to get to this point the ps ram issue is now resolved as march i don't know what that was this person actually done this so i pinged it on july and micro dev micro dev like comes and goes a bit so um but just totally fine they're they're not a paid contributor they're just a community member um but this is me kind of wanting to pick it up so i might be able to leave this open and um push to it we could try that it's in micro devs update update idf branch so let's try to pull it down here's my obs output this is what's happening when i try to do cat cam if you're an obs expert uh this is like this is why cat cam is not working as i'm getting this device or resource busy what's causing that okay so let's find a new tab here how about this one and i was working on this usb reset stuff do i have stuff that i actually need to change it looks like i do so i'm gonna just before i move i'm gonna get this stuff i was just doing some debugging this is debugging this is debugging this is a fix these are better error messages debugging to be debugging i think i'm actually gonna leave this and see if i can't just move branches because like one one is the imx port and one is the idf the esp port so hopefully it'll be okay um so let's do get remote the and i've actually like i have a lot of different remotes this is why this is a good example of why i recommend uh recommend naming your remotes based on the person uh who's it is so we're just gonna fetch from microdove and we're going to get switched to update idea these local changes would be rewritten and commit or stash them okay i'll just i'll just commit stage stage stage stage stage and then get commit and it's on this usb reset branch i need to be on my virtual environment so that the check works oh and it's gonna add these translations my signing key oh can i not move that off the screen i'm getting a password password password password paste it and i'm gonna just highlight this copy something into my paste buffer so i don't actually accidentally paste my my password on screen all right so i've committed and now let's do the switch again let's see get log and we're in a repository sorry uh my sub models are pretty messed up unexpected maker asks is there a reason dan is stopping at 119 and not 120 for the merge for nine point out just curious so the plan is that dan will do 119 and then jeff's gonna do 120 uh because one of the major changes that is in 120 is something that jeff's aware of i see my my camera is frozen comes back to life that's a lovely you can like the stream is looks okay just the mange well is not happy bill says sub modules are always messed up i just nuked them every time i go that between vastly different ranches yet it's annoying but my sub modules sometimes have more state than that which is unfortunate like i sometimes i'll actually like do edits in sub modules i think we'll crash if i unplug yeah see so here's an example where like i have changes in in tiny usb i'll climb under my desk in just a moment and unplug the camera it looks like it's still there we go now if only i could get cat cam working cat cam come back i can just does not like me i don't know what the deal is yeah my sub modules got so messed up all right well at least face cam is fixed maybe maybe i just need to get a mange well again okay mo air decay am i like half on i wonder what branch i'm actually on you know what i always say and i haven't said this in a long time because i have been streaming get status get status is your friend like because it's in a world where it has fetched sub modules even though it's like fetch all sub modules now so like the first thing i'm planning on doing is rebasing the changes that uh migrate of main so what does get status show get status shows fatal not a get not a get repository great huh that's everything you're doing in several times yeah well get status is not helpful if if you get some module speed descriptor fix get status yes it does we are in like because we're not on the re usb reset let's see if the switch works now there now we now we switched fully and our get status is a lot cleaner so now if we get log so we can see micro devs commits oh co-authored by kasey web nice oh and there's a merge let's do uh how adventurous how adventurous do we feel should we uh get fetch ate a fruit and let's do get rebase i ate a fruit main so that doesn't look too bad so these look like all of um just look like all of uh the commits from micro dev or the folks that were working on it um this is the like interactive part of the rebase thing so um what is rebase rebase is uh what i like to think of is like replaying or redoing commits on top of a new uh a new branch or like these changes were originally done in september of 22 on an old version of circuit python so now what we're going to do is we're going to take all the all the commits that happened after that and we're going to just redo them on top of the newer version of circuit python the current version of circuit python so that's what rebase does and it also gives you um this uh interface where you can change the structure of the commits as well um let's take a look at that okay so now what we're doing is we're we're somewhere in the middle so we could see like oh we're trying to auto merge all this stuff and now we're stopped so again um and you could see they could not apply five ebd and i guess we don't have our list but if we get status we can see that um the last command done is update to 5.0 and now we've got all of these other things to do quarky says think of rebase as control c control v on git history yeah kind of mark says rebase is how i always end up messing up my prs and having to ask dan for help oh and it looks like my camera died again didn't doesn't it i wonder if it's too hot in here for the camera maybe that's the problem um it's in the 80s i could turn the fan on but i don't know maybe i need to slap a heat sink on the on the converter anyway that's pretty much what i look like right now i'm maybe if i talk later i'll do it again um most i do before checking out is cherry pick rebase is essentially cherry picking in order so like this is the equivalent of cherry picking that first commit and now i have to deal with the consequences of it but it's important to note that like if you see and get status like it has this new message of like you're in the middle of a rebase like here's how to get out of this special state is to like skip this patch or abort an abort will set me back to the original thing um yeah so you so unexpected maker says same i constantly screw up my repos and i had to kill them in the refork my github food is very very poor so the thing with uh github and like prs and stuff is like you could always so a branch is just a like label for a particular commit um and so if you mess a branch up you can just like change it to point to something else that is correct um so we like i'll see people like open a pr and close a pr and open a pr and close a pr it's like no no no like you don't you can leave the pr open because a pr will follow um a pr will follow a branch let me just get back so now what we have to do is we have to do all of the merge uh fixes for these and i always get confused it says this says deleted by them i get really confused about which is ours and theirs like some terms that they use as well so that really definitely screws me up um all of these frozen things these are all sub modules i'm actually just going to ignore those ignore those for now what if i wanted those to not show up i would do like the make fetch sub modules sort of thing um but i know that i don't actually have to do anything for those what it's really asking me to enter to do is like these uh this set of stuff is what i need to be focused on so let's take a look and see uh supply merge will show us here what we have to do so and this is like a a tiny preview of what um dan is having to deal with these a lot of get reset i almost never get reset yeah some modules are tough i yeah definitely in that boat um that one looks kind of hard hello sorry from ukraine this live protomatter one i might actually run make fetch all sub modules just to so we should have this command because it's in the newer version of circuit by thought so this should handle the protomatter one for example like protomatter is not something that we're going to want to change for so this looks like a kind of a duplicate where um esp cameras support was added but under a different name and here again is like a parallel display camera conflict so we'll have to take a look at that if there's a to do in v5 remove the following include okay so we'll we'll take a a quicker look at actually you know what you know what i haven't looked at before what happens if we click resolve oh okay so here is i think this looks like kind of a different main and this looks like it looks like uh the the idf update what do we do now i'm new we're merging we're updating uh the the branch that updates the esp idf to 5.0 and sorry my camp's frozen i don't know why why it stopped working but i could turn it off and you can see a little bit more on my desk but i basically look like that looking at the screen um okay so we can just confirm confirm this by going to the source and we are in uh ports expressive cmake lists so let's go ports expressive oh i hit the wrong one ports expressive cmake lists so this is the current version so this here's a here's a change seven months ago that's move esp 32 camera to esp camera so this is one of the things we're seeing that a two points out that there's a act command that can run github actions locally i just push in i do the another push another push another push detached head scares all i know my cam needs a reset let me i'll do it again but i don't think it's it may not last long i've already done it once how about now there we go no audio means no closed captions i think you can hear me now confirm the the cat was meowing at me too so we see if cat cam works should we go go for broke oh my gosh cat cam works something happened we've got cat cam working we've got audio working again i don't know what happens thank you tim for confirming you can hear and see me vini you're gonna jump up actually both cats are what was it last week last week i had zero cats in with me and now i've got both oh that's lovely if i tap my hand come on come up i'll hold her up once she jumps on my lap and then you can see both kitties ready cat this is a cat interlude for everyone i'm thinking about moving out of this space at home into a separate office space and i'm i'm a little sad that i won't have cats to deal with although that's both a blessing and a curse it's kind of annoying as well oh i feel like i should be on this screen so you can see spook in the cat cam meow could you hear her all right um i am thinking i need to get a a heat sink for that like because i like i'm using a you an hdmi to usb converter for the main camera and it's been freezing up and i think it is because it's uh because it's getting too hot like it's 83 in here now and it's already gone down like half a degree with the fan on um yeah my cat commercial break um i have a feeling because i i don't usually have that problem with it i bet that's what it was it's just like i touched it i touched the converter when i plugged it back in it was like it's definitely hot so i need to have a fan on it or something um okay and uh that that concludes the message from our cat sponsor i don't know uh okay so let's go back to the desktop and now we know that like this circuit pi esp camera is is the way that we want to go for naming so we are going to take this bit and then um we're going to want this bit over here so how do i pick can you tell i haven't used this i haven't used this before how do i how do i pick it next conflict previous conflict next conflict it's probably a hotkey or this there we go gotta click the arrow okay so that's what we want saving stage let's go to the protomatter one let's not do let's do this make file one so again here we see parallel display and esp camera so this is a little bit trickier in the sense that we want this bit but we want to call it esp camera instead so we want this change here we're getting this change anyway but this also adds parallel display support which apparently we didn't have this change looks okay so there's more changes here so what we'll do is we'll go through do all this merge stuff and then um we'll start building it and see if it builds yeah so i don't know what to do about this build one let's just collapse that for now for the protomatter one let's look at this ulp change and here we are ulp dot risk five versus all of this other stuff and i'm going to assume it's the other stuff that we want this seems too simplistic to me um which yeah so these are the changes that were made for these are the changes that were made for 5.0 titus says oh my gosh parallel display support for esp s3 is on my need list great okay so let's see so here's um here's a to-do so let's um again like do a little bit of both where we do that because we do want this error i'm not implemented but we'll do the rest of the to-do although it seems to me that like yeah i guess we might want to check for the ulp as well that's okay this is this is just a first pass next one i don't see enable ulp interrupt again this looks much newer than this so we'll take that one save and stage next one save def config i wonder why that was deleted this is esp 32 defaults should we try that it's possible we'll want to delete it but i think we just want this left version and if not we can fix it save and stage let's resolve the config port stuff so we want the newer version of esp camera this we want and this we want save and stage again is stk config file auto-generated comment on top i didn't i wrote a script that tries to take so when you do a build it matches all the stk configs together and i made a script that tried to piece it out so k config is designed to just like have one version for whatever but what we want is like stuff that's common to all esp boards should be configured commonly and stuff that's for a specific board should be configured for specific boards and so there's a script that i wrote that splits out stk config settings into like different files that we then mesh together during our our board builds um so that in particular esp 32 one seems like it should have stuff in it but it's probably it's probably an easy shortcut to just empty the file because there will still be defaults they're just not laid out laid out anywhere for that so it's possible we'll just find that we we're setting stuff that doesn't exist in 5.0 okay so let's do a get status again and see where we're at so we have both modified so esp idf obviously we want to change to be the newer one um esp 32 camera we probably don't want to take the newer one we want to take the current one and then i2s lcd driver is deleted and that seems right to me that it should be deleted let's try that so i am going to go through all the commits and then see what what breaks um esp idf so this is just different commits i don't know how we select each side so what we'll do is we'll so those are sub modules and sub modules are tricky they're actually like files that have the commit in them and you've got to decide which commit to be at but you so if we go to ports expressive esp idf and just get log we're in circuit python customizations which is the head of circuit python 5.0 so let's see if we can find where this is because micro dev 1 esp idf is my guess it's either there actually you know what we could do is we could do get remote origins okay so we're in adafruit esp idf and then let's take a look at that branch branch so we have our 4.4 circuit python branch and there should be a branch for 5.0 but i'm tempted generally to move move us over to the plain one i don't know yeah so here's the here's the circuit python v5 thing so if we go to commits so here's an update to 501 and then here's the circuit python customizations commit that is what we're checked out to which we could just leave but let's let's just take a peek like this is a fix that i thought should be fixed um yep interesting okay so those are things that still apply um i think we just leave it as is so let's just leave it as as is for now and get kind of like the existing thing working and my camera froze again i don't know why it's so hot right now um i bet cat camp doesn't work anymore either okay so let's move out of this and it's where we want so we're just going to go get ad esp idf now let's move into 32 camera i wonder if i have a heat sink that could help handy i wonder if there's a fan in there maybe there's like a fan in there that died or got full of cat hair could be the case okay so what where are we here merge upstream updates for esp camera i bet that's not what we want so it was on commit adb because i know jess been doing a lot of work here so here's adb so we are behind so let's just copy this and we'll we'll uh check it out that's fine so both modified esp camera but i think that's what we want so we'll get ad we could always redo any of this okay so protomatter is wrong and again this is a like we probably don't want to be on like this is just old protomatter so we could just go to if we start going python ports or no live protomatter and you can see this is the start of the commit so we'll just copy that and we'll change it check it out okay and that looks right this is why it's pretty common if somebody does do a rebase um they may mess up and i may mess up the sub modules um okay so everything here looks good except for this build file has changed i'm not sure why let's look at it this way so circumvented that did a bunch of externalization of things so this is upstream this is the branch this upload artifact i don't think we want anything from the right hand side fetch id of tool cache this says i tried to i gave up trying to do esp builds on windows and that's so that's where we're at is that this is oh no this is build expressive but this is the windows build oh that's not well lined up i don't think it's there then okay that's just the complex we're on okay so that looks good we'll go with it now if i was good i'd try to build but i i'm not that good i think i'm more willing to just do it at the end like fix up stuff at the end so let's do rebase continue and that's what we'll call it still and so now we've moved forward in time if we do get status and sorry i'm frozen trying to think of it if i can think of where i would have a the heat sink assuming that's the problem now it's a lot cooler here it's just not oh there we go again and you can still hear me weird like last last week it didn't have a problem okay so now we have a bunch of board config changes that we're gonna have to merge in together it's interesting that there's a deleted by us board config but my guess is that these are all those same sort of methodical changes audio and video is still working great thank you dcd okay so we did update the idf we did conditionally include esp ps ram now we're trying to update to 501 okay so let's keep on well let's just say resolve esp 32 s2 spy ram support i assume we want that but i don't actually know like is that a thing in 5.0 or not so maybe what we do is we do what i said we were gonna do and let's move it over we want to save it but i don't want to stage it yet save without staging and let's just which board was that that was probably this first one okay so we're in a new esp idf so let's go esp express it and let's actually deactivate our current virtual environment because we're gonna do install actually we have to export idf path and then do in well i'm gonna do install because i haven't done this for 5.0 yet so this will download all the new tooling gcc 11 then nice that's better than gcc 8 now we'll do export and now let's try make board equals okay so we'll just do we have a couple um actually we could do pip install our requirements dev so there's some so the esp idf sets up its own python environment for building esp stuff but then there's also requirements that circuit python uses um so that's what i just installed is the circuit python stuff we need so it looks like it's just like checking more stuff out because it's it's like getting my copy of esp idf 5 actually up to date which is good how are folks doing do we think cat cams working still yep awesome spook is sleeping there that that is spook cam and then then is up over there and there's the fan so that now that's min cam okay so we didn't succeed the following python requirements are not satisfied requirement is cryptography less than 36.1 and greater than 2.1 but we have cryptography 4.41.03 installed and i wonder if that's something that we just um yeah we just installed it um cryptography 3602 i don't know why but let's see if we can't um downgrade pip install cryptography less than 36.1 maybe that'll work does not turn pi open ssl 23.2 requires cryptography greater than 38 so maybe we do did requirements dev change for this branch i don't i don't know i don't think so so i wonder if we can figure out it looks it seems like we're gonna have to use an older version of pi open ssl which generally i don't like to do i'd like to be on the newest one but it's this like what i think it was the javascript world where like you have all these pin dependencies and you just gotta figure out how to like work together 23.2 so cryptography maximum version has been increased to 41 maximum version has increased maximum version minimum version is now 35 minimum version here is now 38 and we need to use 36 so we actually want to use this is the version that we want to use for pi open ssl and the two equals or one requirement already satisfied but now we should be able to say let's see if it works externally managed environment is being enforced yeah it's tough no such file esp 32 s2 ulp does not exist anymore yes it does so fd is a a rest replacement for find i think so i wonder what we want i wonder if we want sock ulp or esp 32 s2 ulp so this is an imx land but well we'll get switched over here forks expressive common how esp ulp ulp.c ulp and ulp risk 5 yeah fd2 says i'm kind of liking some of the rest replacements for traditional tools specifically finding grep yeah i i think it's super cool that they're they're redoing and modernizing a lot of command my tools like i use rg now instead of ag and rg is ripgrep and i think it's written in rust as well okay so i think this is just a missing include so if we go to our make file and we see that we're missing ulp risk five where did we find it we found it in ulp fsm include so my guess was there two and you know stuff like this is stuff that could be fixed no problem unexpected maker have a good weekend too thanks for stopping by see i'm getting a little ahead of myself and so like i'm not sure whether i should continue to fix issues some contents of esp s2 ulp risk 5 has been merged with ulp risk 5.h include the latter to avoid the warning i wonder if this is fixed in a further let's take a peek at um like is this a new fix or something that i'm gonna have to fix anyway so we look at commits here yeah like we're not we're not down here to this update ulp implementation thing yet yeah so here's the change we just did and here's changes here so okay so we don't want to do this like this is taken care of later um so let's we really need to get through this merge first so we can do these and we'll we'll take our best guess and keep going so this is what we're going to discard actually leaving that would mean that we get a conflict we might get a conflict later which is annoying um okay so let's ram support we'll just save it and stage it cancel huh i wrote a somewhat nifty search wrapper using rg plus fzf great for large code bases or even searching through all your repos when you're looking for snippet what is fzf i don't know what that is i know rg but i don't know the other one resolve l what post name oh oh dang it i hit the wrong button so here we have esp camera optimization flags i don't know why that is this must be a size issue so let's just do that camera update and we'll include the patreon size fzf is a completion ui written and go so let's just do the same thing here we'll use the newer version but update the camera it's totally possible that like later on it'll update the camera like maybe there's a commit that does that here fuzzy search which is hmm s3 spam sram support this looks like more like i want i don't know we'll probably end up regenerating all of these anyway this is just a first pass just a first pass just a first pass fix it later i'm gonna get real good at this merge thing and then there is one that i hit stage that i shouldn't have so it probably has the diff markers in it still yeah this one unstage and oh come on okay feather s342 we need to go in here and do boards feather s342 just line down delete it there we go so we'll stage that as well we'll get status it up we'll see that we don't have anything unmerged and now we'll rebase continue fuzzy finder yeah let's take a look at that after the stream all right where are we at now three commands done nine remaining get status changes not stage for commit this is we don't have anything why so why did it stop us if there's changes that are not stage for commit i see the previous cherry pick is now empty possibly due to come conflict resolution if you wish to commit it anyway use get commit allow empty otherwise use get rebase skip um so the command we did is conditionally include esp ram and fix sdk config so it seems like that's been fixed already so i think we just want to get respace again we can always fix it later so here we have partition table changes probably don't want to do oh it's growing it to be bigger i wonder if it didn't i'm going to do the current version because if we could figure out how to get it oh i hit stage again um because that's something i'd really like to test so that that's something that i would like to do later because i really would prefer it if we didn't have to change it so anytime we change the partition table we risk like corrupting the user file system on the update so that's something and i don't want to take lightly to do so i'd rather like not do that if i could get away then okay so this is partitions for megabyte know you have two right so it would affect a bunch of boards config partitions for know you have two i just i don't we have to do it later we can do it later like we can always go back and reference the older commits as well status and now we can and now we have more problems so this is update you'll be implementation so this added s2 and s3 sport by giving it the generic i think we want i think we want this side and halt okay so here's halt based on this looks better to me because it is checking the the the right uh state machines or the right elp implementation is being used so again this is like the esp32 defaults being like trimmed down to basically nothing again which where are the elp settings on the left here they know where to be found this is why like changes like this are like the ulp hardly anybody uses so like will somebody even notice if we accidentally break it i think we're good and i'm moving still now we're in refactor board config is this before i wrote that script it might be let's say resolve and then we take both this one and sure so this doesn't this is spy ram and this is default ps ram my guess would be this way you pondered if this was before you wrote that script check history yeah i could i could or i could just keep going let's do this one but also this one my guess is that we're going to run that script anyway um to regenerate everything i think i even yeah it's been a while let's look that up browser window so what i'm thinking of is great get going down it's back the tool and the script i'm talking about is ports expressive tools update stk config or generate yeah update seems right right yeah so this this has like these settings go in the opt file these settings go in the target file and these are like prefixes board settings flash settings the least settings so we'll probably we may want to even update this for for esp5 as well let's keep going i don't know how many we have left get status we're ready to go so most of them most of them auto merged um we have three remaining which is good get rebased continue i don't care you know usually what i'll do is if i do want to redo the commits i'll actually get it all like smoothly rebasing first and then um and then i can go back and and juggle them if i want to i'm not super personicity about like structure of commits and stuff so okay what's this do this is oh it's rgb matrix uses supervisor allocation and the other thing we'll do is like once i'm done with this merge we'll then go back we can go back and look at all the changes as well let's just keep going s2 and s3 and tada we are now updated i think i'm gonna actually call this well i'm gonna leave the branch name because if i push it to micro devs repo which i might be able to do and that'll update the pr which would be kind of nice i'm gonna want to keep the same branch name okay so since we're here we've got about 20 25 minutes left um let's go back to the board that we were trying to build and just see what happens um so we've we've gotten up to date with the 5.0 update and now we can take a look at this so so here's the same morning that we had earlier i'm just going to do a quick clean just to make sure that i'm on the very latest version of the branch also note that i'm not doing any j command um i've never had great luck doing j which does some make tasks in parallel i've never had great luck with that on the espidf so i'm just like not doing that so it's going to take longer to compile but it's also going to um make it uh make it less likely to break like make it more reliable okay so we're still having this problem that we're uh getting this warning which is fine warnings are great um let's go to common how ulp and maybe we just try to it's saying it's being a little vague because it's saying don't include esp32 s2 you'll be risk 5 just do you'll be risk 5 so you'll be risk pppp so there's a regular include uh specific include the generic one my guess is they want us to do this so it's probably like that and it's probably that for s2 and s3 that have the risk 5s we could just include it once but because i don't like that esp32 doesn't have that mode i'd rather just have it per target so let's see if that fixes it the weather display crashed i just hit reset again i really like this weather display but i don't catch all the errors that cause problems so i have to hit the reset button okay unknown type name here's the i showed this on show and tell before is it gonna show me oh sorry that's cats gonna show the weather is it gonna show a crash right now to hit the reset button it's possible that the like weather service i'm using is down i don't see anything there i was hoping it was gonna magically appear so crashed must be having a problem key error okay so we're in parallel bus and we don't have the itus lcd config or handle types so let's go there parallel bus oh but we're in the dot h and this is trying to include itus lcd driver so let's just do a wrap and it looks like it is in oh interesting esp iot solution components bus i did see that so we are including itus lcd driver remember seeing it oh you know what there's this that could be the problem it could be including the wrong file delete it and try again desktop thank you at least you got some cat cam inka blinka one of the things about key error handler is that it'll just say here which is not as helpful as the actual trace back without the error handler sometimes yeah i did get a trace back to um you know discord now you can search for rep fcf search on gab it's the top result i'll keep trying to get this to refresh so you can see what i'm doing see my nice weather this way it looks like it's going i could do v1 let's show us what it's compiling so these are the sorts of things that it would normally do in parallel but to be on the safe side i like to do it not in parallel with the esp idea okay we have about 20 minutes left so let me know if you have any questions unknown target esp idf mdns live mdns dot a what could that be about so there's two things to look at for what libraries get linked or that get built for the idf one is there's a big long string somewhere here a big long list of stuff so here we see binary wi-fi blobs binary blobs and then components and i bet mdns is in here yeah there's mdns there and then components link add bluetooth add you will be add esp or ps ram expand it out to the actual directory and then expanded ends up in the link committed here so the question is is like did mdns move or did we fail to actually build it cmake lists so i think this component list is similar it also has mdns although here look extra component directories esp protocols components and dns so i wonder if um i wonder if the location of the live a moved so what if we do an fd live mdns dot a that should be in build some of these newer tools will ignore ignore get ignored stuff that are s2 viewers tft esp don't didn't get built at all i wonder if we could do that's the search a little string insert a rex you can also use glob not that all right so let's take a look in esp idf components so it looks like yeah so there's no mdns directory anymore so it's not going to be there we saw it in esp it was esp protocol wasn't it esp protocols oh you know what i think they moved it out of they moved it into a different directory so here's protocols yeah so does that mean that i accidentally or artificially restricted it so we're doing the reverse tft s2 reverse tft esp idf and esp iot solution i don't know where or how we would have built that make files so let's uh the next thing i want to look at is my fd tool is a re-implementation of find basically like done in rest i think if i remember it it's not a wrap around find it's a replacement for find so let's take a look at um this commit here there's a lot of changes i only update once every 15 minutes yep i only do once every 15 minutes as well i catch some errors just not all of them yeah so the esp protocols they split it out they they did some reorganization as here and then in the make file how do we handle the mdns changes so there's this new include path well display hmm so that's interesting there's no mdns stuff here maybe it doesn't work thanks fatty too yeah get lesson there's lots to learn to get you never stop it's kind of a giant thing all right we've got 15 more minutes or so if you have questions feel free to ask them otherwise i'm just trying to figure out this mdns stuff how do i get an mdns library out of this it's in protocols you know p stuff yeah the thing yeah just thinking about like they make they expect the world to be cmake and i've resisted cmake one because i don't like the language that it has and two it uh i want to if we switch build systems i want to do everything i really like that although our system is super clunky it's pretty clunky is like uniformly clunky um that that makes any sense and so like it's just a lot of work oh it's interesting so here's here's a cryptography limitation here in requirements as well so let's figure out how i am building these components link i actually need to build them too idf cmake targets that sounds promising idf cmake targets ninja idf idf cmake targets so it just looks like this was never the protocol stuff was never actually supported so what we need to do is idf stamp and ninja c build esp idf where is do we call i think we call cmake yeah here we go so we call cmake to build the sdk config with all of our defaults so we need to do something similar for protocols to get the protocol cmake s dot sdk convict let's take a look at this protocol thing you know there's the esp camera thing too probably has the a similar structure protocol esp protocols see like cmake will like merge these all all these things together like there's this extra protocol thing i wonder if the ninja file has it so there's two steps that we do we're doing the cmake call to generate the sdk config and to set everything up and then like cmake is two steps it's like one step generates a ninja file and the second step actually runs ninja ninja you can think of it kind of make but like even lower level so sdk config my guess is so let's look in this build ninja file which is super big i assume and let's just look for mdns nothing so like there's a cmake list thing that that we set to include the extra component but our cmake command is not these are all certified stuff yeah so we're not getting any and i did clean so i would expect that like this cmake list inclusion extra components durrs i wonder if yeah so that's how esp 32 camera is being built as well is by setting extra components component durrs so if we look at esp protocols components mdns cmake lists and think mdns networking socket i would think that it would show up in there i don't know i think it's time for me to take a break and come back to it with fresh eyes uh one thing i will do is i will i'm going to fix i'll commit these two to get add the common howl thing and it's a removal bill has a good question will there be some methodology in retesting everything or do we just wait for issues i'll do like smoke tests but the true test will be getting people to try it um so like i'll make sure all the boards build i'll make sure that everything that i think should work will work um and then i will uh then i'll commit it because i think we had this conversation before it is like it is really valuable to get it out to the broader audience to get tested so main main with like early early 9.0 is not going to be stable and that's just the way it is um you know we won't call we won't make a release and call it stable until until we uh we feel confident that it is pretty stable i appreciate you bill says i will be using main as always so expect me to find a few new at least yep yeah that's super helpful i really appreciate it and you know i find a lot of stuff too because i'm always working off main so there's already uh like there's this toy hacker board that is like inside the toy and really you need what the web workflow for um there are some changes that lumeur wants to make for that so that's a project that i can do after this update that will kind of like keep uh keep me working in esp land and and just like testing it and i'm sure like issues will come around that as well but that's a good question there's definitely a world where we could be more thought mathematical about it um but testing with external devices is like quite tricky automated testing with external devices is quite tricky uh it's something i'd love to see but uh yeah it's really hard especially when it's all networking stuff hmm dj debon three says 8.0 was really really buggy and an important part of smoothing it out was all the testing reports i don't remember 8.0 being that buggy but maybe i was gone i fully expect 9.0 to be pretty buggy because we're updating the idf and we're gonna be updating circuit python so that's gonna be a lot of churn soon but it'll be good okay let's go back here thank you all for joining me for this deep dive reminder next week will be at 2 30 so a half an hour later than normal um as part of circuit python day festivities that start early in the morning uh seven pacific 10 eastern um so keep an eye out for all that we'll have a day full of streaming stuff and uh yeah i'm excited that cat cam is back and uh keep keep an eye on my github and the discords uh if you want to follow along with this idf update stuff uh if you want to support me and you can do so by supporting adafruit by going to adafruit.com and purchasing hardware there one thing that's new and i think it is released but um maybe not i don't always keep track uh there is a metro s3 um which is super handy for development um so if you're uh into the expressive world like this sort of thing could be quite handy um so check that out if you want to support me and adafruit uh if you want to join me and a bunch of other cool people on discord you can go to the url adafru.it slash discord and check that out we'll see you there and with that uh i will switch to cat cam and maybe i'll move it so i can pet spook uh although i can pet into uh and we'll see you next week let's go to cats and maybe i'll pet both cats i was hooked you should have seen the the look here that spook gave me when she started meowing saying what you're gonna pet her that's cat time thank you all we'll see you next week


Well, it was the win they needed in Saturday's victory over Concordia, Saint Paul, but the Beavers will be the first to tell you that there's still a lot of games left in ball to be played. Next up, a trip to Sioux Falls to take on August stand on the Vikings also one-on-one this season, topping Minot State last week. And with the Bees are looking to build off the offensive breakout last week and continue the Game Greeds hot start. That includes seven picks and six of them from their starting safeties, John Vogler and Evan Tompkins. All three of his interceptions, Vogler has came in man coverage and that's just a testament to who he is. He's a fast physical freak, that good luck throwing it on him if he's covering you. And my interceptions have been more less athletic, I'll say, coming in zone coverage. John's a big strong kid and can run and he certainly can cover a lot of ground back there. And Evan's just really savvy and smart. So he puts himself in position to make a lot of plays and he's played really extremely well in all the first two games. So week number three tomorrow at Augustana, one p.m. kickoff in Sioux Falls. If you've enjoyed this segment of Lakeland News, please consider making a tax deductible contribution to Lakeland Public Television.